System

Abstract

A system is provided.SOLUTION: An information collection system includes means for receiving materials and data, means for converting the received materials and data into a designated format, means for extracting metadata of the materials and data and generating an index, means for receiving a query in a natural language, means for analyzing the received query, means for searching for corresponding materials and data, and means for generating a response based on a search result and transmitting the response to a predetermined terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In many organizations, documents and data are scattered across different locations, making it difficult to quickly find the information you need. This results in a lot of time being spent gathering information and searching for documents, reducing work efficiency. It also creates a situation where information is dependent on specific individuals (personal knowledge), making it difficult to share knowledge. Furthermore, tasks such as handovers become more complicated, making it difficult to maintain the knowledge level of the entire organization. The present invention aims to solve these problems by providing an information gathering system that improves work efficiency and promotes knowledge sharing by making it easier to gather information and search for documents. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides an information collection system including the following means: means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata about the materials and data and generating an index, means for receiving queries in natural language, means for analyzing the received queries and searching for corresponding materials and data, and means for generating a response based on the search results and sending it to a specified terminal. The system also provides means for analyzing materials and data using a generative model and generating a response in natural language based on the results. Furthermore, the system includes means that operates in a cloud environment and allows users to directly upload materials and data from shared folders or cloud storage. This enables unified and efficient management of materials and data scattered across multiple locations and rapid information collection through a natural language interactive format.

[0006] "Means for receiving materials and data" refers to the function that enables the system to receive materials and data uploaded by users.

[0007] "Means for converting into a specified format" refers to the function of converting received materials and data into a format that is easy to handle within the system.

[0008] "Means for extracting metadata and generating indexes" refers to the function of extracting attribute information of received materials and data (e.g., creation date, creator, file type, etc.) and creating indexes to enhance search functions.

[0009] "Means for receiving inquiries in natural language" refers to a function that allows a user to input a question in natural language into the system and for the system to receive the question.

[0010] "Means for analyzing received inquiries and searching for corresponding materials and data" refers to the function of analyzing user questions entered in natural language and searching a database for materials and data related to those questions.

[0011] "Means for generating a response based on the search results and sending it to a specified terminal" refers to the function of creating a response to the user based on the information in the searched materials or data and sending that response to the user's terminal.

[0012] "Means of analyzing materials and data using generative models" refers to the function of analyzing the content of received materials and data using an AI model (e.g., GPT, BERT, etc.).

[0013] "Means for generating responses in natural language" refers to a function that uses a generative model to generate appropriate responses in natural language to questions from users.

[0014] "Means for operating in a cloud environment" refers to the functionality that enables this system to be operated on an online cloud server.

[0015] "Means for enabling direct upload of materials and data" refers to a function that allows users to directly upload materials and data to the system from a shared folder or cloud storage. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to an information collection system that can efficiently manage materials and data and quickly acquire information through inquiries in natural language. The system includes the following means.

[0038] The server receives materials and data uploaded by users. For example, a user selects materials and data from a shared folder or cloud storage using a terminal and uploads them to the system. At the same time, the server receives format information for the materials and data and prepares to analyze the content of the received materials and data.

[0039] The server converts the received material or data into the specified format, for example, converting a PDF file to text, and the converted data is then ready for further analysis and indexing within the system.

[0040] The server then extracts metadata from the documents and data, creating an index. This metadata, such as creation date, author, and file type, is stored in a database to improve search performance, laying the foundation for users to quickly find the information they need later.

[0041] A user inputs a query in natural language from a terminal (e.g., "Tell me about the progress of this month's project"). This query is sent to a server. The server analyzes the received query and uses an AI model (e.g., GPT or BERT) to search for corresponding materials and data. After understanding the query and searching for relevant data, the server generates a response based on the information obtained. This response is expressed in natural language and is in a format that is easy for the user to understand.

[0042] The response is sent to the user's device, allowing the user to obtain the information immediately. For example, if user B asks, "What is the progress of this month's project?", the server extracts progress data from the project documents, generates a response saying, "This month's progress is 70% complete," and sends it to user B's device.

[0043] This system operates in a cloud environment and allows users to directly upload documents and data from shared folders or cloud storage. This allows documents and data to be managed in a unified manner even if they are scattered in different locations. It also facilitates information sharing, which has the effect of improving the knowledge level of the entire organization.

[0044] For example, Person A uploads project documents to Google Drive, and the server receives and stores them in a database. After that, Person B makes a query in natural language, and the server analyzes the query, extracts the necessary information, and generates a response.

[0045] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The user accesses a shared folder or cloud storage from their device, selects the materials or data to upload, and then uses the designated interface to upload the selected files to the AI ​​Data Navigator.

[0049] Step 2:

[0050] When a user clicks the upload button on their device, the device sends the selected materials and data to the server, along with metadata such as the file type and file path.

[0051] Step 3:

[0052] The server receives the materials and data sent by the user. After receiving the data, the server determines the file type and converts it to the specified format (e.g., converting PDF to text) if necessary.

[0053] Step 4:

[0054] The server extracts metadata from the converted data, such as creation date, author, file type, etc. This metadata is stored in a database, generating an index for efficient subsequent searches.

[0055] Step 5:

[0056] A user uses a terminal to input a query in natural language, for example, a specific question such as "How is the project progressing this month?" The user sends this query to the server.

[0057] Step 6:

[0058] The server parses the incoming natural language query, using an AI model (e.g., GPT or BERT) to understand the meaning of the query and identify the appropriate database entry.

[0059] Step 7:

[0060] Based on the analysis results, the server retrieves relevant materials and data from the database, which is then further analyzed as needed and used to generate an appropriate response to the query.

[0061] Step 8:

[0062] The server generates a natural language response based on the search results, such as a specific answer like "Last month's progress was 50%, and this month's progress is 70% complete."

[0063] Step 9:

[0064] The server then sends the generated response to the user's terminal, where the user immediately receives the response and obtains the information in response to the query.

[0065] Step 10:

[0066] Users can check the responses sent on their devices and obtain the necessary information. For example, when Person B checks the "This month's project progress," he can plan his next action based on the displayed information.

[0067] This series of processes allows users to easily and quickly access the necessary materials and data and obtain appropriate information.

[0068] Example 1

[0069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0070] Traditional methods for managing materials and data make it difficult to uniformly manage data in different formats, making them inefficient for searching and retrieving. They also lacked a system that could quickly provide appropriate information in response to natural language queries, forcing users to go through multiple steps to find the information they needed. Furthermore, they lacked the functionality for uploading and managing data in a cloud environment, making it difficult to centrally handle materials from multiple data sources.

[0071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0072] In this invention, the server includes: means for receiving documents and data; means for converting the received documents and data into a specified format; means for extracting metadata from the documents and data and generating an index; means for receiving queries in natural language; means for analyzing the received queries and searching for corresponding documents and data; means for generating a response based on the search results and transmitting it to a specified terminal; means for receiving format information for the documents and data; means for performing format conversion using OCR technology; means for analyzing the query content using a generative AI model; and means for expressing the generated response in natural language. This allows users to centrally manage and search documents and data in different formats, enabling prompt and appropriate information provision based on natural language queries. It also facilitates data uploading and automatic management in a cloud environment, enabling efficient management and use of documents and data.

[0073] "Means for receiving materials and data" refers to the function of receiving data files of various formats sent by users via the network to the server.

[0074] "Means for converting received material or data into a specified format" means software or algorithms for converting received material or data into a consistent form.

[0075] "Means for extracting metadata of materials and data and generating indexes" refers to the ability to extract important attribute information from data files (e.g., creation date, creator, file type) and store it in a searchable format.

[0076] "Means for receiving inquiries in natural language" refers to the function of the server receiving questions or requests entered by the user in natural language.

[0077] "Means of analyzing received inquiries and searching for corresponding materials and data" refers to the ability to use technologies such as generative AI models to understand the content of inquiries and search for relevant materials and databases.

[0078] "Means for generating a response based on the search results and sending it to a specified terminal" refers to the function of generating a response in natural language based on the information obtained from the search and sending it to the user's terminal.

[0079] "Means for receiving format information of materials or data" refers to a function for receiving information about the format of received materials or data (e.g., file type, number of pages, size).

[0080] "Means for format conversion using OCR technology" refers to a function that uses OCR (optical character recognition) technology to convert characters in images or PDFs into text.

[0081] "Means of analyzing the content of inquiries using a generative AI model" refers to the function of understanding and analyzing the content of user inquiries using a generative AI model capable of natural language processing (e.g., GPT, BERT).

[0082] "Means for expressing the generated response in natural language" refers to the ability to generate answers to queries in natural language that is easy for humans to understand.

[0083] The present invention relates to an information collection system that improves the efficiency of managing materials and data and enables rapid acquisition of information through inquiries in natural language. This system is mainly composed of a server, terminals, and users.

[0084] The server has a means for receiving materials and data uploaded by users. Specifically, when a user uses a terminal to select materials or data from a shared folder or cloud storage and uploads them to the system, the server receives these materials and data via an HTTP request. The server also simultaneously receives format information for the materials and data, and uses this information to prepare for analysis.

[0085] The received material or data is converted into the specified format using OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six), and the converted data is put into text format, ready for further analysis and indexing within the system.

[0086] Next, the server has the means to extract metadata about the materials and data and generate an index. Extracted metadata includes creation date, author, file type, important keywords, etc. This metadata is extracted using a machine learning model (e.g., SpaCy) and stored in a database (e.g., MySQL®, ElasticSearch®). This lays the foundation for users to quickly search for the information they need later.

[0087] The user inputs a query in natural language from their device and sends it to the server. An example query might be, "What's the progress on this month's project?" The server receives the query and analyzes it using a generative AI model (e.g., GPT-4 (registered trademark), BERT). Based on the analyzed query, the server searches for appropriate materials and data.

[0088] The server generates a response in natural language based on the search results. The response is in a format that is easy for the user to understand. For example, it may generate a response such as "This month's progress is 70% complete" and send it to the user's device.

[0089] This system operates in a cloud environment and provides a means for users to directly upload documents and data from shared folders or cloud storage, allowing documents and data to be managed in a unified manner even if they are scattered in different locations, facilitating information sharing.

[0090] As a concrete example, Person A uploads project materials to cloud storage, and the server receives and stores them in a database. If Person B then makes a natural language inquiry such as "Tell me about this month's project progress," the server analyzes the inquiry using a generative AI model, extracts the necessary information, generates a response, and sends it to Person B's device.

[0091] An example prompt is:

[0092] "Receive uploaded materials from the shared folder and record the format information."

[0093] "Convert the uploaded PDF file into text data and create an index."

[0094] "Parse queries entered in natural language, search for relevant data, and generate responses."

[0095] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[0096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0097] Step 1:

[0098] Uploading data and receiving formatting information

[0099] Users use their devices to select materials and data from shared folders or cloud storage and upload them to the system.

[0100] The server receives an upload request from the user as an HTTP request, saves the materials and data in temporary storage on the server, and simultaneously records the format information of the materials and data (e.g., PDF, Word, Excel, etc.) as metadata.

[0101] Input: Files of materials and data uploaded by users

[0102] Output: Files and their metadata stored in temporary storage

[0103] Step 2:

[0104] Data format conversion

[0105] The server converts the received material or data into the specified format. For example, the server may use OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six) to convert a PDF file into text data.

[0106] Input: Files and metadata of materials and data stored in temporary storage

[0107] Output: A file converted to text data

[0108] Step 3:

[0109] Metadata extraction and indexing

[0110] The server extracts metadata from documents and data and generates an index. To do this, it uses machine learning models (e.g., SpaCy) to extract important information (e.g., creation date, author, keywords, etc.) and stores this information in a database (e.g., MySQL, Elasticsearch).

[0111] Input: A file converted to text data

[0112] Output: Metadata and index information stored in a database

[0113] Step 4:

[0114] Receiving user inquiries

[0115] The user inputs a query in natural language (e.g., "Please tell me about the progress of this month's project") from the terminal and sends it to the system.

[0116] The server receives this query as an HTTP POST request and temporarily stores its contents.

[0117] Input: A natural language query entered by the user

[0118] Output: Query content saved in temporary storage

[0119] Step 5:

[0120] Inquiry analysis and data search

[0121] The server analyzes the received query using a generative AI model (e.g., GPT-4, BERT), and uses the index information to search for relevant materials and data based on the analysis.

[0122] Input: Query content stored in temporary storage and index information stored in the database

[0123] Output: Analysis results and searched materials and data

[0124] Step 6:

[0125] Response generation and transmission

[0126] The server generates a response in natural language based on the search results using a generative AI model. The generated response is sent to the user's device as an HTTP response, where the user can view the response.

[0127] Input: Analysis results and searched materials and data

[0128] Output: A natural language response and the response sent to the user's device

[0129] Through these steps, the system enables management of materials and data, and rapid information search and provision using natural language.

[0130] (Application example 1)

[0131] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] In conventional information gathering systems, users had to go through multiple steps to obtain the specific materials or data they were looking for, which reduced search efficiency. Furthermore, online shopping sites lacked a means for users to quickly obtain product information, which led to the problem of taking a long time to make a purchasing decision.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0134] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for providing an assistant function that allows users to easily collect product information. This allows users to quickly and accurately obtain product information through inquiries in natural language, thereby improving search efficiency and speeding up purchasing decisions.

[0135] "Materials and Data" refers to information and content uploaded by Users, including in a variety of formats such as PDF, text, and images.

[0136] "Means for receiving" refers to the function for transferring materials and data from the user to the server.

[0137] "Means for converting into a specified format" refers to a function that processes received materials or data to convert them into a specific format.

[0138] "Means for extracting metadata and generating indexes" refers to the function of extracting attribute information of materials and data and creating indexes based on that information to facilitate searches.

[0139] "Means for receiving inquiries in natural language" refers to a function for receiving questions or requests from users in natural language format.

[0140] "Means for searching for corresponding materials and data" refers to the function of searching for related materials and data from a database based on a received inquiry.

[0141] "Means for generating a response and sending it to a specified terminal" refers to the function of creating an answer based on the search results and sending it to the user's terminal.

[0142] "Means for providing assistance functions" refers to functions that assist users in easily gathering product information, particularly on online platforms such as shopping sites.

[0143] This invention provides an information collection system that improves the efficiency of document and data management and enables rapid information acquisition through natural language inquiries. This system includes functions for receiving documents and data, format conversion, metadata extraction, index generation, natural language analysis, data search, response generation and transmission, and assistant functions.

[0144] Hardware and software used

[0145] Cloud storage: Use Google Drive or Amazon S3 to store materials and data.

[0146] AI model: OpenAI's (registered trademark) GPT-4 is used for question analysis and answer generation.

[0147] Database: MySQL and MongoDB are used to manage metadata and product information.

[0148] API: RESTful API is used for communication between the application and the server.

[0149] Frontend: Develop a cross-platform smartphone app using React Native.

[0150] Processing flow

[0151] 1. Uploading data:

[0152] The server receives the upload of materials and data selected by the user from cloud storage (e.g., Google Drive). This includes the step of specifying and sending materials and data through an interface as an operation performed by the user in the application.

[0153] 2. Data Conversion and Storage:

[0154] The server converts the received materials and data into the specified format (e.g. PDF to text), and the converted data is stored in a database.

[0155] 3. Metadata extraction and indexing:

[0156] The server extracts metadata (e.g., creation date, creator, product category) from the materials and data, and generates an index, which is stored in a database to enable efficient searches.

[0157] 4. Natural Language Analysis and Data Retrieval:

[0158] The user sends a query in natural language from the application (e.g., "What are the popular products this week?"), and the server analyzes the query using the GPT-4 model and searches the database for corresponding materials and data.

[0159] 5. Response generation and transmission:

[0160] Based on the search results, the server generates a response in natural language format (e.g., "This week's popular products are: Product A, Product B, Product C.") and sends it to the user's terminal.

[0161] Specific examples

[0162] For example, a user opens a smartphone app and uses the chatbot function to type, "What are the popular products this week?" The server receives the request and uses the GPT-4 model to parse the question. It then retrieves information about this week's popular products from a database and uses the GPT-4 model to generate a response for the user. The response is then displayed on the user's smartphone.

[0163] Prompt Sentence Examples

[0164] An example prompt is:

[0165] User Question: "What are the popular items this week?"

[0166] The database contains:

[0167] Popular Item 1: Item A, Other Information...

[0168] Popular Item 2: Item B, Other Information...

[0169] Generated response: "This week's popular products are: Product A, Product B, Product C."

[0170] This system allows users to quickly and accurately obtain product information through natural language inquiries, improving search efficiency and speeding up purchasing decisions.

[0171] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0172] Step 1:

[0173] The user accesses cloud storage (e.g., Google Drive) from their device, selects materials or data, and uploads them. The server receives this request and retrieves the materials or data. The input is the received material data, and the output is the material data waiting to be converted.

[0174] Step 2:

[0175] The server converts the received documents and data into the specified format. For example, to convert a PDF file into a text file, PDF parsing software is used. The input is the received document data, and the output is the text format document data.

[0176] Step 3:

[0177] The server extracts metadata from the converted materials and data. For example, it analyzes and extracts information such as the document creation date, author, and file type. The input is text-formatted material data, and the output is material data with metadata added.

[0178] Step 4:

[0179] The server generates an index based on the extracted metadata and stores it in a database. An index generation algorithm is used to improve search performance. The input is the material data with metadata, and the output is the index and the material data with the index.

[0180] Step 5:

[0181] The user inputs and sends a query in natural language format (e.g., "What are the popular products this week?") from the terminal. The server receives this query. The input is the user's query in natural language, and the output is the query data waiting to be analyzed.

[0182] Step 6:

[0183] The server analyzes the received query and uses a generative AI model (e.g., GPT-4) to understand the intent of the question. This analysis generates a search query. The input is the query data, and the output is the generated search query.

[0184] Step 7:

[0185] The server searches the database using the search query to retrieve the corresponding materials and data. The input is the search query, and the output is the search result data.

[0186] Step 8:

[0187] The server uses a generative AI model to generate a response in natural language based on the search results. For example, it extracts related product information from the search results and creates a response in a format that is easy for the user to understand. The input is the search result data, and the output is a response in natural language.

[0188] Step 9:

[0189] The server generates a response and sends it to the user's terminal. The user checks the response on the terminal and obtains the necessary information. The input is a response in natural language format, and the output is a response message displayed on the terminal.

[0190] This series of steps allows users to quickly and accurately obtain materials and data (especially product information on online shopping sites) through natural language inquiries.

[0191] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0192] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[0193] The server receives documents and data uploaded by users. Users use their devices to access shared folders or cloud storage and select files to upload. The server receives the files and converts them into a format that is easy to handle internally. For example, converting a PDF file into text.

[0194] The server extracts metadata from the received data, including creation date, author, file type, etc., which is used to generate an index and stored in a database.

[0195] When a user inputs a query in natural language from their device, the server analyzes the query. An AI model (e.g., GPT or BERT) is used for the analysis. The emotion engine then recognizes the emotion contained in the user's query. For example, if a user inputs, "I'm very troubled by this problem. I want to know a solution as soon as possible," the emotion engine will detect that the user is confused.

[0196] The server searches the database based on the analysis results and retrieves relevant information. A response is generated based on the retrieved information, taking into account the emotion recognition results of the emotion engine. For example, if a user is in trouble, the server generates a response that includes encouraging words such as, "Don't worry. Here's some helpful information."

[0197] The generated response is sent from the server to the user's device. The user checks the response and plans or executes the next action based on the information obtained. For example, if Person B asks, "Please tell me the progress of the project this month," the server will extract progress data from the documents and respond, "Thank you for your hard work. Progress is currently at 70%."

[0198] The emotion engine stores user emotional data as a history and references it for future inquiries, allowing the system to continuously understand the user's state and provide more personalized responses.

[0199] This system operates in a cloud environment, allowing users to directly upload documents and data from shared folders or cloud storage, enabling the integrated management of distributed documents and data, and providing necessary information quickly and efficiently.

[0200] For example, Person A uploads an important meeting report to a shared folder, and Person B makes a query in natural language based on that report. The server uses an emotion engine to recognize Person B's emotion, generates a response in an appropriate tone, and sends it to Person B.

[0201] In this way, the present invention realizes an information gathering system that automates the management of materials and data and uses an emotion engine to provide information gathering in a more personalized, interactive manner using natural language.

[0202] The processing flow will be explained below.

[0203] Step 1:

[0204] The user uses a device to access a shared folder or cloud storage, selects the materials or data to upload, and clicks the "Upload" button on the interface to send the files to the system.

[0205] Step 2:

[0206] The device sends the selected materials and data to the server, along with metadata such as the file type and file path. Once the file upload is complete, the device displays a confirmation message on the server.

[0207] Step 3:

[0208] The server stores the documents and data received from the user. Depending on the type of data received, it converts it into a format that is easy to handle internally. For example, it runs a process to convert a PDF file into text.

[0209] Step 4:

[0210] The server extracts metadata from the converted data, such as creation date, creator, file type, etc. The extracted metadata is stored in a database and used to generate a search index.

[0211] Step 5:

[0212] The user inputs a query in natural language using a terminal. For example, a question such as "What is the progress of this month's project?" is input into the chat box and sent.

[0213] Step 6:

[0214] The terminal sends the user's query to the server, which receives the query and begins preparing it for analysis.

[0215] Step 7:

[0216] The server uses an AI model (e.g., GPT or BERT) to analyze the query, extract intent from the query, and identify relevant keywords.

[0217] Step 8:

[0218] The server uses an emotion engine to analyze the emotions contained in the user's query, for example, identifying emotions such as "confusion" or "urgency" from the user's text.

[0219] Step 9:

[0220] Based on the analysis results, the server searches the database for relevant materials and data, and extracts the information that best suits the query.

[0221] Step 10:

[0222] The server generates a response based on the search results and the analysis of the emotion engine. For example, if the user is confused, the server creates a response such as, "Don't worry, the following information may be helpful."

[0223] Step 11:

[0224] The server sends the generated response to the user's terminal, which then displays the response to the user, allowing the user to confirm the answer.

[0225] Step 12:

[0226] The user checks the response sent on the terminal and obtains the necessary information. For example, if user B asks "How is the project progressing this month?", he or she receives the response "Thank you for your hard work. The current progress is 70%."

[0227] Step 13:

[0228] The user's emotional information analyzed by the emotion engine is stored as historical data, which can be referenced during future inquiries to provide a more personalized response.

[0229] This series of processes allows users to easily and quickly access the materials and data they need and receive appropriate information along with responses that take their emotions into consideration, thereby streamlining the information gathering process and improving user satisfaction.

[0230] Example 2

[0231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Conventional information gathering systems generate uniform responses without considering the user's feelings, making it difficult to increase user satisfaction. In addition, the management of distributed materials and data is cumbersome, making it difficult to provide necessary information quickly and efficiently.

[0233] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0234] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving queries in natural language, means for analyzing the received queries and searching for corresponding materials and data, means for recognizing emotions based on the analysis results, means for adjusting and generating a response based on the emotion recognition results, and means for transmitting the generated response to a specified terminal. This makes it possible to provide personalized responses that take the user's emotions into consideration, efficiently manage distributed materials and data, and quickly respond to user requests.

[0235] "Materials and data" refers to various types of information, such as text, images, and video, that the system receives and processes.

[0236] "Means for receiving" refers to the function by which the server obtains materials and data uploaded by users.

[0237] "Means for converting into a specified format" refers to the function of analyzing received materials and data and converting them into a format that is easy to handle internally.

[0238] "Metadata" refers to attribute information of materials or data, including, for example, creation date, creator, file type, and the like.

[0239] "Means for extracting metadata and generating indexes" refers to the function of extracting metadata from materials and data and storing it in a database in an easily searchable format.

[0240] A "natural language query" refers to an input form in which a user requests information from a system using everyday language.

[0241] The "means for receiving a query" refers to a function that enables the server to receive a query in natural language input by a user.

[0242] "Means for analyzing inquiries and retrieving corresponding materials and data" refers to the function of using a generative AI model to understand natural language input and search for corresponding materials and data from a database.

[0243] The "means for recognizing emotions" refers to the function of the emotion engine for analyzing the user's emotions contained in the query.

[0244] "Means for adjusting and generating a response based on the emotion recognition result" refers to a function for creating a response in an appropriate tone based on the user's emotion recognized by the emotion engine.

[0245] The "means for transmitting a response to a predetermined terminal" refers to a function for delivering the generated response to the terminal used by the user.

[0246] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[0247] This system consists of a server, a device, cloud storage, a generative AI model, and an emotion engine. The specific roles and processes of each element are described below.

[0248] Hardware and software used

[0249] Hardware: Servers, user devices (PCs, smartphones, etc.)

[0250] Software: Cloud storage (Google Drive, Dropbox, etc.), generative AI models (GPT-3 (registered trademark), BERT), emotion engine

[0251] Details of data processing and calculation

[0252] 1. Upload your data

[0253] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[0254] After the upload is complete, the device sends a notification to the server.

[0255] The server receives this notification and checks for the existence of the new file.

[0256] 2. Data transformation and metadata extraction

[0257] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[0258] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index. For example, from "Project Progress Report.pdf" it obtains the metadata "Creation date: October 1, 2023, Creator: Person A, File type: PDF."

[0259] The server stores the generated indexes and data in a database.

[0260] 3. Natural Language Inquiry Analysis

[0261] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[0262] The terminal transmits the input query to the server.

[0263] The server receives this query and analyzes it using a generative AI model (e.g., GPT or BERT), extracting keywords such as "project progress."

[0264] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[0265] 4. Generating the Response

[0266] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[0267] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[0268] 5. Sending a response and notifying the user

[0269] The server sends the generated response to the user's terminal.

[0270] The device receives this response and notifies the user, for example, by displaying a pop-up notification or email with the message "Thank you for your hard work. Progress is currently at 70%."

[0271] 6. Emotion data history storage

[0272] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[0273] The server can refer to this history during future queries and use it to provide more personalized responses.

[0274] Examples of concrete examples and prompts

[0275] Examples:

[0276] A user uploads a file called "Project Progress Report.pdf" to Google Drive.

[0277] The server receives the file, converts it into text data, extracts the metadata "Creation date: October 1, 2023, Creator: Mr. A, File type: PDF", and stores it in a database.

[0278] When a user asks from their device, "Tell me about this month's project progress," the server uses the generated AI model to analyze and retrieve information about the "project progress."

[0279] The server uses an emotion engine to recognize that the user is confused, generates a response saying, "Thank you for your hard work. Progress is currently at 70%," and sends this to the user's device.

[0280] Finally, the server stores the user's emotional data as a history and uses it for future queries.

[0281] Example prompt sentence:

[0282] "Tell me how this month's project is going."

[0283] "I'm very troubled by this issue. I want to know what to do as soon as possible."

[0284] As described above, the present invention automates the management of materials and data, and utilizes an emotion engine to provide personalized information gathering in a natural language dialogue format.

[0285] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0286] Step 1:

[0287] Uploading data

[0288] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[0289] Input: A file (e.g. PDF) uploaded by the user.

[0290] Output: Files saved in cloud storage.

[0291] What happens: A user opens the Google Drive app or web interface, selects a file, and clicks the upload button.

[0292] Step 2:

[0293] Data transformation and metadata extraction

[0294] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[0295] Input: A file stored in cloud storage.

[0296] Output: The converted text data.

[0297] Specific operation: The server converts the PDF file into text using a PDF parser.

[0298] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index.

[0299] Input: The converted text data.

[0300] Output: Extracted metadata and generated index.

[0301] What happens: The server uses a text parser to extract metadata from the text data, retrieving information such as creation date, author, and file type.

[0302] Step 3:

[0303] Receiving natural language inquiries

[0304] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[0305] Input: A natural language query typed by the user (e.g., How is the project progressing this month?).

[0306] Output: Query data from the terminal to the server.

[0307] Specific actions: A user uses a chat app or web form on their device to enter an inquiry and click the send button.

[0308] Step 4:

[0309] Query analysis and emotion recognition

[0310] The server analyzes the received query and extracts keywords using a generative AI model (e.g., GPT or BERT).

[0311] Input: Query data sent from the terminal.

[0312] Output: Extracted keywords (e.g., project progress).

[0313] What happens: The server uses a natural language processing engine to analyze the query text and identify keywords.

[0314] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[0315] Input: The parsed query text.

[0316] Output: Recognized emotion data (e.g., distressed).

[0317] What it does: The server uses a sentiment analysis algorithm to read the user's emotions from the text.

[0318] Step 5:

[0319] Generate and send a response

[0320] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[0321] Input: Extracted keywords and metadata index.

[0322] Output: Retrieved information (e.g., progress 70%).

[0323] What happens: The server uses an SQL query to retrieve the relevant information from the database.

[0324] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[0325] Input: Retrieved information and sentiment data.

[0326] Output: The generated response message.

[0327] How it works: The server uses a template-based response generation system to generate responses by adding phrases that correspond to the emotion.

[0328] The server sends the generated response to the user's terminal.

[0329] Input: The generated response message.

[0330] Output: Data sent to the terminal.

[0331] Specific operation: The server uses network communication to send the generated response to the specified user terminal.

[0332] Step 6:

[0333] Emotion data history storage

[0334] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[0335] Input: Recognized emotion data.

[0336] Output: Emotion history stored in a database.

[0337] Specific operation: The server executes an SQL command to insert emotion data into the database.

[0338] (Application example 2)

[0339] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] Conventional information gathering systems have difficulty responding appropriately and quickly to inquiries made by users in natural language. They also lack the ability to recognize users' emotions and generate appropriate responses, making it difficult to improve user satisfaction. Food delivery services, in particular, are required to respond to a variety of user emotions, but no effective systems exist.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata of the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for recognizing emotions and adjusting the response based on the analysis results. This makes it possible to provide an appropriate and prompt response according to the user's emotions and improve user satisfaction.

[0342] "Materials and Data" means any document or information in any form that a User uploads to the System.

[0343] "Means for receiving" refers to the functions and mechanisms for taking information from users into the system.

[0344] "Means for converting into a format" refers to the functions and processes for converting received materials and data into a format that is easy for the system to handle.

[0345] "Metadata" is supplementary information about materials and data, including creation date, creator, file type, etc.

[0346] "Means for generating an index" refers to the function or process of creating an index based on the metadata of materials or data to facilitate searching.

[0347] "Natural language inquiries" refer to questions or requests that users make to the system in everyday language.

[0348] "Means for parsing" refers to the functions or techniques that process information to understand the received natural language query and determine an appropriate response.

[0349] "Searching means" refers to the functions and processes for locating relevant materials and data within the system based on an analyzed query.

[0350] "Means for generating a response" refers to the functions or processes that create a response to the user based on the search results or analysis results.

[0351] "Means for sending to a predetermined terminal" refers to the functionality and protocols for sending the generated response to the device used by the user.

[0352] "Means for recognizing emotions and adjusting responses based on the analysis results" refers to a function or engine that analyzes the emotions contained in a user's inquiry and changes the response in tone and content accordingly.

[0353] This invention provides an information collection system for improving the daily user experience of food delivery services. This system receives and manages documents and data, analyzes natural language inquiries, recognizes emotions, and generates optimal responses based on the results, which are then sent to a designated terminal.

[0354] Specifically, the following hardware and software are used.

[0355] Hardware and software used

[0356] Hardware: Smartphone (iOS or ANDROID device)

[0357] Backend servers: Node.js, Express.js, MySQL

[0358] Frontend: React Native

[0359] Generative model: OpenAI GPT-4

[0360] Emotion Recognition Model: Hugging Face Emotion Analysis Model

[0361] System program processing

[0362] 1. Means of receiving materials and data:

[0363] The server receives information and data from the user, including food delivery menu information and restaurant details.

[0364] 2. How to convert to the format:

[0365] The server converts the received materials and data into an easy-to-handle format (for example, text format) internally.

[0366] 3. How to extract metadata and generate indexes:

[0367] The server extracts metadata such as creation date, creator, and file type from the materials and data, generates an index based on that information, and stores it in a database.

[0368] 4. Means for receiving natural language queries:

[0369] Users use natural language to make inquiries through a smartphone app, for example, by typing, "Where is the best pizza today?"

[0370] 5. Analysis Method:

[0371] The server uses OpenAI GPT-4 to analyze and understand the received query, and at the same time, it uses the Hugging Face emotion analysis model to recognize the user's emotions.

[0372] 6. Search by:

[0373] The server searches the database based on the analysis results and index information to obtain the most appropriate materials and data.

[0374] 7. Means of generating a response:

[0375] Based on the acquired information, the system generates a response that reflects the emotion recognition results. For example, if the user is confused, the system will adjust the response to say, "Don't worry. We recommend the following pizza restaurant."

[0376] 8. Means of sending to a given terminal:

[0377] The final response is converted to JSON format and sent to the user's smartphone app, where the user can view the response.

[0378] Specific examples

[0379] 1. User Case:

[0380] A user asks, "I want to eat some good pizza, where can I order it?"

[0381] The server receives this query and performs natural language analysis and emotion recognition.

[0382] The server searches the indexed database for the best pizza place information.

[0383] If emotion recognition reveals that the user is confused, the system generates a response such as, "Don't worry, we have pizza recommendations from the restaurants below."

[0384] The response is sent to the user's smartphone, where they can view it in the app.

[0385] Prompt Sentence Examples

[0386] "Where can I order fried shrimp?"

[0387] "Can you recommend a Chinese restaurant?"

[0388] "Can you tell me about some cheap and delicious bento lunches around here?"

[0389] This invention allows users to receive appropriate responses that take their emotions into consideration when using food delivery services, improving overall satisfaction.

[0390] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0391] Step 1:

[0392] The server receives information and data from users. Specifically, users upload information such as menu items and restaurant details using a smartphone app. The input information and data is received by the server's backend (Node.js and Express.js).

[0393] Step 2:

[0394] The server analyzes the received documents and data and converts them into the specified format. This includes converting PDF files into text data, for example. Specifically, it converts the received data into a text format that is easy to handle internally.

[0395] Step 3:

[0396] The server extracts metadata about the documents and data and generates an index. This metadata includes creation date, author, file type, etc. This metadata is extracted and stored as an index in a MySQL database. The index allows for efficient searches later.

[0397] Step 4:

[0398] The server receives a natural language query that the user enters into the app. For example, the natural language query might be, "Where's the best pizza?" This query is then sent to the server.

[0399] Step 5:

[0400] The server analyzes the received query using an AI model (OpenAI GPT-4). At the same time, it recognizes the user's emotions using an emotion recognition model (Hugging Face emotion analysis model). The input query is analyzed to identify emotions such as whether the user is confused or expectant.

[0401] Step 6:

[0402] The server searches the database based on the analysis results and index information. It understands the query from the analysis results and searches the database (MySQL) for relevant information. For example, it searches for and retrieves information about pizza.

[0403] Step 7:

[0404] The server generates the optimal response based on the emotion recognition results. Specifically, it generates a response with a tone and content that matches the user's emotions based on the acquired information. For example, if the user is confused, it creates a response that gives a sense of reassurance.

[0405] Step 8:

[0406] The server sends the generated response to the specified device (smartphone app). The response is converted to JSON format and sent to the user's smartphone app. The user can check the response through the app.

[0407] This series of processes enables the system to provide quick and appropriate responses to user inquiries, thereby improving user satisfaction with food delivery services.

[0408] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0409] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0410] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0411] [Second embodiment]

[0412] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0413] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0414] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0415] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0416] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0417] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0418] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0419] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0420] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0421] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0422] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0423] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0424] The present invention relates to an information collection system that can efficiently manage materials and data and quickly acquire information through inquiries in natural language. The system includes the following means.

[0425] The server receives materials and data uploaded by users. For example, a user selects materials and data from a shared folder or cloud storage using a terminal and uploads them to the system. At the same time, the server receives format information for the materials and data and prepares to analyze the content of the received materials and data.

[0426] The server converts the received material or data into the specified format, for example, converting a PDF file to text, and the converted data is then ready for further analysis and indexing within the system.

[0427] The server then extracts metadata from the documents and data, creating an index. This metadata, such as creation date, author, and file type, is stored in a database to improve search performance, laying the foundation for users to quickly find the information they need later.

[0428] A user inputs a query in natural language from a terminal (e.g., "Tell me about the progress of this month's project"). This query is sent to a server. The server analyzes the received query and uses an AI model (e.g., GPT or BERT) to search for corresponding materials and data. After understanding the query and searching for relevant data, the server generates a response based on the information obtained. This response is expressed in natural language and is in a format that is easy for the user to understand.

[0429] The response is sent to the user's device, allowing the user to obtain the information immediately. For example, if user B asks, "What is the progress of this month's project?", the server extracts progress data from the project documents, generates a response saying, "This month's progress is 70% complete," and sends it to user B's device.

[0430] This system operates in a cloud environment and allows users to directly upload documents and data from shared folders or cloud storage. This allows documents and data to be managed in a unified manner even if they are scattered in different locations. It also facilitates information sharing, which has the effect of improving the knowledge level of the entire organization.

[0431] For example, Person A uploads project documents to Google Drive, the server receives the documents and stores them in a database, and Person B then makes a query in natural language. The server then analyzes the query, extracts the necessary information, and generates a response.

[0432] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[0433] The processing flow will be explained below.

[0434] Step 1:

[0435] The user accesses a shared folder or cloud storage from their device, selects the materials or data to upload, and then uses the designated interface to upload the selected files to the AI ​​Data Navigator.

[0436] Step 2:

[0437] When a user clicks the upload button on their device, the device sends the selected materials and data to the server, along with metadata such as the file type and file path.

[0438] Step 3:

[0439] The server receives the materials and data sent by the user. After receiving the data, the server determines the file type and converts it to the specified format (e.g., converting PDF to text) if necessary.

[0440] Step 4:

[0441] The server extracts metadata from the converted data, such as creation date, author, file type, etc. This metadata is stored in a database, generating an index for efficient subsequent searches.

[0442] Step 5:

[0443] A user uses a terminal to input a query in natural language, for example, a specific question such as "How is the project progressing this month?" The user sends this query to the server.

[0444] Step 6:

[0445] The server parses the incoming natural language query, using an AI model (e.g., GPT or BERT) to understand the meaning of the query and identify the appropriate database entry.

[0446] Step 7:

[0447] Based on the analysis results, the server retrieves relevant materials and data from the database, which is then further analyzed as needed and used to generate an appropriate response to the query.

[0448] Step 8:

[0449] The server generates a natural language response based on the search results, such as a specific answer like "Last month's progress was 50%, and this month's progress is 70% complete."

[0450] Step 9:

[0451] The server then sends the generated response to the user's terminal, where the user immediately receives the response and obtains the information in response to the query.

[0452] Step 10:

[0453] Users can check the responses sent on their devices and obtain the necessary information. For example, when Person B checks the "This month's project progress," he can plan his next action based on the displayed information.

[0454] This series of processes allows users to easily and quickly access the necessary materials and data and obtain appropriate information.

[0455] Example 1

[0456] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0457] Traditional methods for managing materials and data make it difficult to uniformly manage data in different formats, making them inefficient for searching and retrieving. They also lacked a system that could quickly provide appropriate information in response to natural language queries, forcing users to go through multiple steps to find the information they needed. Furthermore, they lacked the functionality for uploading and managing data in a cloud environment, making it difficult to centrally handle materials from multiple data sources.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0459] In this invention, the server includes: means for receiving documents and data; means for converting the received documents and data into a specified format; means for extracting metadata from the documents and data and generating an index; means for receiving queries in natural language; means for analyzing the received queries and searching for corresponding documents and data; means for generating a response based on the search results and transmitting it to a specified terminal; means for receiving format information for the documents and data; means for performing format conversion using OCR technology; means for analyzing the query content using a generative AI model; and means for expressing the generated response in natural language. This allows users to centrally manage and search documents and data in different formats, enabling prompt and appropriate information provision based on natural language queries. It also facilitates data uploading and automatic management in a cloud environment, enabling efficient management and use of documents and data.

[0460] "Means for receiving materials and data" refers to the function of receiving data files of various formats sent by users via the network to the server.

[0461] "Means for converting received material or data into a specified format" means software or algorithms for converting received material or data into a consistent form.

[0462] "Means for extracting metadata of materials and data and generating indexes" refers to the ability to extract important attribute information from data files (e.g., creation date, creator, file type) and store it in a searchable format.

[0463] "Means for receiving inquiries in natural language" refers to the function of the server receiving questions or requests entered by the user in natural language.

[0464] "Means of analyzing received inquiries and searching for corresponding materials and data" refers to the ability to use technologies such as generative AI models to understand the content of inquiries and search for relevant materials and databases.

[0465] "Means for generating a response based on the search results and sending it to a specified terminal" refers to the function of generating a response in natural language based on the information obtained from the search and sending it to the user's terminal.

[0466] "Means for receiving format information of materials or data" refers to a function for receiving information about the format of received materials or data (e.g., file type, number of pages, size).

[0467] "Means for format conversion using OCR technology" refers to a function that uses OCR (optical character recognition) technology to convert characters in images or PDFs into text.

[0468] "Means of analyzing the content of inquiries using a generative AI model" refers to the function of understanding and analyzing the content of user inquiries using a generative AI model capable of natural language processing (e.g., GPT, BERT).

[0469] "Means for expressing the generated response in natural language" refers to the ability to generate answers to queries in natural language that is easy for humans to understand.

[0470] The present invention relates to an information collection system that improves the efficiency of managing materials and data and enables rapid acquisition of information through inquiries in natural language. This system is mainly composed of a server, terminals, and users.

[0471] The server has a means for receiving materials and data uploaded by users. Specifically, when a user uses a terminal to select materials or data from a shared folder or cloud storage and uploads them to the system, the server receives these materials and data via an HTTP request. The server also simultaneously receives format information for the materials and data, and uses this information to prepare for analysis.

[0472] The received material or data is converted into the specified format using OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six), and the converted data is put into text format, ready for further analysis and indexing within the system.

[0473] The server then has the means to extract metadata about the materials and data and generate an index. Extracted metadata includes creation date, author, file type, important keywords, etc. This metadata is extracted using machine learning models (e.g., SpaCy) and stored in a database (e.g., MySQL, Elasticsearch). This provides a foundation for users to quickly search for the information they need later.

[0474] The user inputs a query in natural language from their device and sends it to the server. An example query might be, "What's the progress on this month's project?" The server receives the query and analyzes it using a generative AI model (e.g., GPT-4, BERT). Based on the analyzed query, the server searches for appropriate materials and data.

[0475] The server generates a response in natural language based on the search results. The response is in a format that is easy for the user to understand. For example, it may generate a response such as "This month's progress is 70% complete" and send it to the user's device.

[0476] This system operates in a cloud environment and provides a means for users to directly upload documents and data from shared folders or cloud storage, allowing documents and data to be managed in a unified manner even if they are scattered in different locations, facilitating information sharing.

[0477] As a concrete example, Person A uploads project materials to cloud storage, and the server receives and stores them in a database. If Person B then makes a natural language inquiry such as "Tell me about this month's project progress," the server analyzes the inquiry using a generative AI model, extracts the necessary information, generates a response, and sends it to Person B's device.

[0478] An example prompt is:

[0479] "Receive uploaded materials from the shared folder and record the format information."

[0480] "Convert the uploaded PDF file into text data and create an index."

[0481] "Parse queries entered in natural language, search for relevant data, and generate responses."

[0482] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[0483] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0484] Step 1:

[0485] Uploading data and receiving formatting information

[0486] Users use their devices to select materials and data from shared folders or cloud storage and upload them to the system.

[0487] The server receives an upload request from the user as an HTTP request, saves the materials and data in temporary storage on the server, and simultaneously records the format information of the materials and data (e.g., PDF, Word, Excel, etc.) as metadata.

[0488] Input: Files of materials and data uploaded by users

[0489] Output: Files and their metadata stored in temporary storage

[0490] Step 2:

[0491] Data format conversion

[0492] The server converts the received material or data into the specified format. For example, the server may use OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six) to convert a PDF file into text data.

[0493] Input: Files and metadata of materials and data stored in temporary storage

[0494] Output: A file converted to text data

[0495] Step 3:

[0496] Metadata extraction and indexing

[0497] The server extracts metadata from documents and data and generates an index. To do this, it uses machine learning models (e.g., SpaCy) to extract important information (e.g., creation date, author, keywords, etc.) and stores this information in a database (e.g., MySQL, Elasticsearch).

[0498] Input: A file converted to text data

[0499] Output: Metadata and index information stored in a database

[0500] Step 4:

[0501] Receiving user inquiries

[0502] The user inputs a query in natural language (e.g., "Please tell me about the progress of this month's project") from the terminal and sends it to the system.

[0503] The server receives this query as an HTTP POST request and temporarily stores its contents.

[0504] Input: A natural language query entered by the user

[0505] Output: Query content saved in temporary storage

[0506] Step 5:

[0507] Inquiry analysis and data search

[0508] The server analyzes the received query using a generative AI model (e.g., GPT-4, BERT), and uses the index information to search for relevant materials and data based on the analysis.

[0509] Input: Query content stored in temporary storage and index information stored in the database

[0510] Output: Analysis results and searched materials and data

[0511] Step 6:

[0512] Response generation and transmission

[0513] The server generates a response in natural language based on the search results using a generative AI model. The generated response is sent to the user's device as an HTTP response, where the user can view the response.

[0514] Input: Analysis results and searched materials and data

[0515] Output: A natural language response and the response sent to the user's device

[0516] Through these steps, the system enables management of materials and data, and rapid information search and provision using natural language.

[0517] (Application example 1)

[0518] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0519] In conventional information gathering systems, users had to go through multiple steps to obtain the specific materials or data they were looking for, which reduced search efficiency. Furthermore, online shopping sites lacked a means for users to quickly obtain product information, which led to the problem of taking a long time to make a purchasing decision.

[0520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0521] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for providing an assistant function that allows users to easily collect product information. This allows users to quickly and accurately obtain product information through inquiries in natural language, thereby improving search efficiency and speeding up purchasing decisions.

[0522] "Materials and Data" refers to information and content uploaded by Users, including in a variety of formats such as PDF, text, and images.

[0523] "Means for receiving" refers to the function for transferring materials and data from the user to the server.

[0524] "Means for converting into a specified format" refers to a function that processes received materials or data to convert them into a specific format.

[0525] "Means for extracting metadata and generating indexes" refers to the function of extracting attribute information of materials and data and creating indexes based on that information to facilitate searches.

[0526] "Means for receiving inquiries in natural language" refers to a function for receiving questions or requests from users in natural language format.

[0527] "Means for searching for corresponding materials and data" refers to the function of searching for related materials and data from a database based on a received inquiry.

[0528] "Means for generating a response and sending it to a specified terminal" refers to the function of creating an answer based on the search results and sending it to the user's terminal.

[0529] "Means for providing assistance functions" refers to functions that assist users in easily gathering product information, particularly on online platforms such as shopping sites.

[0530] This invention provides an information collection system that improves the efficiency of document and data management and enables rapid information acquisition through natural language inquiries. This system includes functions for receiving documents and data, format conversion, metadata extraction, index generation, natural language analysis, data search, response generation and transmission, and assistant functions.

[0531] Hardware and software used

[0532] Cloud storage: Use Google Drive or Amazon S3 to store materials and data.

[0533] AI model: OpenAI's GPT-4 is used for question analysis and answer generation.

[0534] Database: MySQL and MongoDB are used to manage metadata and product information.

[0535] API: RESTful API is used for communication between the application and the server.

[0536] Frontend: Develop a cross-platform smartphone app using React Native.

[0537] Processing flow

[0538] 1. Uploading data:

[0539] The server receives the upload of materials and data selected by the user from cloud storage (e.g., Google Drive). This includes the step of specifying and sending materials and data through an interface as an operation performed by the user in the application.

[0540] 2. Data Conversion and Storage:

[0541] The server converts the received materials and data into the specified format (e.g. PDF to text), and the converted data is stored in a database.

[0542] 3. Metadata extraction and indexing:

[0543] The server extracts metadata (e.g., creation date, creator, product category) from the materials and data, and generates an index, which is stored in a database to enable efficient searches.

[0544] 4. Natural Language Analysis and Data Retrieval:

[0545] The user sends a query in natural language from the application (e.g., "What are the popular products this week?"), and the server analyzes the query using the GPT-4 model and searches the database for corresponding materials and data.

[0546] 5. Response generation and transmission:

[0547] Based on the search results, the server generates a response in natural language format (e.g., "This week's popular products are: Product A, Product B, Product C.") and sends it to the user's terminal.

[0548] Specific examples

[0549] For example, a user opens a smartphone app and uses the chatbot function to type, "What are the popular products this week?" The server receives the request and uses the GPT-4 model to parse the question. It then retrieves information about this week's popular products from a database and uses the GPT-4 model to generate a response for the user. The response is then displayed on the user's smartphone.

[0550] Prompt Sentence Examples

[0551] An example prompt is:

[0552] User Question: "What are the popular items this week?"

[0553] The database contains:

[0554] Popular Item 1: Item A, Other Information...

[0555] Popular Item 2: Item B, Other Information...

[0556] Generated response: "This week's popular products are: Product A, Product B, Product C."

[0557] This system allows users to quickly and accurately obtain product information through natural language inquiries, improving search efficiency and speeding up purchasing decisions.

[0558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0559] Step 1:

[0560] The user accesses cloud storage (e.g., Google Drive) from their device, selects materials or data, and uploads them. The server receives this request and retrieves the materials or data. The input is the received material data, and the output is the material data waiting to be converted.

[0561] Step 2:

[0562] The server converts the received documents and data into the specified format. For example, to convert a PDF file into a text file, PDF parsing software is used. The input is the received document data, and the output is the text format document data.

[0563] Step 3:

[0564] The server extracts metadata from the converted materials and data. For example, it analyzes and extracts information such as the document creation date, author, and file type. The input is text-formatted material data, and the output is material data with metadata added.

[0565] Step 4:

[0566] The server generates an index based on the extracted metadata and stores it in a database. An index generation algorithm is used to improve search performance. The input is the material data with metadata, and the output is the index and the material data with the index.

[0567] Step 5:

[0568] The user inputs and sends a query in natural language format (e.g., "What are the popular products this week?") from the terminal. The server receives this query. The input is the user's query in natural language, and the output is the query data waiting to be analyzed.

[0569] Step 6:

[0570] The server analyzes the received query and uses a generative AI model (e.g., GPT-4) to understand the intent of the question. This analysis generates a search query. The input is the query data, and the output is the generated search query.

[0571] Step 7:

[0572] The server searches the database using the search query to retrieve the corresponding materials and data. The input is the search query, and the output is the search result data.

[0573] Step 8:

[0574] The server uses a generative AI model to generate a response in natural language based on the search results. For example, it extracts related product information from the search results and creates a response in a format that is easy for the user to understand. The input is the search result data, and the output is a response in natural language.

[0575] Step 9:

[0576] The server generates a response and sends it to the user's terminal. The user checks the response on the terminal and obtains the necessary information. The input is a response in natural language format, and the output is a response message displayed on the terminal.

[0577] This series of steps allows users to quickly and accurately obtain materials and data (especially product information on online shopping sites) through natural language inquiries.

[0578] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0579] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[0580] The server receives documents and data uploaded by users. Users use their devices to access shared folders or cloud storage and select files to upload. The server receives the files and converts them into a format that is easy to handle internally. For example, converting a PDF file into text.

[0581] The server extracts metadata from the received data, including creation date, author, file type, etc., which is used to generate an index and stored in a database.

[0582] When a user inputs a query in natural language from their device, the server analyzes the query. An AI model (e.g., GPT or BERT) is used for the analysis. The emotion engine then recognizes the emotion contained in the user's query. For example, if a user inputs, "I'm very troubled by this problem. I want to know a solution as soon as possible," the emotion engine will detect that the user is confused.

[0583] The server searches the database based on the analysis results and retrieves relevant information. A response is generated based on the retrieved information, taking into account the emotion recognition results of the emotion engine. For example, if a user is in trouble, the server generates a response that includes encouraging words such as, "Don't worry. Here's some helpful information."

[0584] The generated response is sent from the server to the user's device. The user checks the response and plans or executes the next action based on the information obtained. For example, if Person B asks, "Please tell me the progress of the project this month," the server will extract progress data from the documents and respond, "Thank you for your hard work. Progress is currently at 70%."

[0585] The emotion engine stores user emotional data as a history and references it for future inquiries, allowing the system to continuously understand the user's state and provide more personalized responses.

[0586] This system operates in a cloud environment, allowing users to directly upload documents and data from shared folders or cloud storage, enabling the integrated management of distributed documents and data, and providing necessary information quickly and efficiently.

[0587] For example, Person A uploads an important meeting report to a shared folder, and Person B makes a query in natural language based on that report. The server uses an emotion engine to recognize Person B's emotion, generates a response in an appropriate tone, and sends it to Person B.

[0588] In this way, the present invention realizes an information gathering system that automates the management of materials and data and uses an emotion engine to provide information gathering in a more personalized, interactive manner using natural language.

[0589] The processing flow will be explained below.

[0590] Step 1:

[0591] The user uses a device to access a shared folder or cloud storage, selects the materials or data to upload, and clicks the "Upload" button on the interface to send the files to the system.

[0592] Step 2:

[0593] The device sends the selected materials and data to the server, along with metadata such as the file type and file path. Once the file upload is complete, the device displays a confirmation message on the server.

[0594] Step 3:

[0595] The server stores the documents and data received from the user. Depending on the type of data received, it converts it into a format that is easy to handle internally. For example, it runs a process to convert a PDF file into text.

[0596] Step 4:

[0597] The server extracts metadata from the converted data, such as creation date, creator, file type, etc. The extracted metadata is stored in a database and used to generate a search index.

[0598] Step 5:

[0599] The user inputs a query in natural language using a terminal. For example, a question such as "What is the progress of this month's project?" is input into the chat box and sent.

[0600] Step 6:

[0601] The terminal sends the user's query to the server, which receives the query and begins preparing it for analysis.

[0602] Step 7:

[0603] The server uses an AI model (e.g., GPT or BERT) to analyze the query, extract intent from the query, and identify relevant keywords.

[0604] Step 8:

[0605] The server uses an emotion engine to analyze the emotions contained in the user's query, for example, identifying emotions such as "confusion" or "urgency" from the user's text.

[0606] Step 9:

[0607] Based on the analysis results, the server searches the database for relevant materials and data, and extracts the information that best suits the query.

[0608] Step 10:

[0609] The server generates a response based on the search results and the analysis of the emotion engine. For example, if the user is confused, the server creates a response such as, "Don't worry, the following information may be helpful."

[0610] Step 11:

[0611] The server sends the generated response to the user's terminal, which then displays the response to the user, allowing the user to confirm the answer.

[0612] Step 12:

[0613] The user checks the response sent on the terminal and obtains the necessary information. For example, if user B asks "How is the project progressing this month?", he or she receives the response "Thank you for your hard work. The current progress is 70%."

[0614] Step 13:

[0615] The user's emotional information analyzed by the emotion engine is stored as historical data, which can be referenced during future inquiries to provide a more personalized response.

[0616] This series of processes allows users to easily and quickly access the materials and data they need and receive appropriate information along with responses that take their emotions into consideration, thereby streamlining the information gathering process and improving user satisfaction.

[0617] Example 2

[0618] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0619] Conventional information gathering systems generate uniform responses without considering the user's feelings, making it difficult to increase user satisfaction. In addition, the management of distributed materials and data is cumbersome, making it difficult to provide necessary information quickly and efficiently.

[0620] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0621] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving queries in natural language, means for analyzing the received queries and searching for corresponding materials and data, means for recognizing emotions based on the analysis results, means for adjusting and generating a response based on the emotion recognition results, and means for transmitting the generated response to a specified terminal. This makes it possible to provide personalized responses that take the user's emotions into consideration, efficiently manage distributed materials and data, and quickly respond to user requests.

[0622] "Materials and data" refers to various types of information, such as text, images, and video, that the system receives and processes.

[0623] "Means for receiving" refers to the function by which the server obtains materials and data uploaded by users.

[0624] "Means for converting into a specified format" refers to the function of analyzing received materials and data and converting them into a format that is easy to handle internally.

[0625] "Metadata" refers to attribute information of materials or data, including, for example, creation date, creator, file type, and the like.

[0626] "Means for extracting metadata and generating indexes" refers to the function of extracting metadata from materials and data and storing it in a database in an easily searchable format.

[0627] A "natural language query" refers to an input form in which a user requests information from a system using everyday language.

[0628] The "means for receiving a query" refers to a function that enables the server to receive a query in natural language input by a user.

[0629] "Means for analyzing inquiries and retrieving corresponding materials and data" refers to the function of using a generative AI model to understand natural language input and search for corresponding materials and data from a database.

[0630] The "means for recognizing emotions" refers to the function of the emotion engine for analyzing the user's emotions contained in the query.

[0631] "Means for adjusting and generating a response based on the emotion recognition result" refers to a function for creating a response in an appropriate tone based on the user's emotion recognized by the emotion engine.

[0632] The "means for transmitting a response to a predetermined terminal" refers to a function for delivering the generated response to the terminal used by the user.

[0633] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[0634] This system consists of a server, a device, cloud storage, a generative AI model, and an emotion engine. The specific roles and processes of each element are described below.

[0635] Hardware and software used

[0636] Hardware: Servers, user devices (PCs, smartphones, etc.)

[0637] Software: Cloud storage (Google Drive, Dropbox, etc.), generative AI models (GPT-3, BERT), sentiment engine

[0638] Details of data processing and calculation

[0639] 1. Upload your data

[0640] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[0641] After the upload is complete, the device sends a notification to the server.

[0642] The server receives this notification and checks for the existence of the new file.

[0643] 2. Data transformation and metadata extraction

[0644] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[0645] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index. For example, from "Project Progress Report.pdf" it obtains the metadata "Creation date: October 1, 2023, Creator: Person A, File type: PDF."

[0646] The server stores the generated indexes and data in a database.

[0647] 3. Natural Language Inquiry Analysis

[0648] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[0649] The terminal transmits the input query to the server.

[0650] The server receives this query and analyzes it using a generative AI model (e.g., GPT or BERT), extracting keywords such as "project progress."

[0651] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[0652] 4. Generating the Response

[0653] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[0654] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[0655] 5. Sending a response and notifying the user

[0656] The server sends the generated response to the user's terminal.

[0657] The device receives this response and notifies the user, for example, by displaying a pop-up notification or email with the message "Thank you for your hard work. Progress is currently at 70%."

[0658] 6. Emotion data history storage

[0659] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[0660] The server can refer to this history during future queries and use it to provide more personalized responses.

[0661] Examples of concrete examples and prompts

[0662] Examples:

[0663] A user uploads a file called "Project Progress Report.pdf" to Google Drive.

[0664] The server receives the file, converts it into text data, extracts the metadata "Creation date: October 1, 2023, Creator: Mr. A, File type: PDF", and stores it in a database.

[0665] When a user asks from their device, "Tell me about this month's project progress," the server uses the generated AI model to analyze and retrieve information about the "project progress."

[0666] The server uses an emotion engine to recognize that the user is confused, generates a response saying, "Thank you for your hard work. Progress is currently at 70%," and sends this to the user's device.

[0667] Finally, the server stores the user's emotional data as a history and uses it for future queries.

[0668] Example prompt sentence:

[0669] "Tell me how this month's project is going."

[0670] "I'm very troubled by this issue. I want to know what to do as soon as possible."

[0671] As described above, the present invention automates the management of materials and data, and utilizes an emotion engine to provide personalized information gathering in a natural language dialogue format.

[0672] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0673] Step 1:

[0674] Uploading data

[0675] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[0676] Input: A file (e.g. PDF) uploaded by the user.

[0677] Output: Files saved in cloud storage.

[0678] What happens: A user opens the Google Drive app or web interface, selects a file, and clicks the upload button.

[0679] Step 2:

[0680] Data transformation and metadata extraction

[0681] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[0682] Input: A file stored in cloud storage.

[0683] Output: The converted text data.

[0684] Specific operation: The server converts the PDF file into text using a PDF parser.

[0685] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index.

[0686] Input: The converted text data.

[0687] Output: Extracted metadata and generated index.

[0688] What happens: The server uses a text parser to extract metadata from the text data, retrieving information such as creation date, author, and file type.

[0689] Step 3:

[0690] Receiving natural language inquiries

[0691] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[0692] Input: A natural language query typed by the user (e.g., How is the project progressing this month?).

[0693] Output: Query data from the terminal to the server.

[0694] Specific actions: A user uses a chat app or web form on their device to enter an inquiry and click the send button.

[0695] Step 4:

[0696] Query analysis and emotion recognition

[0697] The server analyzes the received query and extracts keywords using a generative AI model (e.g., GPT or BERT).

[0698] Input: Query data sent from the terminal.

[0699] Output: Extracted keywords (e.g., project progress).

[0700] What happens: The server uses a natural language processing engine to analyze the query text and identify keywords.

[0701] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[0702] Input: The parsed query text.

[0703] Output: Recognized emotion data (e.g., distressed).

[0704] What it does: The server uses a sentiment analysis algorithm to read the user's emotions from the text.

[0705] Step 5:

[0706] Generate and send a response

[0707] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[0708] Input: Extracted keywords and metadata index.

[0709] Output: Retrieved information (e.g., progress 70%).

[0710] What happens: The server uses an SQL query to retrieve the relevant information from the database.

[0711] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[0712] Input: Retrieved information and sentiment data.

[0713] Output: The generated response message.

[0714] How it works: The server uses a template-based response generation system to generate responses by adding phrases that correspond to the emotion.

[0715] The server sends the generated response to the user's terminal.

[0716] Input: The generated response message.

[0717] Output: Data sent to the terminal.

[0718] Specific operation: The server uses network communication to send the generated response to the specified user terminal.

[0719] Step 6:

[0720] Emotion data history storage

[0721] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[0722] Input: Recognized emotion data.

[0723] Output: Emotion history stored in a database.

[0724] Specific operation: The server executes an SQL command to insert emotion data into the database.

[0725] (Application example 2)

[0726] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0727] Conventional information gathering systems have difficulty responding appropriately and quickly to inquiries made by users in natural language. They also lack the ability to recognize users' emotions and generate appropriate responses, making it difficult to improve user satisfaction. Food delivery services, in particular, are required to respond to a variety of user emotions, but no effective systems exist.

[0728] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata of the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for recognizing emotions and adjusting the response based on the analysis results. This makes it possible to provide an appropriate and prompt response according to the user's emotions and improve user satisfaction.

[0729] "Materials and Data" means any document or information in any form that a User uploads to the System.

[0730] "Means for receiving" refers to the functions and mechanisms for taking information from users into the system.

[0731] "Means for converting into a format" refers to the functions and processes for converting received materials and data into a format that is easy for the system to handle.

[0732] "Metadata" is supplementary information about materials and data, including creation date, creator, file type, etc.

[0733] "Means for generating an index" refers to the function or process of creating an index based on the metadata of materials or data to facilitate searching.

[0734] "Natural language inquiries" refer to questions or requests that users make to the system in everyday language.

[0735] "Means for parsing" refers to the functions or techniques that process information to understand the received natural language query and determine an appropriate response.

[0736] "Searching means" refers to the functions and processes for locating relevant materials and data within the system based on an analyzed query.

[0737] "Means for generating a response" refers to the functions or processes that create a response to the user based on the search results or analysis results.

[0738] "Means for sending to a predetermined terminal" refers to the functionality and protocols for sending the generated response to the device used by the user.

[0739] "Means for recognizing emotions and adjusting responses based on the analysis results" refers to a function or engine that analyzes the emotions contained in a user's inquiry and changes the response in tone and content accordingly.

[0740] This invention provides an information collection system for improving the daily user experience of food delivery services. This system receives and manages documents and data, analyzes natural language inquiries, recognizes emotions, and generates optimal responses based on the results, which are then sent to a designated terminal.

[0741] Specifically, the following hardware and software are used.

[0742] Hardware and software used

[0743] Hardware: Smartphone (iOS or Android device)

[0744] Backend servers: Node.js, Express.js, MySQL

[0745] Frontend: React Native

[0746] Generative model: OpenAI GPT-4

[0747] Emotion Recognition Model: Hugging Face Emotion Analysis Model

[0748] System program processing

[0749] 1. Means of receiving materials and data:

[0750] The server receives information and data from the user, including food delivery menu information and restaurant details.

[0751] 2. How to convert to the format:

[0752] The server converts the received materials and data into an easy-to-handle format (for example, text format) internally.

[0753] 3. How to extract metadata and generate indexes:

[0754] The server extracts metadata such as creation date, creator, and file type from the materials and data, generates an index based on that information, and stores it in a database.

[0755] 4. Means for receiving natural language queries:

[0756] Users use natural language to make inquiries through a smartphone app, for example, by typing, "Where is the best pizza today?"

[0757] 5. Analysis Method:

[0758] The server uses OpenAI GPT-4 to analyze and understand the received query, and at the same time, it uses the Hugging Face emotion analysis model to recognize the user's emotions.

[0759] 6. Search by:

[0760] The server searches the database based on the analysis results and index information to obtain the most appropriate materials and data.

[0761] 7. Means of generating a response:

[0762] Based on the acquired information, the system generates a response that reflects the emotion recognition results. For example, if the user is confused, the system will adjust the response to say, "Don't worry. We recommend the following pizza restaurant."

[0763] 8. Means of sending to a given terminal:

[0764] The final response is converted to JSON format and sent to the user's smartphone app, where the user can view the response.

[0765] Specific examples

[0766] 1. User Case:

[0767] A user asks, "I want to eat some good pizza, where can I order it?"

[0768] The server receives this query and performs natural language analysis and emotion recognition.

[0769] The server searches the indexed database for the best pizza place information.

[0770] If emotion recognition reveals that the user is confused, the system generates a response such as, "Don't worry, we have pizza recommendations from the restaurants below."

[0771] The response is sent to the user's smartphone, where they can view it in the app.

[0772] Prompt Sentence Examples

[0773] "Where can I order fried shrimp?"

[0774] "Can you recommend a Chinese restaurant?"

[0775] "Can you tell me about some cheap and delicious bento lunches around here?"

[0776] This invention allows users to receive appropriate responses that take their emotions into consideration when using food delivery services, improving overall satisfaction.

[0777] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0778] Step 1:

[0779] The server receives information and data from users. Specifically, users upload information such as menu items and restaurant details using a smartphone app. The input information and data is received by the server's backend (Node.js and Express.js).

[0780] Step 2:

[0781] The server analyzes the received documents and data and converts them into the specified format. This includes converting PDF files into text data, for example. Specifically, it converts the received data into a text format that is easy to handle internally.

[0782] Step 3:

[0783] The server extracts metadata about the documents and data and generates an index. This metadata includes creation date, author, file type, etc. This metadata is extracted and stored as an index in a MySQL database. The index allows for efficient searches later.

[0784] Step 4:

[0785] The server receives a natural language query that the user enters into the app. For example, the natural language query might be, "Where's the best pizza?" This query is then sent to the server.

[0786] Step 5:

[0787] The server analyzes the received query using an AI model (OpenAI GPT-4). At the same time, it recognizes the user's emotions using an emotion recognition model (Hugging Face emotion analysis model). The input query is analyzed to identify emotions such as whether the user is confused or expectant.

[0788] Step 6:

[0789] The server searches the database based on the analysis results and index information. It understands the query from the analysis results and searches the database (MySQL) for relevant information. For example, it searches for and retrieves information about pizza.

[0790] Step 7:

[0791] The server generates the optimal response based on the emotion recognition results. Specifically, it generates a response with a tone and content that matches the user's emotions based on the acquired information. For example, if the user is confused, it creates a response that gives a sense of reassurance.

[0792] Step 8:

[0793] The server sends the generated response to the specified device (smartphone app). The response is converted to JSON format and sent to the user's smartphone app. The user can check the response through the app.

[0794] This series of processes enables the system to provide quick and appropriate responses to user inquiries, thereby improving user satisfaction with food delivery services.

[0795] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0796] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0797] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0798] [Third embodiment]

[0799] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0800] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0801] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0802] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0803] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0804] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0805] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0806] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0807] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0808] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0809] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0810] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0811] The present invention relates to an information collection system that can efficiently manage materials and data and quickly acquire information through inquiries in natural language. The system includes the following means.

[0812] The server receives materials and data uploaded by users. For example, a user selects materials and data from a shared folder or cloud storage using a terminal and uploads them to the system. At the same time, the server receives format information for the materials and data and prepares to analyze the content of the received materials and data.

[0813] The server converts the received material or data into the specified format, for example, converting a PDF file to text, and the converted data is then ready for further analysis and indexing within the system.

[0814] The server then extracts metadata from the documents and data, creating an index. This metadata, such as creation date, author, and file type, is stored in a database to improve search performance, laying the foundation for users to quickly find the information they need later.

[0815] A user inputs a query in natural language from a terminal (e.g., "Tell me about the progress of this month's project"). This query is sent to a server. The server analyzes the received query and uses an AI model (e.g., GPT or BERT) to search for corresponding materials and data. After understanding the query and searching for relevant data, the server generates a response based on the information obtained. This response is expressed in natural language and is in a format that is easy for the user to understand.

[0816] The response is sent to the user's device, allowing the user to obtain the information immediately. For example, if user B asks, "What is the progress of this month's project?", the server extracts progress data from the project documents, generates a response saying, "This month's progress is 70% complete," and sends it to user B's device.

[0817] This system operates in a cloud environment and allows users to directly upload documents and data from shared folders or cloud storage. This allows documents and data to be managed in a unified manner even if they are scattered in different locations. It also facilitates information sharing, which has the effect of improving the knowledge level of the entire organization.

[0818] For example, Person A uploads project documents to Google Drive, the server receives the documents and stores them in a database, and Person B then makes a query in natural language. The server then analyzes the query, extracts the necessary information, and generates a response.

[0819] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[0820] The processing flow will be explained below.

[0821] Step 1:

[0822] The user accesses a shared folder or cloud storage from their device, selects the materials or data to upload, and then uses the designated interface to upload the selected files to the AI ​​Data Navigator.

[0823] Step 2:

[0824] When a user clicks the upload button on their device, the device sends the selected materials and data to the server, along with metadata such as the file type and file path.

[0825] Step 3:

[0826] The server receives the materials and data sent by the user. After receiving the data, the server determines the file type and converts it to the specified format (e.g., converting PDF to text) if necessary.

[0827] Step 4:

[0828] The server extracts metadata from the converted data, such as creation date, author, file type, etc. This metadata is stored in a database, generating an index for efficient subsequent searches.

[0829] Step 5:

[0830] A user uses a terminal to input a query in natural language, for example, a specific question such as "How is the project progressing this month?" The user sends this query to the server.

[0831] Step 6:

[0832] The server parses the incoming natural language query, using an AI model (e.g., GPT or BERT) to understand the meaning of the query and identify the appropriate database entry.

[0833] Step 7:

[0834] Based on the analysis results, the server retrieves relevant materials and data from the database, which is then further analyzed as needed and used to generate an appropriate response to the query.

[0835] Step 8:

[0836] The server generates a natural language response based on the search results, such as a specific answer like "Last month's progress was 50%, and this month's progress is 70% complete."

[0837] Step 9:

[0838] The server then sends the generated response to the user's terminal, where the user immediately receives the response and obtains the information in response to the query.

[0839] Step 10:

[0840] Users can check the responses sent on their devices and obtain the necessary information. For example, when Person B checks the "This month's project progress," he can plan his next action based on the displayed information.

[0841] This series of processes allows users to easily and quickly access the necessary materials and data and obtain appropriate information.

[0842] Example 1

[0843] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0844] Traditional methods for managing materials and data make it difficult to uniformly manage data in different formats, making them inefficient for searching and retrieving. They also lacked a system that could quickly provide appropriate information in response to natural language queries, forcing users to go through multiple steps to find the information they needed. Furthermore, they lacked the functionality for uploading and managing data in a cloud environment, making it difficult to centrally handle materials from multiple data sources.

[0845] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0846] In this invention, the server includes: means for receiving documents and data; means for converting the received documents and data into a specified format; means for extracting metadata from the documents and data and generating an index; means for receiving queries in natural language; means for analyzing the received queries and searching for corresponding documents and data; means for generating a response based on the search results and transmitting it to a specified terminal; means for receiving format information for the documents and data; means for performing format conversion using OCR technology; means for analyzing the query content using a generative AI model; and means for expressing the generated response in natural language. This allows users to centrally manage and search documents and data in different formats, enabling prompt and appropriate information provision based on natural language queries. It also facilitates data uploading and automatic management in a cloud environment, enabling efficient management and use of documents and data.

[0847] "Means for receiving materials and data" refers to the function of receiving data files of various formats sent by users via the network to the server.

[0848] "Means for converting received material or data into a specified format" means software or algorithms for converting received material or data into a consistent form.

[0849] "Means for extracting metadata of materials and data and generating indexes" refers to the ability to extract important attribute information from data files (e.g., creation date, creator, file type) and store it in a searchable format.

[0850] "Means for receiving inquiries in natural language" refers to the function of the server receiving questions or requests entered by the user in natural language.

[0851] "Means of analyzing received inquiries and searching for corresponding materials and data" refers to the ability to use technologies such as generative AI models to understand the content of inquiries and search for relevant materials and databases.

[0852] "Means for generating a response based on the search results and sending it to a specified terminal" refers to the function of generating a response in natural language based on the information obtained from the search and sending it to the user's terminal.

[0853] "Means for receiving format information of materials or data" refers to a function for receiving information about the format of received materials or data (e.g., file type, number of pages, size).

[0854] "Means for format conversion using OCR technology" refers to a function that uses OCR (optical character recognition) technology to convert characters in images or PDFs into text.

[0855] "Means of analyzing the content of inquiries using a generative AI model" refers to the function of understanding and analyzing the content of user inquiries using a generative AI model capable of natural language processing (e.g., GPT, BERT).

[0856] "Means for expressing the generated response in natural language" refers to the ability to generate answers to queries in natural language that is easy for humans to understand.

[0857] The present invention relates to an information collection system that improves the efficiency of managing materials and data and enables rapid acquisition of information through inquiries in natural language. This system is mainly composed of a server, terminals, and users.

[0858] The server has a means for receiving materials and data uploaded by users. Specifically, when a user uses a terminal to select materials or data from a shared folder or cloud storage and uploads them to the system, the server receives these materials and data via an HTTP request. The server also simultaneously receives format information for the materials and data, and uses this information to prepare for analysis.

[0859] The received material or data is converted into the specified format using OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six), and the converted data is put into text format, ready for further analysis and indexing within the system.

[0860] The server then has the means to extract metadata about the materials and data and generate an index. Extracted metadata includes creation date, author, file type, important keywords, etc. This metadata is extracted using machine learning models (e.g., SpaCy) and stored in a database (e.g., MySQL, Elasticsearch). This provides a foundation for users to quickly search for the information they need later.

[0861] The user inputs a query in natural language from their device and sends it to the server. An example query might be, "What's the progress on this month's project?" The server receives the query and analyzes it using a generative AI model (e.g., GPT-4, BERT). Based on the analyzed query, the server searches for appropriate materials and data.

[0862] The server generates a response in natural language based on the search results. The response is in a format that is easy for the user to understand. For example, it may generate a response such as "This month's progress is 70% complete" and send it to the user's device.

[0863] This system operates in a cloud environment and provides a means for users to directly upload documents and data from shared folders or cloud storage, allowing documents and data to be managed in a unified manner even if they are scattered in different locations, facilitating information sharing.

[0864] As a concrete example, Person A uploads project materials to cloud storage, and the server receives and stores them in a database. If Person B then makes a natural language inquiry such as "Tell me about this month's project progress," the server analyzes the inquiry using a generative AI model, extracts the necessary information, generates a response, and sends it to Person B's device.

[0865] An example prompt is:

[0866] "Receive uploaded materials from the shared folder and record the format information."

[0867] "Convert the uploaded PDF file into text data and create an index."

[0868] "Parse queries entered in natural language, search for relevant data, and generate responses."

[0869] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[0870] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0871] Step 1:

[0872] Uploading data and receiving formatting information

[0873] Users use their devices to select materials and data from shared folders or cloud storage and upload them to the system.

[0874] The server receives an upload request from the user as an HTTP request, saves the materials and data in temporary storage on the server, and simultaneously records the format information of the materials and data (e.g., PDF, Word, Excel, etc.) as metadata.

[0875] Input: Files of materials and data uploaded by users

[0876] Output: Files and their metadata stored in temporary storage

[0877] Step 2:

[0878] Data format conversion

[0879] The server converts the received material or data into the specified format. For example, the server may use OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six) to convert a PDF file into text data.

[0880] Input: Files and metadata of materials and data stored in temporary storage

[0881] Output: A file converted to text data

[0882] Step 3:

[0883] Metadata extraction and indexing

[0884] The server extracts metadata from documents and data and generates an index. To do this, it uses machine learning models (e.g., SpaCy) to extract important information (e.g., creation date, author, keywords, etc.) and stores this information in a database (e.g., MySQL, Elasticsearch).

[0885] Input: A file converted to text data

[0886] Output: Metadata and index information stored in a database

[0887] Step 4:

[0888] Receiving user inquiries

[0889] The user inputs a query in natural language (e.g., "Please tell me about the progress of this month's project") from the terminal and sends it to the system.

[0890] The server receives this query as an HTTP POST request and temporarily stores its contents.

[0891] Input: A natural language query entered by the user

[0892] Output: Query content saved in temporary storage

[0893] Step 5:

[0894] Inquiry analysis and data search

[0895] The server analyzes the received query using a generative AI model (e.g., GPT-4, BERT), and uses the index information to search for relevant materials and data based on the analysis.

[0896] Input: Query content stored in temporary storage and index information stored in the database

[0897] Output: Analysis results and searched materials and data

[0898] Step 6:

[0899] Response generation and transmission

[0900] The server generates a response in natural language based on the search results using a generative AI model. The generated response is sent to the user's device as an HTTP response, where the user can view the response.

[0901] Input: Analysis results and searched materials and data

[0902] Output: A natural language response and the response sent to the user's device

[0903] Through these steps, the system enables management of materials and data, and rapid information search and provision using natural language.

[0904] (Application example 1)

[0905] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0906] In conventional information gathering systems, users had to go through multiple steps to obtain the specific materials or data they were looking for, which reduced search efficiency. Furthermore, online shopping sites lacked a means for users to quickly obtain product information, which led to the problem of taking a long time to make a purchasing decision.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0908] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for providing an assistant function that allows users to easily collect product information. This allows users to quickly and accurately obtain product information through inquiries in natural language, thereby improving search efficiency and speeding up purchasing decisions.

[0909] "Materials and Data" refers to information and content uploaded by Users, including in a variety of formats such as PDF, text, and images.

[0910] "Means for receiving" refers to the function for transferring materials and data from the user to the server.

[0911] "Means for converting into a specified format" refers to a function that processes received materials or data to convert them into a specific format.

[0912] "Means for extracting metadata and generating indexes" refers to the function of extracting attribute information of materials and data and creating indexes based on that information to facilitate searches.

[0913] "Means for receiving inquiries in natural language" refers to a function for receiving questions or requests from users in natural language format.

[0914] "Means for searching for corresponding materials and data" refers to the function of searching for related materials and data from a database based on a received inquiry.

[0915] "Means for generating a response and sending it to a specified terminal" refers to the function of creating an answer based on the search results and sending it to the user's terminal.

[0916] "Means for providing assistance functions" refers to functions that assist users in easily gathering product information, particularly on online platforms such as shopping sites.

[0917] This invention provides an information collection system that improves the efficiency of document and data management and enables rapid information acquisition through natural language inquiries. This system includes functions for receiving documents and data, format conversion, metadata extraction, index generation, natural language analysis, data search, response generation and transmission, and assistant functions.

[0918] Hardware and software used

[0919] Cloud storage: Use Google Drive or Amazon S3 to store materials and data.

[0920] AI model: OpenAI's GPT-4 is used for question analysis and answer generation.

[0921] Database: MySQL and MongoDB are used to manage metadata and product information.

[0922] API: RESTful API is used for communication between the application and the server.

[0923] Frontend: Develop a cross-platform smartphone app using React Native.

[0924] Processing flow

[0925] 1. Uploading data:

[0926] The server receives the upload of materials and data selected by the user from cloud storage (e.g., Google Drive). This includes the step of specifying and sending materials and data through an interface as an operation performed by the user in the application.

[0927] 2. Data Conversion and Storage:

[0928] The server converts the received materials and data into the specified format (e.g. PDF to text), and the converted data is stored in a database.

[0929] 3. Metadata extraction and indexing:

[0930] The server extracts metadata (e.g., creation date, creator, product category) from the materials and data, and generates an index, which is stored in a database to enable efficient searches.

[0931] 4. Natural Language Analysis and Data Retrieval:

[0932] The user sends a query in natural language from the application (e.g., "What are the popular products this week?"), and the server analyzes the query using the GPT-4 model and searches the database for corresponding materials and data.

[0933] 5. Response generation and transmission:

[0934] Based on the search results, the server generates a response in natural language format (e.g., "This week's popular products are: Product A, Product B, Product C.") and sends it to the user's terminal.

[0935] Specific examples

[0936] For example, a user opens a smartphone app and uses the chatbot function to type, "What are the popular products this week?" The server receives the request and uses the GPT-4 model to parse the question. It then retrieves information about this week's popular products from a database and uses the GPT-4 model to generate a response for the user. The response is then displayed on the user's smartphone.

[0937] Prompt Sentence Examples

[0938] An example prompt is:

[0939] User Question: "What are the popular items this week?"

[0940] The database contains:

[0941] Popular Item 1: Item A, Other Information...

[0942] Popular Item 2: Item B, Other Information...

[0943] Generated response: "This week's popular products are: Product A, Product B, Product C."

[0944] This system allows users to quickly and accurately obtain product information through natural language inquiries, improving search efficiency and speeding up purchasing decisions.

[0945] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0946] Step 1:

[0947] The user accesses cloud storage (e.g., Google Drive) from their device, selects materials or data, and uploads them. The server receives this request and retrieves the materials or data. The input is the received material data, and the output is the material data waiting to be converted.

[0948] Step 2:

[0949] The server converts the received documents and data into the specified format. For example, to convert a PDF file into a text file, PDF parsing software is used. The input is the received document data, and the output is the text format document data.

[0950] Step 3:

[0951] The server extracts metadata from the converted materials and data. For example, it analyzes and extracts information such as the document creation date, author, and file type. The input is text-formatted material data, and the output is material data with metadata added.

[0952] Step 4:

[0953] The server generates an index based on the extracted metadata and stores it in a database. An index generation algorithm is used to improve search performance. The input is the material data with metadata, and the output is the index and the material data with the index.

[0954] Step 5:

[0955] The user inputs and sends a query in natural language format (e.g., "What are the popular products this week?") from the terminal. The server receives this query. The input is the user's query in natural language, and the output is the query data waiting to be analyzed.

[0956] Step 6:

[0957] The server analyzes the received query and uses a generative AI model (e.g., GPT-4) to understand the intent of the question. This analysis generates a search query. The input is the query data, and the output is the generated search query.

[0958] Step 7:

[0959] The server searches the database using the search query to retrieve the corresponding materials and data. The input is the search query, and the output is the search result data.

[0960] Step 8:

[0961] The server uses a generative AI model to generate a response in natural language based on the search results. For example, it extracts related product information from the search results and creates a response in a format that is easy for the user to understand. The input is the search result data, and the output is a response in natural language.

[0962] Step 9:

[0963] The server generates a response and sends it to the user's terminal. The user checks the response on the terminal and obtains the necessary information. The input is a response in natural language format, and the output is a response message displayed on the terminal.

[0964] This series of steps allows users to quickly and accurately obtain materials and data (especially product information on online shopping sites) through natural language inquiries.

[0965] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0966] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[0967] The server receives documents and data uploaded by users. Users use their devices to access shared folders or cloud storage and select files to upload. The server receives the files and converts them into a format that is easy to handle internally. For example, converting a PDF file into text.

[0968] The server extracts metadata from the received data, including creation date, author, file type, etc., which is used to generate an index and stored in a database.

[0969] When a user inputs a query in natural language from their device, the server analyzes the query. An AI model (e.g., GPT or BERT) is used for the analysis. The emotion engine then recognizes the emotion contained in the user's query. For example, if a user inputs, "I'm very troubled by this problem. I want to know a solution as soon as possible," the emotion engine will detect that the user is confused.

[0970] The server searches the database based on the analysis results and retrieves relevant information. A response is generated based on the retrieved information, taking into account the emotion recognition results of the emotion engine. For example, if a user is in trouble, the server generates a response that includes encouraging words such as, "Don't worry. Here's some helpful information."

[0971] The generated response is sent from the server to the user's device. The user checks the response and plans or executes the next action based on the information obtained. For example, if Person B asks, "Please tell me the progress of the project this month," the server will extract progress data from the documents and respond, "Thank you for your hard work. Progress is currently at 70%."

[0972] The emotion engine stores user emotional data as a history and references it for future inquiries, allowing the system to continuously understand the user's state and provide more personalized responses.

[0973] This system operates in a cloud environment, allowing users to directly upload documents and data from shared folders or cloud storage, enabling the integrated management of distributed documents and data, and providing necessary information quickly and efficiently.

[0974] For example, Person A uploads an important meeting report to a shared folder, and Person B makes a query in natural language based on that report. The server uses an emotion engine to recognize Person B's emotion, generates a response in an appropriate tone, and sends it to Person B.

[0975] In this way, the present invention realizes an information gathering system that automates the management of materials and data and uses an emotion engine to provide information gathering in a more personalized, interactive manner using natural language.

[0976] The processing flow will be explained below.

[0977] Step 1:

[0978] The user uses a device to access a shared folder or cloud storage, selects the materials or data to upload, and clicks the "Upload" button on the interface to send the files to the system.

[0979] Step 2:

[0980] The device sends the selected materials and data to the server, along with metadata such as the file type and file path. Once the file upload is complete, the device displays a confirmation message on the server.

[0981] Step 3:

[0982] The server stores the documents and data received from the user. Depending on the type of data received, it converts it into a format that is easy to handle internally. For example, it runs a process to convert a PDF file into text.

[0983] Step 4:

[0984] The server extracts metadata from the converted data, such as creation date, creator, file type, etc. The extracted metadata is stored in a database and used to generate a search index.

[0985] Step 5:

[0986] The user inputs a query in natural language using a terminal. For example, a question such as "What is the progress of this month's project?" is input into the chat box and sent.

[0987] Step 6:

[0988] The terminal sends the user's query to the server, which receives the query and begins preparing it for analysis.

[0989] Step 7:

[0990] The server uses an AI model (e.g., GPT or BERT) to analyze the query, extract intent from the query, and identify relevant keywords.

[0991] Step 8:

[0992] The server uses an emotion engine to analyze the emotions contained in the user's query, for example, identifying emotions such as "confusion" or "urgency" from the user's text.

[0993] Step 9:

[0994] Based on the analysis results, the server searches the database for relevant materials and data, and extracts the information that best suits the query.

[0995] Step 10:

[0996] The server generates a response based on the search results and the analysis of the emotion engine. For example, if the user is confused, the server creates a response such as, "Don't worry, the following information may be helpful."

[0997] Step 11:

[0998] The server sends the generated response to the user's terminal, which then displays the response to the user, allowing the user to confirm the answer.

[0999] Step 12:

[1000] The user checks the response sent on the terminal and obtains the necessary information. For example, if user B asks "How is the project progressing this month?", he or she receives the response "Thank you for your hard work. The current progress is 70%."

[1001] Step 13:

[1002] The user's emotional information analyzed by the emotion engine is stored as historical data, which can be referenced during future inquiries to provide a more personalized response.

[1003] This series of processes allows users to easily and quickly access the materials and data they need and receive appropriate information along with responses that take their emotions into consideration, thereby streamlining the information gathering process and improving user satisfaction.

[1004] Example 2

[1005] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1006] Conventional information gathering systems generate uniform responses without considering the user's feelings, making it difficult to increase user satisfaction. In addition, the management of distributed materials and data is cumbersome, making it difficult to provide necessary information quickly and efficiently.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1008] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving queries in natural language, means for analyzing the received queries and searching for corresponding materials and data, means for recognizing emotions based on the analysis results, means for adjusting and generating a response based on the emotion recognition results, and means for transmitting the generated response to a specified terminal. This makes it possible to provide personalized responses that take the user's emotions into consideration, efficiently manage distributed materials and data, and quickly respond to user requests.

[1009] "Materials and data" refers to various types of information, such as text, images, and video, that the system receives and processes.

[1010] "Means for receiving" refers to the function by which the server obtains materials and data uploaded by users.

[1011] "Means for converting into a specified format" refers to the function of analyzing received materials and data and converting them into a format that is easy to handle internally.

[1012] "Metadata" refers to attribute information of materials or data, including, for example, creation date, creator, file type, and the like.

[1013] "Means for extracting metadata and generating indexes" refers to the function of extracting metadata from materials and data and storing it in a database in an easily searchable format.

[1014] A "natural language query" refers to an input form in which a user requests information from a system using everyday language.

[1015] The "means for receiving a query" refers to a function that enables the server to receive a query in natural language input by a user.

[1016] "Means for analyzing inquiries and retrieving corresponding materials and data" refers to the function of using a generative AI model to understand natural language input and search for corresponding materials and data from a database.

[1017] The "means for recognizing emotions" refers to the function of the emotion engine for analyzing the user's emotions contained in the query.

[1018] "Means for adjusting and generating a response based on the emotion recognition result" refers to a function for creating a response in an appropriate tone based on the user's emotion recognized by the emotion engine.

[1019] The "means for transmitting a response to a predetermined terminal" refers to a function for delivering the generated response to the terminal used by the user.

[1020] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[1021] This system consists of a server, a device, cloud storage, a generative AI model, and an emotion engine. The specific roles and processes of each element are described below.

[1022] Hardware and software used

[1023] Hardware: Servers, user devices (PCs, smartphones, etc.)

[1024] Software: Cloud storage (Google Drive, Dropbox, etc.), generative AI models (GPT-3, BERT), sentiment engine

[1025] Details of data processing and calculation

[1026] 1. Upload your data

[1027] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[1028] After the upload is complete, the device sends a notification to the server.

[1029] The server receives this notification and checks for the existence of the new file.

[1030] 2. Data transformation and metadata extraction

[1031] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[1032] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index. For example, from "Project Progress Report.pdf" it obtains the metadata "Creation date: October 1, 2023, Creator: Person A, File type: PDF."

[1033] The server stores the generated indexes and data in a database.

[1034] 3. Natural Language Inquiry Analysis

[1035] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[1036] The terminal transmits the input query to the server.

[1037] The server receives this query and analyzes it using a generative AI model (e.g., GPT or BERT), extracting keywords such as "project progress."

[1038] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[1039] 4. Generating the Response

[1040] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[1041] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[1042] 5. Sending a response and notifying the user

[1043] The server sends the generated response to the user's terminal.

[1044] The device receives this response and notifies the user, for example, by displaying a pop-up notification or email with the message "Thank you for your hard work. Progress is currently at 70%."

[1045] 6. Emotion data history storage

[1046] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[1047] The server can refer to this history during future queries and use it to provide more personalized responses.

[1048] Examples of concrete examples and prompts

[1049] Examples:

[1050] A user uploads a file called "Project Progress Report.pdf" to Google Drive.

[1051] The server receives the file, converts it into text data, extracts the metadata "Creation date: October 1, 2023, Creator: Mr. A, File type: PDF", and stores it in a database.

[1052] When a user asks from their device, "Tell me about this month's project progress," the server uses the generated AI model to analyze and retrieve information about the "project progress."

[1053] The server uses an emotion engine to recognize that the user is confused, generates a response saying, "Thank you for your hard work. Progress is currently at 70%," and sends this to the user's device.

[1054] Finally, the server stores the user's emotional data as a history and uses it for future queries.

[1055] Example prompt sentence:

[1056] "Tell me how this month's project is going."

[1057] "I'm very troubled by this issue. I want to know what to do as soon as possible."

[1058] As described above, the present invention automates the management of materials and data, and utilizes an emotion engine to provide personalized information gathering in a natural language dialogue format.

[1059] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1060] Step 1:

[1061] Uploading data

[1062] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[1063] Input: A file (e.g. PDF) uploaded by the user.

[1064] Output: Files saved in cloud storage.

[1065] What happens: A user opens the Google Drive app or web interface, selects a file, and clicks the upload button.

[1066] Step 2:

[1067] Data transformation and metadata extraction

[1068] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[1069] Input: A file stored in cloud storage.

[1070] Output: The converted text data.

[1071] Specific operation: The server converts the PDF file into text using a PDF parser.

[1072] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index.

[1073] Input: The converted text data.

[1074] Output: Extracted metadata and generated index.

[1075] What happens: The server uses a text parser to extract metadata from the text data, retrieving information such as creation date, author, and file type.

[1076] Step 3:

[1077] Receiving natural language inquiries

[1078] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[1079] Input: A natural language query typed by the user (e.g., How is the project progressing this month?).

[1080] Output: Query data from the terminal to the server.

[1081] Specific actions: A user uses a chat app or web form on their device to enter an inquiry and click the send button.

[1082] Step 4:

[1083] Query analysis and emotion recognition

[1084] The server analyzes the received query and extracts keywords using a generative AI model (e.g., GPT or BERT).

[1085] Input: Query data sent from the terminal.

[1086] Output: Extracted keywords (e.g., project progress).

[1087] What happens: The server uses a natural language processing engine to analyze the query text and identify keywords.

[1088] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[1089] Input: The parsed query text.

[1090] Output: Recognized emotion data (e.g., distressed).

[1091] What it does: The server uses a sentiment analysis algorithm to read the user's emotions from the text.

[1092] Step 5:

[1093] Generate and send a response

[1094] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[1095] Input: Extracted keywords and metadata index.

[1096] Output: Retrieved information (e.g., progress 70%).

[1097] What happens: The server uses an SQL query to retrieve the relevant information from the database.

[1098] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[1099] Input: Retrieved information and sentiment data.

[1100] Output: The generated response message.

[1101] How it works: The server uses a template-based response generation system to generate responses by adding phrases that correspond to the emotion.

[1102] The server sends the generated response to the user's terminal.

[1103] Input: The generated response message.

[1104] Output: Data sent to the terminal.

[1105] Specific operation: The server uses network communication to send the generated response to the specified user terminal.

[1106] Step 6:

[1107] Emotion data history storage

[1108] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[1109] Input: Recognized emotion data.

[1110] Output: Emotion history stored in a database.

[1111] Specific operation: The server executes an SQL command to insert emotion data into the database.

[1112] (Application example 2)

[1113] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1114] Conventional information gathering systems have difficulty responding appropriately and quickly to inquiries made by users in natural language. They also lack the ability to recognize users' emotions and generate appropriate responses, making it difficult to improve user satisfaction. Food delivery services, in particular, are required to respond to a variety of user emotions, but no effective systems exist.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata of the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for recognizing emotions and adjusting the response based on the analysis results. This makes it possible to provide an appropriate and prompt response according to the user's emotions and improve user satisfaction.

[1116] "Materials and Data" means any document or information in any form that a User uploads to the System.

[1117] "Means for receiving" refers to the functions and mechanisms for taking information from users into the system.

[1118] "Means for converting into a format" refers to the functions and processes for converting received materials and data into a format that is easy for the system to handle.

[1119] "Metadata" is supplementary information about materials and data, including creation date, creator, file type, etc.

[1120] "Means for generating an index" refers to the function or process of creating an index based on the metadata of materials or data to facilitate searching.

[1121] "Natural language inquiries" refer to questions or requests that users make to the system in everyday language.

[1122] "Means for parsing" refers to the functions or techniques that process information to understand the received natural language query and determine an appropriate response.

[1123] "Searching means" refers to the functions and processes for locating relevant materials and data within the system based on an analyzed query.

[1124] "Means for generating a response" refers to the functions or processes that create a response to the user based on the search results or analysis results.

[1125] "Means for sending to a predetermined terminal" refers to the functionality and protocols for sending the generated response to the device used by the user.

[1126] "Means for recognizing emotions and adjusting responses based on the analysis results" refers to a function or engine that analyzes the emotions contained in a user's inquiry and changes the response in tone and content accordingly.

[1127] This invention provides an information collection system for improving the daily user experience of food delivery services. This system receives and manages documents and data, analyzes natural language inquiries, recognizes emotions, and generates optimal responses based on the results, which are then sent to a designated terminal.

[1128] Specifically, the following hardware and software are used.

[1129] Hardware and software used

[1130] Hardware: Smartphone (iOS or Android device)

[1131] Backend servers: Node.js, Express.js, MySQL

[1132] Frontend: React Native

[1133] Generative model: OpenAI GPT-4

[1134] Emotion Recognition Model: Hugging Face Emotion Analysis Model

[1135] System program processing

[1136] 1. Means of receiving materials and data:

[1137] The server receives information and data from the user, including food delivery menu information and restaurant details.

[1138] 2. How to convert to the format:

[1139] The server converts the received materials and data into an easy-to-handle format (for example, text format) internally.

[1140] 3. How to extract metadata and generate indexes:

[1141] The server extracts metadata such as creation date, creator, and file type from the materials and data, generates an index based on that information, and stores it in a database.

[1142] 4. Means for receiving natural language queries:

[1143] Users use natural language to make inquiries through a smartphone app, for example, by typing, "Where is the best pizza today?"

[1144] 5. Analysis Method:

[1145] The server uses OpenAI GPT-4 to analyze and understand the received query, and at the same time, it uses the Hugging Face emotion analysis model to recognize the user's emotions.

[1146] 6. Search by:

[1147] The server searches the database based on the analysis results and index information to obtain the most appropriate materials and data.

[1148] 7. Means of generating a response:

[1149] Based on the acquired information, the system generates a response that reflects the emotion recognition results. For example, if the user is confused, the system will adjust the response to say, "Don't worry. We recommend the following pizza restaurant."

[1150] 8. Means of sending to a given terminal:

[1151] The final response is converted to JSON format and sent to the user's smartphone app, where the user can view the response.

[1152] Specific examples

[1153] 1. User Case:

[1154] A user asks, "I want to eat some good pizza, where can I order it?"

[1155] The server receives this query and performs natural language analysis and emotion recognition.

[1156] The server searches the indexed database for the best pizza place information.

[1157] If emotion recognition reveals that the user is confused, the system generates a response such as, "Don't worry, we have pizza recommendations from the restaurants below."

[1158] The response is sent to the user's smartphone, where they can view it in the app.

[1159] Prompt Sentence Examples

[1160] "Where can I order fried shrimp?"

[1161] "Can you recommend a Chinese restaurant?"

[1162] "Can you tell me about some cheap and delicious bento lunches around here?"

[1163] This invention allows users to receive appropriate responses that take their emotions into consideration when using food delivery services, improving overall satisfaction.

[1164] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1165] Step 1:

[1166] The server receives information and data from users. Specifically, users upload information such as menu items and restaurant details using a smartphone app. The input information and data is received by the server's backend (Node.js and Express.js).

[1167] Step 2:

[1168] The server analyzes the received documents and data and converts them into the specified format. This includes converting PDF files into text data, for example. Specifically, it converts the received data into a text format that is easy to handle internally.

[1169] Step 3:

[1170] The server extracts metadata about the documents and data and generates an index. This metadata includes creation date, author, file type, etc. This metadata is extracted and stored as an index in a MySQL database. The index allows for efficient searches later.

[1171] Step 4:

[1172] The server receives a natural language query that the user enters into the app. For example, the natural language query might be, "Where's the best pizza?" This query is then sent to the server.

[1173] Step 5:

[1174] The server analyzes the received query using an AI model (OpenAI GPT-4). At the same time, it recognizes the user's emotions using an emotion recognition model (Hugging Face emotion analysis model). The input query is analyzed to identify emotions such as whether the user is confused or expectant.

[1175] Step 6:

[1176] The server searches the database based on the analysis results and index information. It understands the query from the analysis results and searches the database (MySQL) for relevant information. For example, it searches for and retrieves information about pizza.

[1177] Step 7:

[1178] The server generates the optimal response based on the emotion recognition results. Specifically, it generates a response with a tone and content that matches the user's emotions based on the acquired information. For example, if the user is confused, it creates a response that gives a sense of reassurance.

[1179] Step 8:

[1180] The server sends the generated response to the specified device (smartphone app). The response is converted to JSON format and sent to the user's smartphone app. The user can check the response through the app.

[1181] This series of processes enables the system to provide quick and appropriate responses to user inquiries, thereby improving user satisfaction with food delivery services.

[1182] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1183] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1184] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1185] [Fourth embodiment]

[1186] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1187] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1188] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1189] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1190] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1191] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1192] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1193] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1194] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1195] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1196] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1197] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1198] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1199] The present invention relates to an information collection system that can efficiently manage materials and data and quickly acquire information through inquiries in natural language. The system includes the following means.

[1200] The server receives materials and data uploaded by users. For example, a user selects materials and data from a shared folder or cloud storage using a terminal and uploads them to the system. At the same time, the server receives format information for the materials and data and prepares to analyze the content of the received materials and data.

[1201] The server converts the received material or data into the specified format, for example, converting a PDF file to text, and the converted data is then ready for further analysis and indexing within the system.

[1202] The server then extracts metadata from the documents and data, creating an index. This metadata, such as creation date, author, and file type, is stored in a database to improve search performance, laying the foundation for users to quickly find the information they need later.

[1203] A user inputs a query in natural language from a terminal (e.g., "Tell me about the progress of this month's project"). This query is sent to a server. The server analyzes the received query and uses an AI model (e.g., GPT or BERT) to search for corresponding materials and data. After understanding the query and searching for relevant data, the server generates a response based on the information obtained. This response is expressed in natural language and is in a format that is easy for the user to understand.

[1204] The response is sent to the user's device, allowing the user to obtain the information immediately. For example, if user B asks, "What is the progress of this month's project?", the server extracts progress data from the project documents, generates a response saying, "This month's progress is 70% complete," and sends it to user B's device.

[1205] This system operates in a cloud environment and allows users to directly upload documents and data from shared folders or cloud storage. This allows documents and data to be managed in a unified manner even if they are scattered in different locations. It also facilitates information sharing, which has the effect of improving the knowledge level of the entire organization.

[1206] For example, Person A uploads project documents to Google Drive, the server receives the documents and stores them in a database, and Person B then makes a query in natural language. The server then analyzes the query, extracts the necessary information, and generates a response.

[1207] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[1208] The processing flow will be explained below.

[1209] Step 1:

[1210] The user accesses a shared folder or cloud storage from their device, selects the materials or data to upload, and then uses the designated interface to upload the selected files to the AI ​​Data Navigator.

[1211] Step 2:

[1212] When a user clicks the upload button on their device, the device sends the selected materials and data to the server, along with metadata such as the file type and file path.

[1213] Step 3:

[1214] The server receives the materials and data sent by the user. After receiving the data, the server determines the file type and converts it to the specified format (e.g., converting PDF to text) if necessary.

[1215] Step 4:

[1216] The server extracts metadata from the converted data, such as creation date, author, file type, etc. This metadata is stored in a database, generating an index for efficient subsequent searches.

[1217] Step 5:

[1218] A user uses a terminal to input a query in natural language, for example, a specific question such as "How is the project progressing this month?" The user sends this query to the server.

[1219] Step 6:

[1220] The server parses the incoming natural language query, using an AI model (e.g., GPT or BERT) to understand the meaning of the query and identify the appropriate database entry.

[1221] Step 7:

[1222] Based on the analysis results, the server retrieves relevant materials and data from the database, which is then further analyzed as needed and used to generate an appropriate response to the query.

[1223] Step 8:

[1224] The server generates a natural language response based on the search results, such as a specific answer like "Last month's progress was 50%, and this month's progress is 70% complete."

[1225] Step 9:

[1226] The server then sends the generated response to the user's terminal, where the user immediately receives the response and obtains the information in response to the query.

[1227] Step 10:

[1228] Users can check the responses sent on their devices and obtain the necessary information. For example, when Person B checks the "This month's project progress," he can plan his next action based on the displayed information.

[1229] This series of processes allows users to easily and quickly access the necessary materials and data and obtain appropriate information.

[1230] Example 1

[1231] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1232] Traditional methods for managing materials and data make it difficult to uniformly manage data in different formats, making them inefficient for searching and retrieving. They also lacked a system that could quickly provide appropriate information in response to natural language queries, forcing users to go through multiple steps to find the information they needed. Furthermore, they lacked the functionality for uploading and managing data in a cloud environment, making it difficult to centrally handle materials from multiple data sources.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1234] In this invention, the server includes: means for receiving documents and data; means for converting the received documents and data into a specified format; means for extracting metadata from the documents and data and generating an index; means for receiving queries in natural language; means for analyzing the received queries and searching for corresponding documents and data; means for generating a response based on the search results and transmitting it to a specified terminal; means for receiving format information for the documents and data; means for performing format conversion using OCR technology; means for analyzing the query content using a generative AI model; and means for expressing the generated response in natural language. This allows users to centrally manage and search documents and data in different formats, enabling prompt and appropriate information provision based on natural language queries. It also facilitates data uploading and automatic management in a cloud environment, enabling efficient management and use of documents and data.

[1235] "Means for receiving materials and data" refers to the function of receiving data files of various formats sent by users via the network to the server.

[1236] "Means for converting received material or data into a specified format" means software or algorithms for converting received material or data into a consistent form.

[1237] "Means for extracting metadata of materials and data and generating indexes" refers to the ability to extract important attribute information from data files (e.g., creation date, creator, file type) and store it in a searchable format.

[1238] "Means for receiving inquiries in natural language" refers to the function of the server receiving questions or requests entered by the user in natural language.

[1239] "Means of analyzing received inquiries and searching for corresponding materials and data" refers to the ability to use technologies such as generative AI models to understand the content of inquiries and search for relevant materials and databases.

[1240] "Means for generating a response based on the search results and sending it to a specified terminal" refers to the function of generating a response in natural language based on the information obtained from the search and sending it to the user's terminal.

[1241] "Means for receiving format information of materials or data" refers to a function for receiving information about the format of received materials or data (e.g., file type, number of pages, size).

[1242] "Means for format conversion using OCR technology" refers to a function that uses OCR (optical character recognition) technology to convert characters in images or PDFs into text.

[1243] "Means of analyzing the content of inquiries using a generative AI model" refers to the function of understanding and analyzing the content of user inquiries using a generative AI model capable of natural language processing (e.g., GPT, BERT).

[1244] "Means for expressing the generated response in natural language" refers to the ability to generate answers to queries in natural language that is easy for humans to understand.

[1245] The present invention relates to an information collection system that improves the efficiency of managing materials and data and enables rapid acquisition of information through inquiries in natural language. This system is mainly composed of a server, terminals, and users.

[1246] The server has a means for receiving materials and data uploaded by users. Specifically, when a user uses a terminal to select materials or data from a shared folder or cloud storage and uploads them to the system, the server receives these materials and data via an HTTP request. The server also simultaneously receives format information for the materials and data, and uses this information to prepare for analysis.

[1247] The received material or data is converted into the specified format using OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six), and the converted data is put into text format, ready for further analysis and indexing within the system.

[1248] The server then has the means to extract metadata about the materials and data and generate an index. Extracted metadata includes creation date, author, file type, important keywords, etc. This metadata is extracted using machine learning models (e.g., SpaCy) and stored in a database (e.g., MySQL, Elasticsearch). This provides a foundation for users to quickly search for the information they need later.

[1249] The user inputs a query in natural language from their device and sends it to the server. An example query might be, "What's the progress on this month's project?" The server receives the query and analyzes it using a generative AI model (e.g., GPT-4, BERT). Based on the analyzed query, the server searches for appropriate materials and data.

[1250] The server generates a response in natural language based on the search results. The response is in a format that is easy for the user to understand. For example, it may generate a response such as "This month's progress is 70% complete" and send it to the user's device.

[1251] This system operates in a cloud environment and provides a means for users to directly upload documents and data from shared folders or cloud storage, allowing documents and data to be managed in a unified manner even if they are scattered in different locations, facilitating information sharing.

[1252] As a concrete example, Person A uploads project materials to cloud storage, and the server receives and stores them in a database. If Person B then makes a natural language inquiry such as "Tell me about this month's project progress," the server analyzes the inquiry using a generative AI model, extracts the necessary information, generates a response, and sends it to Person B's device.

[1253] An example prompt is:

[1254] "Receive uploaded materials from the shared folder and record the format information."

[1255] "Convert the uploaded PDF file into text data and create an index."

[1256] "Parse queries entered in natural language, search for relevant data, and generate responses."

[1257] In this way, the present invention provides an information collection system for automating the management of materials and data and collecting information in an interactive manner using natural language.

[1258] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1259] Step 1:

[1260] Uploading data and receiving formatting information

[1261] Users use their devices to select materials and data from shared folders or cloud storage and upload them to the system.

[1262] The server receives an upload request from the user as an HTTP request, saves the materials and data in temporary storage on the server, and simultaneously records the format information of the materials and data (e.g., PDF, Word, Excel, etc.) as metadata.

[1263] Input: Files of materials and data uploaded by users

[1264] Output: Files and their metadata stored in temporary storage

[1265] Step 2:

[1266] Data format conversion

[1267] The server converts the received material or data into the specified format. For example, the server may use OCR technology (e.g., Tesseract OCR engine) or a PDF parser library (e.g., pdfminer.six) to convert a PDF file into text data.

[1268] Input: Files and metadata of materials and data stored in temporary storage

[1269] Output: A file converted to text data

[1270] Step 3:

[1271] Metadata extraction and indexing

[1272] The server extracts metadata from documents and data and generates an index. To do this, it uses machine learning models (e.g., SpaCy) to extract important information (e.g., creation date, author, keywords, etc.) and stores this information in a database (e.g., MySQL, Elasticsearch).

[1273] Input: A file converted to text data

[1274] Output: Metadata and index information stored in a database

[1275] Step 4:

[1276] Receiving user inquiries

[1277] The user inputs a query in natural language (e.g., "Please tell me about the progress of this month's project") from the terminal and sends it to the system.

[1278] The server receives this query as an HTTP POST request and temporarily stores its contents.

[1279] Input: A natural language query entered by the user

[1280] Output: Query content saved in temporary storage

[1281] Step 5:

[1282] Inquiry analysis and data search

[1283] The server analyzes the received query using a generative AI model (e.g., GPT-4, BERT), and uses the index information to search for relevant materials and data based on the analysis.

[1284] Input: Query content stored in temporary storage and index information stored in the database

[1285] Output: Analysis results and searched materials and data

[1286] Step 6:

[1287] Response generation and transmission

[1288] The server generates a response in natural language based on the search results using a generative AI model. The generated response is sent to the user's device as an HTTP response, where the user can view the response.

[1289] Input: Analysis results and searched materials and data

[1290] Output: A natural language response and the response sent to the user's device

[1291] Through these steps, the system enables management of materials and data, and rapid information search and provision using natural language.

[1292] (Application example 1)

[1293] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1294] In conventional information gathering systems, users had to go through multiple steps to obtain the specific materials or data they were looking for, which reduced search efficiency. Furthermore, online shopping sites lacked a means for users to quickly obtain product information, which led to the problem of taking a long time to make a purchasing decision.

[1295] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1296] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for providing an assistant function that allows users to easily collect product information. This allows users to quickly and accurately obtain product information through inquiries in natural language, thereby improving search efficiency and speeding up purchasing decisions.

[1297] "Materials and Data" refers to information and content uploaded by Users, including in a variety of formats such as PDF, text, and images.

[1298] "Means for receiving" refers to the function for transferring materials and data from the user to the server.

[1299] "Means for converting into a specified format" refers to a function that processes received materials or data to convert them into a specific format.

[1300] "Means for extracting metadata and generating indexes" refers to the function of extracting attribute information of materials and data and creating indexes based on that information to facilitate searches.

[1301] "Means for receiving inquiries in natural language" refers to a function for receiving questions or requests from users in natural language format.

[1302] "Means for searching for corresponding materials and data" refers to the function of searching for related materials and data from a database based on a received inquiry.

[1303] "Means for generating a response and sending it to a specified terminal" refers to the function of creating an answer based on the search results and sending it to the user's terminal.

[1304] "Means for providing assistance functions" refers to functions that assist users in easily gathering product information, particularly on online platforms such as shopping sites.

[1305] This invention provides an information collection system that improves the efficiency of document and data management and enables rapid information acquisition through natural language inquiries. This system includes functions for receiving documents and data, format conversion, metadata extraction, index generation, natural language analysis, data search, response generation and transmission, and assistant functions.

[1306] Hardware and software used

[1307] Cloud storage: Use Google Drive or Amazon S3 to store materials and data.

[1308] AI model: OpenAI's GPT-4 is used for question analysis and answer generation.

[1309] Database: MySQL and MongoDB are used to manage metadata and product information.

[1310] API: RESTful API is used for communication between the application and the server.

[1311] Frontend: Develop a cross-platform smartphone app using React Native.

[1312] Processing flow

[1313] 1. Uploading data:

[1314] The server receives the upload of materials and data selected by the user from cloud storage (e.g., Google Drive). This includes the step of specifying and sending materials and data through an interface as an operation performed by the user in the application.

[1315] 2. Data Conversion and Storage:

[1316] The server converts the received materials and data into the specified format (e.g. PDF to text), and the converted data is stored in a database.

[1317] 3. Metadata extraction and indexing:

[1318] The server extracts metadata (e.g., creation date, creator, product category) from the materials and data, and generates an index, which is stored in a database to enable efficient searches.

[1319] 4. Natural Language Analysis and Data Retrieval:

[1320] The user sends a query in natural language from the application (e.g., "What are the popular products this week?"), and the server analyzes the query using the GPT-4 model and searches the database for corresponding materials and data.

[1321] 5. Response generation and transmission:

[1322] Based on the search results, the server generates a response in natural language format (e.g., "This week's popular products are: Product A, Product B, Product C.") and sends it to the user's terminal.

[1323] Specific examples

[1324] For example, a user opens a smartphone app and uses the chatbot function to type, "What are the popular products this week?" The server receives the request and uses the GPT-4 model to parse the question. It then retrieves information about this week's popular products from a database and uses the GPT-4 model to generate a response for the user. The response is then displayed on the user's smartphone.

[1325] Prompt Sentence Examples

[1326] An example prompt is:

[1327] User Question: "What are the popular items this week?"

[1328] The database contains:

[1329] Popular Item 1: Item A, Other Information...

[1330] Popular Item 2: Item B, Other Information...

[1331] Generated response: "This week's popular products are: Product A, Product B, Product C."

[1332] This system allows users to quickly and accurately obtain product information through natural language inquiries, improving search efficiency and speeding up purchasing decisions.

[1333] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1334] Step 1:

[1335] The user accesses cloud storage (e.g., Google Drive) from their device, selects materials or data, and uploads them. The server receives this request and retrieves the materials or data. The input is the received material data, and the output is the material data waiting to be converted.

[1336] Step 2:

[1337] The server converts the received documents and data into the specified format. For example, to convert a PDF file into a text file, PDF parsing software is used. The input is the received document data, and the output is the text format document data.

[1338] Step 3:

[1339] The server extracts metadata from the converted materials and data. For example, it analyzes and extracts information such as the document creation date, author, and file type. The input is text-formatted material data, and the output is material data with metadata added.

[1340] Step 4:

[1341] The server generates an index based on the extracted metadata and stores it in a database. An index generation algorithm is used to improve search performance. The input is the material data with metadata, and the output is the index and the material data with the index.

[1342] Step 5:

[1343] The user inputs and sends a query in natural language format (e.g., "What are the popular products this week?") from the terminal. The server receives this query. The input is the user's query in natural language, and the output is the query data waiting to be analyzed.

[1344] Step 6:

[1345] The server analyzes the received query and uses a generative AI model (e.g., GPT-4) to understand the intent of the question. This analysis generates a search query. The input is the query data, and the output is the generated search query.

[1346] Step 7:

[1347] The server searches the database using the search query to retrieve the corresponding materials and data. The input is the search query, and the output is the search result data.

[1348] Step 8:

[1349] The server uses a generative AI model to generate a response in natural language based on the search results. For example, it extracts related product information from the search results and creates a response in a format that is easy for the user to understand. The input is the search result data, and the output is a response in natural language.

[1350] Step 9:

[1351] The server generates a response and sends it to the user's terminal. The user checks the response on the terminal and obtains the necessary information. The input is a response in natural language format, and the output is a response message displayed on the terminal.

[1352] This series of steps allows users to quickly and accurately obtain materials and data (especially product information on online shopping sites) through natural language inquiries.

[1353] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1354] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[1355] The server receives documents and data uploaded by users. Users use their devices to access shared folders or cloud storage and select files to upload. The server receives the files and converts them into a format that is easy to handle internally. For example, converting a PDF file into text.

[1356] The server extracts metadata from the received data, including creation date, author, file type, etc., which is used to generate an index and stored in a database.

[1357] When a user inputs a query in natural language from their device, the server analyzes the query. An AI model (e.g., GPT or BERT) is used for the analysis. The emotion engine then recognizes the emotion contained in the user's query. For example, if a user inputs, "I'm very troubled by this problem. I want to know a solution as soon as possible," the emotion engine will detect that the user is confused.

[1358] The server searches the database based on the analysis results and retrieves relevant information. A response is generated based on the retrieved information, taking into account the emotion recognition results of the emotion engine. For example, if a user is in trouble, the server generates a response that includes encouraging words such as, "Don't worry. Here's some helpful information."

[1359] The generated response is sent from the server to the user's device. The user checks the response and plans or executes the next action based on the information obtained. For example, if Person B asks, "Please tell me the progress of the project this month," the server will extract progress data from the documents and respond, "Thank you for your hard work. Progress is currently at 70%."

[1360] The emotion engine stores user emotional data as a history and references it for future inquiries, allowing the system to continuously understand the user's state and provide more personalized responses.

[1361] This system operates in a cloud environment, allowing users to directly upload documents and data from shared folders or cloud storage, enabling the integrated management of distributed documents and data, and providing necessary information quickly and efficiently.

[1362] For example, Person A uploads an important meeting report to a shared folder, and Person B makes a query in natural language based on that report. The server uses an emotion engine to recognize Person B's emotion, generates a response in an appropriate tone, and sends it to Person B.

[1363] In this way, the present invention realizes an information gathering system that automates the management of materials and data and uses an emotion engine to provide information gathering in a more personalized, interactive manner using natural language.

[1364] The processing flow will be explained below.

[1365] Step 1:

[1366] The user uses a device to access a shared folder or cloud storage, selects the materials or data to upload, and clicks the "Upload" button on the interface to send the files to the system.

[1367] Step 2:

[1368] The device sends the selected materials and data to the server, along with metadata such as the file type and file path. Once the file upload is complete, the device displays a confirmation message on the server.

[1369] Step 3:

[1370] The server stores the documents and data received from the user. Depending on the type of data received, it converts it into a format that is easy to handle internally. For example, it runs a process to convert a PDF file into text.

[1371] Step 4:

[1372] The server extracts metadata from the converted data, such as creation date, creator, file type, etc. The extracted metadata is stored in a database and used to generate a search index.

[1373] Step 5:

[1374] The user inputs a query in natural language using a terminal. For example, a question such as "What is the progress of this month's project?" is input into the chat box and sent.

[1375] Step 6:

[1376] The terminal sends the user's query to the server, which receives the query and begins preparing it for analysis.

[1377] Step 7:

[1378] The server uses an AI model (e.g., GPT or BERT) to analyze the query, extract intent from the query, and identify relevant keywords.

[1379] Step 8:

[1380] The server uses an emotion engine to analyze the emotions contained in the user's query, for example, identifying emotions such as "confusion" or "urgency" from the user's text.

[1381] Step 9:

[1382] Based on the analysis results, the server searches the database for relevant materials and data, and extracts the information that best suits the query.

[1383] Step 10:

[1384] The server generates a response based on the search results and the analysis of the emotion engine. For example, if the user is confused, the server creates a response such as, "Don't worry, the following information may be helpful."

[1385] Step 11:

[1386] The server sends the generated response to the user's terminal, which then displays the response to the user, allowing the user to confirm the answer.

[1387] Step 12:

[1388] The user checks the response sent on the terminal and obtains the necessary information. For example, if user B asks "How is the project progressing this month?", he or she receives the response "Thank you for your hard work. The current progress is 70%."

[1389] Step 13:

[1390] The user's emotional information analyzed by the emotion engine is stored as historical data, which can be referenced during future inquiries to provide a more personalized response.

[1391] This series of processes allows users to easily and quickly access the materials and data they need and receive appropriate information along with responses that take their emotions into consideration, thereby streamlining the information gathering process and improving user satisfaction.

[1392] Example 2

[1393] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1394] Conventional information gathering systems generate uniform responses without considering the user's feelings, making it difficult to increase user satisfaction. In addition, the management of distributed materials and data is cumbersome, making it difficult to provide necessary information quickly and efficiently.

[1395] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1396] In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata from the materials and data and generating an index, means for receiving queries in natural language, means for analyzing the received queries and searching for corresponding materials and data, means for recognizing emotions based on the analysis results, means for adjusting and generating a response based on the emotion recognition results, and means for transmitting the generated response to a specified terminal. This makes it possible to provide personalized responses that take the user's emotions into consideration, efficiently manage distributed materials and data, and quickly respond to user requests.

[1397] "Materials and data" refers to various types of information, such as text, images, and video, that the system receives and processes.

[1398] "Means for receiving" refers to the function by which the server obtains materials and data uploaded by users.

[1399] "Means for converting into a specified format" refers to the function of analyzing received materials and data and converting them into a format that is easy to handle internally.

[1400] "Metadata" refers to attribute information of materials or data, including, for example, creation date, creator, file type, and the like.

[1401] "Means for extracting metadata and generating indexes" refers to the function of extracting metadata from materials and data and storing it in a database in an easily searchable format.

[1402] A "natural language query" refers to an input form in which a user requests information from a system using everyday language.

[1403] The "means for receiving a query" refers to a function that enables the server to receive a query in natural language input by a user.

[1404] "Means for analyzing inquiries and retrieving corresponding materials and data" refers to the function of using a generative AI model to understand natural language input and search for corresponding materials and data from a database.

[1405] The "means for recognizing emotions" refers to the function of the emotion engine for analyzing the user's emotions contained in the query.

[1406] "Means for adjusting and generating a response based on the emotion recognition result" refers to a function for creating a response in an appropriate tone based on the user's emotion recognized by the emotion engine.

[1407] The "means for transmitting a response to a predetermined terminal" refers to a function for delivering the generated response to the terminal used by the user.

[1408] This invention relates to an information gathering system for efficient management of materials and data, and for rapid responses to inquiries in natural language. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusts the response, the system aims to provide more appropriate and satisfying responses.

[1409] This system consists of a server, a device, cloud storage, a generative AI model, and an emotion engine. The specific roles and processes of each element are described below.

[1410] Hardware and software used

[1411] Hardware: Servers, user devices (PCs, smartphones, etc.)

[1412] Software: Cloud storage (Google Drive, Dropbox, etc.), generative AI models (GPT-3, BERT), sentiment engine

[1413] Details of data processing and calculation

[1414] 1. Upload your data

[1415] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[1416] After the upload is complete, the device sends a notification to the server.

[1417] The server receives this notification and checks for the existence of the new file.

[1418] 2. Data transformation and metadata extraction

[1419] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[1420] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index. For example, from "Project Progress Report.pdf" it obtains the metadata "Creation date: October 1, 2023, Creator: Person A, File type: PDF."

[1421] The server stores the generated indexes and data in a database.

[1422] 3. Natural Language Inquiry Analysis

[1423] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[1424] The terminal transmits the input query to the server.

[1425] The server receives this query and analyzes it using a generative AI model (e.g., GPT or BERT), extracting keywords such as "project progress."

[1426] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[1427] 4. Generating the Response

[1428] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[1429] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[1430] 5. Sending a response and notifying the user

[1431] The server sends the generated response to the user's terminal.

[1432] The device receives this response and notifies the user, for example, by displaying a pop-up notification or email with the message "Thank you for your hard work. Progress is currently at 70%."

[1433] 6. Emotion data history storage

[1434] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[1435] The server can refer to this history during future queries and use it to provide more personalized responses.

[1436] Examples of concrete examples and prompts

[1437] Examples:

[1438] A user uploads a file called "Project Progress Report.pdf" to Google Drive.

[1439] The server receives the file, converts it into text data, extracts the metadata "Creation date: October 1, 2023, Creator: Mr. A, File type: PDF", and stores it in a database.

[1440] When a user asks from their device, "Tell me about this month's project progress," the server uses the generated AI model to analyze and retrieve information about the "project progress."

[1441] The server uses an emotion engine to recognize that the user is confused, generates a response saying, "Thank you for your hard work. Progress is currently at 70%," and sends this to the user's device.

[1442] Finally, the server stores the user's emotional data as a history and uses it for future queries.

[1443] Example prompt sentence:

[1444] "Tell me how this month's project is going."

[1445] "I'm very troubled by this issue. I want to know what to do as soon as possible."

[1446] As described above, the present invention automates the management of materials and data, and utilizes an emotion engine to provide personalized information gathering in a natural language dialogue format.

[1447] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1448] Step 1:

[1449] Uploading data

[1450] Users can use their devices to upload documents and data to cloud storage. For example, they can upload "Project Progress Report.pdf" to Google Drive.

[1451] Input: A file (e.g. PDF) uploaded by the user.

[1452] Output: Files saved in cloud storage.

[1453] What happens: A user opens the Google Drive app or web interface, selects a file, and clicks the upload button.

[1454] Step 2:

[1455] Data transformation and metadata extraction

[1456] The server retrieves the uploaded file from the cloud storage and converts it into a format that is easy to handle internally, for example, converting a PDF file into text data.

[1457] Input: A file stored in cloud storage.

[1458] Output: The converted text data.

[1459] Specific operation: The server converts the PDF file into text using a PDF parser.

[1460] The server extracts metadata (creation date, creator, file type, etc.) from the converted data and generates an index.

[1461] Input: The converted text data.

[1462] Output: Extracted metadata and generated index.

[1463] What happens: The server uses a text parser to extract metadata from the text data, retrieving information such as creation date, author, and file type.

[1464] Step 3:

[1465] Receiving natural language inquiries

[1466] A user inputs a query in natural language from a terminal, for example, "Please tell me about the progress of this month's project."

[1467] Input: A natural language query typed by the user (e.g., How is the project progressing this month?).

[1468] Output: Query data from the terminal to the server.

[1469] Specific actions: A user uses a chat app or web form on their device to enter an inquiry and click the send button.

[1470] Step 4:

[1471] Query analysis and emotion recognition

[1472] The server analyzes the received query and extracts keywords using a generative AI model (e.g., GPT or BERT).

[1473] Input: Query data sent from the terminal.

[1474] Output: Extracted keywords (e.g., project progress).

[1475] What happens: The server uses a natural language processing engine to analyze the query text and identify keywords.

[1476] At the same time, the server uses an emotion engine to analyze the user's emotions, for example, recognizing the emotion "I'm in trouble."

[1477] Input: The parsed query text.

[1478] Output: Recognized emotion data (e.g., distressed).

[1479] What it does: The server uses a sentiment analysis algorithm to read the user's emotions from the text.

[1480] Step 5:

[1481] Generate and send a response

[1482] The server searches the database based on the analysis results to retrieve relevant information, for example, searching for "project progress report" and finding that the progress is 70%.

[1483] Input: Extracted keywords and metadata index.

[1484] Output: Retrieved information (e.g., progress 70%).

[1485] What happens: The server uses an SQL query to retrieve the relevant information from the database.

[1486] The server takes into account the recognition results of the emotion engine and generates a response in an appropriate tone, for example, "Thank you for your hard work. Progress is currently at 70%."

[1487] Input: Retrieved information and sentiment data.

[1488] Output: The generated response message.

[1489] How it works: The server uses a template-based response generation system to generate responses by adding phrases that correspond to the emotion.

[1490] The server sends the generated response to the user's terminal.

[1491] Input: The generated response message.

[1492] Output: Data sent to the terminal.

[1493] Specific operation: The server uses network communication to send the generated response to the specified user terminal.

[1494] Step 6:

[1495] Emotion data history storage

[1496] The server stores the user's emotion data recognized by the emotion engine as history. For example, it stores data such as "Mr. B is in trouble."

[1497] Input: Recognized emotion data.

[1498] Output: Emotion history stored in a database.

[1499] Specific operation: The server executes an SQL command to insert emotion data into the database.

[1500] (Application example 2)

[1501] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1502] Conventional information gathering systems have difficulty responding appropriately and quickly to inquiries made by users in natural language. They also lack the ability to recognize users' emotions and generate appropriate responses, making it difficult to improve user satisfaction. Food delivery services, in particular, are required to respond to a variety of user emotions, but no effective systems exist.

[1503] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving materials and data, means for converting the received materials and data into a specified format, means for extracting metadata of the materials and data and generating an index, means for receiving inquiries in natural language, means for analyzing the received inquiries and searching for corresponding materials and data, means for generating a response based on the search results and sending it to a specified terminal, and means for recognizing emotions and adjusting the response based on the analysis results. This makes it possible to provide an appropriate and prompt response according to the user's emotions and improve user satisfaction.

[1504] "Materials and Data" means any document or information in any form that a User uploads to the System.

[1505] "Means for receiving" refers to the functions and mechanisms for taking information from users into the system.

[1506] "Means for converting into a format" refers to the functions and processes for converting received materials and data into a format that is easy for the system to handle.

[1507] "Metadata" is supplementary information about materials and data, including creation date, creator, file type, etc.

[1508] "Means for generating an index" refers to the function or process of creating an index based on the metadata of materials or data to facilitate searching.

[1509] "Natural language inquiries" refer to questions or requests that users make to the system in everyday language.

[1510] "Means for parsing" refers to the functions or techniques that process information to understand the received natural language query and determine an appropriate response.

[1511] "Searching means" refers to the functions and processes for locating relevant materials and data within the system based on an analyzed query.

[1512] "Means for generating a response" refers to the functions or processes that create a response to the user based on the search results or analysis results.

[1513] "Means for sending to a predetermined terminal" refers to the functionality and protocols for sending the generated response to the device used by the user.

[1514] "Means for recognizing emotions and adjusting responses based on the analysis results" refers to a function or engine that analyzes the emotions contained in a user's inquiry and changes the response in tone and content accordingly.

[1515] This invention provides an information collection system for improving the daily user experience of food delivery services. This system receives and manages documents and data, analyzes natural language inquiries, recognizes emotions, and generates optimal responses based on the results, which are then sent to a designated terminal.

[1516] Specifically, the following hardware and software are used.

[1517] Hardware and software used

[1518] Hardware: Smartphone (iOS or Android device)

[1519] Backend servers: Node.js, Express.js, MySQL

[1520] Frontend: React Native

[1521] Generative model: OpenAI GPT-4

[1522] Emotion Recognition Model: Hugging Face Emotion Analysis Model

[1523] System program processing

[1524] 1. Means of receiving materials and data:

[1525] The server receives information and data from the user, including food delivery menu information and restaurant details.

[1526] 2. How to convert to the format:

[1527] The server converts the received materials and data into an easy-to-handle format (for example, text format) internally.

[1528] 3. How to extract metadata and generate indexes:

[1529] The server extracts metadata such as creation date, creator, and file type from the materials and data, generates an index based on that information, and stores it in a database.

[1530] 4. Means for receiving natural language queries:

[1531] Users use natural language to make inquiries through a smartphone app, for example, by typing, "Where is the best pizza today?"

[1532] 5. Analysis Method:

[1533] The server uses OpenAI GPT-4 to analyze and understand the received query, and at the same time, it uses the Hugging Face emotion analysis model to recognize the user's emotions.

[1534] 6. Search by:

[1535] The server searches the database based on the analysis results and index information to obtain the most appropriate materials and data.

[1536] 7. Means of generating a response:

[1537] Based on the acquired information, the system generates a response that reflects the emotion recognition results. For example, if the user is confused, the system will adjust the response to say, "Don't worry. We recommend the following pizza restaurant."

[1538] 8. Means of sending to a given terminal:

[1539] The final response is converted to JSON format and sent to the user's smartphone app, where the user can view the response.

[1540] Specific examples

[1541] 1. User Case:

[1542] A user asks, "I want to eat some good pizza, where can I order it?"

[1543] The server receives this query and performs natural language analysis and emotion recognition.

[1544] The server searches the indexed database for the best pizza place information.

[1545] If emotion recognition reveals that the user is confused, the system generates a response such as, "Don't worry, we have pizza recommendations from the restaurants below."

[1546] The response is sent to the user's smartphone, where they can view it in the app.

[1547] Prompt Sentence Examples

[1548] "Where can I order fried shrimp?"

[1549] "Can you recommend a Chinese restaurant?"

[1550] "Can you tell me about some cheap and delicious bento lunches around here?"

[1551] This invention allows users to receive appropriate responses that take their emotions into consideration when using food delivery services, improving overall satisfaction.

[1552] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1553] Step 1:

[1554] The server receives information and data from users. Specifically, users upload information such as menu items and restaurant details using a smartphone app. The input information and data is received by the server's backend (Node.js and Express.js).

[1555] Step 2:

[1556] The server analyzes the received documents and data and converts them into the specified format. This includes converting PDF files into text data, for example. Specifically, it converts the received data into a text format that is easy to handle internally.

[1557] Step 3:

[1558] The server extracts metadata about the documents and data and generates an index. This metadata includes creation date, author, file type, etc. This metadata is extracted and stored as an index in a MySQL database. The index allows for efficient searches later.

[1559] Step 4:

[1560] The server receives a natural language query that the user enters into the app. For example, the natural language query might be, "Where's the best pizza?" This query is then sent to the server.

[1561] Step 5:

[1562] The server analyzes the received query using an AI model (OpenAI GPT-4). At the same time, it recognizes the user's emotions using an emotion recognition model (Hugging Face emotion analysis model). The input query is analyzed to identify emotions such as whether the user is confused or expectant.

[1563] Step 6:

[1564] The server searches the database based on the analysis results and index information. It understands the query from the analysis results and searches the database (MySQL) for relevant information. For example, it searches for and retrieves information about pizza.

[1565] Step 7:

[1566] The server generates the optimal response based on the emotion recognition results. Specifically, it generates a response with a tone and content that matches the user's emotions based on the acquired information. For example, if the user is confused, it creates a response that gives a sense of reassurance.

[1567] Step 8:

[1568] The server sends the generated response to the specified device (smartphone app). The response is converted to JSON format and sent to the user's smartphone app. The user can check the response through the app.

[1569] This series of processes enables the system to provide quick and appropriate responses to user inquiries, thereby improving user satisfaction with food delivery services.

[1570] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1571] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1572] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1573] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1574] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1575] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1576] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1577] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1578] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1579] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1580] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1581] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1582] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1583] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1584] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1585] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1586] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1587] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1588] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1589] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1590] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1591] The following is further disclosed regarding the above embodiment.

[1592] (Claim 1)

[1593] means for receiving materials and data;

[1594] a means for converting received materials and data into a specified format;

[1595] A means of extracting metadata from materials and data and generating indexes;

[1596] means for receiving a natural language query;

[1597] A means for analyzing received inquiries and searching for corresponding materials and data;

[1598] means for generating a response based on the search results and transmitting the response to a predetermined terminal;

[1599] Information gathering system including.

[1600] (Claim 2)

[1601] 2. The information collection system according to claim 1, wherein the generative model is used to analyze materials and data, and a response in natural language is generated based on the results.

[1602] (Claim 3)

[1603] 2. The information collection system according to claim 1, which operates in a cloud environment and comprises means for enabling users to directly upload materials and data from shared folders or cloud storage.

[1604] "Example 1"

[1605] (Claim 1)

[1606] means for receiving materials and data;

[1607] a means for converting received materials and data into a specified format;

[1608] A means of extracting metadata from materials and data and generating indexes;

[1609] means for receiving a natural language query;

[1610] A means for analyzing received inquiries and searching for corresponding materials and data;

[1611] means for generating a response based on the search results and transmitting the response to a predetermined terminal;

[1612] means for receiving formatting information for materials or data;

[1613] A means for performing format conversion using OCR technology;

[1614] A means for analyzing the content of the inquiry using a generative AI model;

[1615] a means for expressing the generated response in natural language;

[1616] A system including:

[1617] (Claim 2)

[1618] The system of claim 1, which analyzes materials and data using a generative model and generates responses in natural language based on the results.

[1619] (Claim 3)

[1620] The system of claim 1, which operates in a cloud environment and includes a means for enabling users to directly upload materials and data from shared folders or cloud storage.

[1621] "Application Example 1"

[1622] (Claim 1)

[1623] means for receiving materials and data;

[1624] a means for converting received materials and data into a specified format;

[1625] A means of extracting metadata from materials and data and generating indexes;

[1626] means for receiving a natural language query;

[1627] A means for analyzing received inquiries and searching for corresponding materials and data;

[1628] means for generating a response based on the search results and transmitting the response to a predetermined terminal;

[1629] A means for providing an assistant function that allows a user to easily collect product information;

[1630] A system including:

[1631] (Claim 2)

[1632] The system of claim 1, which analyzes materials and data using a generative model and generates responses in natural language based on the results.

[1633] (Claim 3)

[1634] The system of claim 1, which operates in a cloud environment and includes a means for enabling users to directly upload materials and data from shared folders or cloud storage.

[1635] "Example 2: Combining Emotion Engines"

[1636] (Claim 1)

[1637] means for receiving materials and data;

[1638] a means for converting received materials and data into a specified format;

[1639] A means of extracting metadata from materials and data and generating indexes;

[1640] means for receiving a natural language query;

[1641] A means for analyzing received inquiries and searching for corresponding materials and data;

[1642] A means for recognizing emotions based on the analysis results;

[1643] a means for adjusting and generating a response based on the emotion recognition results;

[1644] means for transmitting the generated response to a predetermined terminal;

[1645] A system including:

[1646] (Claim 2)

[1647] The system of claim 1, which analyzes materials and data using a generative model and generates responses in natural language based on the results.

[1648] (Claim 3)

[1649] The system of claim 1, which operates in a cloud environment and includes a means for enabling users to directly upload materials and data from shared folders or cloud storage.

[1650] "Application example 2 when combining emotion engines"

[1651] (Claim 1)

[1652] means for receiving materials and data;

[1653] a means for converting received materials and data into a specified format;

[1654] A means of extracting metadata from materials and data and generating indexes;

[1655] means for receiving a natural language query;

[1656] A means for analyzing received inquiries and searching for corresponding materials and data;

[1657] means for generating a response based on the search results and transmitting the response to a predetermined terminal;

[1658] a means for recognizing emotions and adjusting responses based on the analysis;

[1659] A system including:

[1660] (Claim 2)

[1661] The system of claim 1, which analyzes materials and data using a generative model and generates responses in natural language based on the results.

[1662] (Claim 3)

[1663] 10. The system of claim 1, which operates in a cloud environment and includes means for enabling users to directly upload materials and data from shared storage. [Explanation of symbols]

[1664] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving materials and data; a means for converting received materials and data into a specified format; A means of extracting metadata from materials and data and generating indexes; means for receiving a natural language query; A means for analyzing received inquiries and searching for corresponding materials and data; means for generating a response based on the search results and transmitting the response to a predetermined terminal; Information gathering system including.

2. 2. The information gathering system according to claim 1, wherein the generative model is used to analyze materials and data, and a response in natural language is generated based on the results.

3. 2. The information collection system according to claim 1, which operates in a cloud environment and comprises means for enabling a user to directly upload materials and data from a shared folder or cloud storage.

Images