system

Abstract

We provide the system. [Solution] A means for the user to input a question or keyword, A means of sending the entered question or keyword to the server, A means of searching the database based on the question or keyword received by the server, A means of organizing search results and providing information in a format that is easy for users to understand, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional operations and business negotiations with trading partners, when questions arise regarding the details of measures or the content of agency notifications, they often cannot be resolved immediately on the spot. It was common to take the information back and provide an answer by phone or email at a later date based on the research results. However, such a method has the problem of reducing business efficiency and causing delays in discussions. The present invention aims to solve such problems and provide a system that improves business efficiency and productivity by immediately resolving questions on the spot.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides the following means: a means for the user to input a question or keyword and send the input content to a server. The server provides a system that includes means for searching a database based on the received question or keyword, organizing the search results, and providing the information in a format that is easy for the user to understand. Furthermore, the server has means for prioritizing the selection of information that is highly relevant to the user's question or keyword, and the system includes means for the terminal to notify the user of the information provided, thereby enabling immediate responses in business negotiations and operations.

[0006] A "user" refers to an individual who operates the system and inputs questions or keywords.

[0007] "Questions or keywords" refer to words or phrases that users enter into the system to search for information.

[0008] "Input method" refers to the interface through which users input questions or keywords and send them to the system.

[0009] "Means of transmission" refers to the communication function used to send the entered question or keyword to the server.

[0010] A "server" refers to a computer system that analyzes incoming questions or keywords, searches a database, and provides information.

[0011] "Means of searching the database based on received questions or keywords" refers to the function that allows the server to search for relevant information within the database.

[0012] A "database" refers to a collection of information where details of policies, agent notifications, and other such information are stored.

[0013] "Means of organizing search results" refers to the function by which the server processes information obtained from the database into a format that is easy for the user to understand.

[0014] "The means for providing information" refers to the function for displaying organized information to the user.

[0015] "The means for preferentially selecting highly relevant information" refers to the function for the server to preferentially select particularly important information from the search results.

[0016] "The means for notifying" refers to the function for notifying the provided information to the user.

Brief Description of Drawings

[0017] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the language used in the following description will be explained.

[0020] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The system of this invention is designed to allow questions regarding the details of policies and the content of agent notifications to be resolved on the spot during daily operations and business negotiations with clients. The program for this system is generated and its operation is described below.

[0039] System-wide configuration

[0040] The system primarily consists of terminals that provide the user interface, servers that process data, and databases that store information. For this system to function, each component must work in coordination with the others.

[0041] Program processing

[0042] 1. The user enters a question or keyword.

[0043] User: Enter questions or keywords into the input field on the device. For example, if a client asks "What are the latest campaign strategies?" during a business meeting, the user would enter "Latest campaign strategies."

[0044] 2. The terminal sends the input to the server.

[0045] Terminal: Sends the entered questions and keywords to the server. Specifically, it sends a request to the server via the API. This request includes metadata such as the user ID and the current date and time, along with the input content.

[0046] 3. The server searches the database.

[0047] Server: Analyzes received questions and keywords and searches the database. The server generates queries based on the keywords received and retrieves relevant information from the database.

[0048] 4. The server organizes the search results.

[0049] Server: Organizes search results and provides information in a user-friendly format. For example, it selects highly relevant information and generates response data by converting it into JSON or XML format. It also adds relevant links and additional explanations as needed.

[0050] 5. The server sends the organized information to the terminal.

[0051] Server: Sends the generated response data to the terminal. This completes the transfer of data from the server to the terminal.

[0052] 6. The device displays the information.

[0053] Terminal: Displays received information in a user-friendly format. For example, the latest campaign details are displayed on the screen. Notifications can also be used to inform users when information is available.

[0054] 7. The user confirms the information.

[0055] User: Review the displayed information and resolve any questions on the spot. For example, a user can immediately explain the latest campaign strategies to a business partner.

[0056] Specific example

[0057] Situation 1: Resolving questions during a business negotiation

[0058] User: During a business meeting, typed "What are the latest campaign strategies?"

[0059] Terminal: Sends the input content to the server.

[0060] Server: Searches the database and extracts information about the latest campaign initiatives.

[0061] Server: Organizes search results and sends them to the terminal.

[0062] Terminal: Display information.

[0063] User: Review the displayed information and provide it to the business partner.

[0064] Situation 2: Inquiry from an agency

[0065] User: "Is there an updated version of the operation manual?"

[0066] Terminal: Sends the input content to the server.

[0067] Server: Search the database and extract information from the latest operation manual.

[0068] Server: Organizes search results and sends them to the terminal.

[0069] Terminal: Display information.

[0070] User: Review the displayed information and immediately provide the agent with a link to the latest operation manual and other relevant information.

[0071] Thus, the system of the present invention improves operational efficiency and productivity by providing information in real time.

[0072] The following describes the processing flow.

[0073] Step 1:

[0074] The user enters a question or keyword.

[0075] User: Enter "Latest Campaign Measures" into the input field on the device.

[0076] Step 2:

[0077] The terminal sends the entered question or keyword to the server.

[0078] Terminal: Sends input data to the server via API. This includes metadata such as user ID and timestamp.

[0079] Step 3:

[0080] The server analyzes the questions and keywords it receives.

[0081] Server: Analyzes incoming data and extracts keywords necessary for searching.

[0082] Step 4:

[0083] The server searches the database.

[0084] Server: Generates database queries based on extracted keywords and searches the database.

[0085] Step 5:

[0086] The server organizes the search results.

[0087] Server: Organizes information retrieved from the database and selects the most relevant information.

[0088] Step 6:

[0089] The server converts the organized information into a format that is easy for the user to understand.

[0090] Server: Converts information into formats such as JSON or XML, adds relevant links and additional explanations, and generates response data.

[0091] Step 7:

[0092] The server sends the organized information to the terminal.

[0093] Server: Sends the generated response data to the terminal as an API response.

[0094] Step 8:

[0095] The device displays the information it has received to the user.

[0096] Terminal: Analyzes received data and displays it on the screen in a user-friendly format.

[0097] Step 9:

[0098] The user reviews the displayed information and resolves any questions.

[0099] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[0100] (Example 1)

[0101] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0102] In today's business environment, there is a growing need for systems that provide reliable information quickly to address the wide range of questions and inquiries that arise during work and business negotiations. However, existing systems make it difficult to resolve questions in real time and access information immediately. Furthermore, when search results are overwhelming, it becomes difficult for users to find the information they need.

[0103] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0104] In this invention, the server includes means for the user to input a question or keyword, means for the terminal to transmit the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for the server to organize the search results and provide the information in a format that is easy for the user to understand, means for the terminal to display the received information, and means for the user to confirm the information. This enables users to resolve questions in real time during business negotiations or work, improving work efficiency and enhancing the user experience.

[0105] A "user" is an entity that uses a system to input questions or keywords and obtain information.

[0106] A "terminal" is a computer device that allows users to input questions or keywords, send them to a server, and display the received information.

[0107] A "server" is a central processing unit that receives questions and keywords sent from terminals, searches a database to organize the information, and returns a response to the terminals.

[0108] A "database" is an information system that stores related information and is used by servers to retrieve data based on search queries.

[0109] A "question" is a question or problem that a user wants to resolve.

[0110] "Keywords" are important words or phrases that users use when entering questions.

[0111] "Searching" is the process by which a server checks data within a database and retrieves relevant information.

[0112] "Information" refers to the data and answers that the server provides based on the user's questions and keywords.

[0113] "Organization" is the process by which a server converts search results retrieved from a database into a format that is easy for users to understand.

[0114] "Provision" refers to the act of a server processing information, sending it to a terminal, and the terminal displaying that information to the user.

[0115] "Display" refers to the act of visually showing information received by a device to the user.

[0116] "Confirmation" refers to the act of a user viewing and understanding the information displayed on their device and taking the necessary action.

[0117] Modes for carrying out the invention

[0118] The system of this invention is designed to quickly and effectively resolve questions that arise during business operations or negotiations. The system operates as follows: the user inputs questions or keywords via a terminal, the server analyzes them, retrieves relevant information from a database, and presents it to the user.

[0119] Hardware and software configuration

[0120] The system consists of the following main elements:

[0121] Device: A device operated by the user, including personal computers, tablets, and smartphones. In addition to browsers, devices can use dedicated applications.

[0122] A server is a central processing unit that handles data processing. It receives questions and keywords, searches databases, and organizes the results. Examples of servers include typical web servers and cloud servers.

[0123] Database: An information system that stores related information, using relational database management systems such as MySQL (registered trademark) or PostgreSQL.

[0124] Software name

[0125] Frontend development: JavaScript frameworks such as React and Vue.js.

[0126] Backend development: Server-side frameworks such as Python and Node.js

[0127] API: Use a RESTful API to send and receive data between the device and the server.

[0128] Database: MySQL, PostgreSQL, or other RDBMS

[0129] Program Processing Description

[0130] 1. The user enters a question or keyword.

[0131] Users enter questions or keywords into the input fields on their terminals while negotiating with clients or performing tasks. An example of this prompt text is the question, "What are the latest campaign initiatives?"

[0132] 2. Send the entered question to the server.

[0133] The terminal sends the entered question or keyword to the server. A POST request is used via the API, and the request includes the question or keyword, user ID, and current date and time.

[0134] 3. The server searches the database.

[0135] The server analyzes the received questions and keywords and searches the database. Based on these keywords, the server generates SQL queries and retrieves relevant information from the database.

[0136] 4. Organize search results and provide information.

[0137] The server organizes the retrieved search results and provides the information in a format that is easy for the user to understand. For example, it converts them to JSON format, selects the most relevant information, and generates response data.

[0138] 5. The device displays the information.

[0139] The terminal displays information received from the server in a user-friendly format. Users can view search results through the terminal's screen.

[0140] 6. The user confirms the information.

[0141] Users can review the displayed information and resolve any questions on the spot. For example, they can immediately explain the latest campaign initiatives displayed to their business partners.

[0142] Specific examples of operation

[0143] Situation 1: Resolving questions during a business negotiation

[0144] User: "What are the latest campaign initiatives?" is typed into the terminal.

[0145] Terminal: Sends the input content to the server.

[0146] Server: Searches the database and extracts the necessary information.

[0147] Server: Organizes search results and sends them to the terminal.

[0148] Terminal: Displays information.

[0149] User: Review the displayed information and explain it to the business partner.

[0150] Situation 2: Responding to inquiries from agencies

[0151] User: "Is there an updated version of the latest operation manual?"

[0152] Terminal: Sends the input content to the server.

[0153] Server: Searches the database and retrieves manual update information.

[0154] Server: Organizes search results and sends them to the terminal.

[0155] Terminal: Displays information.

[0156] User: Review the displayed information and provide the agent with a link to the latest operation manual.

[0157] Example of a prompt

[0158] By inputting prompts into the generative AI model, you can obtain answers to questions about the system's behavior. For example:

[0159] "If you have any questions about this system, please enter the following: 'What are the latest campaign initiatives?'"

[0160] The system of this invention improves operational efficiency and productivity by providing information in real time, and enables users to quickly resolve their questions.

[0161] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0162] Step 1:

[0163] The user enters a question or keyword.

[0164] Specific action: While negotiating with a client or performing tasks, the user enters questions or keywords into the input field on the terminal. For example, they might enter, "What are the latest campaign strategies?"

[0165] Input: The user enters questions or keywords in text format into the device.

[0166] Output: Text data of the questions and keywords entered into the terminal is generated.

[0167] Step 2:

[0168] The terminal sends the input to the server.

[0169] Specific operation: The terminal sends the entered question or keyword to the server. Using a POST request via the API, the question or keyword, user ID, and current date and time are sent to the server in JSON format.

[0170] Input: User-entered question or keyword, User ID, current date and time

[0171] Output: API request to the server (JSON format)

[0172] Step 3:

[0173] The server analyzes the questions and keywords it receives.

[0174] Specific operation: The server analyzes the received request data and extracts questions and keywords. Text analysis algorithms and natural language processing techniques are used for the analysis process.

[0175] Input: API request data that reached the server

[0176] Output: Text data of analyzed questions and keywords

[0177] Step 4:

[0178] The server searches the database.

[0179] Specific operation: Based on the analyzed questions and keywords, the server generates SQL queries against the database and searches the database. A database management system (e.g., MySQL, PostgreSQL) is used for the search process.

[0180] Input: Analyzed questions and keywords

[0181] Output: Search results (raw data) retrieved from the database.

[0182] Step 5:

[0183] The server organizes the search results.

[0184] Specific operation: The server organizes search results into a format that is easy for the user to understand. Data formatting is done using data filtering and ranking algorithms. The organized data is often converted to JSON format.

[0185] Input: Search results retrieved from the database

[0186] Output: Organized search results (response data in JSON format)

[0187] Step 6:

[0188] The server sends the organized information to the terminal.

[0189] Specific operation: The server sends organized response data to the terminal. JSON data is sent using the HTTP response.

[0190] Input: Organized search results (response data in JSON format)

[0191] Output: HTTP response to the terminal

[0192] Step 7:

[0193] The device displays the information.

[0194] Specific operation: The terminal displays information received from the server in a user-friendly format. For example, a UI designed with HTML and CSS is used to display retrieved information on a web page. Notifications can also be used to inform the user when information is available.

[0195] Input: Search results received from the server (response data in JSON format)

[0196] Output: Information displayed to the user (text and links on the web page)

[0197] Step 8:

[0198] The user verifies the information.

[0199] Specific actions: Users view information through their device screen and take the next action as needed. For example, during a business negotiation, the user can move to the next step by sharing details of the latest campaign with their business partner.

[0200] Input: Information displayed on the device

[0201] Output: User understanding and next actions (e.g., advancing the deal)

[0202] In this way, the entire system's processing progresses with specific inputs and outputs at each step, enabling quick answers to user questions.

[0203] (Application Example 1)

[0204] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0205] Currently, in brick-and-mortar stores, staff are required to quickly access and provide product and campaign information to customers. However, searching for this information is time-consuming, making immediate responses difficult. This leads to decreased customer satisfaction and inefficiencies in operations. To solve this problem, a system is needed that can quickly acquire and provide necessary information in real time.

[0206] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0207] In this invention, the server includes means for a user to input a question or keyword, means for transmitting the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for organizing the search results and providing the information in a format that is easy for staff to understand, means for the terminal to display the information organized based on the search to the staff, and means for the terminal to notify the staff of the provided information. This enables staff in physical stores to respond quickly and accurately to customer questions.

[0208] A "user" is someone who uses this system to input questions or keywords.

[0209] A "question or keyword" refers to a string of characters or words that a user enters to specifically indicate the information they want to know.

[0210] A "server" is an information processing device that receives questions or keywords sent by users, searches a database based on them, organizes the search results, and provides the information.

[0211] A "database" is a collection of information stored within a system, and it is the source of information that a server searches.

[0212] A "search method" refers to the function or process by which a server retrieves relevant information from a database based on a question or keyword it receives.

[0213] "Organization methods" refer to the functions and processes that a server uses to convert search results into a format that is easy for users and staff to understand.

[0214] "Information provision means" refers to the functions and processes that the server uses to communicate the search results it has compiled to users and staff.

[0215] A "terminal" refers to a device such as a computer, smartphone, or smart glasses that users or staff use to input questions or keywords, or to display information from a server.

[0216] "Display means" refers to the function or process by which a terminal visually displays information sent from a server.

[0217] "Notification means" refers to the functions or processes that a device uses to inform users or staff about the provision of information.

[0218] System Overview

[0219] This invention provides a system for enabling staff in physical stores to respond quickly and accurately to customer inquiries. The system is implemented using a terminal for inputting questions and keywords, a server for processing the input, and a database for providing search results.

[0220] Hardware and software configuration

[0221] 1. Hardware

[0222] Terminal: This refers to a smartphone (Android® or iOS) or smart glasses (e.g., Google® Glass®) used by staff who are users of the system.

[0223] Server: A cloud-based server that receives, processes, and retrieves information. Examples include AWS (registered trademark) and Google Cloud.

[0224] 2. Software

[0225] Mobile application: Installed on smartphones or smart glasses used by staff. Development is done using Android Studio (Java® or Kotlin) or Xcode (Swift).

[0226] Server software: Developed using Node.js, providing a RESTful API.

[0227] Database: We manage information using MySQL or Firebase.

[0228] Program processing

[0229] When a user enters a question or keyword using a device such as a smartphone or smart glasses, the content is sent to the server. The server analyzes the received question or keyword and searches its database. It then organizes the search results and provides the information in a format that is easy for the user to understand.

[0230] Data processing and calculations

[0231] Data reception and transmission: Questions and keywords entered by the user on their device are sent to the server via the API. The request includes not only the question and keywords, but also metadata such as the user ID and the date and time the request was made.

[0232] Data retrieval: The server analyzes the received keywords and searches the database for relevant information. This part utilizes natural language processing techniques to generate the optimal query for the keywords.

[0233] Data organization: Acquired data is organized on the server and converted to JSON or XML format. This ensures that information is provided in a format that is easy for users to understand.

[0234] Information provision: The organized information is sent back to the terminal via the API, notified to the staff, and displayed on the terminal's screen.

[0235] Specific example

[0236] Example 1: Customer Service

[0237] Situation: A customer asks, "What's the latest campaign information?"

[0238] Instructions: The staff member enters "campaign information" into their smartphone.

[0239] Processing: The smartphone sends the input content to the server. The server searches and organizes the latest campaign information from the database and sends the results to the smartphone.

[0240] Display: Campaign information is displayed on smartphones. Staff provide this information to customers.

[0241] Example 2: In-store event announcement

[0242] Situation: A customer asks, "What's the next in-store event?"

[0243] Instructions: The staff member enters "in-store event" into the smart glasses.

[0244] Processing: The smart glasses send the input to the server. The server searches and organizes event information from the database and sends the results back to the smart glasses.

[0245] Display: Smart glasses will display information about upcoming in-store events. Staff will then inform customers of this information.

[0246] Example of a prompt

[0247] "Please tell me the details of the next in-store event."

[0248] "Please display the latest campaign information."

[0249] This invention aims to improve the responsiveness of in-store staff, enhance customer satisfaction, and increase operational efficiency by quickly acquiring and providing necessary information in real time.

[0250] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0251] Step 1:

[0252] The user enters a question or keyword.

[0253] Users input questions or keywords using their smartphones or smart glasses. For example, if a customer asks, "What are the latest campaign details?", the user would input "campaign details". This input is then sent to the server in the form of an API request, as described later.

[0254] Input: Customer question "What are the latest campaign details?"

[0255] Output: Input data "Campaign Information" to a smartphone or smart glasses.

[0256] Step 2:

[0257] The terminal sends input to the server.

[0258] The terminal sends the entered questions and keywords to the server. Specifically, it sends an HTTP POST request to the API endpoint, and the request includes metadata such as the user ID and the current date and time along with the input content.

[0259] Input: Questions or keywords entered on the device, such as "campaign information".

[0260] Output: API request sent to the server (questions, keywords + metadata)

[0261] Step 3:

[0262] The server searches the database.

[0263] The server analyzes the received questions and keywords and searches the database. It uses a generative AI model to perform natural language processing and generate the optimal query. Based on the query, it retrieves relevant information from the database.

[0264] Input: API request received by the server (question or keyword + metadata)

[0265] Output: Raw data (related information) retrieved from the database.

[0266] Step 4:

[0267] The server organizes the search results.

[0268] The server organizes the acquired data and converts it into a user-friendly format. Specifically, it selects highly relevant information and converts it into JSON or XML format to generate response data. Relevant links and additional explanations are also added as needed.

[0269] Input: Raw data retrieved from the database (related information)

[0270] Output: Organized response data (in JSON or XML format)

[0271] Step 5:

[0272] The server sends the organized information to the terminal.

[0273] The server sends the generated response data to the terminal. This ensures that the necessary information is provided to the user's terminal in real time.

[0274] Input: Response data generated within the server (organized information)

[0275] Output: Response data sent to the terminal

[0276] Step 6:

[0277] The device displays information

[0278] The device displays received information in a user-friendly format. The displayed content includes detailed information and related links. Notifications are also used to inform the user that information has been provided.

[0279] Input: Response data sent from the server

[0280] Output: Information displayed on the terminal (campaign information and related links)

[0281] Step 7:

[0282] The user checks the information

[0283] The user checks the displayed information and provides it to the customer on the spot. For example, the latest campaign information can be immediately explained to the customer.

[0284] Input: Information displayed on the terminal (campaign information and related links)

[0285] Output: The latest campaign information provided to the customer

[0286] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0287] The system of the present invention aims to immediately resolve doubts in business and negotiations and improve the efficiency and productivity of business. In addition, by combining an emotion engine that recognizes the user's emotion, it aims to further enhance the appropriateness of the information provided.

[0288] Configuration of the entire system

[0289] The system is configured as follows:

[0290] 1. A terminal that provides a user interface

[0291] 2. A server that processes information

[0292] 3. A database that stores information

[0293] 4. Emotion engine that recognizes user emotions

[0294] Program processing

[0295] 1. The user enters a question or keyword.

[0296] User: Enter questions or keywords such as "What are the latest campaign strategies?" into the device.

[0297] 2. The terminal sends the input to the server.

[0298] Terminal: Sends input data to the emotion engine to analyze the user's emotions. Then, it sends questions, keywords, and emotion data to the server.

[0299] 3. The server searches the database.

[0300] Server: Searches the database based on received questions, keywords, and sentiment data. Sentiment data is also considered when selecting highly relevant information.

[0301] 4. The server organizes the search results.

[0302] Server: Organizes information retrieved from the database and adjusts the format and priority of the information based on sentiment data. For example, if a user is confused, it prioritizes providing more detailed and specific information.

[0303] 5. The server sends the organized information to the terminal.

[0304] Server: Sends organized information to the terminal. The information is provided in JSON or XML format and includes additional explanations and related links.

[0305] 6. The device displays the information.

[0306] Terminal: Displays the received information to the user. Based on the emotional data, the interface design and notification method are also adjusted.

[0307] 7. The user checks the information and resolves doubts

[0308] User: Can check the displayed information and immediately explain it to the negotiation partner.

[0309] Specific example

[0310] Situation 1: Doubt resolution and emotion recognition during negotiation

[0311] 1. User: During the negotiation, enter "What are the latest campaign measures?" At that time, the user's voice and expression are analyzed by the emotion engine, and it is recognized that the user is a little nervous.

[0312] 2. Terminal: Sends the doubt to the server together with the analyzed emotional data.

[0313] 3. Server: Searches the database and selects campaign measure information in a format that is easy to understand, taking into account the nervous user.

[0314] 4. Server: Organizes the selected information in detail and sends it to the terminal.

[0315] 5. Terminal: Because the user is nervous, the information is displayed using easy-to-understand color display and large fonts.

[0316] 6. User: Looks at the displayed information and immediately explains it to the negotiation partner, enabling good communication.

[0317] ] Situation 2: Inquiry from the agency and emotion recognition

[0318] 1. User: "Is there an updated version of the operation manual?" At this point, the user's voice and facial expressions are analyzed by the emotion engine, and it is recognized that the user is confused.

[0319] 2. Terminal: Sends the question to the server along with the analyzed sentiment data.

[0320] 3. Server: Search the database and prioritize selecting user manuals with more detailed instructions for users who are confused.

[0321] 4. Server: Organizes the selected information and sends it to the terminal.

[0322] 5. Device: Displays information in an easy-to-understand format, and provides supplementary explanations as needed.

[0323] 6. User: Can review the displayed information and explain to the agent how to provide the latest operation manual.

[0324] Thus, the system of the present invention, which combines an emotion engine, enables the provision of appropriate information according to the user's situation, thereby significantly improving the efficiency and quality of business operations and negotiations.

[0325] The following describes the processing flow.

[0326] Step 1:

[0327] The user enters a question or keyword.

[0328] User: Enter "What are the latest campaign measures?" into the input field on the device.

[0329] Step 2:

[0330] The device sends the entered question or keyword to the emotion engine.

[0331] Terminal: Captures user voice and facial expressions along with input content, and sends this to the emotion engine.

[0332] Step 3:

[0333] The emotion engine analyzes the user's emotions.

[0334] Emotion Engine: Analyzes captured audio and facial expression data to recognize emotions such as tension or confusion in the user.

[0335] Step 4:

[0336] The emotion engine sends the analysis results to the terminal.

[0337] Emotion Engine: Sends emotion analysis results back to the device. For example, it may include information such as "the user is feeling nervous."

[0338] Step 5:

[0339] The device sends the sentiment analysis results, along with any questions or keywords, to the server.

[0340] Terminal: Sends user input and sentiment analysis results to the server via API.

[0341] Step 6:

[0342] The server analyzes the questions, keywords, and sentiment data it receives.

[0343] Server: Analyzes questions and keywords, and generates optimal queries considering sentiment data.

[0344] Step 7:

[0345] The server searches the database.

[0346] Server: Searches the database based on the generated query and retrieves the most relevant information.

[0347] Step 8:

[0348] The server organizes the search results.

[0349] Server: Organizes the acquired information and adjusts the priority and display method of the information, taking into account the sentiment analysis results. For example, it provides information in a format that is particularly easy to understand for users who are feeling anxious.

[0350] Step 9:

[0351] The server sends the organized information to the terminal.

[0352] Server: Sends organized information to the terminal in JSON or XML format.

[0353] Step 10:

[0354] The device displays the information it has received to the user.

[0355] Terminal: Analyzes received data and displays it in a user-friendly format. For example, it might use large fonts or color displays.

[0356] Step 11:

[0357] The user reviews the displayed information and resolves any questions.

[0358] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[0359] (Example 2)

[0360] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0361] Traditional systems can provide information based on user-inputted questions and keywords, but they fail to consider the user's emotional state, which can result in providing information that is optimal for the user. In particular, in business negotiations and operations, user emotions significantly influence information comprehension and decision-making, so ignoring this can lead to problems such as decreased work efficiency and reduced quality of communication.

[0362] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0363] In this invention, the server includes means for transmitting a user-inputted question or keyword to an emotion recognition engine to generate emotion data; means for transmitting the generated emotion data along with the question or keyword to the server; means for searching a database based on the received question or keyword and emotion data; and means for organizing the search results based on the emotion data and providing the information in a format that is easy for the user to understand. This makes it possible to provide appropriate information according to the user's emotional state.

[0364] A "user" refers to an entity that uses a system to input questions or keywords and obtain information.

[0365] A "question" refers to a question that a user enters as information they want to know or a problem they want to solve.

[0366] A "keyword" refers to a word or phrase that a user enters to search for specific information.

[0367] An "emotion recognition engine" refers to a software or hardware system that analyzes a user's voice and facial expressions and generates data on their emotional state.

[0368] "Emotional data" refers to data generated by an emotion recognition engine that indicates the user's current emotional state.

[0369] A "server" refers to a central processing unit that analyzes received data, searches databases, and provides information.

[0370] A "database" refers to a data storage system that systematically stores information and manages it in a searchable format.

[0371] "Searching" refers to the process by which a server examines information within a database and extracts data that matches specific criteria.

[0372] "Providing information" refers to presenting data that is highly relevant to the questions or keywords entered by the user.

[0373] The system of this invention aims to improve the efficiency and productivity of work by immediately resolving questions in business operations and negotiations. Furthermore, by combining it with an emotion engine that recognizes user emotions, it aims to further enhance the appropriateness of the information provided.

[0374] System Configuration

[0375] The system consists of the following elements:

[0376] 1. A terminal that provides a user interface.

[0377] Hardware required: Windows PC, Mac, mobile devices (iOS, Android)

[0378] Software to use: Browser (Chrome, Safari, etc.)

[0379] 2. Servers that process information

[0380] Hardware used: Linux (registered trademark) server

[0381] Software used: Apache(registered trademark) HTTP Server

[0382] 3. Database for storing information

[0383] Hardware used: Hard disk, SSD

[0384] Software used: MySQL, PostgreSQL

[0385] 4. Emotion engine that recognizes user emotions

[0386] Hardware used: Camera, microphone

[0387] Software used: Microsoft® Azure® Emotion API, IBM Watson® Tone Analyzer

[0388] Program processing

[0389] 1. The user enters a question or keyword.

[0390] Users input questions or keywords on their own devices. For example, during a business meeting, they might type "What are the latest campaign strategies?" using the keyboard. Alternatively, they might use voice input to ask "What are the latest campaign strategies?"

[0391] 2. The device sends the input to the emotion engine.

[0392] The device receives the entered question or keyword and sends that data to the emotion engine. The emotion engine analyzes the user's voice tone and facial expressions and generates the user's emotion data (e.g., tension, confusion, joy, etc.).

[0393] 3. The device sends question and sentiment data to the server.

[0394] The device sends emotion data returned from the emotion engine, along with questions and keywords entered by the user, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[0395] 4. The server searches the database.

[0396] The server analyzes the received questions, keywords, and sentiment data, and searches a MySQL or PostgreSQL database. The server generates SQL queries to extract relevant information.

[0397] 5. The server organizes the search results.

[0398] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking sentiment data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious. Specifically, it performs filtering based on the importance of the information and the user's emotions.

[0399] 6. The server sends the organized information to the terminal.

[0400] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[0401] 7. The device displays the information.

[0402] The device displays received information to the user. Based on emotional data, the interface design and display methods are adjusted. For example, color displays and larger fonts are used to alleviate tension.

[0403] 8. Users verify information and resolve their questions.

[0404] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[0405] Specific example

[0406] 1. Resolving questions and recognizing emotions during business negotiations

[0407] User: During a business meeting, the user voice-inputs "What are the latest campaign strategies?" into their laptop. The tone of voice and facial expression are captured by the camera.

[0408] Terminal: Sends input data to the emotion engine and retrieves emotional data that indicates tension. Next, it sends the question along with the analyzed data to the server.

[0409] Server: Searches the database and selects easy-to-understand campaign strategy information.

[0410] Server: Organizes information and sends it to the terminal.

[0411] Terminal: Displays information using large fonts and colored highlighting.

[0412] User: Confirm information and immediately explain it to the business partner. Share detailed materials to facilitate smooth business negotiations.

[0413] 2. Inquiries from agencies and emotional recognition

[0414] User: "Is there an updated version of the operation manual?" is typed on the smartphone. Voice input and facial expression capture via camera are used.

[0415] Terminal: Sends input data to the emotion engine and retrieves emotion data that analyzes the user as confused. Next, it sends the emotion data and question to the server.

[0416] Server: Search the database and prioritize selecting user manuals that provide detailed instructions for users who are experiencing difficulties.

[0417] Server: Organizes information and sends it to the terminal.

[0418] Device: Displays information in an easy-to-understand format, and links to video tutorials if necessary.

[0419] User: Review the information and provide detailed instructions to the agency on how to provide the latest operation manual. Share video tutorials to enhance understanding.

[0420] In this way, by enabling the provision of appropriate information tailored to the user's emotional state, it is possible to significantly improve work efficiency and the quality of communication.

[0421] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0422] Step 1:

[0423] The user enters a question or keyword.

[0424] Specifically, the user enters a question, such as "What are the latest campaign initiatives?", using a text area or voice input on their device. The input data is stored on the device in its initial format.

[0425] input:

[0426] User questions and keywords (e.g., "What are the latest campaign strategies?")

[0427] output:

[0428] Initial format input data

[0429] Step 2:

[0430] The device sends input to the emotion engine.

[0431] Specifically, the device captures the entered text data as well as the user's voice and facial expressions, and sends them to an emotion recognition engine. Microsoft Azure Emotion API and IBM Watson Tone Analyzer are examples of such systems. The emotion engine analyzes the user's voice tone and facial expressions to generate emotion data.

[0432] input:

[0433] Initial format input data

[0434] User voice and facial expression data

[0435] output:

[0436] User emotion data (e.g., "nervous")

[0437] Step 3:

[0438] The device sends question and emotion data to the server.

[0439] After the emotion data is generated, the device sends that data, along with any questions or keywords, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[0440] input:

[0441] Initial format input data

[0442] Emotional data

[0443] output:

[0444] Question and sentiment data formatted in JSON format

[0445] Step 4:

[0446] The server searches the database.

[0447] The server analyzes the received questions, keywords, and sentiment data, and searches MySQL or PostgreSQL databases based on this analysis. The server generates SQL queries to extract the most relevant information.

[0448] input:

[0449] Question and sentiment data formatted in JSON format

[0450] output:

[0451] Related information extracted based on SQL queries

[0452] Step 5:

[0453] The server organizes the search results.

[0454] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking emotional data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious.

[0455] input:

[0456] Related information extracted from the database

[0457] Emotional data

[0458] output:

[0459] Organized information (including diagrams, graphs, and explanatory text)

[0460] Step 6:

[0461] The server sends the organized information to the terminal.

[0462] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[0463] input:

[0464] Organized information

[0465] output:

[0466] Information formatted in JSON or XML format

[0467] Step 7:

[0468] The device displays the information.

[0469] The device displays received information to the user. Based on emotional data, it adjusts the interface design and notification methods. For example, it uses color displays and large fonts to alleviate tension.

[0470] input:

[0471] Information formatted in JSON or XML format

[0472] Emotional data

[0473] output:

[0474] Adjusted information displayed to the user

[0475] Step 8:

[0476] Users can verify information and resolve their questions.

[0477] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[0478] input:

[0479] Adjusted information

[0480] output:

[0481] Understood information

[0482] Use in business negotiations and explanations.

[0483] (Application Example 2)

[0484] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0485] In modern business and negotiations, it is essential to quickly resolve questions and provide appropriate information. However, there is a lack of systems that can immediately answer user questions, and there are no mechanisms to adjust the way information is delivered based on emotions. As a result, user satisfaction and the efficiency of negotiations suffer.

[0486] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions using an emotion engine, means for adjusting the format and priority of the information provided according to the user's emotions, and means for displaying the information using a smart device. This makes it possible to answer the user's questions in real time while providing information in an optimal form according to the user's emotions.

[0487] A "user" is an individual or legal entity that uses the system and obtains information by entering questions or keywords.

[0488] A "question" refers to a specific question or point of confusion that a user has, and is information that the user seeks to resolve from the system.

[0489] A "keyword" is a specific word or phrase entered by the user, which serves as the basis for the system to search for information.

[0490] A "server" is a computer device that receives user input, searches a database, and provides relevant information.

[0491] A "database" is a collection of diverse information that is organized and stored, and is what a server searches for.

[0492] An "emotion engine" is a technology that analyzes a user's emotions and adjusts how information is provided within the system based on the results.

[0493] "Emotions" or "emotional data" refers to information that represents the user's psychological state and is the subject of analysis and consideration by the system.

[0494] A "smart device" is a device that connects to a network and can display and operate digital information. Specific examples include smart glasses and head-mounted displays.

[0495] "Information format" refers to the composition, design, font size, color scheme, and other elements of how information provided to users is visually displayed.

[0496] "Priority" refers to the order and importance of information presented to the user, and is adjusted based on the user's emotions.

[0497] "Organization" refers to the process of compiling information retrieved from a database into a format that is easy for users to understand.

[0498] "Providing" means the act of a system displaying organized information on a user's device, making it accessible to the user.

[0499] "Real-time" means processing user input and emotions instantly and providing results immediately.

[0500] This invention is a system in which a user inputs questions or keywords using a smart device and obtains relevant information in real time. The system is implemented with the following configuration.

[0501] Hardware configuration

[0502] Device: A device such as smart glasses or a head-mounted display. This device has a built-in microphone and camera that can capture the user's voice and facial expressions.

[0503] Server: The server computer performs tasks such as searching the database and running the sentiment engine.

[0504] Database: Database software such as MySQL or PostgreSQL is used to store and manage necessary information.

[0505] Software Configuration

[0506] Emotion Engine: Uses Microsoft Azure Cognitive Services, Affectiva, and other tools to analyze user emotions in real time.

[0507] Database search: A software module for searching for relevant information from a database.

[0508] Display Control: A software module for displaying searched information on smart glasses.

[0509] Operation flow

[0510] 1. The user enters a question.

[0511] Enter questions or keywords such as "What are the latest camera features?" into the device via voice or text.

[0512] 2. Analysis of emotions

[0513] The device transmits voice and facial expression data to an emotion engine, which analyzes the user's emotions. This analysis includes information such as whether the user is nervous or excited.

[0514] 3. Sending to the server

[0515] The device sends questions, keywords, and sentiment data to the server.

[0516] 4. Information Search

[0517] The server searches the database based on questions and keywords and retrieves relevant information. It then adjusts the format and priority of the information based on sentiment data.

[0518] 5. Display of Information

[0519] The organized information is sent to the device and displayed on the smart glasses in a format optimized for the user's emotions, such as using large fonts or illustrations.

[0520] Specific example

[0521] For example, while a user is wearing smart glasses, a customer might ask, "What are the latest features of this camera?" The user then inputs the question by voice through the smart glasses.

[0522] The emotion engine analyzes the customer's behavior and determines that they are slightly nervous. Based on this information, the server prioritizes selecting the latest feature information from the database, presenting it in an easy-to-understand format. This information is then displayed on the smart glasses' screen using large fonts and diagrams, allowing the user to receive information quickly and accurately.

[0523] Example of a prompt

[0524] "Could you give me an example of a system that analyzes a customer's feelings of tension or anxiety when they ask a question about a product, and then displays the latest product information on smart glasses using easy-to-understand diagrams and large fonts?"

[0525] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0526] Step 1:

[0527] The user enters a question or keyword into the device. The user uses a smart device such as smart glasses to input a question or keyword, such as "What are the latest camera features?", via voice or text. The data provided by the user here is the text input of the question or keyword.

[0528] Step 2:

[0529] The device sends the entered questions and keywords to the emotion engine. The device uses its built-in microphone and camera to collect the user's voice and facial expression data and sends it to the emotion engine. The emotion engine receives this data and analyzes the user's psychological state (e.g., tension, excitement). The input data consists of voice data and facial expression data, and the output is the user's emotional state.

[0530] Step 3:

[0531] The device sends questions, keywords, and emotion data acquired by the terminal to the server. The emotion information analyzed by the emotion engine, along with the questions and keywords in text format, are sent to the server together. This allows the server to process the data while considering the user's psychological state. The input data consists of questions, keywords, and emotion data; the output is the transmission of this data to the server.

[0532] Step 4:

[0533] The server searches the database based on the questions, keywords, and sentiment data it receives. The server uses a database search module to find the information most relevant to the user's input. Furthermore, it adjusts the format and priority of the information provided based on the sentiment data. The input data consists of questions, keywords, and sentiment data; the output is a list of relevant information.

[0534] Step 5:

[0535] The server organizes search results and provides information in the most appropriate format according to the user's emotional state. The server organizes the retrieved search results and formats the information in an easy-to-understand format (e.g., diagrams, color display, large font) based on the emotional data. This ensures that information is provided in a format that is easy for the user to understand. The input data consists of a list of related information and emotional data, and the output is formatted information.

[0536] Step 6:

[0537] The terminal displays organized information on the smart device. The terminal receives formatted information from the server and displays it on the smart glasses' display. Appropriate information is provided in response to the user's questions, taking into account emotions such as tension and excitement. The input data is formatted information, and the output is the information displayed in the user's field of vision.

[0538] Step 7:

[0539] Users review the displayed information and utilize it in business negotiations and work. Users check the information displayed on their smart glasses and use it to explain things to their business partners. This allows users to provide information quickly and appropriately. The input data is the information displayed on the smart device, and the output is the smooth progress of business negotiations and work.

[0540] Through the steps described above, this invention realizes a system that instantly resolves user questions and provides appropriate information according to their emotions.

[0541] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0542] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0543] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0544] [Second Embodiment]

[0545] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0546] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0547] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0548] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0549] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0551] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0552] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0553] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0554] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0555] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0556] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0557] The system of this invention is designed to allow questions regarding the details of policies and the content of agent notifications to be resolved on the spot during daily operations and business negotiations with clients. The program for this system is generated and its operation is described below.

[0558] System-wide configuration

[0559] The system primarily consists of terminals that provide the user interface, servers that process data, and databases that store information. For this system to function, each component must work in coordination with the others.

[0560] Program processing

[0561] 1. The user enters a question or keyword.

[0562] User: Enter questions or keywords into the input field on the device. For example, if a client asks "What are the latest campaign strategies?" during a business meeting, the user would enter "Latest campaign strategies."

[0563] 2. The terminal sends the input to the server.

[0564] Terminal: Sends the entered questions and keywords to the server. Specifically, it sends a request to the server via the API. This request includes metadata such as the user ID and the current date and time, along with the input content.

[0565] 3. The server searches the database.

[0566] Server: Analyzes received questions and keywords and searches the database. The server generates queries based on the keywords received and retrieves relevant information from the database.

[0567] 4. The server organizes the search results.

[0568] Server: Organizes search results and provides information in a user-friendly format. For example, it selects highly relevant information and generates response data by converting it into JSON or XML format. It also adds relevant links and additional explanations as needed.

[0569] 5. The server sends the organized information to the terminal.

[0570] Server: Sends the generated response data to the terminal. This completes the transfer of data from the server to the terminal.

[0571] 6. The device displays the information.

[0572] Terminal: Displays received information in a user-friendly format. For example, the latest campaign details are displayed on the screen. Notifications can also be used to inform users when information is available.

[0573] 7. The user confirms the information.

[0574] User: Review the displayed information and resolve any questions on the spot. For example, a user can immediately explain the latest campaign strategies to a business partner.

[0575] Specific example

[0576] Situation 1: Resolving questions during a business negotiation

[0577] User: During a business meeting, typed "What are the latest campaign strategies?"

[0578] Terminal: Sends the input content to the server.

[0579] Server: Searches the database and extracts information about the latest campaign initiatives.

[0580] Server: Organizes search results and sends them to the terminal.

[0581] Terminal: Display information.

[0582] User: Review the displayed information and provide it to the business partner.

[0583] Situation 2: Inquiry from an agency

[0584] User: "Is there an updated version of the operation manual?"

[0585] Terminal: Sends the input content to the server.

[0586] Server: Search the database and extract information from the latest operation manual.

[0587] Server: Organizes search results and sends them to the terminal.

[0588] Terminal: Display information.

[0589] User: Review the displayed information and immediately provide the agent with a link to the latest operation manual and other relevant information.

[0590] Thus, the system of the present invention improves operational efficiency and productivity by providing information in real time.

[0591] The following describes the processing flow.

[0592] Step 1:

[0593] The user enters a question or keyword.

[0594] User: Enter "Latest Campaign Measures" into the input field on the device.

[0595] Step 2:

[0596] The terminal sends the entered question or keyword to the server.

[0597] Terminal: Sends input data to the server via API. This includes metadata such as user ID and timestamp.

[0598] Step 3:

[0599] The server analyzes the questions and keywords it receives.

[0600] Server: Analyzes incoming data and extracts keywords necessary for searching.

[0601] Step 4:

[0602] The server searches the database.

[0603] Server: Generates database queries based on extracted keywords and searches the database.

[0604] Step 5:

[0605] The server organizes the search results.

[0606] Server: Organizes information retrieved from the database and selects the most relevant information.

[0607] Step 6:

[0608] The server converts the organized information into a format that is easy for the user to understand.

[0609] Server: Converts information into formats such as JSON or XML, adds relevant links and additional explanations, and generates response data.

[0610] Step 7:

[0611] The server sends the organized information to the terminal.

[0612] Server: Sends the generated response data to the terminal as an API response.

[0613] Step 8:

[0614] The device displays the information it has received to the user.

[0615] Terminal: Analyzes received data and displays it on the screen in a user-friendly format.

[0616] Step 9:

[0617] The user reviews the displayed information and resolves any questions.

[0618] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[0619] (Example 1)

[0620] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0621] In today's business environment, there is a growing need for systems that provide reliable information quickly to address the wide range of questions and inquiries that arise during work and business negotiations. However, existing systems make it difficult to resolve questions in real time and access information immediately. Furthermore, when search results are overwhelming, it becomes difficult for users to find the information they need.

[0622] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0623] In this invention, the server includes means for the user to input a question or keyword, means for the terminal to transmit the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for the server to organize the search results and provide the information in a format that is easy for the user to understand, means for the terminal to display the received information, and means for the user to confirm the information. This enables users to resolve questions in real time during business negotiations or work, improving work efficiency and enhancing the user experience.

[0624] A "user" is an entity that uses a system to input questions or keywords and obtain information.

[0625] A "terminal" is a computer device that allows users to input questions or keywords, send them to a server, and display the received information.

[0626] A "server" is a central processing unit that receives questions and keywords sent from terminals, searches a database to organize the information, and returns a response to the terminals.

[0627] A "database" is an information system that stores related information and is used by servers to retrieve data based on search queries.

[0628] A "question" is a question or problem that a user wants to resolve.

[0629] "Keywords" are important words or phrases that users use when entering questions.

[0630] "Searching" is the process by which a server checks data within a database and retrieves relevant information.

[0631] "Information" refers to the data and answers that the server provides based on the user's questions and keywords.

[0632] "Organization" is the process by which a server converts search results retrieved from a database into a format that is easy for users to understand.

[0633] "Provision" refers to the act of a server processing information, sending it to a terminal, and the terminal displaying that information to the user.

[0634] "Display" refers to the act of visually showing information received by a device to the user.

[0635] "Confirmation" refers to the act of a user viewing and understanding the information displayed on their device and taking the necessary action.

[0636] Modes for carrying out the invention

[0637] The system of this invention is designed to quickly and effectively resolve questions that arise during business operations or negotiations. The system operates as follows: the user inputs questions or keywords via a terminal, the server analyzes them, retrieves relevant information from a database, and presents it to the user.

[0638] Hardware and software configuration

[0639] The system consists of the following main elements:

[0640] Device: A device operated by the user, including personal computers, tablets, and smartphones. In addition to browsers, devices can use dedicated applications.

[0641] A server is a central processing unit that handles data processing. It receives questions and keywords, searches databases, and organizes the results. Examples of servers include typical web servers and cloud servers.

[0642] Database: An information system that stores related information, using relational database management systems such as MySQL or PostgreSQL.

[0643] Software name

[0644] Frontend development: JavaScript frameworks such as React and Vue.js

[0645] Backend development: Server-side frameworks such as Python and Node.js

[0646] API: Use a RESTful API to send and receive data between the device and the server.

[0647] Database: MySQL, PostgreSQL, or other RDBMS

[0648] Program Processing Description

[0649] 1. The user enters a question or keyword.

[0650] Users enter questions or keywords into the input fields on their terminals while negotiating with clients or performing tasks. An example of this prompt text is the question, "What are the latest campaign initiatives?"

[0651] 2. Send the entered question to the server.

[0652] The terminal sends the entered question or keyword to the server. A POST request is used via the API, and the request includes the question or keyword, user ID, and current date and time.

[0653] 3. The server searches the database.

[0654] The server analyzes the received questions and keywords and searches the database. Based on these keywords, the server generates SQL queries and retrieves relevant information from the database.

[0655] 4. Organize search results and provide information.

[0656] The server organizes the retrieved search results and provides the information in a format that is easy for the user to understand. For example, it converts them to JSON format, selects the most relevant information, and generates response data.

[0657] 5. The device displays the information.

[0658] The terminal displays information received from the server in a user-friendly format. Users can view search results through the terminal's screen.

[0659] 6. The user confirms the information.

[0660] Users can review the displayed information and resolve any questions on the spot. For example, they can immediately explain the latest campaign initiatives displayed to their business partners.

[0661] Specific examples of operation

[0662] Situation 1: Resolving questions during a business negotiation

[0663] User: "What are the latest campaign initiatives?" is typed into the terminal.

[0664] Terminal: Sends the input content to the server.

[0665] Server: Searches the database and extracts the necessary information.

[0666] Server: Organizes search results and sends them to the terminal.

[0667] Terminal: Displays information.

[0668] User: Review the displayed information and explain it to the business partner.

[0669] Situation 2: Responding to inquiries from agencies

[0670] User: "Is there an updated version of the latest operation manual?"

[0671] Terminal: Sends the input content to the server.

[0672] Server: Searches the database and retrieves manual update information.

[0673] Server: Organizes search results and sends them to the terminal.

[0674] Terminal: Displays information.

[0675] User: Review the displayed information and provide the agent with a link to the latest operation manual.

[0676] Example of a prompt

[0677] By inputting prompts into the generative AI model, you can obtain answers to questions about the system's behavior. For example:

[0678] "If you have any questions about this system, please enter the following: 'What are the latest campaign initiatives?'"

[0679] The system of this invention improves operational efficiency and productivity by providing information in real time, and enables users to quickly resolve their questions.

[0680] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0681] Step 1:

[0682] The user enters a question or keyword.

[0683] Specific action: While negotiating with a client or performing tasks, the user enters questions or keywords into the input field on the terminal. For example, they might enter, "What are the latest campaign strategies?"

[0684] Input: The user enters questions or keywords in text format into the device.

[0685] Output: Text data of the questions and keywords entered into the terminal is generated.

[0686] Step 2:

[0687] The terminal sends the input to the server.

[0688] Specific operation: The terminal sends the entered question or keyword to the server. Using a POST request via the API, the question or keyword, user ID, and current date and time are sent to the server in JSON format.

[0689] Input: User-entered question or keyword, User ID, current date and time

[0690] Output: API request to the server (JSON format)

[0691] Step 3:

[0692] The server analyzes the questions and keywords it receives.

[0693] Specific operation: The server analyzes the received request data and extracts questions and keywords. Text analysis algorithms and natural language processing techniques are used for the analysis process.

[0694] Input: API request data that reached the server

[0695] Output: Text data of analyzed questions and keywords

[0696] Step 4:

[0697] The server searches the database.

[0698] Specific operation: Based on the analyzed questions and keywords, the server generates SQL queries against the database and searches the database. A database management system (e.g., MySQL, PostgreSQL) is used for the search process.

[0699] Input: Analyzed questions and keywords

[0700] Output: Search results (raw data) retrieved from the database.

[0701] Step 5:

[0702] The server organizes the search results.

[0703] Specific operation: The server organizes search results into a format that is easy for the user to understand. Data formatting is done using data filtering and ranking algorithms. The organized data is often converted to JSON format.

[0704] Input: Search results retrieved from the database

[0705] Output: Organized search results (response data in JSON format)

[0706] Step 6:

[0707] The server sends the organized information to the terminal.

[0708] Specific operation: The server sends organized response data to the terminal. JSON data is sent using the HTTP response.

[0709] Input: Organized search results (response data in JSON format)

[0710] Output: HTTP response to the terminal

[0711] Step 7:

[0712] The device displays the information.

[0713] Specific operation: The terminal displays information received from the server in a user-friendly format. For example, a UI designed with HTML and CSS is used to display retrieved information on a web page. Notifications can also be used to inform the user when information is available.

[0714] Input: Search results received from the server (response data in JSON format)

[0715] Output: Information displayed to the user (text and links on the web page)

[0716] Step 8:

[0717] The user verifies the information.

[0718] Specific actions: Users view information through their device screen and take the next action as needed. For example, during a business negotiation, the user can move to the next step by sharing details of the latest campaign with their business partner.

[0719] Input: Information displayed on the device

[0720] Output: User understanding and next actions (e.g., advancing the deal)

[0721] In this way, the entire system's processing progresses with specific inputs and outputs at each step, enabling quick answers to user questions.

[0722] (Application Example 1)

[0723] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0724] Currently, in brick-and-mortar stores, staff are required to quickly access and provide product and campaign information to customers. However, searching for this information is time-consuming, making immediate responses difficult. This leads to decreased customer satisfaction and inefficiencies in operations. To solve this problem, a system is needed that can quickly acquire and provide necessary information in real time.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0726] In this invention, the server includes means for a user to input a question or keyword, means for transmitting the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for organizing the search results and providing the information in a format that is easy for staff to understand, means for the terminal to display the information organized based on the search to the staff, and means for the terminal to notify the staff of the provided information. This enables staff in physical stores to respond quickly and accurately to customer questions.

[0727] A "user" is someone who uses this system to input questions or keywords.

[0728] A "question or keyword" refers to a string of characters or words that a user enters to specifically indicate the information they want to know.

[0729] A "server" is an information processing device that receives questions or keywords sent by users, searches a database based on them, organizes the search results, and provides the information.

[0730] A "database" is a collection of information stored within a system, and it is the source of information that a server searches.

[0731] A "search method" refers to the function or process by which a server retrieves relevant information from a database based on a question or keyword it receives.

[0732] "Organization methods" refer to the functions and processes that a server uses to convert search results into a format that is easy for users and staff to understand.

[0733] "Information provision means" refers to the functions and processes that the server uses to communicate the search results it has compiled to users and staff.

[0734] A "terminal" refers to a device such as a computer, smartphone, or smart glasses that users or staff use to input questions or keywords, or to display information from a server.

[0735] "Display means" refers to the function or process by which a terminal visually displays information sent from a server.

[0736] "Notification means" refers to the functions or processes that a device uses to inform users or staff about the provision of information.

[0737] System Overview

[0738] This invention provides a system for enabling staff in physical stores to respond quickly and accurately to customer inquiries. The system is implemented using a terminal for inputting questions and keywords, a server for processing the input, and a database for providing search results.

[0739] Hardware and software configuration

[0740] 1. Hardware

[0741] Terminal: This refers to a smartphone (Android or iOS) or smart glasses (e.g., Google Glass) used by staff members who are users of the system.

[0742] Server: A cloud-based server that receives, processes, and retrieves information. Examples include AWS and Google Cloud.

[0743] 2. Software

[0744] Mobile application: Installed on smartphones or smart glasses used by staff. Development is done using Android Studio (Java or Kotlin) or Xcode (Swift).

[0745] Server software: Developed using Node.js, providing a RESTful API.

[0746] Database: We manage information using MySQL or Firebase.

[0747] Program processing

[0748] When a user enters a question or keyword using a device such as a smartphone or smart glasses, the content is sent to the server. The server analyzes the received question or keyword and searches its database. It then organizes the search results and provides the information in a format that is easy for the user to understand.

[0749] Data processing and calculations

[0750] Data reception and transmission: Questions and keywords entered by the user on their device are sent to the server via the API. The request includes not only the question and keywords, but also metadata such as the user ID and the date and time the request was made.

[0751] Data retrieval: The server analyzes the received keywords and searches the database for relevant information. This part utilizes natural language processing techniques to generate the optimal query for the keywords.

[0752] Data organization: Acquired data is organized on the server and converted to JSON or XML format. This ensures that information is provided in a format that is easy for users to understand.

[0753] Information provision: The organized information is sent back to the terminal via the API, notified to the staff, and displayed on the terminal's screen.

[0754] Specific example

[0755] Example 1: Customer Service

[0756] Situation: A customer asks, "What's the latest campaign information?"

[0757] Instructions: The staff member enters "campaign information" into their smartphone.

[0758] Processing: The smartphone sends the input content to the server. The server searches and organizes the latest campaign information from the database and sends the results to the smartphone.

[0759] Display: Campaign information is displayed on smartphones. Staff provide this information to customers.

[0760] Example 2: In-store event announcement

[0761] Situation: A customer asks, "What's the next in-store event?"

[0762] Instructions: The staff member enters "in-store event" into the smart glasses.

[0763] Processing: The smart glasses send the input to the server. The server searches and organizes event information from the database and sends the results back to the smart glasses.

[0764] Display: Smart glasses will display information about upcoming in-store events. Staff will then inform customers of this information.

[0765] Example of a prompt

[0766] "Please tell me the details of the next in-store event."

[0767] "Please display the latest campaign information."

[0768] This invention aims to improve the responsiveness of in-store staff, enhance customer satisfaction, and increase operational efficiency by quickly acquiring and providing necessary information in real time.

[0769] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0770] Step 1:

[0771] The user enters a question or keyword.

[0772] Users input questions or keywords using their smartphones or smart glasses. For example, if a customer asks, "What are the latest campaign details?", the user would input "campaign details". This input is then sent to the server in the form of an API request, as described later.

[0773] Input: Customer question "What are the latest campaign details?"

[0774] Output: Input data "Campaign Information" to a smartphone or smart glasses.

[0775] Step 2:

[0776] The terminal sends input to the server.

[0777] The terminal sends the entered questions and keywords to the server. Specifically, it sends an HTTP POST request to the API endpoint, and the request includes metadata such as the user ID and the current date and time along with the input content.

[0778] Input: Questions or keywords entered on the device, such as "campaign information".

[0779] Output: API request sent to the server (questions, keywords + metadata)

[0780] Step 3:

[0781] The server searches the database.

[0782] The server analyzes the received questions and keywords and searches the database. It uses a generative AI model to perform natural language processing and generate the optimal query. Based on the query, it retrieves relevant information from the database.

[0783] Input: API request received by the server (question or keyword + metadata)

[0784] Output: Raw data (related information) retrieved from the database.

[0785] Step 4:

[0786] The server organizes the search results.

[0787] The server organizes the acquired data and converts it into a user-friendly format. Specifically, it selects highly relevant information and converts it into JSON or XML format to generate response data. Relevant links and additional explanations are also added as needed.

[0788] Input: Raw data retrieved from the database (related information)

[0789] Output: Organized response data (in JSON or XML format)

[0790] Step 5:

[0791] The server sends the organized information to the terminal.

[0792] The server sends the generated response data to the terminal. This ensures that the necessary information is provided to the user's terminal in real time.

[0793] Input: Response data generated within the server (organized information)

[0794] Output: Response data sent to the terminal

[0795] Step 6:

[0796] The device displays information

[0797] The device displays received information in a user-friendly format. The displayed content includes detailed information and related links. Notifications are also used to inform the user that information has been provided.

[0798] Input: Response data sent from the server

[0799] Output: Information displayed on the device (campaign information and related links)

[0800] Step 7:

[0801] The user confirms the information.

[0802] Users can review the displayed information and provide it to customers on the spot. For example, they can instantly explain the latest campaign information to customers.

[0803] Input: Information displayed on the device (campaign information and related links)

[0804] Output: Latest campaign information provided to customers

[0805] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0806] The system of this invention aims to resolve questions immediately during business operations and negotiations, thereby improving operational efficiency and productivity. Furthermore, by combining it with an emotion engine that recognizes user emotions, it aims to further enhance the appropriateness of the information provided.

[0807] System-wide configuration

[0808] The system was configured as follows:

[0809] 1. A terminal that provides a user interface.

[0810] 2. Servers that process information

[0811] 3. Database for storing information

[0812] 4. Emotion engine that recognizes user emotions

[0813] Program processing

[0814] 1. The user enters a question or keyword.

[0815] User: Enter questions or keywords such as "What are the latest campaign strategies?" into the device.

[0816] 2. The terminal sends the input to the server.

[0817] Terminal: Sends input data to the emotion engine to analyze the user's emotions. Then, it sends questions, keywords, and emotion data to the server.

[0818] 3. The server searches the database.

[0819] Server: Searches the database based on received questions, keywords, and sentiment data. Sentiment data is also considered when selecting highly relevant information.

[0820] 4. The server organizes the search results.

[0821] Server: Organizes information retrieved from the database and adjusts the format and priority of the information based on sentiment data. For example, if a user is confused, it prioritizes providing more detailed and specific information.

[0822] 5. The server sends the organized information to the terminal.

[0823] Server: Sends organized information to the terminal. The information is provided in JSON or XML format and includes additional explanations and related links.

[0824] 6. The device displays the information.

[0825] Terminal: Displays received information to the user. Based on sentiment data, the interface design and notification methods are also adjusted.

[0826] 7. Users verify information and resolve their questions.

[0827] User: Can review the displayed information and immediately explain it to the business partner.

[0828] Specific example

[0829] Situation 1: Resolving questions and recognizing emotions during a business negotiation

[0830] 1. User: During a business meeting, the user types "What are the latest campaign strategies?". At that time, the user's voice and facial expressions are analyzed by an emotion engine, and it is recognized that the user is slightly nervous.

[0831] 2. Terminal: Sends the question to the server along with the analyzed sentiment data.

[0832] 3. Server: Search the database and select campaign information in an easy-to-understand format, taking into consideration the needs of anxious users.

[0833] 4. Server: Organizes the selected information in detail and sends it to the terminal.

[0834] 5. Device: Since users may be stressed, display information clearly using color and large fonts.

[0835] 6. User: They can view the displayed information, immediately explain it to their business partner, and communicate effectively.

[0836] Situation 2: Inquiry from an agency and emotional recognition

[0837] 1. User: "Is there an updated version of the operation manual?" At this point, the user's voice and facial expressions are analyzed by the emotion engine, and it is recognized that the user is confused.

[0838] 2. Terminal: Sends the question to the server along with the analyzed sentiment data.

[0839] 3. Server: Search the database and prioritize selecting user manuals with more detailed instructions for users who are confused.

[0840] 4. Server: Organizes the selected information and sends it to the terminal.

[0841] 5. Device: Displays information in an easy-to-understand format, and provides supplementary explanations as needed.

[0842] 6. User: Can review the displayed information and explain to the agent how to provide the latest operation manual.

[0843] Thus, the system of the present invention, which combines an emotion engine, enables the provision of appropriate information according to the user's situation, thereby significantly improving the efficiency and quality of business operations and negotiations.

[0844] The following describes the processing flow.

[0845] Step 1:

[0846] The user enters a question or keyword.

[0847] User: Enter "What are the latest campaign measures?" into the input field on the device.

[0848] Step 2:

[0849] The device sends the entered question or keyword to the emotion engine.

[0850] Terminal: Captures user voice and facial expressions along with input content, and sends this to the emotion engine.

[0851] Step 3:

[0852] The emotion engine analyzes the user's emotions.

[0853] Emotion Engine: Analyzes captured audio and facial expression data to recognize emotions such as tension or confusion in the user.

[0854] Step 4:

[0855] The emotion engine sends the analysis results to the terminal.

[0856] Emotion Engine: Sends emotion analysis results back to the device. For example, it may include information such as "the user is feeling nervous."

[0857] Step 5:

[0858] The device sends the sentiment analysis results, along with any questions or keywords, to the server.

[0859] Terminal: Sends user input and sentiment analysis results to the server via API.

[0860] Step 6:

[0861] The server analyzes the questions, keywords, and sentiment data it receives.

[0862] Server: Analyzes questions and keywords, and generates optimal queries considering sentiment data.

[0863] Step 7:

[0864] The server searches the database.

[0865] Server: Searches the database based on the generated query and retrieves the most relevant information.

[0866] Step 8:

[0867] The server organizes the search results.

[0868] Server: Organizes the acquired information and adjusts the priority and display method of the information, taking into account the sentiment analysis results. For example, it provides information in a format that is particularly easy to understand for users who are feeling anxious.

[0869] Step 9:

[0870] The server sends the organized information to the terminal.

[0871] Server: Sends organized information to the terminal in JSON or XML format.

[0872] Step 10:

[0873] The device displays the information it has received to the user.

[0874] Terminal: Analyzes received data and displays it in a user-friendly format. For example, it might use large fonts or color displays.

[0875] Step 11:

[0876] The user reviews the displayed information and resolves any questions.

[0877] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[0878] (Example 2)

[0879] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0880] Traditional systems can provide information based on user-inputted questions and keywords, but they fail to consider the user's emotional state, which can result in providing information that is optimal for the user. In particular, in business negotiations and operations, user emotions significantly influence information comprehension and decision-making, so ignoring this can lead to problems such as decreased work efficiency and reduced quality of communication.

[0881] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0882] In this invention, the server includes means for transmitting a user-inputted question or keyword to an emotion recognition engine to generate emotion data; means for transmitting the generated emotion data along with the question or keyword to the server; means for searching a database based on the received question or keyword and emotion data; and means for organizing the search results based on the emotion data and providing the information in a format that is easy for the user to understand. This makes it possible to provide appropriate information according to the user's emotional state.

[0883] A "user" refers to an entity that uses a system to input questions or keywords and obtain information.

[0884] A "question" refers to a question that a user enters as information they want to know or a problem they want to solve.

[0885] A "keyword" refers to a word or phrase that a user enters to search for specific information.

[0886] An "emotion recognition engine" refers to a software or hardware system that analyzes a user's voice and facial expressions and generates data on their emotional state.

[0887] "Emotional data" refers to data generated by an emotion recognition engine that indicates the user's current emotional state.

[0888] A "server" refers to a central processing unit that analyzes received data, searches databases, and provides information.

[0889] A "database" refers to a data storage system that systematically stores information and manages it in a searchable format.

[0890] "Searching" refers to the process by which a server examines information within a database and extracts data that matches specific criteria.

[0891] "Providing information" refers to presenting data that is highly relevant to the questions or keywords entered by the user.

[0892] The system of this invention aims to improve the efficiency and productivity of work by immediately resolving questions in business operations and negotiations. Furthermore, by combining it with an emotion engine that recognizes user emotions, it aims to further enhance the appropriateness of the information provided.

[0893] System Configuration

[0894] The system consists of the following elements:

[0895] 1. A terminal that provides a user interface.

[0896] Hardware required: Windows PC, Mac, mobile devices (iOS, Android)

[0897] Software to use: Browser (Chrome, Safari, etc.)

[0898] 2. Servers that process information

[0899] Hardware used: Linux server

[0900] Software used: Apache HTTP Server

[0901] 3. Database for storing information

[0902] Hardware used: Hard disk, SSD

[0903] Software used: MySQL, PostgreSQL

[0904] 4. Emotion engine that recognizes user emotions

[0905] Hardware used: Camera, microphone

[0906] Software used: Microsoft Azure Emotion API, IBM Watson Tone Analyzer

[0907] Program processing

[0908] 1. The user enters a question or keyword.

[0909] Users input questions or keywords on their own devices. For example, during a business meeting, they might type "What are the latest campaign strategies?" using the keyboard. Alternatively, they might use voice input to ask "What are the latest campaign strategies?"

[0910] 2. The device sends the input to the emotion engine.

[0911] The device receives the entered question or keyword and sends that data to the emotion engine. The emotion engine analyzes the user's voice tone and facial expressions and generates the user's emotion data (e.g., tension, confusion, joy, etc.).

[0912] 3. The device sends question and sentiment data to the server.

[0913] The device sends emotion data returned from the emotion engine, along with questions and keywords entered by the user, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[0914] 4. The server searches the database.

[0915] The server analyzes the received questions, keywords, and sentiment data, and searches a MySQL or PostgreSQL database. The server generates SQL queries to extract relevant information.

[0916] 5. The server organizes the search results.

[0917] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking sentiment data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious. Specifically, it performs filtering based on the importance of the information and the user's emotions.

[0918] 6. The server sends the organized information to the terminal.

[0919] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[0920] 7. The device displays the information.

[0921] The device displays received information to the user. Based on emotional data, the interface design and display methods are adjusted. For example, color displays and larger fonts are used to alleviate tension.

[0922] 8. Users verify information and resolve their questions.

[0923] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[0924] Specific example

[0925] 1. Resolving questions and recognizing emotions during business negotiations

[0926] User: During a business meeting, the user voice-inputs "What are the latest campaign strategies?" into their laptop. The tone of voice and facial expression are captured by the camera.

[0927] Terminal: Sends input data to the emotion engine and retrieves emotional data that indicates tension. Next, it sends the question along with the analyzed data to the server.

[0928] Server: Searches the database and selects easy-to-understand campaign strategy information.

[0929] Server: Organizes information and sends it to the terminal.

[0930] Terminal: Displays information using large fonts and colored highlighting.

[0931] User: Confirm information and immediately explain it to the business partner. Share detailed materials to facilitate smooth business negotiations.

[0932] 2. Inquiries from agencies and emotional recognition

[0933] User: "Is there an updated version of the operation manual?" is typed on the smartphone. Voice input and facial expression capture via camera are used.

[0934] Terminal: Sends input data to the emotion engine and retrieves emotion data that analyzes the user as confused. Next, it sends the emotion data and question to the server.

[0935] Server: Search the database and prioritize selecting user manuals that provide detailed instructions for users who are experiencing difficulties.

[0936] Server: Organizes information and sends it to the terminal.

[0937] Device: Displays information in an easy-to-understand format, and links to video tutorials if necessary.

[0938] User: Review the information and provide detailed instructions to the agency on how to provide the latest operation manual. Share video tutorials to enhance understanding.

[0939] In this way, by enabling the provision of appropriate information tailored to the user's emotional state, it is possible to significantly improve work efficiency and the quality of communication.

[0940] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0941] Step 1:

[0942] The user enters a question or keyword.

[0943] Specifically, the user enters a question, such as "What are the latest campaign initiatives?", using a text area or voice input on their device. The input data is stored on the device in its initial format.

[0944] input:

[0945] User questions and keywords (e.g., "What are the latest campaign strategies?")

[0946] output:

[0947] Initial format input data

[0948] Step 2:

[0949] The device sends input to the emotion engine.

[0950] Specifically, the device captures the entered text data as well as the user's voice and facial expressions, and sends them to an emotion recognition engine. Microsoft Azure Emotion API and IBM Watson Tone Analyzer are examples of such systems. The emotion engine analyzes the user's voice tone and facial expressions to generate emotion data.

[0951] input:

[0952] Initial format input data

[0953] User voice and facial expression data

[0954] output:

[0955] User emotion data (e.g., "nervous")

[0956] Step 3:

[0957] The device sends question and emotion data to the server.

[0958] After the emotion data is generated, the device sends that data, along with any questions or keywords, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[0959] input:

[0960] Initial format input data

[0961] Emotional data

[0962] output:

[0963] Question and sentiment data formatted in JSON format

[0964] Step 4:

[0965] The server searches the database.

[0966] The server analyzes the received questions, keywords, and sentiment data, and searches MySQL or PostgreSQL databases based on this analysis. The server generates SQL queries to extract the most relevant information.

[0967] input:

[0968] Question and sentiment data formatted in JSON format

[0969] output:

[0970] Related information extracted based on SQL queries

[0971] Step 5:

[0972] The server organizes the search results.

[0973] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking emotional data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious.

[0974] input:

[0975] Related information extracted from the database

[0976] Emotional data

[0977] output:

[0978] Organized information (including diagrams, graphs, and explanatory text)

[0979] Step 6:

[0980] The server sends the organized information to the terminal.

[0981] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[0982] input:

[0983] Organized information

[0984] output:

[0985] Information formatted in JSON or XML format

[0986] Step 7:

[0987] The device displays the information.

[0988] The device displays received information to the user. Based on emotional data, it adjusts the interface design and notification methods. For example, it uses color displays and large fonts to alleviate tension.

[0989] input:

[0990] Information formatted in JSON or XML format

[0991] Emotional data

[0992] output:

[0993] Adjusted information displayed to the user

[0994] Step 8:

[0995] Users can verify information and resolve their questions.

[0996] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[0997] input:

[0998] Adjusted information

[0999] output:

[1000] Understood information

[1001] Use in business negotiations and explanations.

[1002] (Application Example 2)

[1003] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1004] In modern business and negotiations, it is essential to quickly resolve questions and provide appropriate information. However, there is a lack of systems that can immediately answer user questions, and there are no mechanisms to adjust the way information is delivered based on emotions. As a result, user satisfaction and the efficiency of negotiations suffer.

[1005] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions using an emotion engine, means for adjusting the format and priority of the information provided according to the user's emotions, and means for displaying the information using a smart device. This makes it possible to answer the user's questions in real time while providing information in an optimal form according to the user's emotions.

[1006] A "user" is an individual or legal entity that uses the system and obtains information by entering questions or keywords.

[1007] A "question" refers to a specific question or point of confusion that a user has, and is information that the user seeks to resolve from the system.

[1008] A "keyword" is a specific word or phrase entered by the user, which serves as the basis for the system to search for information.

[1009] A "server" is a computer device that receives user input, searches a database, and provides relevant information.

[1010] A "database" is a collection of diverse information that is organized and stored, and is what a server searches for.

[1011] An "emotion engine" is a technology that analyzes a user's emotions and adjusts how information is provided within the system based on the results.

[1012] "Emotions" or "emotional data" refers to information that represents the user's psychological state and is the subject of analysis and consideration by the system.

[1013] A "smart device" is a device that connects to a network and can display and operate digital information. Specific examples include smart glasses and head-mounted displays.

[1014] "Information format" refers to the composition, design, font size, color scheme, and other elements of how information provided to users is visually displayed.

[1015] "Priority" refers to the order and importance of information presented to the user, and is adjusted based on the user's emotions.

[1016] "Organization" refers to the process of compiling information retrieved from a database into a format that is easy for users to understand.

[1017] "Providing" means the act of a system displaying organized information on a user's device, making it accessible to the user.

[1018] "Real-time" means processing user input and emotions instantly and providing results immediately.

[1019] This invention is a system in which a user inputs questions or keywords using a smart device and obtains relevant information in real time. The system is implemented with the following configuration.

[1020] Hardware configuration

[1021] Device: A device such as smart glasses or a head-mounted display. This device has a built-in microphone and camera that can capture the user's voice and facial expressions.

[1022] Server: The server computer performs tasks such as searching the database and running the sentiment engine.

[1023] Database: Database software such as MySQL or PostgreSQL is used to store and manage necessary information.

[1024] Software Configuration

[1025] Emotion Engine: Uses Microsoft Azure Cognitive Services, Affectiva, and other tools to analyze user emotions in real time.

[1026] Database search: A software module for searching for relevant information from a database.

[1027] Display Control: A software module for displaying searched information on smart glasses.

[1028] Operation flow

[1029] 1. The user enters a question.

[1030] Enter questions or keywords such as "What are the latest camera features?" into the device via voice or text.

[1031] 2. Analysis of emotions

[1032] The device transmits voice and facial expression data to an emotion engine, which analyzes the user's emotions. This analysis includes information such as whether the user is nervous or excited.

[1033] 3. Sending to the server

[1034] The device sends questions, keywords, and sentiment data to the server.

[1035] 4. Information Search

[1036] The server searches the database based on questions and keywords and retrieves relevant information. It then adjusts the format and priority of the information based on sentiment data.

[1037] 5. Display of Information

[1038] The organized information is sent to the device and displayed on the smart glasses in a format optimized for the user's emotions, such as using large fonts or illustrations.

[1039] Specific example

[1040] For example, while a user is wearing smart glasses, a customer might ask, "What are the latest features of this camera?" The user then inputs the question by voice through the smart glasses.

[1041] The emotion engine analyzes the customer's behavior and determines that they are slightly nervous. Based on this information, the server prioritizes selecting the latest feature information from the database, presenting it in an easy-to-understand format. This information is then displayed on the smart glasses' screen using large fonts and diagrams, allowing the user to receive information quickly and accurately.

[1042] Example of a prompt

[1043] "Could you give me an example of a system that analyzes a customer's feelings of tension or anxiety when they ask a question about a product, and then displays the latest product information on smart glasses using easy-to-understand diagrams and large fonts?"

[1044] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1045] Step 1:

[1046] The user enters a question or keyword into the device. The user uses a smart device such as smart glasses to input a question or keyword, such as "What are the latest camera features?", via voice or text. The data provided by the user here is the text input of the question or keyword.

[1047] Step 2:

[1048] The device sends the entered questions and keywords to the emotion engine. The device uses its built-in microphone and camera to collect the user's voice and facial expression data and sends it to the emotion engine. The emotion engine receives this data and analyzes the user's psychological state (e.g., tension, excitement). The input data consists of voice data and facial expression data, and the output is the user's emotional state.

[1049] Step 3:

[1050] The device sends questions, keywords, and emotion data acquired by the terminal to the server. The emotion information analyzed by the emotion engine, along with the questions and keywords in text format, are sent to the server together. This allows the server to process the data while considering the user's psychological state. The input data consists of questions, keywords, and emotion data; the output is the transmission of this data to the server.

[1051] Step 4:

[1052] The server searches the database based on the questions, keywords, and sentiment data it receives. The server uses a database search module to find the information most relevant to the user's input. Furthermore, it adjusts the format and priority of the information provided based on the sentiment data. The input data consists of questions, keywords, and sentiment data; the output is a list of relevant information.

[1053] Step 5:

[1054] The server organizes search results and provides information in the most appropriate format according to the user's emotional state. The server organizes the retrieved search results and formats the information in an easy-to-understand format (e.g., diagrams, color display, large font) based on the emotional data. This ensures that information is provided in a format that is easy for the user to understand. The input data consists of a list of related information and emotional data, and the output is formatted information.

[1055] Step 6:

[1056] The terminal displays organized information on the smart device. The terminal receives formatted information from the server and displays it on the smart glasses' display. Appropriate information is provided in response to the user's questions, taking into account emotions such as tension and excitement. The input data is formatted information, and the output is the information displayed in the user's field of vision.

[1057] Step 7:

[1058] Users review the displayed information and utilize it in business negotiations and work. Users check the information displayed on their smart glasses and use it to explain things to their business partners. This allows users to provide information quickly and appropriately. The input data is the information displayed on the smart device, and the output is the smooth progress of business negotiations and work.

[1059] Through the steps described above, this invention realizes a system that instantly resolves user questions and provides appropriate information according to their emotions.

[1060] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1061] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1062] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1063] [Third Embodiment]

[1064] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1065] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1066] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1067] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1068] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1069] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1070] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1071] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1072] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1073] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1074] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1075] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1076] The system of this invention is designed to allow questions regarding the details of policies and the content of agent notifications to be resolved on the spot during daily operations and business negotiations with clients. The program for this system is generated and its operation is described below.

[1077] System-wide configuration

[1078] The system primarily consists of terminals that provide the user interface, servers that process data, and databases that store information. For this system to function, each component must work in coordination with the others.

[1079] Program processing

[1080] 1. The user enters a question or keyword.

[1081] User: Enter questions or keywords into the input field on the device. For example, if a client asks "What are the latest campaign strategies?" during a business meeting, the user would enter "Latest campaign strategies."

[1082] 2. The terminal sends the input to the server.

[1083] Terminal: Sends the entered questions and keywords to the server. Specifically, it sends a request to the server via the API. This request includes metadata such as the user ID and the current date and time, along with the input content.

[1084] 3. The server searches the database.

[1085] Server: Analyzes received questions and keywords and searches the database. The server generates queries based on the keywords received and retrieves relevant information from the database.

[1086] 4. The server organizes the search results.

[1087] Server: Organizes search results and provides information in a user-friendly format. For example, it selects highly relevant information and generates response data by converting it into JSON or XML format. It also adds relevant links and additional explanations as needed.

[1088] 5. The server sends the organized information to the terminal.

[1089] Server: Sends the generated response data to the terminal. This completes the transfer of data from the server to the terminal.

[1090] 6. The device displays the information.

[1091] Terminal: Displays received information in a user-friendly format. For example, the latest campaign details are displayed on the screen. Notifications can also be used to inform users when information is available.

[1092] 7. The user confirms the information.

[1093] User: Review the displayed information and resolve any questions on the spot. For example, a user can immediately explain the latest campaign strategies to a business partner.

[1094] Specific example

[1095] Situation 1: Resolving questions during a business negotiation

[1096] User: During a business meeting, typed "What are the latest campaign strategies?"

[1097] Terminal: Sends the input content to the server.

[1098] Server: Searches the database and extracts information about the latest campaign initiatives.

[1099] Server: Organizes search results and sends them to the terminal.

[1100] Terminal: Display information.

[1101] User: Review the displayed information and provide it to the business partner.

[1102] Situation 2: Inquiry from an agency

[1103] User: "Is there an updated version of the operation manual?"

[1104] Terminal: Sends the input content to the server.

[1105] Server: Search the database and extract information from the latest operation manual.

[1106] Server: Organizes search results and sends them to the terminal.

[1107] Terminal: Display information.

[1108] User: Review the displayed information and immediately provide the agent with a link to the latest operation manual and other relevant information.

[1109] Thus, the system of the present invention improves operational efficiency and productivity by providing information in real time.

[1110] The following describes the processing flow.

[1111] Step 1:

[1112] The user enters a question or keyword.

[1113] User: Enter "Latest Campaign Measures" into the input field on the device.

[1114] Step 2:

[1115] The terminal sends the entered question or keyword to the server.

[1116] Terminal: Sends input data to the server via API. This includes metadata such as user ID and timestamp.

[1117] Step 3:

[1118] The server analyzes the questions and keywords it receives.

[1119] Server: Analyzes incoming data and extracts keywords necessary for searching.

[1120] Step 4:

[1121] The server searches the database.

[1122] Server: Generates database queries based on extracted keywords and searches the database.

[1123] Step 5:

[1124] The server organizes the search results.

[1125] Server: Organizes information retrieved from the database and selects the most relevant information.

[1126] Step 6:

[1127] The server converts the organized information into a format that is easy for the user to understand.

[1128] Server: Converts information into formats such as JSON or XML, adds relevant links and additional explanations, and generates response data.

[1129] Step 7:

[1130] The server sends the organized information to the terminal.

[1131] Server: Sends the generated response data to the terminal as an API response.

[1132] Step 8:

[1133] The device displays the information it has received to the user.

[1134] Terminal: Analyzes received data and displays it on the screen in a user-friendly format.

[1135] Step 9:

[1136] The user reviews the displayed information and resolves any questions.

[1137] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[1138] (Example 1)

[1139] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1140] In today's business environment, there is a growing need for systems that provide reliable information quickly to address the wide range of questions and inquiries that arise during work and business negotiations. However, existing systems make it difficult to resolve questions in real time and access information immediately. Furthermore, when search results are overwhelming, it becomes difficult for users to find the information they need.

[1141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1142] In this invention, the server includes means for the user to input a question or keyword, means for the terminal to transmit the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for the server to organize the search results and provide the information in a format that is easy for the user to understand, means for the terminal to display the received information, and means for the user to confirm the information. This enables users to resolve questions in real time during business negotiations or work, improving work efficiency and enhancing the user experience.

[1143] A "user" is an entity that uses a system to input questions or keywords and obtain information.

[1144] A "terminal" is a computer device that allows users to input questions or keywords, send them to a server, and display the received information.

[1145] A "server" is a central processing unit that receives questions and keywords sent from terminals, searches a database to organize the information, and returns a response to the terminals.

[1146] A "database" is an information system that stores related information and is used by servers to retrieve data based on search queries.

[1147] A "question" is a question or problem that a user wants to resolve.

[1148] "Keywords" are important words or phrases that users use when entering questions.

[1149] "Searching" is the process by which a server checks data within a database and retrieves relevant information.

[1150] "Information" refers to the data and answers that the server provides based on the user's questions and keywords.

[1151] "Organization" is the process by which a server converts search results retrieved from a database into a format that is easy for users to understand.

[1152] "Provision" refers to the act of a server processing information, sending it to a terminal, and the terminal displaying that information to the user.

[1153] "Display" refers to the act of visually showing information received by a device to the user.

[1154] "Confirmation" refers to the act of a user viewing and understanding the information displayed on their device and taking the necessary action.

[1155] Modes for carrying out the invention

[1156] The system of this invention is designed to quickly and effectively resolve questions that arise during business operations or negotiations. The system operates as follows: the user inputs questions or keywords via a terminal, the server analyzes them, retrieves relevant information from a database, and presents it to the user.

[1157] Hardware and software configuration

[1158] The system consists of the following main elements:

[1159] Device: A device operated by the user, including personal computers, tablets, and smartphones. In addition to browsers, devices can use dedicated applications.

[1160] A server is a central processing unit that handles data processing. It receives questions and keywords, searches databases, and organizes the results. Examples of servers include typical web servers and cloud servers.

[1161] Database: An information system that stores related information, using relational database management systems such as MySQL or PostgreSQL.

[1162] Software name

[1163] Frontend development: JavaScript frameworks such as React and Vue.js

[1164] Backend development: Server-side frameworks such as Python and Node.js

[1165] API: Use a RESTful API to send and receive data between the device and the server.

[1166] Database: MySQL, PostgreSQL, or other RDBMS

[1167] Program Processing Description

[1168] 1. The user enters a question or keyword.

[1169] Users enter questions or keywords into the input fields on their terminals while negotiating with clients or performing tasks. An example of this prompt text is the question, "What are the latest campaign initiatives?"

[1170] 2. Send the entered question to the server.

[1171] The terminal sends the entered question or keyword to the server. A POST request is used via the API, and the request includes the question or keyword, user ID, and current date and time.

[1172] 3. The server searches the database.

[1173] The server analyzes the received questions and keywords and searches the database. Based on these keywords, the server generates SQL queries and retrieves relevant information from the database.

[1174] 4. Organize search results and provide information.

[1175] The server organizes the retrieved search results and provides the information in a format that is easy for the user to understand. For example, it converts them to JSON format, selects the most relevant information, and generates response data.

[1176] 5. The device displays the information.

[1177] The terminal displays information received from the server in a user-friendly format. Users can view search results through the terminal's screen.

[1178] 6. The user confirms the information.

[1179] Users can review the displayed information and resolve any questions on the spot. For example, they can immediately explain the latest campaign initiatives displayed to their business partners.

[1180] Specific examples of operation

[1181] Situation 1: Resolving questions during a business negotiation

[1182] User: "What are the latest campaign initiatives?" is typed into the terminal.

[1183] Terminal: Sends the input content to the server.

[1184] Server: Searches the database and extracts the necessary information.

[1185] Server: Organizes search results and sends them to the terminal.

[1186] Terminal: Displays information.

[1187] User: Review the displayed information and explain it to the business partner.

[1188] Situation 2: Responding to inquiries from agencies

[1189] User: "Is there an updated version of the latest operation manual?"

[1190] Terminal: Sends the input content to the server.

[1191] Server: Searches the database and retrieves manual update information.

[1192] Server: Organizes search results and sends them to the terminal.

[1193] Terminal: Displays information.

[1194] User: Review the displayed information and provide the agent with a link to the latest operation manual.

[1195] Example of a prompt

[1196] By inputting prompts into the generative AI model, you can obtain answers to questions about the system's behavior. For example:

[1197] "If you have any questions about this system, please enter the following: 'What are the latest campaign initiatives?'"

[1198] The system of this invention improves operational efficiency and productivity by providing information in real time, and enables users to quickly resolve their questions.

[1199] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1200] Step 1:

[1201] The user enters a question or keyword.

[1202] Specific action: While negotiating with a client or performing tasks, the user enters questions or keywords into the input field on the terminal. For example, they might enter, "What are the latest campaign strategies?"

[1203] Input: The user enters questions or keywords in text format into the device.

[1204] Output: Text data of the questions and keywords entered into the terminal is generated.

[1205] Step 2:

[1206] The terminal sends the input to the server.

[1207] Specific operation: The terminal sends the entered question or keyword to the server. Using a POST request via the API, the question or keyword, user ID, and current date and time are sent to the server in JSON format.

[1208] Input: User-entered question or keyword, User ID, current date and time

[1209] Output: API request to the server (JSON format)

[1210] Step 3:

[1211] The server analyzes the questions and keywords it receives.

[1212] Specific operation: The server analyzes the received request data and extracts questions and keywords. Text analysis algorithms and natural language processing techniques are used for the analysis process.

[1213] Input: API request data that reached the server

[1214] Output: Text data of analyzed questions and keywords

[1215] Step 4:

[1216] The server searches the database.

[1217] Specific operation: Based on the analyzed questions and keywords, the server generates SQL queries against the database and searches the database. A database management system (e.g., MySQL, PostgreSQL) is used for the search process.

[1218] Input: Analyzed questions and keywords

[1219] Output: Search results (raw data) retrieved from the database.

[1220] Step 5:

[1221] The server organizes the search results.

[1222] Specific operation: The server organizes search results into a format that is easy for the user to understand. Data formatting is done using data filtering and ranking algorithms. The organized data is often converted to JSON format.

[1223] Input: Search results retrieved from the database

[1224] Output: Organized search results (response data in JSON format)

[1225] Step 6:

[1226] The server sends the organized information to the terminal.

[1227] Specific operation: The server sends organized response data to the terminal. JSON data is sent using the HTTP response.

[1228] Input: Organized search results (response data in JSON format)

[1229] Output: HTTP response to the terminal

[1230] Step 7:

[1231] The device displays the information.

[1232] Specific operation: The terminal displays information received from the server in a user-friendly format. For example, a UI designed with HTML and CSS is used to display retrieved information on a web page. Notifications can also be used to inform the user when information is available.

[1233] Input: Search results received from the server (response data in JSON format)

[1234] Output: Information displayed to the user (text and links on the web page)

[1235] Step 8:

[1236] The user verifies the information.

[1237] Specific actions: Users view information through their device screen and take the next action as needed. For example, during a business negotiation, the user can move to the next step by sharing details of the latest campaign with their business partner.

[1238] Input: Information displayed on the device

[1239] Output: User understanding and next actions (e.g., advancing the deal)

[1240] In this way, the entire system's processing progresses with specific inputs and outputs at each step, enabling quick answers to user questions.

[1241] (Application Example 1)

[1242] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1243] Currently, in brick-and-mortar stores, staff are required to quickly access and provide product and campaign information to customers. However, searching for this information is time-consuming, making immediate responses difficult. This leads to decreased customer satisfaction and inefficiencies in operations. To solve this problem, a system is needed that can quickly acquire and provide necessary information in real time.

[1244] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1245] In this invention, the server includes means for a user to input a question or keyword, means for transmitting the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for organizing the search results and providing the information in a format that is easy for staff to understand, means for the terminal to display the information organized based on the search to the staff, and means for the terminal to notify the staff of the provided information. This enables staff in physical stores to respond quickly and accurately to customer questions.

[1246] A "user" is someone who uses this system to input questions or keywords.

[1247] A "question or keyword" refers to a string of characters or words that a user enters to specifically indicate the information they want to know.

[1248] A "server" is an information processing device that receives questions or keywords sent by users, searches a database based on them, organizes the search results, and provides the information.

[1249] A "database" is a collection of information stored within a system, and it is the source of information that a server searches.

[1250] A "search method" refers to the function or process by which a server retrieves relevant information from a database based on a question or keyword it receives.

[1251] "Organization methods" refer to the functions and processes that a server uses to convert search results into a format that is easy for users and staff to understand.

[1252] "Information provision means" refers to the functions and processes that the server uses to communicate the search results it has compiled to users and staff.

[1253] A "terminal" refers to a device such as a computer, smartphone, or smart glasses that users or staff use to input questions or keywords, or to display information from a server.

[1254] "Display means" refers to the function or process by which a terminal visually displays information sent from a server.

[1255] "Notification means" refers to the functions or processes that a device uses to inform users or staff about the provision of information.

[1256] System Overview

[1257] This invention provides a system for enabling staff in physical stores to respond quickly and accurately to customer inquiries. The system is implemented using a terminal for inputting questions and keywords, a server for processing the input, and a database for providing search results.

[1258] Hardware and software configuration

[1259] 1. Hardware

[1260] Terminal: This refers to a smartphone (Android or iOS) or smart glasses (e.g., Google Glass) used by staff members who are users of the system.

[1261] Server: A cloud-based server that receives, processes, and retrieves information. Examples include AWS and Google Cloud.

[1262] 2. Software

[1263] Mobile application: Installed on smartphones or smart glasses used by staff. Development is done using Android Studio (Java or Kotlin) or Xcode (Swift).

[1264] Server software: Developed using Node.js, providing a RESTful API.

[1265] Database: We manage information using MySQL or Firebase.

[1266] Program processing

[1267] When a user enters a question or keyword using a device such as a smartphone or smart glasses, the content is sent to the server. The server analyzes the received question or keyword and searches its database. It then organizes the search results and provides the information in a format that is easy for the user to understand.

[1268] Data processing and calculations

[1269] Data reception and transmission: Questions and keywords entered by the user on their device are sent to the server via the API. The request includes not only the question and keywords, but also metadata such as the user ID and the date and time the request was made.

[1270] Data retrieval: The server analyzes the received keywords and searches the database for relevant information. This part utilizes natural language processing techniques to generate the optimal query for the keywords.

[1271] Data organization: Acquired data is organized on the server and converted to JSON or XML format. This ensures that information is provided in a format that is easy for users to understand.

[1272] Information provision: The organized information is sent back to the terminal via the API, notified to the staff, and displayed on the terminal's screen.

[1273] Specific example

[1274] Example 1: Customer Service

[1275] Situation: A customer asks, "What's the latest campaign information?"

[1276] Instructions: The staff member enters "campaign information" into their smartphone.

[1277] Processing: The smartphone sends the input content to the server. The server searches and organizes the latest campaign information from the database and sends the results to the smartphone.

[1278] Display: Campaign information is displayed on smartphones. Staff provide this information to customers.

[1279] Example 2: In-store event announcement

[1280] Situation: A customer asks, "What's the next in-store event?"

[1281] Instructions: The staff member enters "in-store event" into the smart glasses.

[1282] Processing: The smart glasses send the input to the server. The server searches and organizes event information from the database and sends the results back to the smart glasses.

[1283] Display: Smart glasses will display information about upcoming in-store events. Staff will then inform customers of this information.

[1284] Example of a prompt

[1285] "Please tell me the details of the next in-store event."

[1286] "Please display the latest campaign information."

[1287] This invention aims to improve the responsiveness of in-store staff, enhance customer satisfaction, and increase operational efficiency by quickly acquiring and providing necessary information in real time.

[1288] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1289] Step 1:

[1290] The user enters a question or keyword.

[1291] Users input questions or keywords using their smartphones or smart glasses. For example, if a customer asks, "What are the latest campaign details?", the user would input "campaign details". This input is then sent to the server in the form of an API request, as described later.

[1292] Input: Customer question "What are the latest campaign details?"

[1293] Output: Input data "Campaign Information" to a smartphone or smart glasses.

[1294] Step 2:

[1295] The terminal sends input to the server.

[1296] The terminal sends the entered questions and keywords to the server. Specifically, it sends an HTTP POST request to the API endpoint, and the request includes metadata such as the user ID and the current date and time along with the input content.

[1297] Input: Questions or keywords entered on the device, such as "campaign information".

[1298] Output: API request sent to the server (questions, keywords + metadata)

[1299] Step 3:

[1300] The server searches the database.

[1301] The server analyzes the received questions and keywords and searches the database. It uses a generative AI model to perform natural language processing and generate the optimal query. Based on the query, it retrieves relevant information from the database.

[1302] Input: API request received by the server (question or keyword + metadata)

[1303] Output: Raw data (related information) retrieved from the database.

[1304] Step 4:

[1305] The server organizes the search results.

[1306] The server organizes the acquired data and converts it into a user-friendly format. Specifically, it selects highly relevant information and converts it into JSON or XML format to generate response data. Relevant links and additional explanations are also added as needed.

[1307] Input: Raw data retrieved from the database (related information)

[1308] Output: Organized response data (in JSON or XML format)

[1309] Step 5:

[1310] The server sends the organized information to the terminal.

[1311] The server sends the generated response data to the terminal. This ensures that the necessary information is provided to the user's terminal in real time.

[1312] Input: Response data generated within the server (organized information)

[1313] Output: Response data sent to the terminal

[1314] Step 6:

[1315] The device displays information

[1316] The device displays received information in a user-friendly format. The displayed content includes detailed information and related links. Notifications are also used to inform the user that information has been provided.

[1317] Input: Response data sent from the server

[1318] Output: Information displayed on the device (campaign information and related links)

[1319] Step 7:

[1320] The user confirms the information.

[1321] Users can review the displayed information and provide it to customers on the spot. For example, they can instantly explain the latest campaign information to customers.

[1322] Input: Information displayed on the device (campaign information and related links)

[1323] Output: Latest campaign information provided to customers

[1324] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1325] The system of this invention aims to resolve questions immediately during business operations and negotiations, thereby improving operational efficiency and productivity. Furthermore, by combining it with an emotion engine that recognizes user emotions, it aims to further enhance the appropriateness of the information provided.

[1326] System-wide configuration

[1327] The system was configured as follows:

[1328] 1. A terminal that provides a user interface.

[1329] 2. Servers that process information

[1330] 3. Database for storing information

[1331] 4. Emotion engine that recognizes user emotions

[1332] Program processing

[1333] 1. The user enters a question or keyword.

[1334] User: Enter questions or keywords such as "What are the latest campaign strategies?" into the device.

[1335] 2. The terminal sends the input to the server.

[1336] Terminal: Sends input data to the emotion engine to analyze the user's emotions. Then, it sends questions, keywords, and emotion data to the server.

[1337] 3. The server searches the database.

[1338] Server: Searches the database based on received questions, keywords, and sentiment data. Sentiment data is also considered when selecting highly relevant information.

[1339] 4. The server organizes the search results.

[1340] Server: Organizes information retrieved from the database and adjusts the format and priority of the information based on sentiment data. For example, if a user is confused, it prioritizes providing more detailed and specific information.

[1341] 5. The server sends the organized information to the terminal.

[1342] Server: Sends organized information to the terminal. The information is provided in JSON or XML format and includes additional explanations and related links.

[1343] 6. The device displays the information.

[1344] Terminal: Displays received information to the user. Based on sentiment data, the interface design and notification methods are also adjusted.

[1345] 7. Users verify information and resolve their questions.

[1346] User: Can review the displayed information and immediately explain it to the business partner.

[1347] Specific example

[1348] Situation 1: Resolving questions and recognizing emotions during a business negotiation

[1349] 1. User: During a business meeting, the user types "What are the latest campaign strategies?". At that time, the user's voice and facial expressions are analyzed by an emotion engine, and it is recognized that the user is slightly nervous.

[1350] 2. Terminal: Sends the question to the server along with the analyzed sentiment data.

[1351] 3. Server: Search the database and select campaign information in an easy-to-understand format, taking into consideration the needs of anxious users.

[1352] 4. Server: Organizes the selected information in detail and sends it to the terminal.

[1353] 5. Device: Since users may be stressed, display information clearly using color and large fonts.

[1354] 6. User: They can view the displayed information, immediately explain it to their business partner, and communicate effectively.

[1355] Situation 2: Inquiry from an agency and emotional recognition

[1356] 1. User: "Is there an updated version of the operation manual?" At this point, the user's voice and facial expressions are analyzed by the emotion engine, and it is recognized that the user is confused.

[1357] 2. Terminal: Sends the question to the server along with the analyzed sentiment data.

[1358] 3. Server: Search the database and prioritize selecting user manuals with more detailed instructions for users who are confused.

[1359] 4. Server: Organizes the selected information and sends it to the terminal.

[1360] 5. Device: Displays information in an easy-to-understand format, and provides supplementary explanations as needed.

[1361] 6. User: Can review the displayed information and explain to the agent how to provide the latest operation manual.

[1362] Thus, the system of the present invention, which combines an emotion engine, enables the provision of appropriate information according to the user's situation, thereby significantly improving the efficiency and quality of business operations and negotiations.

[1363] The following describes the processing flow.

[1364] Step 1:

[1365] The user enters a question or keyword.

[1366] User: Enter "What are the latest campaign measures?" into the input field on the device.

[1367] Step 2:

[1368] The device sends the entered question or keyword to the emotion engine.

[1369] Terminal: Captures user voice and facial expressions along with input content, and sends this to the emotion engine.

[1370] Step 3:

[1371] The emotion engine analyzes the user's emotions.

[1372] Emotion Engine: Analyzes captured audio and facial expression data to recognize emotions such as tension or confusion in the user.

[1373] Step 4:

[1374] The emotion engine sends the analysis results to the terminal.

[1375] Emotion Engine: Sends emotion analysis results back to the device. For example, it may include information such as "the user is feeling nervous."

[1376] Step 5:

[1377] The device sends the sentiment analysis results, along with any questions or keywords, to the server.

[1378] Terminal: Sends user input and sentiment analysis results to the server via API.

[1379] Step 6:

[1380] The server analyzes the questions, keywords, and sentiment data it receives.

[1381] Server: Analyzes questions and keywords, and generates optimal queries considering sentiment data.

[1382] Step 7:

[1383] The server searches the database.

[1384] Server: Searches the database based on the generated query and retrieves the most relevant information.

[1385] Step 8:

[1386] The server organizes the search results.

[1387] Server: Organizes the acquired information and adjusts the priority and display method of the information, taking into account the sentiment analysis results. For example, it provides information in a format that is particularly easy to understand for users who are feeling anxious.

[1388] Step 9:

[1389] The server sends the organized information to the terminal.

[1390] Server: Sends organized information to the terminal in JSON or XML format.

[1391] Step 10:

[1392] The device displays the information it has received to the user.

[1393] Terminal: Analyzes received data and displays it in a user-friendly format. For example, it might use large fonts or color displays.

[1394] Step 11:

[1395] The user reviews the displayed information and resolves any questions.

[1396] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[1397] (Example 2)

[1398] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1399] Traditional systems can provide information based on user-inputted questions and keywords, but they fail to consider the user's emotional state, which can result in providing information that is optimal for the user. In particular, in business negotiations and operations, user emotions significantly influence information comprehension and decision-making, so ignoring this can lead to problems such as decreased work efficiency and reduced quality of communication.

[1400] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1401] In this invention, the server includes means for transmitting a user-inputted question or keyword to an emotion recognition engine to generate emotion data; means for transmitting the generated emotion data along with the question or keyword to the server; means for searching a database based on the received question or keyword and emotion data; and means for organizing the search results based on the emotion data and providing the information in a format that is easy for the user to understand. This makes it possible to provide appropriate information according to the user's emotional state.

[1402] A "user" refers to an entity that uses a system to input questions or keywords and obtain information.

[1403] A "question" refers to a question that a user enters as information they want to know or a problem they want to solve.

[1404] A "keyword" refers to a word or phrase that a user enters to search for specific information.

[1405] An "emotion recognition engine" refers to a software or hardware system that analyzes a user's voice and facial expressions and generates data on their emotional state.

[1406] "Emotional data" refers to data generated by an emotion recognition engine that indicates the user's current emotional state.

[1407] A "server" refers to a central processing unit that analyzes received data, searches databases, and provides information.

[1408] A "database" refers to a data storage system that systematically stores information and manages it in a searchable format.

[1409] "Searching" refers to the process by which a server examines information within a database and extracts data that matches specific criteria.

[1410] "Providing information" refers to presenting data that is highly relevant to the questions or keywords entered by the user.

[1411] The system of this invention aims to improve the efficiency and productivity of work by immediately resolving questions in business operations and negotiations. Furthermore, by combining it with an emotion engine that recognizes user emotions, it aims to further enhance the appropriateness of the information provided.

[1412] System Configuration

[1413] The system consists of the following elements:

[1414] 1. A terminal that provides a user interface.

[1415] Hardware required: Windows PC, Mac, mobile devices (iOS, Android)

[1416] Software to use: Browser (Chrome, Safari, etc.)

[1417] 2. Servers that process information

[1418] Hardware used: Linux server

[1419] Software used: Apache HTTP Server

[1420] 3. Database for storing information

[1421] Hardware used: Hard disk, SSD

[1422] Software used: MySQL, PostgreSQL

[1423] 4. Emotion engine that recognizes user emotions

[1424] Hardware used: Camera, microphone

[1425] Software used: Microsoft Azure Emotion API, IBM Watson Tone Analyzer

[1426] Program processing

[1427] 1. The user enters a question or keyword.

[1428] Users input questions or keywords on their own devices. For example, during a business meeting, they might type "What are the latest campaign strategies?" using the keyboard. Alternatively, they might use voice input to ask "What are the latest campaign strategies?"

[1429] 2. The device sends the input to the emotion engine.

[1430] The device receives the entered question or keyword and sends that data to the emotion engine. The emotion engine analyzes the user's voice tone and facial expressions and generates the user's emotion data (e.g., tension, confusion, joy, etc.).

[1431] 3. The device sends question and sentiment data to the server.

[1432] The device sends emotion data returned from the emotion engine, along with questions and keywords entered by the user, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[1433] 4. The server searches the database.

[1434] The server analyzes the received questions, keywords, and sentiment data, and searches a MySQL or PostgreSQL database. The server generates SQL queries to extract relevant information.

[1435] 5. The server organizes the search results.

[1436] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking sentiment data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious. Specifically, it performs filtering based on the importance of the information and the user's emotions.

[1437] 6. The server sends the organized information to the terminal.

[1438] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[1439] 7. The device displays the information.

[1440] The device displays received information to the user. Based on emotional data, the interface design and display methods are adjusted. For example, color displays and larger fonts are used to alleviate tension.

[1441] 8. Users verify information and resolve their questions.

[1442] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[1443] Specific example

[1444] 1. Resolving questions and recognizing emotions during business negotiations

[1445] User: During a business meeting, the user voice-inputs "What are the latest campaign strategies?" into their laptop. The tone of voice and facial expression are captured by the camera.

[1446] Terminal: Sends input data to the emotion engine and retrieves emotional data that indicates tension. Next, it sends the question along with the analyzed data to the server.

[1447] Server: Searches the database and selects easy-to-understand campaign strategy information.

[1448] Server: Organizes information and sends it to the terminal.

[1449] Terminal: Displays information using large fonts and colored highlighting.

[1450] User: Confirm information and immediately explain it to the business partner. Share detailed materials to facilitate smooth business negotiations.

[1451] 2. Inquiries from agencies and emotional recognition

[1452] User: "Is there an updated version of the operation manual?" is typed on the smartphone. Voice input and facial expression capture via camera are used.

[1453] Terminal: Sends input data to the emotion engine and retrieves emotion data that analyzes the user as confused. Next, it sends the emotion data and question to the server.

[1454] Server: Search the database and prioritize selecting user manuals that provide detailed instructions for users who are experiencing difficulties.

[1455] Server: Organizes information and sends it to the terminal.

[1456] Device: Displays information in an easy-to-understand format, and links to video tutorials if necessary.

[1457] User: Review the information and provide detailed instructions to the agency on how to provide the latest operation manual. Share video tutorials to enhance understanding.

[1458] In this way, by enabling the provision of appropriate information tailored to the user's emotional state, it is possible to significantly improve work efficiency and the quality of communication.

[1459] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1460] Step 1:

[1461] The user enters a question or keyword.

[1462] Specifically, the user enters a question, such as "What are the latest campaign initiatives?", using a text area or voice input on their device. The input data is stored on the device in its initial format.

[1463] input:

[1464] User questions and keywords (e.g., "What are the latest campaign strategies?")

[1465] output:

[1466] Initial format input data

[1467] Step 2:

[1468] The device sends input to the emotion engine.

[1469] Specifically, the device captures the entered text data as well as the user's voice and facial expressions, and sends them to an emotion recognition engine. Microsoft Azure Emotion API and IBM Watson Tone Analyzer are examples of such systems. The emotion engine analyzes the user's voice tone and facial expressions to generate emotion data.

[1470] input:

[1471] Initial format input data

[1472] User voice and facial expression data

[1473] output:

[1474] User emotion data (e.g., "nervous")

[1475] Step 3:

[1476] The device sends question and emotion data to the server.

[1477] After the emotion data is generated, the device sends that data, along with any questions or keywords, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[1478] input:

[1479] Initial format input data

[1480] Emotional data

[1481] output:

[1482] Question and sentiment data formatted in JSON format

[1483] Step 4:

[1484] The server searches the database.

[1485] The server analyzes the received questions, keywords, and sentiment data, and searches MySQL or PostgreSQL databases based on this analysis. The server generates SQL queries to extract the most relevant information.

[1486] input:

[1487] Question and sentiment data formatted in JSON format

[1488] output:

[1489] Related information extracted based on SQL queries

[1490] Step 5:

[1491] The server organizes the search results.

[1492] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking emotional data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious.

[1493] input:

[1494] Related information extracted from the database

[1495] Emotional data

[1496] output:

[1497] Organized information (including diagrams, graphs, and explanatory text)

[1498] Step 6:

[1499] The server sends the organized information to the terminal.

[1500] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[1501] input:

[1502] Organized information

[1503] output:

[1504] Information formatted in JSON or XML format

[1505] Step 7:

[1506] The device displays the information.

[1507] The device displays received information to the user. Based on emotional data, it adjusts the interface design and notification methods. For example, it uses color displays and large fonts to alleviate tension.

[1508] input:

[1509] Information formatted in JSON or XML format

[1510] Emotional data

[1511] output:

[1512] Adjusted information displayed to the user

[1513] Step 8:

[1514] Users can verify information and resolve their questions.

[1515] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[1516] input:

[1517] Adjusted information

[1518] output:

[1519] Understood information

[1520] Use in business negotiations and explanations.

[1521] (Application Example 2)

[1522] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1523] In modern business and negotiations, it is essential to quickly resolve questions and provide appropriate information. However, there is a lack of systems that can immediately answer user questions, and there are no mechanisms to adjust the way information is delivered based on emotions. As a result, user satisfaction and the efficiency of negotiations suffer.

[1524] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions using an emotion engine, means for adjusting the format and priority of the information provided according to the user's emotions, and means for displaying the information using a smart device. This makes it possible to answer the user's questions in real time while providing information in an optimal form according to the user's emotions.

[1525] A "user" is an individual or legal entity that uses the system and obtains information by entering questions or keywords.

[1526] A "question" refers to a specific question or point of confusion that a user has, and is information that the user seeks to resolve from the system.

[1527] A "keyword" is a specific word or phrase entered by the user, which serves as the basis for the system to search for information.

[1528] A "server" is a computer device that receives user input, searches a database, and provides relevant information.

[1529] A "database" is a collection of diverse information that is organized and stored, and is what a server searches for.

[1530] An "emotion engine" is a technology that analyzes a user's emotions and adjusts how information is provided within the system based on the results.

[1531] "Emotions" or "emotional data" refers to information that represents the user's psychological state and is the subject of analysis and consideration by the system.

[1532] A "smart device" is a device that connects to a network and can display and operate digital information. Specific examples include smart glasses and head-mounted displays.

[1533] "Information format" refers to the composition, design, font size, color scheme, and other elements of how information provided to users is visually displayed.

[1534] "Priority" refers to the order and importance of information presented to the user, and is adjusted based on the user's emotions.

[1535] "Organization" refers to the process of compiling information retrieved from a database into a format that is easy for users to understand.

[1536] "Providing" means the act of a system displaying organized information on a user's device, making it accessible to the user.

[1537] "Real-time" means processing user input and emotions instantly and providing results immediately.

[1538] This invention is a system in which a user inputs questions or keywords using a smart device and obtains relevant information in real time. The system is implemented with the following configuration.

[1539] Hardware configuration

[1540] Device: A device such as smart glasses or a head-mounted display. This device has a built-in microphone and camera that can capture the user's voice and facial expressions.

[1541] Server: The server computer performs tasks such as searching the database and running the sentiment engine.

[1542] Database: Database software such as MySQL or PostgreSQL is used to store and manage necessary information.

[1543] Software Configuration

[1544] Emotion Engine: Uses Microsoft Azure Cognitive Services, Affectiva, and other tools to analyze user emotions in real time.

[1545] Database search: A software module for searching for relevant information from a database.

[1546] Display Control: A software module for displaying searched information on smart glasses.

[1547] Operation flow

[1548] 1. The user enters a question.

[1549] Enter questions or keywords such as "What are the latest camera features?" into the device via voice or text.

[1550] 2. Analysis of emotions

[1551] The device transmits voice and facial expression data to an emotion engine, which analyzes the user's emotions. This analysis includes information such as whether the user is nervous or excited.

[1552] 3. Sending to the server

[1553] The device sends questions, keywords, and sentiment data to the server.

[1554] 4. Information Search

[1555] The server searches the database based on questions and keywords and retrieves relevant information. It then adjusts the format and priority of the information based on sentiment data.

[1556] 5. Display of Information

[1557] The organized information is sent to the device and displayed on the smart glasses in a format optimized for the user's emotions, such as using large fonts or illustrations.

[1558] Specific example

[1559] For example, while a user is wearing smart glasses, a customer might ask, "What are the latest features of this camera?" The user then inputs the question by voice through the smart glasses.

[1560] The emotion engine analyzes the customer's behavior and determines that they are slightly nervous. Based on this information, the server prioritizes selecting the latest feature information from the database, presenting it in an easy-to-understand format. This information is then displayed on the smart glasses' screen using large fonts and diagrams, allowing the user to receive information quickly and accurately.

[1561] Example of a prompt

[1562] "Could you give me an example of a system that analyzes a customer's feelings of tension or anxiety when they ask a question about a product, and then displays the latest product information on smart glasses using easy-to-understand diagrams and large fonts?"

[1563] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1564] Step 1:

[1565] The user enters a question or keyword into the device. The user uses a smart device such as smart glasses to input a question or keyword, such as "What are the latest camera features?", via voice or text. The data provided by the user here is the text input of the question or keyword.

[1566] Step 2:

[1567] The device sends the entered questions and keywords to the emotion engine. The device uses its built-in microphone and camera to collect the user's voice and facial expression data and sends it to the emotion engine. The emotion engine receives this data and analyzes the user's psychological state (e.g., tension, excitement). The input data consists of voice data and facial expression data, and the output is the user's emotional state.

[1568] Step 3:

[1569] The device sends questions, keywords, and emotion data acquired by the terminal to the server. The emotion information analyzed by the emotion engine, along with the questions and keywords in text format, are sent to the server together. This allows the server to process the data while considering the user's psychological state. The input data consists of questions, keywords, and emotion data; the output is the transmission of this data to the server.

[1570] Step 4:

[1571] The server searches the database based on the questions, keywords, and sentiment data it receives. The server uses a database search module to find the information most relevant to the user's input. Furthermore, it adjusts the format and priority of the information provided based on the sentiment data. The input data consists of questions, keywords, and sentiment data; the output is a list of relevant information.

[1572] Step 5:

[1573] The server organizes search results and provides information in the most appropriate format according to the user's emotional state. The server organizes the retrieved search results and formats the information in an easy-to-understand format (e.g., diagrams, color display, large font) based on the emotional data. This ensures that information is provided in a format that is easy for the user to understand. The input data consists of a list of related information and emotional data, and the output is formatted information.

[1574] Step 6:

[1575] The terminal displays organized information on the smart device. The terminal receives formatted information from the server and displays it on the smart glasses' display. Appropriate information is provided in response to the user's questions, taking into account emotions such as tension and excitement. The input data is formatted information, and the output is the information displayed in the user's field of vision.

[1576] Step 7:

[1577] Users review the displayed information and utilize it in business negotiations and work. Users check the information displayed on their smart glasses and use it to explain things to their business partners. This allows users to provide information quickly and appropriately. The input data is the information displayed on the smart device, and the output is the smooth progress of business negotiations and work.

[1578] Through the steps described above, this invention realizes a system that instantly resolves user questions and provides appropriate information according to their emotions.

[1579] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1580] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1581] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1582] [Fourth Embodiment]

[1583] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1584] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1585] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1586] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1587] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1588] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1589] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1590] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1591] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1592] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1593] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1594] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1595] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1596] The system of this invention is designed to allow questions regarding the details of policies and the content of agent notifications to be resolved on the spot during daily operations and business negotiations with clients. The program for this system is generated and its operation is described below.

[1597] System-wide configuration

[1598] The system primarily consists of terminals that provide the user interface, servers that process data, and databases that store information. For this system to function, each component must work in coordination with the others.

[1599] Program processing

[1600] 1. The user enters a question or keyword.

[1601] User: Enter questions or keywords into the input field on the device. For example, if a client asks "What are the latest campaign strategies?" during a business meeting, the user would enter "Latest campaign strategies."

[1602] 2. The terminal sends the input to the server.

[1603] Terminal: Sends the entered questions and keywords to the server. Specifically, it sends a request to the server via the API. This request includes metadata such as the user ID and the current date and time, along with the input content.

[1604] 3. The server searches the database.

[1605] Server: Analyzes received questions and keywords and searches the database. The server generates queries based on the keywords received and retrieves relevant information from the database.

[1606] 4. The server organizes the search results.

[1607] Server: Organizes search results and provides information in a user-friendly format. For example, it selects highly relevant information and generates response data by converting it into JSON or XML format. It also adds relevant links and additional explanations as needed.

[1608] 5. The server sends the organized information to the terminal.

[1609] Server: Sends the generated response data to the terminal. This completes the transfer of data from the server to the terminal.

[1610] 6. The device displays the information.

[1611] Terminal: Displays received information in a user-friendly format. For example, the latest campaign details are displayed on the screen. Notifications can also be used to inform users when information is available.

[1612] 7. The user confirms the information.

[1613] User: Review the displayed information and resolve any questions on the spot. For example, a user can immediately explain the latest campaign strategies to a business partner.

[1614] Specific example

[1615] Situation 1: Resolving questions during a business negotiation

[1616] User: During a business meeting, typed "What are the latest campaign strategies?"

[1617] Terminal: Sends the input content to the server.

[1618] Server: Searches the database and extracts information about the latest campaign initiatives.

[1619] Server: Organizes search results and sends them to the terminal.

[1620] Terminal: Display information.

[1621] User: Review the displayed information and provide it to the business partner.

[1622] Situation 2: Inquiry from an agency

[1623] User: "Is there an updated version of the operation manual?"

[1624] Terminal: Sends the input content to the server.

[1625] Server: Search the database and extract information from the latest operation manual.

[1626] Server: Organizes search results and sends them to the terminal.

[1627] Terminal: Display information.

[1628] User: Review the displayed information and immediately provide the agent with a link to the latest operation manual and other relevant information.

[1629] Thus, the system of the present invention improves operational efficiency and productivity by providing information in real time.

[1630] The following describes the processing flow.

[1631] Step 1:

[1632] The user enters a question or keyword.

[1633] User: Enter "Latest Campaign Measures" into the input field on the device.

[1634] Step 2:

[1635] The terminal sends the entered question or keyword to the server.

[1636] Terminal: Sends input data to the server via API. This includes metadata such as user ID and timestamp.

[1637] Step 3:

[1638] The server analyzes the questions and keywords it receives.

[1639] Server: Analyzes incoming data and extracts keywords necessary for searching.

[1640] Step 4:

[1641] The server searches the database.

[1642] Server: Generates database queries based on extracted keywords and searches the database.

[1643] Step 5:

[1644] The server organizes the search results.

[1645] Server: Organizes information retrieved from the database and selects the most relevant information.

[1646] Step 6:

[1647] The server converts the organized information into a format that is easy for the user to understand.

[1648] Server: Converts information into formats such as JSON or XML, adds relevant links and additional explanations, and generates response data.

[1649] Step 7:

[1650] The server sends the organized information to the terminal.

[1651] Server: Sends the generated response data to the terminal as an API response.

[1652] Step 8:

[1653] The device displays the information it has received to the user.

[1654] Terminal: Analyzes received data and displays it on the screen in a user-friendly format.

[1655] Step 9:

[1656] The user reviews the displayed information and resolves any questions.

[1657] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[1658] (Example 1)

[1659] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1660] In today's business environment, there is a growing need for systems that provide reliable information quickly to address the wide range of questions and inquiries that arise during work and business negotiations. However, existing systems make it difficult to resolve questions in real time and access information immediately. Furthermore, when search results are overwhelming, it becomes difficult for users to find the information they need.

[1661] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1662] In this invention, the server includes means for the user to input a question or keyword, means for the terminal to transmit the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for the server to organize the search results and provide the information in a format that is easy for the user to understand, means for the terminal to display the received information, and means for the user to confirm the information. This enables users to resolve questions in real time during business negotiations or work, improving work efficiency and enhancing the user experience.

[1663] A "user" is an entity that uses a system to input questions or keywords and obtain information.

[1664] A "terminal" is a computer device that allows users to input questions or keywords, send them to a server, and display the received information.

[1665] A "server" is a central processing unit that receives questions and keywords sent from terminals, searches a database to organize the information, and returns a response to the terminals.

[1666] A "database" is an information system that stores related information and is used by servers to retrieve data based on search queries.

[1667] A "question" is a question or problem that a user wants to resolve.

[1668] "Keywords" are important words or phrases that users use when entering questions.

[1669] "Searching" is the process by which a server checks data within a database and retrieves relevant information.

[1670] "Information" refers to the data and answers that the server provides based on the user's questions and keywords.

[1671] "Organization" is the process by which a server converts search results retrieved from a database into a format that is easy for users to understand.

[1672] "Provision" refers to the act of a server processing information, sending it to a terminal, and the terminal displaying that information to the user.

[1673] "Display" refers to the act of visually showing information received by a device to the user.

[1674] "Confirmation" refers to the act of a user viewing and understanding the information displayed on their device and taking the necessary action.

[1675] Modes for carrying out the invention

[1676] The system of this invention is designed to quickly and effectively resolve questions that arise during business operations or negotiations. The system operates as follows: the user inputs questions or keywords via a terminal, the server analyzes them, retrieves relevant information from a database, and presents it to the user.

[1677] Hardware and software configuration

[1678] The system consists of the following main elements:

[1679] Device: A device operated by the user, including personal computers, tablets, and smartphones. In addition to browsers, devices can use dedicated applications.

[1680] A server is a central processing unit that handles data processing. It receives questions and keywords, searches databases, and organizes the results. Examples of servers include typical web servers and cloud servers.

[1681] Database: An information system that stores related information, using relational database management systems such as MySQL or PostgreSQL.

[1682] Software name

[1683] Frontend development: JavaScript frameworks such as React and Vue.js

[1684] Backend development: Server-side frameworks such as Python and Node.js

[1685] API: Use a RESTful API to send and receive data between the device and the server.

[1686] Database: MySQL, PostgreSQL, or other RDBMS

[1687] Program Processing Description

[1688] 1. The user enters a question or keyword.

[1689] Users enter questions or keywords into the input fields on their terminals while negotiating with clients or performing tasks. An example of this prompt text is the question, "What are the latest campaign initiatives?"

[1690] 2. Send the entered question to the server.

[1691] The terminal sends the entered question or keyword to the server. A POST request is used via the API, and the request includes the question or keyword, user ID, and current date and time.

[1692] 3. The server searches the database.

[1693] The server analyzes the received questions and keywords and searches the database. Based on these keywords, the server generates SQL queries and retrieves relevant information from the database.

[1694] 4. Organize search results and provide information.

[1695] The server organizes the retrieved search results and provides the information in a format that is easy for the user to understand. For example, it converts them to JSON format, selects the most relevant information, and generates response data.

[1696] 5. The device displays the information.

[1697] The terminal displays information received from the server in a user-friendly format. Users can view search results through the terminal's screen.

[1698] 6. The user confirms the information.

[1699] Users can review the displayed information and resolve any questions on the spot. For example, they can immediately explain the latest campaign initiatives displayed to their business partners.

[1700] Specific examples of operation

[1701] Situation 1: Resolving questions during a business negotiation

[1702] User: "What are the latest campaign initiatives?" is typed into the terminal.

[1703] Terminal: Sends the input content to the server.

[1704] Server: Searches the database and extracts the necessary information.

[1705] Server: Organizes search results and sends them to the terminal.

[1706] Terminal: Displays information.

[1707] User: Review the displayed information and explain it to the business partner.

[1708] Situation 2: Responding to inquiries from agencies

[1709] User: "Is there an updated version of the latest operation manual?"

[1710] Terminal: Sends the input content to the server.

[1711] Server: Searches the database and retrieves manual update information.

[1712] Server: Organizes search results and sends them to the terminal.

[1713] Terminal: Displays information.

[1714] User: Review the displayed information and provide the agent with a link to the latest operation manual.

[1715] Example of a prompt

[1716] By inputting prompts into the generative AI model, you can obtain answers to questions about the system's behavior. For example:

[1717] "If you have any questions about this system, please enter the following: 'What are the latest campaign initiatives?'"

[1718] The system of this invention improves operational efficiency and productivity by providing information in real time, and enables users to quickly resolve their questions.

[1719] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1720] Step 1:

[1721] The user enters a question or keyword.

[1722] Specific action: While negotiating with a client or performing tasks, the user enters questions or keywords into the input field on the terminal. For example, they might enter, "What are the latest campaign strategies?"

[1723] Input: The user enters questions or keywords in text format into the device.

[1724] Output: Text data of the questions and keywords entered into the terminal is generated.

[1725] Step 2:

[1726] The terminal sends the input to the server.

[1727] Specific operation: The terminal sends the entered question or keyword to the server. Using a POST request via the API, the question or keyword, user ID, and current date and time are sent to the server in JSON format.

[1728] Input: User-entered question or keyword, User ID, current date and time

[1729] Output: API request to the server (JSON format)

[1730] Step 3:

[1731] The server analyzes the questions and keywords it receives.

[1732] Specific operation: The server analyzes the received request data and extracts questions and keywords. Text analysis algorithms and natural language processing techniques are used for the analysis process.

[1733] Input: API request data that reached the server

[1734] Output: Text data of analyzed questions and keywords

[1735] Step 4:

[1736] The server searches the database.

[1737] Specific operation: Based on the analyzed questions and keywords, the server generates SQL queries against the database and searches the database. A database management system (e.g., MySQL, PostgreSQL) is used for the search process.

[1738] Input: Analyzed questions and keywords

[1739] Output: Search results (raw data) retrieved from the database.

[1740] Step 5:

[1741] The server organizes the search results.

[1742] Specific operation: The server organizes search results into a format that is easy for the user to understand. Data formatting is done using data filtering and ranking algorithms. The organized data is often converted to JSON format.

[1743] Input: Search results retrieved from the database

[1744] Output: Organized search results (response data in JSON format)

[1745] Step 6:

[1746] The server sends the organized information to the terminal.

[1747] Specific operation: The server sends organized response data to the terminal. JSON data is sent using the HTTP response.

[1748] Input: Organized search results (response data in JSON format)

[1749] Output: HTTP response to the terminal

[1750] Step 7:

[1751] The device displays the information.

[1752] Specific operation: The terminal displays information received from the server in a user-friendly format. For example, a UI designed with HTML and CSS is used to display retrieved information on a web page. Notifications can also be used to inform the user when information is available.

[1753] Input: Search results received from the server (response data in JSON format)

[1754] Output: Information displayed to the user (text and links on the web page)

[1755] Step 8:

[1756] The user verifies the information.

[1757] Specific actions: Users view information through their device screen and take the next action as needed. For example, during a business negotiation, the user can move to the next step by sharing details of the latest campaign with their business partner.

[1758] Input: Information displayed on the device

[1759] Output: User understanding and next actions (e.g., advancing the deal)

[1760] In this way, the entire system's processing progresses with specific inputs and outputs at each step, enabling quick answers to user questions.

[1761] (Application Example 1)

[1762] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1763] Currently, in brick-and-mortar stores, staff are required to quickly access and provide product and campaign information to customers. However, searching for this information is time-consuming, making immediate responses difficult. This leads to decreased customer satisfaction and inefficiencies in operations. To solve this problem, a system is needed that can quickly acquire and provide necessary information in real time.

[1764] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1765] In this invention, the server includes means for a user to input a question or keyword, means for transmitting the input question or keyword to the server, means for the server to search a database based on the received question or keyword, means for organizing the search results and providing the information in a format that is easy for staff to understand, means for the terminal to display the information organized based on the search to the staff, and means for the terminal to notify the staff of the provided information. This enables staff in physical stores to respond quickly and accurately to customer questions.

[1766] A "user" is someone who uses this system to input questions or keywords.

[1767] A "question or keyword" refers to a string of characters or words that a user enters to specifically indicate the information they want to know.

[1768] A "server" is an information processing device that receives questions or keywords sent by users, searches a database based on them, organizes the search results, and provides the information.

[1769] A "database" is a collection of information stored within a system, and it is the source of information that a server searches.

[1770] A "search method" refers to the function or process by which a server retrieves relevant information from a database based on a question or keyword it receives.

[1771] "Organization methods" refer to the functions and processes that a server uses to convert search results into a format that is easy for users and staff to understand.

[1772] "Information provision means" refers to the functions and processes that the server uses to communicate the search results it has compiled to users and staff.

[1773] A "terminal" refers to a device such as a computer, smartphone, or smart glasses that users or staff use to input questions or keywords, or to display information from a server.

[1774] "Display means" refers to the function or process by which a terminal visually displays information sent from a server.

[1775] "Notification means" refers to the functions or processes that a device uses to inform users or staff about the provision of information.

[1776] System Overview

[1777] This invention provides a system for enabling staff in physical stores to respond quickly and accurately to customer inquiries. The system is implemented using a terminal for inputting questions and keywords, a server for processing the input, and a database for providing search results.

[1778] Hardware and software configuration

[1779] 1. Hardware

[1780] Terminal: This refers to a smartphone (Android or iOS) or smart glasses (e.g., Google Glass) used by staff members who are users of the system.

[1781] Server: A cloud-based server that receives, processes, and retrieves information. Examples include AWS and Google Cloud.

[1782] 2. Software

[1783] Mobile application: Installed on smartphones or smart glasses used by staff. Development is done using Android Studio (Java or Kotlin) or Xcode (Swift).

[1784] Server software: Developed using Node.js, providing a RESTful API.

[1785] Database: We manage information using MySQL or Firebase.

[1786] Program processing

[1787] When a user enters a question or keyword using a device such as a smartphone or smart glasses, the content is sent to the server. The server analyzes the received question or keyword and searches its database. It then organizes the search results and provides the information in a format that is easy for the user to understand.

[1788] Data processing and calculations

[1789] Data reception and transmission: Questions and keywords entered by the user on their device are sent to the server via the API. The request includes not only the question and keywords, but also metadata such as the user ID and the date and time the request was made.

[1790] Data retrieval: The server analyzes the received keywords and searches the database for relevant information. This part utilizes natural language processing techniques to generate the optimal query for the keywords.

[1791] Data organization: Acquired data is organized on the server and converted to JSON or XML format. This ensures that information is provided in a format that is easy for users to understand.

[1792] Information provision: The organized information is sent back to the terminal via the API, notified to the staff, and displayed on the terminal's screen.

[1793] Specific example

[1794] Example 1: Customer Service

[1795] Situation: A customer asks, "What's the latest campaign information?"

[1796] Instructions: The staff member enters "campaign information" into their smartphone.

[1797] Processing: The smartphone sends the input content to the server. The server searches and organizes the latest campaign information from the database and sends the results to the smartphone.

[1798] Display: Campaign information is displayed on smartphones. Staff provide this information to customers.

[1799] Example 2: In-store event announcement

[1800] Situation: A customer asks, "What's the next in-store event?"

[1801] Instructions: The staff member enters "in-store event" into the smart glasses.

[1802] Processing: The smart glasses send the input to the server. The server searches and organizes event information from the database and sends the results back to the smart glasses.

[1803] Display: Smart glasses will display information about upcoming in-store events. Staff will then inform customers of this information.

[1804] Example of a prompt

[1805] "Please tell me the details of the next in-store event."

[1806] "Please display the latest campaign information."

[1807] This invention aims to improve the responsiveness of in-store staff, enhance customer satisfaction, and increase operational efficiency by quickly acquiring and providing necessary information in real time.

[1808] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1809] Step 1:

[1810] The user enters a question or keyword.

[1811] Users input questions or keywords using their smartphones or smart glasses. For example, if a customer asks, "What are the latest campaign details?", the user would input "campaign details". This input is then sent to the server in the form of an API request, as described later.

[1812] Input: Customer question "What are the latest campaign details?"

[1813] Output: Input data "Campaign Information" to a smartphone or smart glasses.

[1814] Step 2:

[1815] The terminal sends input to the server.

[1816] The terminal sends the entered questions and keywords to the server. Specifically, it sends an HTTP POST request to the API endpoint, and the request includes metadata such as the user ID and the current date and time along with the input content.

[1817] Input: Questions or keywords entered on the device, such as "campaign information".

[1818] Output: API request sent to the server (questions, keywords + metadata)

[1819] Step 3:

[1820] The server searches the database.

[1821] The server analyzes the received questions and keywords and searches the database. It uses a generative AI model to perform natural language processing and generate the optimal query. Based on the query, it retrieves relevant information from the database.

[1822] Input: API request received by the server (question or keyword + metadata)

[1823] Output: Raw data (related information) retrieved from the database.

[1824] Step 4:

[1825] The server organizes the search results.

[1826] The server organizes the acquired data and converts it into a user-friendly format. Specifically, it selects highly relevant information and converts it into JSON or XML format to generate response data. Relevant links and additional explanations are also added as needed.

[1827] Input: Raw data retrieved from the database (related information)

[1828] Output: Organized response data (in JSON or XML format)

[1829] Step 5:

[1830] The server sends the organized information to the terminal.

[1831] The server sends the generated response data to the terminal. This ensures that the necessary information is provided to the user's terminal in real time.

[1832] Input: Response data generated within the server (organized information)

[1833] Output: Response data sent to the terminal

[1834] Step 6:

[1835] The device displays information

[1836] The device displays received information in a user-friendly format. The displayed content includes detailed information and related links. Notifications are also used to inform the user that information has been provided.

[1837] Input: Response data sent from the server

[1838] Output: Information displayed on the device (campaign information and related links)

[1839] Step 7:

[1840] The user confirms the information.

[1841] Users can review the displayed information and provide it to customers on the spot. For example, they can instantly explain the latest campaign information to customers.

[1842] Input: Information displayed on the device (campaign information and related links)

[1843] Output: Latest campaign information provided to customers

[1844] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1845] The system of this invention aims to resolve questions immediately during business operations and negotiations, thereby improving operational efficiency and productivity. Furthermore, by combining it with an emotion engine that recognizes user emotions, it aims to further enhance the appropriateness of the information provided.

[1846] System-wide configuration

[1847] The system was configured as follows:

[1848] 1. A terminal that provides a user interface.

[1849] 2. Servers that process information

[1850] 3. Database for storing information

[1851] 4. Emotion engine that recognizes user emotions

[1852] Program processing

[1853] 1. The user enters a question or keyword.

[1854] User: Enter questions or keywords such as "What are the latest campaign strategies?" into the device.

[1855] 2. The terminal sends the input to the server.

[1856] Terminal: Sends input data to the emotion engine to analyze the user's emotions. Then, it sends questions, keywords, and emotion data to the server.

[1857] 3. The server searches the database.

[1858] Server: Searches the database based on received questions, keywords, and sentiment data. Sentiment data is also considered when selecting highly relevant information.

[1859] 4. The server organizes the search results.

[1860] Server: Organizes information retrieved from the database and adjusts the format and priority of the information based on sentiment data. For example, if a user is confused, it prioritizes providing more detailed and specific information.

[1861] 5. The server sends the organized information to the terminal.

[1862] Server: Sends organized information to the terminal. The information is provided in JSON or XML format and includes additional explanations and related links.

[1863] 6. The device displays the information.

[1864] Terminal: Displays received information to the user. Based on sentiment data, the interface design and notification methods are also adjusted.

[1865] 7. Users verify information and resolve their questions.

[1866] User: Can review the displayed information and immediately explain it to the business partner.

[1867] Specific example

[1868] Situation 1: Resolving questions and recognizing emotions during a business negotiation

[1869] 1. User: During a business meeting, the user types "What are the latest campaign strategies?". At that time, the user's voice and facial expressions are analyzed by an emotion engine, and it is recognized that the user is slightly nervous.

[1870] 2. Terminal: Sends the question to the server along with the analyzed sentiment data.

[1871] 3. Server: Search the database and select campaign information in an easy-to-understand format, taking into consideration the needs of anxious users.

[1872] 4. Server: Organizes the selected information in detail and sends it to the terminal.

[1873] 5. Device: Since users may be stressed, display information clearly using color and large fonts.

[1874] 6. User: They can view the displayed information, immediately explain it to their business partner, and communicate effectively.

[1875] Situation 2: Inquiry from an agency and emotional recognition

[1876] 1. User: "Is there an updated version of the operation manual?" At this point, the user's voice and facial expressions are analyzed by the emotion engine, and it is recognized that the user is confused.

[1877] 2. Terminal: Sends the question to the server along with the analyzed sentiment data.

[1878] 3. Server: Search the database and prioritize selecting user manuals with more detailed instructions for users who are confused.

[1879] 4. Server: Organizes the selected information and sends it to the terminal.

[1880] 5. Device: Displays information in an easy-to-understand format, and provides supplementary explanations as needed.

[1881] 6. User: Can review the displayed information and explain to the agent how to provide the latest operation manual.

[1882] Thus, the system of the present invention, which combines an emotion engine, enables the provision of appropriate information according to the user's situation, thereby significantly improving the efficiency and quality of business operations and negotiations.

[1883] The following describes the processing flow.

[1884] Step 1:

[1885] The user enters a question or keyword.

[1886] User: Enter "What are the latest campaign measures?" into the input field on the device.

[1887] Step 2:

[1888] The device sends the entered question or keyword to the emotion engine.

[1889] Terminal: Captures user voice and facial expressions along with input content, and sends this to the emotion engine.

[1890] Step 3:

[1891] The emotion engine analyzes the user's emotions.

[1892] Emotion Engine: Analyzes captured audio and facial expression data to recognize emotions such as tension or confusion in the user.

[1893] Step 4:

[1894] The emotion engine sends the analysis results to the terminal.

[1895] Emotion Engine: Sends emotion analysis results back to the device. For example, it may include information such as "the user is feeling nervous."

[1896] Step 5:

[1897] The device sends the sentiment analysis results, along with any questions or keywords, to the server.

[1898] Terminal: Sends user input and sentiment analysis results to the server via API.

[1899] Step 6:

[1900] The server analyzes the questions, keywords, and sentiment data it receives.

[1901] Server: Analyzes questions and keywords, and generates optimal queries considering sentiment data.

[1902] Step 7:

[1903] The server searches the database.

[1904] Server: Searches the database based on the generated query and retrieves the most relevant information.

[1905] Step 8:

[1906] The server organizes the search results.

[1907] Server: Organizes the acquired information and adjusts the priority and display method of the information, taking into account the sentiment analysis results. For example, it provides information in a format that is particularly easy to understand for users who are feeling anxious.

[1908] Step 9:

[1909] The server sends the organized information to the terminal.

[1910] Server: Sends organized information to the terminal in JSON or XML format.

[1911] Step 10:

[1912] The device displays the information it has received to the user.

[1913] Terminal: Analyzes received data and displays it in a user-friendly format. For example, it might use large fonts or color displays.

[1914] Step 11:

[1915] The user reviews the displayed information and resolves any questions.

[1916] User: Review the displayed information on "Latest Campaign Measures" and explain it to the business partner on the spot.

[1917] (Example 2)

[1918] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1919] Traditional systems can provide information based on user-inputted questions and keywords, but they fail to consider the user's emotional state, which can result in providing information that is optimal for the user. In particular, in business negotiations and operations, user emotions significantly influence information comprehension and decision-making, so ignoring this can lead to problems such as decreased work efficiency and reduced quality of communication.

[1920] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1921] In this invention, the server includes means for transmitting a user-inputted question or keyword to an emotion recognition engine to generate emotion data; means for transmitting the generated emotion data along with the question or keyword to the server; means for searching a database based on the received question or keyword and emotion data; and means for organizing the search results based on the emotion data and providing the information in a format that is easy for the user to understand. This makes it possible to provide appropriate information according to the user's emotional state.

[1922] A "user" refers to an entity that uses a system to input questions or keywords and obtain information.

[1923] A "question" refers to a question that a user enters as information they want to know or a problem they want to solve.

[1924] A "keyword" refers to a word or phrase that a user enters to search for specific information.

[1925] An "emotion recognition engine" refers to a software or hardware system that analyzes a user's voice and facial expressions and generates data on their emotional state.

[1926] "Emotional data" refers to data generated by an emotion recognition engine that indicates the user's current emotional state.

[1927] A "server" refers to a central processing unit that analyzes received data, searches databases, and provides information.

[1928] A "database" refers to a data storage system that systematically stores information and manages it in a searchable format.

[1929] "Searching" refers to the process by which a server examines information within a database and extracts data that matches specific criteria.

[1930] "Providing information" refers to presenting data that is highly relevant to the questions or keywords entered by the user.

[1931] The system of this invention aims to improve the efficiency and productivity of work by immediately resolving questions in business operations and negotiations. Furthermore, by combining it with an emotion engine that recognizes user emotions, it aims to further enhance the appropriateness of the information provided.

[1932] System Configuration

[1933] The system consists of the following elements:

[1934] 1. A terminal that provides a user interface.

[1935] Hardware required: Windows PC, Mac, mobile devices (iOS, Android)

[1936] Software to use: Browser (Chrome, Safari, etc.)

[1937] 2. Servers that process information

[1938] Hardware used: Linux server

[1939] Software used: Apache HTTP Server

[1940] 3. Database for storing information

[1941] Hardware used: Hard disk, SSD

[1942] Software used: MySQL, PostgreSQL

[1943] 4. Emotion engine that recognizes user emotions

[1944] Hardware used: Camera, microphone

[1945] Software used: Microsoft Azure Emotion API, IBM Watson Tone Analyzer

[1946] Program processing

[1947] 1. The user enters a question or keyword.

[1948] Users input questions or keywords on their own devices. For example, during a business meeting, they might type "What are the latest campaign strategies?" using the keyboard. Alternatively, they might use voice input to ask "What are the latest campaign strategies?"

[1949] 2. The device sends the input to the emotion engine.

[1950] The device receives the entered question or keyword and sends that data to the emotion engine. The emotion engine analyzes the user's voice tone and facial expressions and generates the user's emotion data (e.g., tension, confusion, joy, etc.).

[1951] 3. The device sends question and sentiment data to the server.

[1952] The device sends emotion data returned from the emotion engine, along with questions and keywords entered by the user, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[1953] 4. The server searches the database.

[1954] The server analyzes the received questions, keywords, and sentiment data, and searches a MySQL or PostgreSQL database. The server generates SQL queries to extract relevant information.

[1955] 5. The server organizes the search results.

[1956] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking sentiment data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious. Specifically, it performs filtering based on the importance of the information and the user's emotions.

[1957] 6. The server sends the organized information to the terminal.

[1958] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[1959] 7. The device displays the information.

[1960] The device displays received information to the user. Based on emotional data, the interface design and display methods are adjusted. For example, color displays and larger fonts are used to alleviate tension.

[1961] 8. Users verify information and resolve their questions.

[1962] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[1963] Specific example

[1964] 1. Resolving questions and recognizing emotions during business negotiations

[1965] User: During a business meeting, the user voice-inputs "What are the latest campaign strategies?" into their laptop. The tone of voice and facial expression are captured by the camera.

[1966] Terminal: Sends input data to the emotion engine and retrieves emotional data that indicates tension. Next, it sends the question along with the analyzed data to the server.

[1967] Server: Searches the database and selects easy-to-understand campaign strategy information.

[1968] Server: Organizes information and sends it to the terminal.

[1969] Terminal: Displays information using large fonts and colored highlighting.

[1970] User: Confirm information and immediately explain it to the business partner. Share detailed materials to facilitate smooth business negotiations.

[1971] 2. Inquiries from agencies and emotional recognition

[1972] User: "Is there an updated version of the operation manual?" is typed on the smartphone. Voice input and facial expression capture via camera are used.

[1973] Terminal: Sends input data to the emotion engine and retrieves emotion data that analyzes the user as confused. Next, it sends the emotion data and question to the server.

[1974] Server: Search the database and prioritize selecting user manuals that provide detailed instructions for users who are experiencing difficulties.

[1975] Server: Organizes information and sends it to the terminal.

[1976] Device: Displays information in an easy-to-understand format, and links to video tutorials if necessary.

[1977] User: Review the information and provide detailed instructions to the agency on how to provide the latest operation manual. Share video tutorials to enhance understanding.

[1978] In this way, by enabling the provision of appropriate information tailored to the user's emotional state, it is possible to significantly improve work efficiency and the quality of communication.

[1979] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1980] Step 1:

[1981] The user enters a question or keyword.

[1982] Specifically, the user enters a question, such as "What are the latest campaign initiatives?", using a text area or voice input on their device. The input data is stored on the device in its initial format.

[1983] input:

[1984] User questions and keywords (e.g., "What are the latest campaign strategies?")

[1985] output:

[1986] Initial format input data

[1987] Step 2:

[1988] The device sends input to the emotion engine.

[1989] Specifically, the device captures the entered text data as well as the user's voice and facial expressions, and sends them to an emotion recognition engine. Microsoft Azure Emotion API and IBM Watson Tone Analyzer are examples of such systems. The emotion engine analyzes the user's voice tone and facial expressions to generate emotion data.

[1990] input:

[1991] Initial format input data

[1992] User voice and facial expression data

[1993] output:

[1994] User emotion data (e.g., "nervous")

[1995] Step 3:

[1996] The device sends question and emotion data to the server.

[1997] After the emotion data is generated, the device sends that data, along with any questions or keywords, to the server. The data is formatted in JSON format and sent using an HTTP POST request.

[1998] input:

[1999] Initial format input data

[2000] Emotional data

[2001] output:

[2002] Question and sentiment data formatted in JSON format

[2003] Step 4:

[2004] The server searches the database.

[2005] The server analyzes the received questions, keywords, and sentiment data, and searches MySQL or PostgreSQL databases based on this analysis. The server generates SQL queries to extract the most relevant information.

[2006] input:

[2007] Question and sentiment data formatted in JSON format

[2008] output:

[2009] Related information extracted based on SQL queries

[2010] Step 5:

[2011] The server organizes the search results.

[2012] The server organizes the results based on information retrieved from the database. It adjusts the format and priority of the information, taking emotional data into consideration. For example, it prioritizes providing information written in simple language to users who are feeling anxious.

[2013] input:

[2014] Related information extracted from the database

[2015] Emotional data

[2016] output:

[2017] Organized information (including diagrams, graphs, and explanatory text)

[2018] Step 6:

[2019] The server sends the organized information to the terminal.

[2020] The server sends organized information to the terminal. The information is formatted in JSON or XML format and includes relevant links and supplementary explanations. It is returned as an HTTP response.

[2021] input:

[2022] Organized information

[2023] output:

[2024] Information formatted in JSON or XML format

[2025] Step 7:

[2026] The device displays the information.

[2027] The device displays received information to the user. Based on emotional data, it adjusts the interface design and notification methods. For example, it uses color displays and large fonts to alleviate tension.

[2028] input:

[2029] Information formatted in JSON or XML format

[2030] Emotional data

[2031] output:

[2032] Adjusted information displayed to the user

[2033] Step 8:

[2034] Users can verify information and resolve their questions.

[2035] Users can review the displayed information and immediately explain it to their business partners. For example, information about campaign strategies might be displayed, allowing them to proceed with negotiations based on that information. Users can also click on supplementary information and related links to view more detailed content.

[2036] input:

[2037] Adjusted information

[2038] output:

[2039] Understood information

[2040] Use in business negotiations and explanations.

[2041] (Application Example 2)

[2042] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2043] In modern business and negotiations, it is essential to quickly resolve questions and provide appropriate information. However, there is a lack of systems that can immediately answer user questions, and there are no mechanisms to adjust the way information is delivered based on emotions. As a result, user satisfaction and the efficiency of negotiations suffer.

[2044] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions using an emotion engine, means for adjusting the format and priority of the information provided according to the user's emotions, and means for displaying the information using a smart device. This makes it possible to answer the user's questions in real time while providing information in an optimal form according to the user's emotions.

[2045] A "user" is an individual or legal entity that uses the system and obtains information by entering questions or keywords.

[2046] A "question" refers to a specific question or point of confusion that a user has, and is information that the user seeks to resolve from the system.

[2047] A "keyword" is a specific word or phrase entered by the user, which serves as the basis for the system to search for information.

[2048] A "server" is a computer device that receives user input, searches a database, and provides relevant information.

[2049] A "database" is a collection of diverse information that is organized and stored, and is what a server searches for.

[2050] An "emotion engine" is a technology that analyzes a user's emotions and adjusts how information is provided within the system based on the results.

[2051] "Emotions" or "emotional data" refers to information that represents the user's psychological state and is the subject of analysis and consideration by the system.

[2052] A "smart device" is a device that connects to a network and can display and operate digital information. Specific examples include smart glasses and head-mounted displays.

[2053] "Information format" refers to the composition, design, font size, color scheme, and other elements of how information provided to users is visually displayed.

[2054] "Priority" refers to the order and importance of information presented to the user, and is adjusted based on the user's emotions.

[2055] "Organization" refers to the process of compiling information retrieved from a database into a format that is easy for users to understand.

[2056] "Providing" means the act of a system displaying organized information on a user's device, making it accessible to the user.

[2057] "Real-time" means processing user input and emotions instantly and providing results immediately.

[2058] This invention is a system in which a user inputs questions or keywords using a smart device and obtains relevant information in real time. The system is implemented with the following configuration.

[2059] Hardware configuration

[2060] Device: A device such as smart glasses or a head-mounted display. This device has a built-in microphone and camera that can capture the user's voice and facial expressions.

[2061] Server: The server computer performs tasks such as searching the database and running the sentiment engine.

[2062] Database: Database software such as MySQL or PostgreSQL is used to store and manage necessary information.

[2063] Software Configuration

[2064] Emotion Engine: Uses Microsoft Azure Cognitive Services, Affectiva, and other tools to analyze user emotions in real time.

[2065] Database search: A software module for searching for relevant information from a database.

[2066] Display Control: A software module for displaying searched information on smart glasses.

[2067] Operation flow

[2068] 1. The user enters a question.

[2069] Enter questions or keywords such as "What are the latest camera features?" into the device via voice or text.

[2070] 2. Analysis of emotions

[2071] The device transmits voice and facial expression data to an emotion engine, which analyzes the user's emotions. This analysis includes information such as whether the user is nervous or excited.

[2072] 3. Sending to the server

[2073] The device sends questions, keywords, and sentiment data to the server.

[2074] 4. Information Search

[2075] The server searches the database based on questions and keywords and retrieves relevant information. It then adjusts the format and priority of the information based on sentiment data.

[2076] 5. Display of Information

[2077] The organized information is sent to the device and displayed on the smart glasses in a format optimized for the user's emotions, such as using large fonts or illustrations.

[2078] Specific example

[2079] For example, while a user is wearing smart glasses, a customer might ask, "What are the latest features of this camera?" The user then inputs the question by voice through the smart glasses.

[2080] The emotion engine analyzes the customer's behavior and determines that they are slightly nervous. Based on this information, the server prioritizes selecting the latest feature information from the database, presenting it in an easy-to-understand format. This information is then displayed on the smart glasses' screen using large fonts and diagrams, allowing the user to receive information quickly and accurately.

[2081] Example of a prompt

[2082] "Could you give me an example of a system that analyzes a customer's feelings of tension or anxiety when they ask a question about a product, and then displays the latest product information on smart glasses using easy-to-understand diagrams and large fonts?"

[2083] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2084] Step 1:

[2085] The user enters a question or keyword into the device. The user uses a smart device such as smart glasses to input a question or keyword, such as "What are the latest camera features?", via voice or text. The data provided by the user here is the text input of the question or keyword.

[2086] Step 2:

[2087] The device sends the entered questions and keywords to the emotion engine. The device uses its built-in microphone and camera to collect the user's voice and facial expression data and sends it to the emotion engine. The emotion engine receives this data and analyzes the user's psychological state (e.g., tension, excitement). The input data consists of voice data and facial expression data, and the output is the user's emotional state.

[2088] Step 3:

[2089] The device sends questions, keywords, and emotion data acquired by the terminal to the server. The emotion information analyzed by the emotion engine, along with the questions and keywords in text format, are sent to the server together. This allows the server to process the data while considering the user's psychological state. The input data consists of questions, keywords, and emotion data; the output is the transmission of this data to the server.

[2090] Step 4:

[2091] The server searches the database based on the questions, keywords, and sentiment data it receives. The server uses a database search module to find the information most relevant to the user's input. Furthermore, it adjusts the format and priority of the information provided based on the sentiment data. The input data consists of questions, keywords, and sentiment data; the output is a list of relevant information.

[2092] Step 5:

[2093] The server organizes search results and provides information in the most appropriate format according to the user's emotional state. The server organizes the retrieved search results and formats the information in an easy-to-understand format (e.g., diagrams, color display, large font) based on the emotional data. This ensures that information is provided in a format that is easy for the user to understand. The input data consists of a list of related information and emotional data, and the output is formatted information.

[2094] Step 6:

[2095] The terminal displays organized information on the smart device. The terminal receives formatted information from the server and displays it on the smart glasses' display. Appropriate information is provided in response to the user's questions, taking into account emotions such as tension and excitement. The input data is formatted information, and the output is the information displayed in the user's field of vision.

[2096] Step 7:

[2097] Users review the displayed information and utilize it in business negotiations and work. Users check the information displayed on their smart glasses and use it to explain things to their business partners. This allows users to provide information quickly and appropriately. The input data is the information displayed on the smart device, and the output is the smooth progress of business negotiations and work.

[2098] Through the steps described above, this invention realizes a system that instantly resolves user questions and provides appropriate information according to their emotions.

[2099] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2100] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2101] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2102] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2103] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2104] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2105] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2106] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2107] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[2108] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2109] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2110] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2111] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2112] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2113] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2114] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2115] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2116] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2117] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2118] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2119] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[2120] The following is further disclosed regarding the embodiments described above.

[2121] (Claim 1)

[2122] A means for the user to enter a question or keyword,

[2123] A means of sending the entered question or keyword to the server,

[2124] A means of searching the database based on the question or keyword received by the server,

[2125] A means of organizing search results and providing information in a format that is easy for users to understand,

[2126] A system that includes this.

[2127] (Claim 2)

[2128] The system according to claim 1, further comprising means for the server to prioritize the selection of information that is highly relevant to the user's question or keyword.

[2129] (Claim 3)

[2130] The system according to claim 1, further comprising means for the terminal to notify the user of information provided to the terminal.

[2131] "Example 1"

[2132] (Claim 1)

[2133] A means for the user to enter a question or keyword,

[2134] A means by which the terminal sends the entered question or keyword to the server,

[2135] A means of searching the database based on the question or keyword received by the server,

[2136] A means by which the server organizes search results and provides information in a format that is easy for the user to understand,

[2137] A means of displaying information received by the terminal,

[2138] Means by which users can verify information,

[2139] A system that includes this.

[2140] (Claim 2)

[2141] The system according to claim 1, further comprising means for the server to prioritize the selection of information that is highly relevant to the user's question or keyword.

[2142] (Claim 3)

[2143] The system according to claim 1, further comprising means for the terminal to notify the user of information provided to the terminal.

[2144] "Application Example 1"

[2145] (Claim 1)

[2146] A means for the user to enter a question or keyword,

[2147] A means of sending the entered question or keyword to the server,

[2148] A means of searching the database based on the question or keyword received by the server,

[2149] A means of organizing search results and providing information in a format that is easy for staff to understand,

[2150] A means by which the terminal displays information organized based on the search to the staff,

[2151] The terminal provides a means for staff to be notified of the information it has provided,

[2152] A system that includes this.

[2153] (Claim 2)

[2154] The system according to claim 1, further comprising means for the server to prioritize the selection of information that is highly relevant to the user's question or keyword.

[2155] (Claim 3)

[2156] The system according to claim 1, further comprising means for obtaining information using a generative AI model based on a prompt statement.

[2157] "Example 2 of combining an emotion engine"

[2158] (Claim 1)

[2159] A means for the user to enter a question or keyword,

[2160] A means for sending an input question or keyword to an emotion recognition engine and generating user emotion data,

[2161] A means of sending a question or keyword to the server along with the generated sentiment data,

[2162] A means for searching a database based on the questions or keywords and sentiment data received by the server,

[2163] A means of organizing search results and providing information in a format that is easy for users to understand, taking into account user sentiment data,

[2164] A system that includes this.

[2165] (Claim 2)

[2166] The system according to claim 1, further comprising means for the server to prioritize the selection of information that is highly relevant to the user's questions or keywords and sentiment data.

[2167] (Claim 3)

[2168] The system according to claim 1, further comprising means for the terminal to adjust and display information provided to the user based on sentiment data.

[2169] "Application example 2 when combining with an emotional engine"

[2170] (Claim 1)

[2171] A means for the user to enter a question or keyword,

[2172] A means of sending the entered question or keyword to the server,

[2173] A means of searching the database based on the question or keyword received by the server,

[2174] A means of organizing search results and providing information in a format that is easy for users to understand,

[2175] A method for analyzing user emotions using an emotion engine,

[2176] A means of adjusting the format and priority of information provided according to the user's emotions,

[2177] A means of displaying information using a smart device,

[2178] A system that includes this.

[2179] (Claim 2)

[2180] The system according to claim 1, further comprising means for the server to prioritize the selection of information that is highly relevant to the user's question or keyword.

[2181] (Claim 3)

[2182] The system according to claim 1, further comprising means for the terminal to notify the user of information provided to the terminal. [Explanation of symbols]

[2183] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for the user to enter a question or keyword, A means of sending the entered question or keyword to the server, A means of searching the database based on the question or keyword received by the server, A means of organizing search results and providing information in a format that is easy for users to understand, A system that includes this.

2. The system according to claim 1, further comprising means for the server to prioritize the selection of information that is highly relevant to the user's question or keyword.

3. The system according to claim 1, further comprising means for the terminal to notify the user of information provided to the terminal.

Images