System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a question and an answer to the question from a user and generating an entry including a time stamp, the question, and the answer; means for storing the generated entry as a history; means for drawing a content of the entry in an image; and means for displaying the entry stored as the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional information search systems manage the history of information that users have researched in text format, which makes it difficult for users to visually recognize and remember specific research content when referring to it later. Furthermore, when conducting multiple researches, it is difficult to determine which entry corresponds to which question, making it difficult for users to efficiently review information. Furthermore, there is a lack of functionality for managing history for each user, making it difficult for users to easily identify their own research history. [Means for solving the problem]

[0005] This invention solves the above-mentioned problems by providing a system including: means for receiving a question and its answer from a user, generating an entry including a timestamp, the question, and the answer; means for saving the generated entry as a history; means for rendering the contents of the entry as an image; and means for displaying the entry saved as an image. Specifically, by recording the question and answer from the user as an entry and saving the entry in image format, the history is managed in a form that is easy to visually recognize and remember later. Furthermore, by including means for adding a user ID to the entry saved as an image, history management for each user is facilitated, allowing users to efficiently refer to their own research history. By rendering the timestamp, question, and answer on the image, it is possible to clarify what was researched at a specific time, improving convenience when reviewing the history later.

[0006] "User" refers to an entity that uses the system to ask questions and receive answers.

[0007] A "question" refers to a specific piece of information or knowledge that a user requests from the system.

[0008] "Answer" refers to the information or knowledge provided by the system in response to a question.

[0009] "Timestamp" refers to time information for recording the date and time when a question and answer were asked.

[0010] An "entry" refers to a unit of recording that includes a timestamp, a question, and an answer.

[0011] "History" refers to a collection of data in which multiple entries are saved and managed in chronological order.

[0012] "Means for rendering into an image" refers to a technique for processing to generate an image file to visually represent the contents of an entry.

[0013] "Means for displaying" refers to the technology or method by which a user can view the saved image.

[0014] "User ID" refers to a unique identifier for identifying an individual user. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system that receives user questions and their answers, saves them as a history, and allows users to visually review them later. How this system is implemented will be specifically described below.

[0037] System Overview

[0038] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to refer to information in a visually easy-to-recognize format, making it easier to remember.

[0039] Program Operation

[0040] 1. Getting user questions and answers

[0041] The user asks the system a question. The device receives the question and sends it to an appropriate system (e.g., generative AI) to get an answer.

[0042] 2. Creating an entry containing a timestamp, question, and answer

[0043] The device adds a timestamp to the acquired question and answer and records it as an entry, which includes the question, the answer, and the date and time it was recorded.

[0044] 3. Save entries as history

[0045] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[0046] 4. Drawing the entry content onto the image

[0047] A means is used to render the contents of the entry, i.e., the timestamp, question, and answer, as an image, specifically by writing the text information to an image file.

[0048] 5. Save the image

[0049] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[0050] 6. Displaying Images

[0051] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[0052] Specific examples

[0053] For example, if a user asks the following question:

[0054] Question 1: "What is AI?"

[0055] Answer 1: "AI stands for Artificial Intelligence."

[0056] Next, the user asks another question:

[0057] Question 2: "What is Machine Learning?"

[0058] Answer 2: "Machine Learning is a subset of AI."

[0059] In this case, each question and answer is assigned a timestamp and created as an entry. This entry is saved as an image with a file name such as "user1234_20230203_120501.png" or "user1234_20230203_121201.png." Users can later review these images to visually confirm when they asked what question and what answer they received.

[0060] The server manages the storage of these images and provides a means for the user to quickly display them when they want to refer to a particular entry, so that the user can easily recall the content of their research at that time.

[0061] The above is an embodiment of the present invention, which provides a system that allows users to visually and efficiently manage and refer to their investigation history.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] The user inputs a question to the system.

[0065] Step 2:

[0066] The device receives questions entered by the user and poses the questions to generative AI or other information provision systems.

[0067] Step 3:

[0068] Generative AI and information provision systems generate answers to questions and return them to the device.

[0069] Step 4:

[0070] The device retrieves the current date and time and formats it as a timestamp.

[0071] python

[0072] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[0073] Step 5:

[0074] The terminal generates an entry that includes a timestamp, a question, and an answer.

[0075] python

[0076] entry = {

[0077] 'timestamp': timestamp,

[0078] 'question': question,

[0079] 'answer': answer

[0080] }

[0081] Step 6:

[0082] The terminal stores the generated entries in its history.

[0083] python

[0084] self.history.append(entry)

[0085] Step 7:

[0086] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[0087] python

[0088] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[0089] draw = ImageDraw.Draw(img)

[0090] font = ImageFont.load_default()

[0091] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[0092] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[0093] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[0094] Step 8:

[0095] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[0096] python

[0097] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[0098] Step 9:

[0099] Saves the image generated by the device to the specified path.

[0100] python

[0101] img.save(img_path)

[0102] Step 10:

[0103] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[0104] Step 11:

[0105] When a user wants to view the history, they send a request from their device to the server. The server searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question and answer they asked.

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] In conventional question-answering systems, there are few ways for users to easily refer to past questions and their answers, making it difficult for users to visually confirm what questions they asked and what the answers were. In addition, storing history in text format lacks visual information, making it difficult to remember and recognize information.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes means for receiving questions and answers from users and transmitting prompt sentences to the generative AI model to obtain answers, means for generating entries by adding timestamps to the obtained questions and answers, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for saving the rendered images, and means for displaying the saved images. This allows users to visually check past questions and answers, making it easier for them to remember and recognize information.

[0111] A "user" is an individual or entity that enters a question into the system and obtains an answer.

[0112] A "question" refers to a question or problem that a user inputs into the system.

[0113] "Answer" refers to the response that a generative AI model generates in response to a user's question.

[0114] A "generative AI model" is an artificial intelligence algorithm or system that receives a prompt and generates an answer based on its content.

[0115] A "prompt" is a command or query that a generative AI model uses to generate an appropriate answer.

[0116] A "timestamp" is a digital mark that indicates a specific date and time, and is information that clarifies the point in time when a question or answer was recorded.

[0117] An "entry" refers to a collection of data that includes a question, an answer, and a timestamp.

[0118] "History" refers to a collection of data in which entries created in the past are organized in chronological order.

[0119] "Rendering to image" refers to the act of converting the contents of an entry into a visually identifiable format and saving it as an image file.

[0120] "User identification information" refers to information for uniquely identifying a specific user, and includes, for example, a user ID.

[0121] "Server" means the central computer that manages and operates the entire system and processes requests from users.

[0122] The present invention relates to a system that records questions and answers posed by users along with timestamps, and allows them to be visually saved and referenced. How this system is implemented is described below.

[0123] System configuration

[0124] 1. Hardware configuration:

[0125] Server: The central computer that manages the entire question-answering system. It is responsible for communicating with the database and the generative AI model.

[0126] Terminal: A device (such as a computer or smartphone) that is directly operated by the user and provides functions for inputting questions, displaying them, and saving them.

[0127] 2. Software configuration:

[0128] Generative AI model: A generative AI model uses an artificial intelligence algorithm to generate answers to user questions. As a specific example, we use a model that uses natural language processing technology.

[0129] Pillow Library: Uses Python's Pillow library to draw entries onto the image.

[0130] Overview of program processing

[0131] 1. Getting user questions and answers

[0132] The user inputs a question into the device, which then sends the question to the generative AI model as a prompt. For example, the question "What is AI?" is converted into the prompt "Generate an answer to the user's question: 'What is AI?'"

[0133] The generative AI model receives the prompt sentence, generates a response such as "AI stands for Artificial Intelligence," and returns it to the device.

[0134] 2. Creating an entry containing a timestamp, question, and answer

[0135] The device adds a timestamp indicating the current date and time to the questions and answers it receives, and organizes each piece of information into an entry.

[0136] 3. Save entries as history

[0137] The device stores the generated entries in a database and organizes them in chronological order.

[0138] 4. Drawing the entry content onto the image

[0139] The terminal uses Python's Pillow library to draw the entry contents (timestamp, question, answer) as an image.

[0140] 5. Save the image

[0141] The drawn image is given a unique name using the user's identification information and a timestamp and saved as a file, for example, in the format "user1234_20231003_120000.png".

[0142] 6. Displaying Images

[0143] If the user wants to view the history, the device will display the saved image, allowing the user to visually review the history of questions and answers.

[0144] Specific operation example

[0145] For example, a user asks the following question:

[0146] Question 1: "What is AI?"

[0147] Answer 1: "AI stands for Artificial Intelligence."

[0148] Prompt: "Generate an answer to the user's question: 'What is AI?'"

[0149] Next, the user asks another question:

[0150] Question 2: "What is Machine Learning?"

[0151] Answer 2: "Machine Learning is a subset of AI."

[0152] Prompt: "Generate an answer to the user's question: 'What is Machine Learning?'"

[0153] These questions and answers are each time-stamped and created as entries. These entries are saved in image format with file names such as "user1234_20231003_120000.png" and "user1234_20231003_121000.png." By referring to these images, users can visually confirm when they asked what questions and what answers they received.

[0154] As described above, the present invention provides a specific means for visually and efficiently managing and referencing a user's question and answer history.

[0155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0156] Step 1:

[0157] The user inputs a question via a terminal.

[0158] Specific behavior:

[0159] The user enters "What is AI?" into the input field on the terminal and presses the send button.

[0160] input:

[0161] A user asks, "What is AI?"

[0162] output:

[0163] The terminal acquires the user's question as text data.

[0164] Step 2:

[0165] The device generates a prompt sentence to send the acquired question to the generative AI model.

[0166] Specific behavior:

[0167] The device converts the question text "What is AI?" into the prompt "Generate an answer to the user's question: 'What is AI?'"

[0168] input:

[0169] The question text obtained from the user: "What is AI?"

[0170] output:

[0171] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[0172] Step 3:

[0173] The device sends the generated prompt to the generative AI model and obtains the answer.

[0174] Specific behavior:

[0175] The device sends the prompt as a request to the generative AI model's API, which then generates the answer "AI stands for Artificial Intelligence." based on the prompt and returns it to the device.

[0176] input:

[0177] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[0178] output:

[0179] The answer returned by the generative AI model: "AI stands for Artificial Intelligence."

[0180] Step 4:

[0181] The terminal adds a timestamp to the acquired question and answer and creates an entry.

[0182] Specific behavior:

[0183] The device obtains the current date and time and creates entry data containing the question "What is AI?", the answer "AI stands for Artificial Intelligence.", and the timestamp "2023 / 10 / 03 12:00:00".

[0184] input:

[0185] Question: "What is AI?" Answer: "AI stands for Artificial Intelligence." Current date and time: "2023 / 10 / 03 12:00:00".

[0186] output:

[0187] Generated entry data (question, answer, timestamp).

[0188] Step 5:

[0189] The terminal stores the created entries in a database as history.

[0190] Specific behavior:

[0191] Insert the entry data (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Timestamp: "2023 / 10 / 03 12:00:00") into the database.

[0192] input:

[0193] The generated entry data.

[0194] output:

[0195] The history of entries in the database, ordered chronologically.

[0196] Step 6:

[0197] The terminal will render the contents of the entry into an image and save it as a file.

[0198] Specific behavior:

[0199] The device uses Python's Pillow library to convert the entry content (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Time Stamp: "2023 / 10 / 03 12:00:00") into an image and saves it with the file name "user1234_20231003_120000.png".

[0200] input:

[0201] Entry data (question, answer, timestamp).

[0202] output:

[0203] The saved image file is "user1234_20231003_120000.png".

[0204] Step 7:

[0205] If the user wants to view the history, the device displays the saved images.

[0206] Specific behavior:

[0207] When the user performs an operation to open the history display screen, the device loads the corresponding image file "user1234_20231003_120000.png" and displays it on the screen.

[0208] input:

[0209] User reference request, image file "user1234_20231003_120000.png".

[0210] output:

[0211] Image content (question, answer, timestamp) displayed on the device screen.

[0212] (Application example 1)

[0213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] In modern virtual stores, there is a problem in that users have few means to inquire about product information and then visually confirm that information later. Specifically, when a user makes multiple inquiries, it is difficult to centrally manage the details of those inquiries and review them visually. This leads to problems such as users having to ask the same questions repeatedly or being unable to efficiently utilize the information they obtain.

[0215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0216] In this invention, the server includes means for receiving questions and answers from users and generating entries including a timestamp, the question, and the answer, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for displaying the entries saved as images, means for recording product information inquiries and their answers in a virtual store, means for uniquely naming the entries using a user ID and a timestamp and saving them in cloud storage, and means for displaying a list of image entries in an album view format, thereby enabling users to visually manage their product inquiry history and efficiently review the information.

[0217] A "user question" is a request for information posed by a user to the system.

[0218] An "answer" is information provided by the system in response to an inquiry.

[0219] A "timestamp" is information that indicates the date and time when the entry was created.

[0220] An "entry" is a record that includes a timestamp, question, and answer data.

[0221] A "history" is a collection of entries created in the past, stored in chronological order.

[0222] An "image" is digital data that visually depicts the contents of an entry.

[0223] A "virtual store" is an online sales platform that offers products and services over the Internet.

[0224] A "user ID" is an identifier that uniquely identifies an individual user.

[0225] "Cloud storage" is a service that stores and makes accessible data on remote servers over the Internet.

[0226] The "album view format" is a user interface format for visually displaying a list of image entries.

[0227] "Uniquely named" means adding a unique name to each entry or file so that the same name is never generated twice.

[0228] The present invention is implemented as a system that allows users to manage and visually review questions and answers posed by users in a virtual store. This system effectively manages questions posed by users to search for product information and their answers, and displays them in an album view format.

[0229] System Overview

[0230] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to visually manage their product inquiry history and easily review the information they have obtained later.

[0231] Hardware and Software Configuration

[0232] This system uses the following hardware and software:

[0233] Hardware: Smartphones, smart glasses

[0234] software:

[0235] Frontend: React Native

[0236] Backend: Python, Flask

[0237] Database: AWS S3, DynamoDB

[0238] AI model: OpenAI GPT-3

[0239] Image processing library: Pillow (Python)

[0240] Details of data processing and data calculation

[0241] Get user questions and answers

[0242] When a user asks the system a question, it is sent to the server via a smartphone or smart glasses, and the server then poses the question to a back-end AI model (GPT-3) to obtain an answer.

[0243] Creating and Saving Entries

[0244] The server generates an entry by adding a timestamp to the question and answer. This entry is saved as an image in an AWS S3 bucket with a unique name and user ID. The image is generated using the Python Pillow library.

[0245] Displaying images

[0246] When a user browses saved entries, the image entries are listed through an album-view style interface on their smartphone or smart glasses. When the user selects a specific entry, its details are displayed on the smart device.

[0247] Specific examples

[0248] For example, if a user asks "What is the battery capacity of this smartphone?" in a virtual store, the system will answer "This smartphone's battery capacity is 4000mAh." This question and answer are generated as an entry with a timestamp and saved in image format. The saved image is uniquely named using the user ID and timestamp and saved in cloud storage.

[0249] Prompt Sentence Examples

[0250] Q: What is the battery capacity of this smartphone?

[0251] A: This smartphone has a battery capacity of 4000mAh.

[0252] This allows users to visually manage product inquiry history and efficiently review information.

[0253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0254] Step 1:

[0255] The user inputs a question using a smartphone or smart glasses. The input question is sent from the device to the server. Input: User's question. Output: Question data sent to the server.

[0256] Step 2:

[0257] The server inputs the received question into a generative AI model (GPT-3) to generate an answer. Input: User's question. Output: Answer from the generative AI model.

[0258] Step 3:

[0259] When the server receives the question and answer, it adds a timestamp to them and creates an entry that includes the question, answer, and the date and time of recording. Input: Question, answer, timestamp. Output: Entry with timestamp.

[0260] Step 4:

[0261] The server converts the generated entries into an image format. Specifically, it draws each entry into an image using Python's Pillow library. Input: Entry with timestamp. Output: Rendered image.

[0262] Step 5:

[0263] The server assigns a unique file name to the drawn image using the user ID and timestamp and uploads it to an AWS S3 bucket. Input: drawn image, user ID, timestamp. Output: image saved in cloud storage.

[0264] Step 6:

[0265] When a user wants to view a saved entry, the device retrieves the image entry from the server in album view format and displays it. Input: User's view request. Output: Image entry displayed in album view format.

[0266] Step 7:

[0267] When the user selects a particular image entry, the terminal displays the details of that image entry. Input: User's image entry selection. Output: Display of details of the selected image entry.

[0268] Through these processing steps, the system visually manages user inquiries in the virtual store and enables efficient review of information.

[0269] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0270] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions, the system also saves the user's emotions in the entry and provides a function to draw an image that includes those emotions.

[0271] System Overview

[0272] This system records user questions and their answers as entries with timestamps, and adds the user's emotions recognized using an emotion engine. These entries are saved in image format, allowing users to view them later visually like an album. In addition, the images also include the user ID and emotion information, making it possible to understand each user's history at a glance.

[0273] Program Operation

[0274] 1. Getting user questions and answers

[0275] The user inputs a question to the system. The device receives the question from the user and sends it to an appropriate system (e.g., generative AI) to obtain an answer.

[0276] 2. Emotion Recognition by Emotion Engine

[0277] The device passes the question and answer to the emotion engine, which then identifies the user's emotion. The emotion engine infers the emotion from the content of the text and the user's input method, and returns the result to the device.

[0278] 3. Generate an entry containing a timestamp, question, answer, and recognized sentiment.

[0279] The device adds a timestamp to the captured question and answer and also generates an entry containing the recognized emotion, including the question, the answer, the recorded date and time, and the user's emotion.

[0280] 4. Saving entries as history

[0281] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[0282] 5. Drawing the entry content onto an image

[0283] The terminal uses a means to render the contents of the entry, i.e., the timestamp, question, answer, and emotion, as an image. Specifically, the terminal writes the text information and emotion information to an image file.

[0284] 6. Save the image

[0285] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[0286] 7. Displaying Images

[0287] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[0288] Specific examples

[0289] For example, if a user asks the following question:

[0290] Question 1: "What is AI?"

[0291] Answer 1: "AI stands for Artificial Intelligence."

[0292] Furthermore, let's say the emotion engine recognizes "interest" in the user's input, and generates the following entry:

[0293] plaintext

[0294] Timestamp: 2023-02-03 12:05:01

[0295] Question: What is AI?

[0296] Answer: AI stands for Artificial Intelligence.

[0297] Emotion: Interest

[0298] Next, the user asks another question:

[0299] Question 2: "What is Machine Learning?"

[0300] Answer 2: "Machine Learning is a subset of AI."

[0301] If the emotion engine recognizes "curiosity" from this input, it will generate an entry like this:

[0302] plaintext

[0303] Timestamp: 2023-02-03 12:12:01

[0304] Question: What is Machine Learning?

[0305] Answer: Machine Learning is a subset of AI.

[0306] Emotion: Curiosity

[0307] These questions and answers are each added with a timestamp and emotional information, and are created as entries. These entries are saved as images with filenames such as "user1234_20230203_120501.png" and "user1234_20230203_121201.png."

[0308] The server manages the storage of these images and provides a means for quickly displaying them when a user wants to refer to a specific entry. By reviewing these images later, users can visually confirm when, what questions were asked, and what emotional state they were in.

[0309] The above is an embodiment of the present invention, which allows management and reference of a user's investigation history to be performed visually and efficiently, and in particular, by including information on the user's emotions, a richer history management system is provided.

[0310] The processing flow will be explained below.

[0311] Step 1:

[0312] The user inputs a question to the system.

[0313] Step 2:

[0314] The device receives questions from the user and poses the questions to the generative AI or information provision system.

[0315] Step 3:

[0316] Generative AI and information provision systems generate answers to questions and return them to the device.

[0317] Step 4:

[0318] The device passes the question and answer to the emotion engine, which identifies the user's emotion. The emotion engine infers the user's emotion from the content of the text and the user's input method, and returns the result to the device.

[0319] Step 5:

[0320] The device retrieves the current date and time and formats it as a timestamp.

[0321] python

[0322] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[0323] Step 6:

[0324] The terminal generates an entry that includes a timestamp, a question, an answer, and a recognized emotion.

[0325] python

[0326] entry = {

[0327] 'timestamp': timestamp,

[0328] 'question': question,

[0329] 'answer': answer,

[0330] 'emotion': emotion

[0331] }

[0332] Step 7:

[0333] The terminal stores the generated entries in its history.

[0334] python

[0335] self.history.append(entry)

[0336] Step 8:

[0337] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[0338] python

[0339] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[0340] draw = ImageDraw.Draw(img)

[0341] font = ImageFont.load_default()

[0342] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[0343] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[0344] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[0345] draw.text((10, 70), f'Emotion: {emotion}', fill=(0, 0, 0), font=font)

[0346] Step 9:

[0347] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[0348] python

[0349] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[0350] Step 10:

[0351] Saves the image generated by the device to the specified path.

[0352] python

[0353] img.save(img_path)

[0354] Step 11:

[0355] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[0356] Step 12:

[0357] When a user wants to view the history, they send a request from their device to the server. The server then searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question asked, the answer, and their emotional state at the time.

[0358] Example 2

[0359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] Conventional history management systems only record questions and answers in text format, which means they cannot take into account the user's emotional state, making it difficult to fully grasp the meaning and value of the history. Furthermore, limited means for visually checking the history make it difficult for users to quickly and intuitively restore past questions and answers. Furthermore, there is a lack of ingenuity to allow users to understand each user's history at a glance.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0362] In this invention, the server includes means for receiving questions and answers from users and inputting them into a generative AI model to obtain answers, means for inputting the questions and answers into an emotion recognition engine to identify the user's emotion, means for generating entries including a timestamp, the question, the answer, and the recognized emotion, means for saving the generated entries as history, means for rendering the contents of the entries as images, and means for displaying the saved entries as images. This allows emotional information to be recorded in the user's question history, making it possible to check the history in a visual format.

[0363] 1. "User" means an entity that uses the system to enter questions and obtain answers.

[0364] 2. A "question" is text that a user enters into a system requesting information.

[0365] 3. "Answer" is the text response generated by a generative AI model in response to a question.

[0366] 4. "Generative AI model" refers to an algorithm or system that generates answers to user questions.

[0367] 5. "Emotion recognition engine" refers to an algorithm or system that analyzes questions and answers to identify a user's emotions.

[0368] 6. "Timestamp" is information that indicates the date and time a particular entry was created.

[0369] 7. "Entry" refers to a collection of data including a question, an answer, a timestamp, and a recognized emotion.

[0370] 8. "History" refers to a record of questions asked by a user in the past, their answers, and any accompanying information saved in chronological order.

[0371] 9. "Image rendering means" refers to processes and techniques for converting textual and emotional information into image files.

[0372] 10. "Entry saved as an image" means the content of an entry saved in image format.

[0373] 11. "User ID" means identification information that uniquely identifies each user.

[0374] 12. "Visual Verification" means any method or technology that allows a User to view an entry stored as an image via a Terminal.

[0375] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides a function to save the user's emotions in the entry and draw an image that includes those emotions.

[0376] System Overview

[0377] The system includes the following main means:

[0378] 1. User question input and answer acquisition method

[0379] 2. Emotion identification method using an emotion recognition engine

[0380] 3. Entry generation means including timestamp, question, answer, and recognized sentiment

[0381] 4. How to save entries as history

[0382] 5. How to draw the entry content into an image

[0383] 6. How to display entries saved as images

[0384] Hardware and software used

[0385] Generative AI models: For example, open-source language models or cloud-based language model APIs (e.g., GPT-3)

[0386] Emotion recognition engine: For example, natural language processing API (e.g., IBM Watson Natural Language Understanding)

[0387] Image generation software: For example, a Python image processing library (e.g., PIL - Python Imaging Library)

[0388] Database: For example, a relational database (e.g., MySQL, PostgreSQL)

[0389] Specific processing of the program

[0390] The server receives the question entered by the user and sends it to the generative AI model as a prompt. The generative AI model returns an answer to the question, and the answer is received by the server. An example of a prompt is "What is AI?"

[0391] The server then sends the question and answer to an emotion recognition engine to identify the user's emotion. The emotion recognition engine analyzes the text content and the user's input patterns to recognize the emotion, for example, "interest."

[0392] The server adds a timestamp to the question, answer, and recognized emotion, creates a single entry, associates it with the user ID, and stores it in a database.

[0393] The server renders the entry's content (timestamp, question, answer, sentiment) into an image. Specifically, it uses a Python image processing library to convert the text information into an image file. For example, consider the following entry:

[0394] plaintext

[0395] Timestamp: 2023-02-03 12:05:01

[0396] Question: What is AI?

[0397] Answer: AI stands for Artificial Intelligence.

[0398] Emotion: Interest

[0399] The image is saved on the server with a unique name that includes the user ID and a timestamp, for example, "user1234_20230203_120501.png."

[0400] When a user wants to view the history, the server retrieves the image file and displays it on the device, allowing the user to visually confirm the question and the emotional state at that time.

[0401] The above is an embodiment of the present invention. By using this system, it becomes possible to visually and efficiently manage and refer to a user's research history. In particular, by including user emotional information, it is possible to provide a richer history management system.

[0402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0403] Step 1:

[0404] The user inputs a question through the terminal. The terminal receives this question and temporarily stores the input in memory. This input is used in later processing. For example, the user inputs the question "What is AI?"

[0405] Step 2:

[0406] The device sends a question from the user to the server. The server passes the received question to the generative AI model as a prompt and asks for an answer. The server calls the API of the generative AI model (e.g., GPT-3), sends the question as a prompt, and receives the answer. The input at this time is the user's question, and the output is the answer from the generative AI model.

[0407] Step 3:

[0408] The server receives the answer from the generative AI model and sends it along with the question to the emotion recognition engine. The server calls the API of the emotion recognition engine (e.g., IBM Watson Natural Language Understanding) to analyze the question and answer and identify the user's emotion. The input is the question and answer, and the output is the recognized emotion.

[0409] Step 4:

[0410] The server generates an entry with the question, answer, recognized emotion, and the current date and time. The server obtains the system time and adds it as a timestamp to the entry. This entry contains the question, answer, timestamp, and user emotion. An example of a generated entry is as follows:

[0411] plaintext

[0412] Timestamp: 2023-02-03 12:05:01

[0413] Question: What is AI?

[0414] Answer: AI stands for Artificial Intelligence.

[0415] Emotion: Interest

[0416] Step 5:

[0417] The server saves the generated entries in a database. The database is configured to save entries for each user in chronological order. The input is the generated entries, and the output is the results saved in the database.

[0418] Step 6:

[0419] The server starts the process of rendering the entry content into an image. The server uses image generation software (e.g., PIL) to convert the entry's text information into an image file. The input is the entry, and the output is the image file.

[0420] Step 7:

[0421] The server saves the generated image file with a unique file name that includes the user ID and a timestamp. An example of a saved image file name is "user1234_20230203_120501.png." The input in this case is the image file, and the output is the path to the saved image file.

[0422] Step 8:

[0423] When a user wants to view the history, the device sends a request to the server. The server retrieves the image path of the corresponding entry from the database and sends it to the device. The device receives this and displays it visually to the user. The input in this case is the user request, and the output is the displayed image.

[0424] The above is the specific processing flow of this system, and the data processing and data calculation required at each step will be explained in detail.

[0425] (Application example 2)

[0426] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0427] Providing efficient and effective customer service through proper management of customer interaction history and emotions is a challenge in modern brick-and-mortar stores. Understanding the content of past customer inquiries and their emotions at the time is particularly important for improving customer satisfaction, but many stores lack the means to systematically record and manage this information. Furthermore, appropriate responses are sometimes required even for customers who are not comfortable with direct conversation. Therefore, it is necessary to provide a system that efficiently manages customer interaction history and emotional information and allows for easy visual reference.

[0428] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question and its answer from a user and generating an entry including a timestamp, the question, and the answer, means for saving the generated entry as a history, means for rendering the content of the entry as an image, means for displaying the entry saved as an image, means for identifying the user's emotion, means for adding the identified emotion to the entry, means for saving questions and answers related to specific products, and means for managing the history of customer service in a physical store. This not only makes it possible to effectively manage and visually reference the interaction history and emotion information with customers, but also enables more appropriate and effective customer service by understanding the content of each customer's past inquiries and their emotions at the time.

[0429] "Questions from users" are inquiries provided to the system by users.

[0430] An "answer" is a response provided by the system to a user's question.

[0431] The "timestamp" is information about the date and time when the entry was created.

[0432] An "entry" is a record that combines a question, an answer, a timestamp, and emotional information.

[0433] A "history" is a collection of entries created in the past.

[0434] The "means for rendering to an image" is a method for converting the contents of an entry into an image format that can be visually displayed.

[0435] The "means for displaying" is a method for displaying entries saved in image format so that the user can visually confirm them.

[0436] The "means for identifying user emotions" is a method for inferring emotions from the user's questions and answers.

[0437] The "means for adding identified emotions to an entry" is a method for adding emotion information to an entry.

[0438] The "means for storing questions and answers related to specific products" is a method for recording inquiries about specific products and the answers to those inquiries.

[0439] "Means for managing customer service history in physical stores" refers to a method for systematically storing and managing customer service history in physical stores.

[0440] "Means for adding a user ID" refers to a method for adding a unique ID to an entry to identify a user.

[0441] The system of the present invention provides a method for efficiently managing and visually referencing customer interaction history and emotional information in a physical store. This system is realized by the following steps.

[0442] The system receives questions and answers from users and generates an entry containing a timestamp, question, and answer. It then saves the generated entry as a history and renders the contents of the entry as an image. This rendered image can be used for future reference by customers and staff. It also incorporates a means for identifying emotions from the user's input and adding the identified emotions to the entry. It also saves questions and answers related to specific products, allowing for detailed recording and management of product purchase history and inquiry details. It also manages customer service history in physical stores, making it possible to visually check what inquiries a specific customer has made in the past.

[0443] Hardware and Software Use

[0444] This system is primarily composed of software for the server, user devices, and emotion engine. The server plays the primary role of storing and managing data. The user devices (smartphones and customer service robots) function as interfaces that receive input from users. The emotion engine uses natural language processing libraries (e.g., spaCy and NLTK). The Python Pillow library is used to draw text onto images.

[0445] 1. Ask questions and get answers:

[0446] The user device receives questions posed by customers in the physical store via an input interface, and sends them to the server-side generative AI model to obtain the appropriate answer.

[0447] 2. Identifying emotions:

[0448] The server analyzes the text content of the questions and answers and identifies the user's emotions using an emotion engine, which uses natural language processing techniques to infer emotions from the input text.

[0449] 3. Create and save the entry:

[0450] The server compiles the question, answer, timestamp, and identified emotion into a single entry and stores it in chronological order as a history.

[0451] 4. Image generation and display:

[0452] The server renders the entry contents as an image so that they can be visually confirmed. The Python Pillow library is used for rendering. The generated image is saved with a unique name using the user ID and a timestamp. When the user wants to refer to the history, this image is displayed on the user's device.

[0453] Examples of specific examples and prompts

[0454] For example, if a customer in a clothing store asks, "Can I wash this shirt?" and the robot replies, "Yes, it can be washed," the emotion engine will identify a "feeling of relief." The resulting entry would look like this:

[0455] entry:

[0456] plaintext

[0457] Timestamp: 2023-10-01 12:00:00

[0458] Question: Can this shirt be washed?

[0459] Answer: Yes, it is washable.

[0460] Emotion: A sense of security

[0461] An example prompt is:

[0462] plaintext

[0463] Prompt sentence to be passed to the emotion engine

[0464] Identify the sentiment in the following text:

[0465] Can this shirt be washed?

[0466] A: Yes, it is washable.

[0467] The information recorded in this way is stored visually as an image, providing a detailed customer history including specific questions, answers, and even emotions at the time, enabling more personalized service for each customer and contributing to improved customer satisfaction.

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] A user inputs a question into a terminal (smartphone or customer service robot) in a physical store. The terminal receives this input and saves it as text data. Specifically, when a user asks, "Can I wash this shirt?", the content is saved in text format on the terminal. Input: User's question (text data), Output: Question saved on the terminal (text data).

[0471] Step 2:

[0472] The device sends a question to the server, which uses the generative AI model to generate an appropriate answer. For example, the device sends the text "Can this shirt be washed?" to the server, and the server uses the generative AI model to obtain the answer "Yes, it can be washed." Input: User's question (text data), Output: Answer from the server (text data).

[0473] Step 3:

[0474] The server passes the captured question and answer text to the emotion engine, which then identifies emotions from this text. Specifically, the emotion engine analyzes the text "Can I wash this shirt?" and "Yes, it can be washed" and identifies the emotion "relief." Input: Question and answer text (text data), Output: Identified emotion (text data).

[0475] Step 4:

[0476] The server compiles the question, answer, timestamp, and identified sentiment into a single entry, for example, the following entry:

[0477] plaintext

[0478] Timestamp: 2023-10-01 12:00:00

[0479] Question: Can this shirt be washed?

[0480] Answer: Yes, it is washable.

[0481] Emotion: A sense of security

[0482] Input: Question, Answer, Timestamp, Sentiment (Text data), Output: Entry (Text data).

[0483] Step 5:

[0484] The server saves the generated entries. The entries are saved in chronological order as history and managed for future reference. For example, the entries are saved in a "history database." Input: entry (text data), output: saved history (database).

[0485] Step 6:

[0486] The server uses the Python Pillow library to render the entry contents into image format. Specifically, it reads the entry's text information, renders it into an image file, and saves it with a unique name. For example, an image file named "user1234_20231001_120000.png" is generated. Input: Entry (text data), Output: Rendered image (image file).

[0487] Step 7:

[0488] When the user wants to view the history, the server displays the saved image on the terminal. This display uses the display function of the terminal. An appropriate user interface is provided so that the user can check the image. For example, the terminal loads the image and displays it on the screen. Input: Saved image (image file), Output: Image displayed on the terminal (visual data).

[0489] In this way, a system is created that manages customer response history and emotional information in physical stores and allows for visual reference.

[0490] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0491] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0492] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0493] [Second embodiment]

[0494] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0495] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0496] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0497] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0498] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0499] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0500] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0501] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0502] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0503] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0504] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0505] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0506] The present invention relates to a system that receives user questions and their answers, saves them as a history, and allows users to visually review them later. How this system is implemented will be specifically described below.

[0507] System Overview

[0508] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to refer to information in a visually easy-to-recognize format, making it easier to remember.

[0509] Program Operation

[0510] 1. Getting user questions and answers

[0511] The user asks the system a question. The device receives the question and sends it to an appropriate system (e.g., generative AI) to get an answer.

[0512] 2. Creating an entry containing a timestamp, question, and answer

[0513] The device adds a timestamp to the acquired question and answer and records it as an entry, which includes the question, the answer, and the date and time it was recorded.

[0514] 3. Save entries as history

[0515] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[0516] 4. Drawing the entry content onto the image

[0517] A means is used to render the contents of the entry, i.e., the timestamp, question, and answer, as an image, specifically by writing the text information to an image file.

[0518] 5. Save the image

[0519] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[0520] 6. Displaying Images

[0521] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[0522] Specific examples

[0523] For example, if a user asks the following question:

[0524] Question 1: "What is AI?"

[0525] Answer 1: "AI stands for Artificial Intelligence."

[0526] Next, the user asks another question:

[0527] Question 2: "What is Machine Learning?"

[0528] Answer 2: "Machine Learning is a subset of AI."

[0529] In this case, each question and answer is assigned a timestamp and created as an entry. This entry is saved as an image with a file name such as "user1234_20230203_120501.png" or "user1234_20230203_121201.png." Users can later review these images to visually confirm when they asked what question and what answer they received.

[0530] The server manages the storage of these images and provides a means for the user to quickly display them when they want to refer to a particular entry, so that the user can easily recall the content of their research at that time.

[0531] The above is an embodiment of the present invention, which provides a system that allows users to visually and efficiently manage and refer to their investigation history.

[0532] The processing flow will be explained below.

[0533] Step 1:

[0534] The user inputs a question to the system.

[0535] Step 2:

[0536] The device receives questions entered by the user and poses the questions to generative AI or other information provision systems.

[0537] Step 3:

[0538] Generative AI and information provision systems generate answers to questions and return them to the device.

[0539] Step 4:

[0540] The device retrieves the current date and time and formats it as a timestamp.

[0541] python

[0542] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[0543] Step 5:

[0544] The terminal generates an entry that includes a timestamp, a question, and an answer.

[0545] python

[0546] entry = {

[0547] 'timestamp': timestamp,

[0548] 'question': question,

[0549] 'answer': answer

[0550] }

[0551] Step 6:

[0552] The terminal stores the generated entries in its history.

[0553] python

[0554] self.history.append(entry)

[0555] Step 7:

[0556] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[0557] python

[0558] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[0559] draw = ImageDraw.Draw(img)

[0560] font = ImageFont.load_default()

[0561] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[0562] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[0563] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[0564] Step 8:

[0565] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[0566] python

[0567] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[0568] Step 9:

[0569] Saves the image generated by the device to the specified path.

[0570] python

[0571] img.save(img_path)

[0572] Step 10:

[0573] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[0574] Step 11:

[0575] When a user wants to view the history, they send a request from their device to the server. The server searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question and answer they asked.

[0576] Example 1

[0577] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] In conventional question-answering systems, there are few ways for users to easily refer to past questions and their answers, making it difficult for users to visually confirm what questions they asked and what the answers were. In addition, storing history in text format lacks visual information, making it difficult to remember and recognize information.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0580] In this invention, the server includes means for receiving questions and answers from users and transmitting prompt sentences to the generative AI model to obtain answers, means for generating entries by adding timestamps to the obtained questions and answers, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for saving the rendered images, and means for displaying the saved images. This allows users to visually check past questions and answers, making it easier for them to remember and recognize information.

[0581] A "user" is an individual or entity that enters a question into the system and obtains an answer.

[0582] A "question" refers to a question or problem that a user inputs into the system.

[0583] "Answer" refers to the response that a generative AI model generates in response to a user's question.

[0584] A "generative AI model" is an artificial intelligence algorithm or system that receives a prompt and generates an answer based on its content.

[0585] A "prompt" is a command or query that a generative AI model uses to generate an appropriate answer.

[0586] A "timestamp" is a digital mark that indicates a specific date and time, and is information that clarifies the point in time when a question or answer was recorded.

[0587] An "entry" refers to a collection of data that includes a question, an answer, and a timestamp.

[0588] "History" refers to a collection of data in which entries created in the past are organized in chronological order.

[0589] "Rendering to image" refers to the act of converting the contents of an entry into a visually identifiable format and saving it as an image file.

[0590] "User identification information" refers to information for uniquely identifying a specific user, and includes, for example, a user ID.

[0591] "Server" means the central computer that manages and operates the entire system and processes requests from users.

[0592] The present invention relates to a system that records questions and answers posed by users along with timestamps, and allows them to be visually saved and referenced. How this system is implemented is described below.

[0593] System configuration

[0594] 1. Hardware configuration:

[0595] Server: The central computer that manages the entire question-answering system. It is responsible for communicating with the database and the generative AI model.

[0596] Terminal: A device (such as a computer or smartphone) that is directly operated by the user and provides functions for inputting questions, displaying them, and saving them.

[0597] 2. Software configuration:

[0598] Generative AI model: A generative AI model uses an artificial intelligence algorithm to generate answers to user questions. As a specific example, we use a model that uses natural language processing technology.

[0599] Pillow Library: Uses Python's Pillow library to draw entries onto the image.

[0600] Overview of program processing

[0601] 1. Getting user questions and answers

[0602] The user inputs a question into the device, which then sends the question to the generative AI model as a prompt. For example, the question "What is AI?" is converted into the prompt "Generate an answer to the user's question: 'What is AI?'"

[0603] The generative AI model receives the prompt sentence, generates a response such as "AI stands for Artificial Intelligence," and returns it to the device.

[0604] 2. Creating an entry containing a timestamp, question, and answer

[0605] The device adds a timestamp indicating the current date and time to the questions and answers it receives, and organizes each piece of information into an entry.

[0606] 3. Save entries as history

[0607] The device stores the generated entries in a database and organizes them in chronological order.

[0608] 4. Drawing the entry content onto the image

[0609] The terminal uses Python's Pillow library to draw the entry contents (timestamp, question, answer) as an image.

[0610] 5. Save the image

[0611] The drawn image is given a unique name using the user's identification information and a timestamp and saved as a file, for example, in the format "user1234_20231003_120000.png".

[0612] 6. Displaying Images

[0613] If the user wants to view the history, the device will display the saved image, allowing the user to visually review the history of questions and answers.

[0614] Specific operation example

[0615] For example, a user asks the following question:

[0616] Question 1: "What is AI?"

[0617] Answer 1: "AI stands for Artificial Intelligence."

[0618] Prompt: "Generate an answer to the user's question: 'What is AI?'"

[0619] Next, the user asks another question:

[0620] Question 2: "What is Machine Learning?"

[0621] Answer 2: "Machine Learning is a subset of AI."

[0622] Prompt: "Generate an answer to the user's question: 'What is Machine Learning?'"

[0623] These questions and answers are each time-stamped and created as entries. These entries are saved in image format with file names such as "user1234_20231003_120000.png" and "user1234_20231003_121000.png." By referring to these images, users can visually confirm when they asked what questions and what answers they received.

[0624] As described above, the present invention provides a specific means for visually and efficiently managing and referencing a user's question and answer history.

[0625] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0626] Step 1:

[0627] The user inputs a question via a terminal.

[0628] Specific behavior:

[0629] The user enters "What is AI?" into the input field on the terminal and presses the send button.

[0630] input:

[0631] A user asks, "What is AI?"

[0632] output:

[0633] The terminal acquires the user's question as text data.

[0634] Step 2:

[0635] The device generates a prompt sentence to send the acquired question to the generative AI model.

[0636] Specific behavior:

[0637] The device converts the question text "What is AI?" into the prompt "Generate an answer to the user's question: 'What is AI?'"

[0638] input:

[0639] The question text obtained from the user: "What is AI?"

[0640] output:

[0641] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[0642] Step 3:

[0643] The device sends the generated prompt to the generative AI model and obtains the answer.

[0644] Specific behavior:

[0645] The device sends the prompt as a request to the generative AI model's API, which then generates the answer "AI stands for Artificial Intelligence." based on the prompt and returns it to the device.

[0646] input:

[0647] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[0648] output:

[0649] The answer returned by the generative AI model: "AI stands for Artificial Intelligence."

[0650] Step 4:

[0651] The terminal adds a timestamp to the acquired question and answer and creates an entry.

[0652] Specific behavior:

[0653] The device obtains the current date and time and creates entry data containing the question "What is AI?", the answer "AI stands for Artificial Intelligence.", and the timestamp "2023 / 10 / 03 12:00:00".

[0654] input:

[0655] Question: "What is AI?" Answer: "AI stands for Artificial Intelligence." Current date and time: "2023 / 10 / 03 12:00:00".

[0656] output:

[0657] Generated entry data (question, answer, timestamp).

[0658] Step 5:

[0659] The terminal stores the created entries in a database as history.

[0660] Specific behavior:

[0661] Insert the entry data (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Timestamp: "2023 / 10 / 03 12:00:00") into the database.

[0662] input:

[0663] The generated entry data.

[0664] output:

[0665] The history of entries in the database, ordered chronologically.

[0666] Step 6:

[0667] The terminal will render the contents of the entry into an image and save it as a file.

[0668] Specific behavior:

[0669] The device uses Python's Pillow library to convert the entry content (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Time Stamp: "2023 / 10 / 03 12:00:00") into an image and saves it with the file name "user1234_20231003_120000.png".

[0670] input:

[0671] Entry data (question, answer, timestamp).

[0672] output:

[0673] The saved image file is "user1234_20231003_120000.png".

[0674] Step 7:

[0675] If the user wants to view the history, the device displays the saved images.

[0676] Specific behavior:

[0677] When the user performs an operation to open the history display screen, the device loads the corresponding image file "user1234_20231003_120000.png" and displays it on the screen.

[0678] input:

[0679] User reference request, image file "user1234_20231003_120000.png".

[0680] output:

[0681] Image content (question, answer, timestamp) displayed on the device screen.

[0682] (Application example 1)

[0683] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0684] In modern virtual stores, there is a problem in that users have few means to inquire about product information and then visually confirm that information later. Specifically, when a user makes multiple inquiries, it is difficult to centrally manage the details of those inquiries and review them visually. This leads to problems such as users having to ask the same questions repeatedly or being unable to efficiently utilize the information they obtain.

[0685] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0686] In this invention, the server includes means for receiving questions and answers from users and generating entries including a timestamp, the question, and the answer, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for displaying the entries saved as images, means for recording product information inquiries and their answers in a virtual store, means for uniquely naming the entries using a user ID and a timestamp and saving them in cloud storage, and means for displaying a list of image entries in an album view format, thereby enabling users to visually manage their product inquiry history and efficiently review the information.

[0687] A "user question" is a request for information posed by a user to the system.

[0688] An "answer" is information provided by the system in response to an inquiry.

[0689] A "timestamp" is information that indicates the date and time when the entry was created.

[0690] An "entry" is a record that includes a timestamp, question, and answer data.

[0691] A "history" is a collection of entries created in the past, stored in chronological order.

[0692] An "image" is digital data that visually depicts the contents of an entry.

[0693] A "virtual store" is an online sales platform that offers products and services over the Internet.

[0694] A "user ID" is an identifier that uniquely identifies an individual user.

[0695] "Cloud storage" is a service that stores and makes accessible data on remote servers over the Internet.

[0696] The "album view format" is a user interface format for visually displaying a list of image entries.

[0697] "Uniquely named" means adding a unique name to each entry or file so that the same name is never generated twice.

[0698] The present invention is implemented as a system that allows users to manage and visually review questions and answers posed by users in a virtual store. This system effectively manages questions posed by users to search for product information and their answers, and displays them in an album view format.

[0699] System Overview

[0700] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to visually manage their product inquiry history and easily review the information they have obtained later.

[0701] Hardware and Software Configuration

[0702] This system uses the following hardware and software:

[0703] Hardware: Smartphones, smart glasses

[0704] software:

[0705] Frontend: React Native

[0706] Backend: Python, Flask

[0707] Database: AWS S3, DynamoDB

[0708] AI model: OpenAI GPT-3

[0709] Image processing library: Pillow (Python)

[0710] Details of data processing and data calculation

[0711] Get user questions and answers

[0712] When a user asks the system a question, it is sent to the server via a smartphone or smart glasses, and the server then poses the question to a back-end AI model (GPT-3) to obtain an answer.

[0713] Creating and Saving Entries

[0714] The server generates an entry by adding a timestamp to the question and answer. This entry is saved as an image in an AWS S3 bucket with a unique name and user ID. The image is generated using the Python Pillow library.

[0715] Displaying images

[0716] When a user browses saved entries, the image entries are listed through an album-view style interface on their smartphone or smart glasses. When the user selects a specific entry, its details are displayed on the smart device.

[0717] Specific examples

[0718] For example, if a user asks "What is the battery capacity of this smartphone?" in a virtual store, the system will answer "This smartphone's battery capacity is 4000mAh." This question and answer are generated as an entry with a timestamp and saved in image format. The saved image is uniquely named using the user ID and timestamp and saved in cloud storage.

[0719] Prompt Sentence Examples

[0720] Q: What is the battery capacity of this smartphone?

[0721] A: This smartphone has a battery capacity of 4000mAh.

[0722] This allows users to visually manage product inquiry history and efficiently review information.

[0723] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0724] Step 1:

[0725] The user inputs a question using a smartphone or smart glasses. The input question is sent from the device to the server. Input: User's question. Output: Question data sent to the server.

[0726] Step 2:

[0727] The server inputs the received question into a generative AI model (GPT-3) to generate an answer. Input: User's question. Output: Answer from the generative AI model.

[0728] Step 3:

[0729] When the server receives the question and answer, it adds a timestamp to them and creates an entry that includes the question, answer, and the date and time of recording. Input: Question, answer, timestamp. Output: Entry with timestamp.

[0730] Step 4:

[0731] The server converts the generated entries into an image format. Specifically, it draws each entry into an image using Python's Pillow library. Input: Entry with timestamp. Output: Rendered image.

[0732] Step 5:

[0733] The server assigns a unique file name to the drawn image using the user ID and timestamp and uploads it to an AWS S3 bucket. Input: drawn image, user ID, timestamp. Output: image saved in cloud storage.

[0734] Step 6:

[0735] When a user wants to view a saved entry, the device retrieves the image entry from the server in album view format and displays it. Input: User's view request. Output: Image entry displayed in album view format.

[0736] Step 7:

[0737] When the user selects a particular image entry, the terminal displays the details of that image entry. Input: User's image entry selection. Output: Display of details of the selected image entry.

[0738] Through these processing steps, the system visually manages user inquiries in the virtual store and enables efficient review of information.

[0739] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0740] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions, the system also saves the user's emotions in the entry and provides a function to draw an image that includes those emotions.

[0741] System Overview

[0742] This system records user questions and their answers as entries with timestamps, and adds the user's emotions recognized using an emotion engine. These entries are saved in image format, allowing users to view them later visually like an album. In addition, the images also include the user ID and emotion information, making it possible to understand each user's history at a glance.

[0743] Program Operation

[0744] 1. Getting user questions and answers

[0745] The user inputs a question to the system. The device receives the question from the user and sends it to an appropriate system (e.g., generative AI) to obtain an answer.

[0746] 2. Emotion Recognition by Emotion Engine

[0747] The device passes the question and answer to the emotion engine, which then identifies the user's emotion. The emotion engine infers the emotion from the content of the text and the user's input method, and returns the result to the device.

[0748] 3. Generate an entry containing a timestamp, question, answer, and recognized sentiment.

[0749] The device adds a timestamp to the captured question and answer and also generates an entry containing the recognized emotion, including the question, the answer, the recorded date and time, and the user's emotion.

[0750] 4. Saving entries as history

[0751] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[0752] 5. Drawing the entry content onto an image

[0753] The terminal uses a means to render the contents of the entry, i.e., the timestamp, question, answer, and emotion, as an image. Specifically, the terminal writes the text information and emotion information to an image file.

[0754] 6. Save the image

[0755] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[0756] 7. Displaying Images

[0757] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[0758] Specific examples

[0759] For example, if a user asks the following question:

[0760] Question 1: "What is AI?"

[0761] Answer 1: "AI stands for Artificial Intelligence."

[0762] Furthermore, let's say the emotion engine recognizes "interest" in the user's input, and generates the following entry:

[0763] plaintext

[0764] Timestamp: 2023-02-03 12:05:01

[0765] Question: What is AI?

[0766] Answer: AI stands for Artificial Intelligence.

[0767] Emotion: Interest

[0768] Next, the user asks another question:

[0769] Question 2: "What is Machine Learning?"

[0770] Answer 2: "Machine Learning is a subset of AI."

[0771] If the emotion engine recognizes "curiosity" from this input, it will generate an entry like this:

[0772] plaintext

[0773] Timestamp: 2023-02-03 12:12:01

[0774] Question: What is Machine Learning?

[0775] Answer: Machine Learning is a subset of AI.

[0776] Emotion: Curiosity

[0777] These questions and answers are each added with a timestamp and emotional information, and are created as entries. These entries are saved as images with filenames such as "user1234_20230203_120501.png" and "user1234_20230203_121201.png."

[0778] The server manages the storage of these images and provides a means for quickly displaying them when a user wants to refer to a specific entry. By reviewing these images later, users can visually confirm when, what questions were asked, and what emotional state they were in.

[0779] The above is an embodiment of the present invention, which allows management and reference of a user's investigation history to be performed visually and efficiently, and in particular, by including information on the user's emotions, a richer history management system is provided.

[0780] The processing flow will be explained below.

[0781] Step 1:

[0782] The user inputs a question to the system.

[0783] Step 2:

[0784] The device receives questions from the user and poses the questions to the generative AI or information provision system.

[0785] Step 3:

[0786] Generative AI and information provision systems generate answers to questions and return them to the device.

[0787] Step 4:

[0788] The device passes the question and answer to the emotion engine, which identifies the user's emotion. The emotion engine infers the user's emotion from the content of the text and the user's input method, and returns the result to the device.

[0789] Step 5:

[0790] The device retrieves the current date and time and formats it as a timestamp.

[0791] python

[0792] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[0793] Step 6:

[0794] The terminal generates an entry that includes a timestamp, a question, an answer, and a recognized emotion.

[0795] python

[0796] entry = {

[0797] 'timestamp': timestamp,

[0798] 'question': question,

[0799] 'answer': answer,

[0800] 'emotion': emotion

[0801] }

[0802] Step 7:

[0803] The terminal stores the generated entries in its history.

[0804] python

[0805] self.history.append(entry)

[0806] Step 8:

[0807] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[0808] python

[0809] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[0810] draw = ImageDraw.Draw(img)

[0811] font = ImageFont.load_default()

[0812] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[0813] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[0814] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[0815] draw.text((10, 70), f'Emotion: {emotion}', fill=(0, 0, 0), font=font)

[0816] Step 9:

[0817] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[0818] python

[0819] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[0820] Step 10:

[0821] Saves the image generated by the device to the specified path.

[0822] python

[0823] img.save(img_path)

[0824] Step 11:

[0825] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[0826] Step 12:

[0827] When a user wants to view the history, they send a request from their device to the server. The server then searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question asked, the answer, and their emotional state at the time.

[0828] Example 2

[0829] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0830] Conventional history management systems only record questions and answers in text format, which means they cannot take into account the user's emotional state, making it difficult to fully grasp the meaning and value of the history. Furthermore, limited means for visually checking the history make it difficult for users to quickly and intuitively restore past questions and answers. Furthermore, there is a lack of ingenuity to allow users to understand each user's history at a glance.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0832] In this invention, the server includes means for receiving questions and answers from users and inputting them into a generative AI model to obtain answers, means for inputting the questions and answers into an emotion recognition engine to identify the user's emotion, means for generating entries including a timestamp, the question, the answer, and the recognized emotion, means for saving the generated entries as history, means for rendering the contents of the entries as images, and means for displaying the saved entries as images. This allows emotional information to be recorded in the user's question history, making it possible to check the history in a visual format.

[0833] 1. "User" means an entity that uses the system to enter questions and obtain answers.

[0834] 2. A "question" is text that a user enters into a system requesting information.

[0835] 3. "Answer" is the text response generated by a generative AI model in response to a question.

[0836] 4. "Generative AI model" refers to an algorithm or system that generates answers to user questions.

[0837] 5. "Emotion recognition engine" refers to an algorithm or system that analyzes questions and answers to identify a user's emotions.

[0838] 6. "Timestamp" is information that indicates the date and time a particular entry was created.

[0839] 7. "Entry" refers to a collection of data including a question, an answer, a timestamp, and a recognized emotion.

[0840] 8. "History" refers to a record of questions asked by a user in the past, their answers, and any accompanying information saved in chronological order.

[0841] 9. "Image rendering means" refers to processes and techniques for converting textual and emotional information into image files.

[0842] 10. "Entry saved as an image" means the content of an entry saved in image format.

[0843] 11. "User ID" means identification information that uniquely identifies each user.

[0844] 12. "Visual Verification" means any method or technology that allows a User to view an entry stored as an image via a Terminal.

[0845] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides a function to save the user's emotions in the entry and draw an image that includes those emotions.

[0846] System Overview

[0847] The system includes the following main means:

[0848] 1. User question input and answer acquisition method

[0849] 2. Emotion identification method using an emotion recognition engine

[0850] 3. Entry generation means including timestamp, question, answer, and recognized sentiment

[0851] 4. How to save entries as history

[0852] 5. How to draw the entry content into an image

[0853] 6. How to display entries saved as images

[0854] Hardware and software used

[0855] Generative AI models: For example, open-source language models or cloud-based language model APIs (e.g., GPT-3)

[0856] Emotion recognition engine: For example, natural language processing API (e.g., IBM Watson Natural Language Understanding)

[0857] Image generation software: For example, a Python image processing library (e.g., PIL - Python Imaging Library)

[0858] Database: For example, a relational database (e.g., MySQL, PostgreSQL)

[0859] Specific processing of the program

[0860] The server receives the question entered by the user and sends it to the generative AI model as a prompt. The generative AI model returns an answer to the question, and the answer is received by the server. An example of a prompt is "What is AI?"

[0861] The server then sends the question and answer to an emotion recognition engine to identify the user's emotion. The emotion recognition engine analyzes the text content and the user's input patterns to recognize the emotion, for example, "interest."

[0862] The server adds a timestamp to the question, answer, and recognized emotion, creates a single entry, associates it with the user ID, and stores it in a database.

[0863] The server renders the entry's content (timestamp, question, answer, sentiment) into an image. Specifically, it uses a Python image processing library to convert the text information into an image file. For example, consider the following entry:

[0864] plaintext

[0865] Timestamp: 2023-02-03 12:05:01

[0866] Question: What is AI?

[0867] Answer: AI stands for Artificial Intelligence.

[0868] Emotion: Interest

[0869] The image is saved on the server with a unique name that includes the user ID and a timestamp, for example, "user1234_20230203_120501.png."

[0870] When a user wants to view the history, the server retrieves the image file and displays it on the device, allowing the user to visually confirm the question and the emotional state at that time.

[0871] The above is an embodiment of the present invention. By using this system, it becomes possible to visually and efficiently manage and refer to a user's research history. In particular, by including user emotional information, it is possible to provide a richer history management system.

[0872] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0873] Step 1:

[0874] The user inputs a question through the terminal. The terminal receives this question and temporarily stores the input in memory. This input is used in later processing. For example, the user inputs the question "What is AI?"

[0875] Step 2:

[0876] The device sends a question from the user to the server. The server passes the received question to the generative AI model as a prompt and asks for an answer. The server calls the API of the generative AI model (e.g., GPT-3), sends the question as a prompt, and receives the answer. The input at this time is the user's question, and the output is the answer from the generative AI model.

[0877] Step 3:

[0878] The server receives the answer from the generative AI model and sends it along with the question to the emotion recognition engine. The server calls the API of the emotion recognition engine (e.g., IBM Watson Natural Language Understanding) to analyze the question and answer and identify the user's emotion. The input is the question and answer, and the output is the recognized emotion.

[0879] Step 4:

[0880] The server generates an entry with the question, answer, recognized emotion, and the current date and time. The server obtains the system time and adds it as a timestamp to the entry. This entry contains the question, answer, timestamp, and user emotion. An example of a generated entry is as follows:

[0881] plaintext

[0882] Timestamp: 2023-02-03 12:05:01

[0883] Question: What is AI?

[0884] Answer: AI stands for Artificial Intelligence.

[0885] Emotion: Interest

[0886] Step 5:

[0887] The server saves the generated entries in a database. The database is configured to save entries for each user in chronological order. The input is the generated entries, and the output is the results saved in the database.

[0888] Step 6:

[0889] The server starts the process of rendering the entry content into an image. The server uses image generation software (e.g., PIL) to convert the entry's text information into an image file. The input is the entry, and the output is the image file.

[0890] Step 7:

[0891] The server saves the generated image file with a unique file name that includes the user ID and a timestamp. An example of a saved image file name is "user1234_20230203_120501.png." The input in this case is the image file, and the output is the path to the saved image file.

[0892] Step 8:

[0893] When a user wants to view the history, the device sends a request to the server. The server retrieves the image path of the corresponding entry from the database and sends it to the device. The device receives this and displays it visually to the user. The input in this case is the user request, and the output is the displayed image.

[0894] The above is the specific processing flow of this system, and the data processing and data calculation required at each step will be explained in detail.

[0895] (Application example 2)

[0896] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0897] Providing efficient and effective customer service through proper management of customer interaction history and emotions is a challenge in modern brick-and-mortar stores. Understanding the content of past customer inquiries and their emotions at the time is particularly important for improving customer satisfaction, but many stores lack the means to systematically record and manage this information. Furthermore, appropriate responses are sometimes required even for customers who are not comfortable with direct conversation. Therefore, it is necessary to provide a system that efficiently manages customer interaction history and emotional information and allows for easy visual reference.

[0898] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question and its answer from a user and generating an entry including a timestamp, the question, and the answer, means for saving the generated entry as a history, means for rendering the content of the entry as an image, means for displaying the entry saved as an image, means for identifying the user's emotion, means for adding the identified emotion to the entry, means for saving questions and answers related to specific products, and means for managing the history of customer service in a physical store. This not only makes it possible to effectively manage and visually reference the interaction history and emotion information with customers, but also enables more appropriate and effective customer service by understanding the content of each customer's past inquiries and their emotions at the time.

[0899] "Questions from users" are inquiries provided to the system by users.

[0900] An "answer" is a response provided by the system to a user's question.

[0901] The "timestamp" is information about the date and time when the entry was created.

[0902] An "entry" is a record that combines a question, an answer, a timestamp, and emotional information.

[0903] A "history" is a collection of entries created in the past.

[0904] The "means for rendering to an image" is a method for converting the contents of an entry into an image format that can be visually displayed.

[0905] The "means for displaying" is a method for displaying entries saved in image format so that the user can visually confirm them.

[0906] The "means for identifying user emotions" is a method for inferring emotions from the user's questions and answers.

[0907] The "means for adding identified emotions to an entry" is a method for adding emotion information to an entry.

[0908] The "means for storing questions and answers related to specific products" is a method for recording inquiries about specific products and the answers to those inquiries.

[0909] "Means for managing customer service history in physical stores" refers to a method for systematically storing and managing customer service history in physical stores.

[0910] "Means for adding a user ID" refers to a method for adding a unique ID to an entry to identify a user.

[0911] The system of the present invention provides a method for efficiently managing and visually referencing customer interaction history and emotional information in a physical store. This system is realized by the following steps.

[0912] The system receives questions and answers from users and generates an entry containing a timestamp, question, and answer. It then saves the generated entry as a history and renders the contents of the entry as an image. This rendered image can be used for future reference by customers and staff. It also incorporates a means for identifying emotions from the user's input and adding the identified emotions to the entry. It also saves questions and answers related to specific products, allowing for detailed recording and management of product purchase history and inquiry details. It also manages customer service history in physical stores, making it possible to visually check what inquiries a specific customer has made in the past.

[0913] Hardware and Software Use

[0914] This system is primarily composed of software for the server, user devices, and emotion engine. The server plays the primary role of storing and managing data. The user devices (smartphones and customer service robots) function as interfaces that receive input from users. The emotion engine uses natural language processing libraries (e.g., spaCy and NLTK). The Python Pillow library is used to draw text onto images.

[0915] 1. Ask questions and get answers:

[0916] The user device receives questions posed by customers in the physical store via an input interface, and sends them to the server-side generative AI model to obtain the appropriate answer.

[0917] 2. Identifying emotions:

[0918] The server analyzes the text content of the questions and answers and identifies the user's emotions using an emotion engine, which uses natural language processing techniques to infer emotions from the input text.

[0919] 3. Create and save the entry:

[0920] The server compiles the question, answer, timestamp, and identified emotion into a single entry and stores it in chronological order as a history.

[0921] 4. Image generation and display:

[0922] The server renders the entry contents as an image so that they can be visually confirmed. The Python Pillow library is used for rendering. The generated image is saved with a unique name using the user ID and a timestamp. When the user wants to refer to the history, this image is displayed on the user's device.

[0923] Examples of specific examples and prompts

[0924] For example, if a customer in a clothing store asks, "Can I wash this shirt?" and the robot replies, "Yes, it can be washed," the emotion engine will identify a "feeling of relief." The resulting entry would look like this:

[0925] entry:

[0926] plaintext

[0927] Timestamp: 2023-10-01 12:00:00

[0928] Question: Can this shirt be washed?

[0929] Answer: Yes, it is washable.

[0930] Emotion: A sense of security

[0931] An example prompt is:

[0932] plaintext

[0933] Prompt sentence to be passed to the emotion engine

[0934] Identify the sentiment in the following text:

[0935] Can this shirt be washed?

[0936] A: Yes, it is washable.

[0937] The information recorded in this way is stored visually as an image, providing a detailed customer history including specific questions, answers, and even emotions at the time, enabling more personalized service for each customer and contributing to improved customer satisfaction.

[0938] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0939] Step 1:

[0940] A user inputs a question into a terminal (smartphone or customer service robot) in a physical store. The terminal receives this input and saves it as text data. Specifically, when a user asks, "Can I wash this shirt?", the content is saved in text format on the terminal. Input: User's question (text data), Output: Question saved on the terminal (text data).

[0941] Step 2:

[0942] The device sends a question to the server, which uses the generative AI model to generate an appropriate answer. For example, the device sends the text "Can this shirt be washed?" to the server, and the server uses the generative AI model to obtain the answer "Yes, it can be washed." Input: User's question (text data), Output: Answer from the server (text data).

[0943] Step 3:

[0944] The server passes the captured question and answer text to the emotion engine, which then identifies emotions from this text. Specifically, the emotion engine analyzes the text "Can I wash this shirt?" and "Yes, it can be washed" and identifies the emotion "relief." Input: Question and answer text (text data), Output: Identified emotion (text data).

[0945] Step 4:

[0946] The server compiles the question, answer, timestamp, and identified sentiment into a single entry, for example, the following entry:

[0947] plaintext

[0948] Timestamp: 2023-10-01 12:00:00

[0949] Question: Can this shirt be washed?

[0950] Answer: Yes, it is washable.

[0951] Emotion: A sense of security

[0952] Input: Question, Answer, Timestamp, Sentiment (Text data), Output: Entry (Text data).

[0953] Step 5:

[0954] The server saves the generated entries. The entries are saved in chronological order as history and managed for future reference. For example, the entries are saved in a "history database." Input: entry (text data), output: saved history (database).

[0955] Step 6:

[0956] The server uses the Python Pillow library to render the entry contents into image format. Specifically, it reads the entry's text information, renders it into an image file, and saves it with a unique name. For example, an image file named "user1234_20231001_120000.png" is generated. Input: Entry (text data), Output: Rendered image (image file).

[0957] Step 7:

[0958] When the user wants to view the history, the server displays the saved image on the terminal. This display uses the display function of the terminal. An appropriate user interface is provided so that the user can check the image. For example, the terminal loads the image and displays it on the screen. Input: Saved image (image file), Output: Image displayed on the terminal (visual data).

[0959] In this way, a system is created that manages customer response history and emotional information in physical stores and allows for visual reference.

[0960] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0961] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0962] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0963] [Third embodiment]

[0964] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0965] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0966] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0967] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0968] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0969] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0970] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0971] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0972] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0973] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0974] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0975] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0976] The present invention relates to a system that receives user questions and their answers, saves them as a history, and allows users to visually review them later. How this system is implemented will be specifically described below.

[0977] System Overview

[0978] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to refer to information in a visually easy-to-recognize format, making it easier to remember.

[0979] Program Operation

[0980] 1. Getting user questions and answers

[0981] The user asks the system a question. The device receives the question and sends it to an appropriate system (e.g., generative AI) to get an answer.

[0982] 2. Creating an entry containing a timestamp, question, and answer

[0983] The device adds a timestamp to the acquired question and answer and records it as an entry, which includes the question, the answer, and the date and time it was recorded.

[0984] 3. Save entries as history

[0985] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[0986] 4. Drawing the entry content onto the image

[0987] A means is used to render the contents of the entry, i.e., the timestamp, question, and answer, as an image, specifically by writing the text information to an image file.

[0988] 5. Save the image

[0989] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[0990] 6. Displaying Images

[0991] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[0992] Specific examples

[0993] For example, if a user asks the following question:

[0994] Question 1: "What is AI?"

[0995] Answer 1: "AI stands for Artificial Intelligence."

[0996] Next, the user asks another question:

[0997] Question 2: "What is Machine Learning?"

[0998] Answer 2: "Machine Learning is a subset of AI."

[0999] In this case, each question and answer is assigned a timestamp and created as an entry. This entry is saved as an image with a file name such as "user1234_20230203_120501.png" or "user1234_20230203_121201.png." Users can later review these images to visually confirm when they asked what question and what answer they received.

[1000] The server manages the storage of these images and provides a means for the user to quickly display them when they want to refer to a particular entry, so that the user can easily recall the content of their research at that time.

[1001] The above is an embodiment of the present invention, which provides a system that allows users to visually and efficiently manage and refer to their investigation history.

[1002] The processing flow will be explained below.

[1003] Step 1:

[1004] The user inputs a question to the system.

[1005] Step 2:

[1006] The device receives questions entered by the user and poses the questions to generative AI or other information provision systems.

[1007] Step 3:

[1008] Generative AI and information provision systems generate answers to questions and return them to the device.

[1009] Step 4:

[1010] The device retrieves the current date and time and formats it as a timestamp.

[1011] python

[1012] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[1013] Step 5:

[1014] The terminal generates an entry that includes a timestamp, a question, and an answer.

[1015] python

[1016] entry = {

[1017] 'timestamp': timestamp,

[1018] 'question': question,

[1019] 'answer': answer

[1020] }

[1021] Step 6:

[1022] The terminal stores the generated entries in its history.

[1023] python

[1024] self.history.append(entry)

[1025] Step 7:

[1026] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[1027] python

[1028] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[1029] draw = ImageDraw.Draw(img)

[1030] font = ImageFont.load_default()

[1031] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[1032] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[1033] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[1034] Step 8:

[1035] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[1036] python

[1037] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[1038] Step 9:

[1039] Saves the image generated by the device to the specified path.

[1040] python

[1041] img.save(img_path)

[1042] Step 10:

[1043] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[1044] Step 11:

[1045] When a user wants to view the history, they send a request from their device to the server. The server searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question and answer they asked.

[1046] Example 1

[1047] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1048] In conventional question-answering systems, there are few ways for users to easily refer to past questions and their answers, making it difficult for users to visually confirm what questions they asked and what the answers were. In addition, storing history in text format lacks visual information, making it difficult to remember and recognize information.

[1049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1050] In this invention, the server includes means for receiving questions and answers from users and transmitting prompt sentences to the generative AI model to obtain answers, means for generating entries by adding timestamps to the obtained questions and answers, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for saving the rendered images, and means for displaying the saved images. This allows users to visually check past questions and answers, making it easier for them to remember and recognize information.

[1051] A "user" is an individual or entity that enters a question into the system and obtains an answer.

[1052] A "question" refers to a question or problem that a user inputs into the system.

[1053] "Answer" refers to the response that a generative AI model generates in response to a user's question.

[1054] A "generative AI model" is an artificial intelligence algorithm or system that receives a prompt and generates an answer based on its content.

[1055] A "prompt" is a command or query that a generative AI model uses to generate an appropriate answer.

[1056] A "timestamp" is a digital mark that indicates a specific date and time, and is information that clarifies the point in time when a question or answer was recorded.

[1057] An "entry" refers to a collection of data that includes a question, an answer, and a timestamp.

[1058] "History" refers to a collection of data in which entries created in the past are organized in chronological order.

[1059] "Rendering to image" refers to the act of converting the contents of an entry into a visually identifiable format and saving it as an image file.

[1060] "User identification information" refers to information for uniquely identifying a specific user, and includes, for example, a user ID.

[1061] "Server" means the central computer that manages and operates the entire system and processes requests from users.

[1062] The present invention relates to a system that records questions and answers posed by users along with timestamps, and allows them to be visually saved and referenced. How this system is implemented is described below.

[1063] System configuration

[1064] 1. Hardware configuration:

[1065] Server: The central computer that manages the entire question-answering system. It is responsible for communicating with the database and the generative AI model.

[1066] Terminal: A device (such as a computer or smartphone) that is directly operated by the user and provides functions for inputting questions, displaying them, and saving them.

[1067] 2. Software configuration:

[1068] Generative AI model: A generative AI model uses an artificial intelligence algorithm to generate answers to user questions. As a specific example, we use a model that uses natural language processing technology.

[1069] Pillow Library: Uses Python's Pillow library to draw entries onto the image.

[1070] Overview of program processing

[1071] 1. Getting user questions and answers

[1072] The user inputs a question into the device, which then sends the question to the generative AI model as a prompt. For example, the question "What is AI?" is converted into the prompt "Generate an answer to the user's question: 'What is AI?'"

[1073] The generative AI model receives the prompt sentence, generates a response such as "AI stands for Artificial Intelligence," and returns it to the device.

[1074] 2. Creating an entry containing a timestamp, question, and answer

[1075] The device adds a timestamp indicating the current date and time to the questions and answers it receives, and organizes each piece of information into an entry.

[1076] 3. Save entries as history

[1077] The device stores the generated entries in a database and organizes them in chronological order.

[1078] 4. Drawing the entry content onto the image

[1079] The terminal uses Python's Pillow library to draw the entry contents (timestamp, question, answer) as an image.

[1080] 5. Save the image

[1081] The drawn image is given a unique name using the user's identification information and a timestamp and saved as a file, for example, in the format "user1234_20231003_120000.png".

[1082] 6. Displaying Images

[1083] If the user wants to view the history, the device will display the saved image, allowing the user to visually review the history of questions and answers.

[1084] Specific operation example

[1085] For example, a user asks the following question:

[1086] Question 1: "What is AI?"

[1087] Answer 1: "AI stands for Artificial Intelligence."

[1088] Prompt: "Generate an answer to the user's question: 'What is AI?'"

[1089] Next, the user asks another question:

[1090] Question 2: "What is Machine Learning?"

[1091] Answer 2: "Machine Learning is a subset of AI."

[1092] Prompt: "Generate an answer to the user's question: 'What is Machine Learning?'"

[1093] These questions and answers are each time-stamped and created as entries. These entries are saved in image format with file names such as "user1234_20231003_120000.png" and "user1234_20231003_121000.png." By referring to these images, users can visually confirm when they asked what questions and what answers they received.

[1094] As described above, the present invention provides a specific means for visually and efficiently managing and referencing a user's question and answer history.

[1095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1096] Step 1:

[1097] The user inputs a question via a terminal.

[1098] Specific behavior:

[1099] The user enters "What is AI?" into the input field on the terminal and presses the send button.

[1100] input:

[1101] A user asks, "What is AI?"

[1102] output:

[1103] The terminal acquires the user's question as text data.

[1104] Step 2:

[1105] The device generates a prompt sentence to send the acquired question to the generative AI model.

[1106] Specific behavior:

[1107] The device converts the question text "What is AI?" into the prompt "Generate an answer to the user's question: 'What is AI?'"

[1108] input:

[1109] The question text obtained from the user: "What is AI?"

[1110] output:

[1111] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[1112] Step 3:

[1113] The device sends the generated prompt to the generative AI model and obtains the answer.

[1114] Specific behavior:

[1115] The device sends the prompt as a request to the generative AI model's API, which then generates the answer "AI stands for Artificial Intelligence." based on the prompt and returns it to the device.

[1116] input:

[1117] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[1118] output:

[1119] The answer returned by the generative AI model: "AI stands for Artificial Intelligence."

[1120] Step 4:

[1121] The terminal adds a timestamp to the acquired question and answer and creates an entry.

[1122] Specific behavior:

[1123] The device obtains the current date and time and creates entry data containing the question "What is AI?", the answer "AI stands for Artificial Intelligence.", and the timestamp "2023 / 10 / 03 12:00:00".

[1124] input:

[1125] Question: "What is AI?" Answer: "AI stands for Artificial Intelligence." Current date and time: "2023 / 10 / 03 12:00:00".

[1126] output:

[1127] Generated entry data (question, answer, timestamp).

[1128] Step 5:

[1129] The terminal stores the created entries in a database as history.

[1130] Specific behavior:

[1131] Insert the entry data (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Timestamp: "2023 / 10 / 03 12:00:00") into the database.

[1132] input:

[1133] The generated entry data.

[1134] output:

[1135] The history of entries in the database, ordered chronologically.

[1136] Step 6:

[1137] The terminal will render the contents of the entry into an image and save it as a file.

[1138] Specific behavior:

[1139] The device uses Python's Pillow library to convert the entry content (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Time Stamp: "2023 / 10 / 03 12:00:00") into an image and saves it with the file name "user1234_20231003_120000.png".

[1140] input:

[1141] Entry data (question, answer, timestamp).

[1142] output:

[1143] The saved image file is "user1234_20231003_120000.png".

[1144] Step 7:

[1145] If the user wants to view the history, the device displays the saved images.

[1146] Specific behavior:

[1147] When the user performs an operation to open the history display screen, the device loads the corresponding image file "user1234_20231003_120000.png" and displays it on the screen.

[1148] input:

[1149] User reference request, image file "user1234_20231003_120000.png".

[1150] output:

[1151] Image content (question, answer, timestamp) displayed on the device screen.

[1152] (Application example 1)

[1153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1154] In modern virtual stores, there is a problem in that users have few means to inquire about product information and then visually confirm that information later. Specifically, when a user makes multiple inquiries, it is difficult to centrally manage the details of those inquiries and review them visually. This leads to problems such as users having to ask the same questions repeatedly or being unable to efficiently utilize the information they obtain.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1156] In this invention, the server includes means for receiving questions and answers from users and generating entries including a timestamp, the question, and the answer, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for displaying the entries saved as images, means for recording product information inquiries and their answers in a virtual store, means for uniquely naming the entries using a user ID and a timestamp and saving them in cloud storage, and means for displaying a list of image entries in an album view format, thereby enabling users to visually manage their product inquiry history and efficiently review the information.

[1157] A "user question" is a request for information posed by a user to the system.

[1158] An "answer" is information provided by the system in response to an inquiry.

[1159] A "timestamp" is information that indicates the date and time when the entry was created.

[1160] An "entry" is a record that includes a timestamp, question, and answer data.

[1161] A "history" is a collection of entries created in the past, stored in chronological order.

[1162] An "image" is digital data that visually depicts the contents of an entry.

[1163] A "virtual store" is an online sales platform that offers products and services over the Internet.

[1164] A "user ID" is an identifier that uniquely identifies an individual user.

[1165] "Cloud storage" is a service that stores and makes accessible data on remote servers over the Internet.

[1166] The "album view format" is a user interface format for visually displaying a list of image entries.

[1167] "Uniquely named" means adding a unique name to each entry or file so that the same name is never generated twice.

[1168] The present invention is implemented as a system that allows users to manage and visually review questions and answers posed by users in a virtual store. This system effectively manages questions posed by users to search for product information and their answers, and displays them in an album view format.

[1169] System Overview

[1170] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to visually manage their product inquiry history and easily review the information they have obtained later.

[1171] Hardware and Software Configuration

[1172] This system uses the following hardware and software:

[1173] Hardware: Smartphones, smart glasses

[1174] software:

[1175] Frontend: React Native

[1176] Backend: Python, Flask

[1177] Database: AWS S3, DynamoDB

[1178] AI model: OpenAI GPT-3

[1179] Image processing library: Pillow (Python)

[1180] Details of data processing and data calculation

[1181] Get user questions and answers

[1182] When a user asks the system a question, it is sent to the server via a smartphone or smart glasses, and the server then poses the question to a back-end AI model (GPT-3) to obtain an answer.

[1183] Creating and Saving Entries

[1184] The server generates an entry by adding a timestamp to the question and answer. This entry is saved as an image in an AWS S3 bucket with a unique name and user ID. The image is generated using the Python Pillow library.

[1185] Displaying images

[1186] When a user browses saved entries, the image entries are listed through an album-view style interface on their smartphone or smart glasses. When the user selects a specific entry, its details are displayed on the smart device.

[1187] Specific examples

[1188] For example, if a user asks "What is the battery capacity of this smartphone?" in a virtual store, the system will answer "This smartphone's battery capacity is 4000mAh." This question and answer are generated as an entry with a timestamp and saved in image format. The saved image is uniquely named using the user ID and timestamp and saved in cloud storage.

[1189] Prompt Sentence Examples

[1190] Q: What is the battery capacity of this smartphone?

[1191] A: This smartphone has a battery capacity of 4000mAh.

[1192] This allows users to visually manage product inquiry history and efficiently review information.

[1193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1194] Step 1:

[1195] The user inputs a question using a smartphone or smart glasses. The input question is sent from the device to the server. Input: User's question. Output: Question data sent to the server.

[1196] Step 2:

[1197] The server inputs the received question into a generative AI model (GPT-3) to generate an answer. Input: User's question. Output: Answer from the generative AI model.

[1198] Step 3:

[1199] When the server receives the question and answer, it adds a timestamp to them and creates an entry that includes the question, answer, and the date and time of recording. Input: Question, answer, timestamp. Output: Entry with timestamp.

[1200] Step 4:

[1201] The server converts the generated entries into an image format. Specifically, it draws each entry into an image using Python's Pillow library. Input: Entry with timestamp. Output: Rendered image.

[1202] Step 5:

[1203] The server assigns a unique file name to the drawn image using the user ID and timestamp and uploads it to an AWS S3 bucket. Input: drawn image, user ID, timestamp. Output: image saved in cloud storage.

[1204] Step 6:

[1205] When a user wants to view a saved entry, the device retrieves the image entry from the server in album view format and displays it. Input: User's view request. Output: Image entry displayed in album view format.

[1206] Step 7:

[1207] When the user selects a particular image entry, the terminal displays the details of that image entry. Input: User's image entry selection. Output: Display of details of the selected image entry.

[1208] Through these processing steps, the system visually manages user inquiries in the virtual store and enables efficient review of information.

[1209] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1210] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions, the system also saves the user's emotions in the entry and provides a function to draw an image that includes those emotions.

[1211] System Overview

[1212] This system records user questions and their answers as entries with timestamps, and adds the user's emotions recognized using an emotion engine. These entries are saved in image format, allowing users to view them later visually like an album. In addition, the images also include the user ID and emotion information, making it possible to understand each user's history at a glance.

[1213] Program Operation

[1214] 1. Getting user questions and answers

[1215] The user inputs a question to the system. The device receives the question from the user and sends it to an appropriate system (e.g., generative AI) to obtain an answer.

[1216] 2. Emotion Recognition by Emotion Engine

[1217] The device passes the question and answer to the emotion engine, which then identifies the user's emotion. The emotion engine infers the emotion from the content of the text and the user's input method, and returns the result to the device.

[1218] 3. Generate an entry containing a timestamp, question, answer, and recognized sentiment.

[1219] The device adds a timestamp to the captured question and answer and also generates an entry containing the recognized emotion, including the question, the answer, the recorded date and time, and the user's emotion.

[1220] 4. Saving entries as history

[1221] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[1222] 5. Drawing the entry content onto an image

[1223] The terminal uses a means to render the contents of the entry, i.e., the timestamp, question, answer, and emotion, as an image. Specifically, the terminal writes the text information and emotion information to an image file.

[1224] 6. Save the image

[1225] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[1226] 7. Displaying Images

[1227] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[1228] Specific examples

[1229] For example, if a user asks the following question:

[1230] Question 1: "What is AI?"

[1231] Answer 1: "AI stands for Artificial Intelligence."

[1232] Furthermore, let's say the emotion engine recognizes "interest" in the user's input, and generates the following entry:

[1233] plaintext

[1234] Timestamp: 2023-02-03 12:05:01

[1235] Question: What is AI?

[1236] Answer: AI stands for Artificial Intelligence.

[1237] Emotion: Interest

[1238] Next, the user asks another question:

[1239] Question 2: "What is Machine Learning?"

[1240] Answer 2: "Machine Learning is a subset of AI."

[1241] If the emotion engine recognizes "curiosity" from this input, it will generate an entry like this:

[1242] plaintext

[1243] Timestamp: 2023-02-03 12:12:01

[1244] Question: What is Machine Learning?

[1245] Answer: Machine Learning is a subset of AI.

[1246] Emotion: Curiosity

[1247] These questions and answers are each added with a timestamp and emotional information, and are created as entries. These entries are saved as images with filenames such as "user1234_20230203_120501.png" and "user1234_20230203_121201.png."

[1248] The server manages the storage of these images and provides a means for quickly displaying them when a user wants to refer to a specific entry. By reviewing these images later, users can visually confirm when, what questions were asked, and what emotional state they were in.

[1249] The above is an embodiment of the present invention, which allows management and reference of a user's investigation history to be performed visually and efficiently, and in particular, by including information on the user's emotions, a richer history management system is provided.

[1250] The processing flow will be explained below.

[1251] Step 1:

[1252] The user inputs a question to the system.

[1253] Step 2:

[1254] The device receives questions from the user and poses the questions to the generative AI or information provision system.

[1255] Step 3:

[1256] Generative AI and information provision systems generate answers to questions and return them to the device.

[1257] Step 4:

[1258] The device passes the question and answer to the emotion engine, which identifies the user's emotion. The emotion engine infers the user's emotion from the content of the text and the user's input method, and returns the result to the device.

[1259] Step 5:

[1260] The device retrieves the current date and time and formats it as a timestamp.

[1261] python

[1262] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[1263] Step 6:

[1264] The terminal generates an entry that includes a timestamp, a question, an answer, and a recognized emotion.

[1265] python

[1266] entry = {

[1267] 'timestamp': timestamp,

[1268] 'question': question,

[1269] 'answer': answer,

[1270] 'emotion': emotion

[1271] }

[1272] Step 7:

[1273] The terminal stores the generated entries in its history.

[1274] python

[1275] self.history.append(entry)

[1276] Step 8:

[1277] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[1278] python

[1279] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[1280] draw = ImageDraw.Draw(img)

[1281] font = ImageFont.load_default()

[1282] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[1283] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[1284] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[1285] draw.text((10, 70), f'Emotion: {emotion}', fill=(0, 0, 0), font=font)

[1286] Step 9:

[1287] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[1288] python

[1289] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[1290] Step 10:

[1291] Saves the image generated by the device to the specified path.

[1292] python

[1293] img.save(img_path)

[1294] Step 11:

[1295] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[1296] Step 12:

[1297] When a user wants to view the history, they send a request from their device to the server. The server then searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question asked, the answer, and their emotional state at the time.

[1298] Example 2

[1299] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1300] Conventional history management systems only record questions and answers in text format, which means they cannot take into account the user's emotional state, making it difficult to fully grasp the meaning and value of the history. Furthermore, limited means for visually checking the history make it difficult for users to quickly and intuitively restore past questions and answers. Furthermore, there is a lack of ingenuity to allow users to understand each user's history at a glance.

[1301] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1302] In this invention, the server includes means for receiving questions and answers from users and inputting them into a generative AI model to obtain answers, means for inputting the questions and answers into an emotion recognition engine to identify the user's emotion, means for generating entries including a timestamp, the question, the answer, and the recognized emotion, means for saving the generated entries as history, means for rendering the contents of the entries as images, and means for displaying the saved entries as images. This allows emotional information to be recorded in the user's question history, making it possible to check the history in a visual format.

[1303] 1. "User" means an entity that uses the system to enter questions and obtain answers.

[1304] 2. A "question" is text that a user enters into a system requesting information.

[1305] 3. "Answer" is the text response generated by a generative AI model in response to a question.

[1306] 4. "Generative AI model" refers to an algorithm or system that generates answers to user questions.

[1307] 5. "Emotion recognition engine" refers to an algorithm or system that analyzes questions and answers to identify a user's emotions.

[1308] 6. "Timestamp" is information that indicates the date and time a particular entry was created.

[1309] 7. "Entry" refers to a collection of data including a question, an answer, a timestamp, and a recognized emotion.

[1310] 8. "History" refers to a record of questions asked by a user in the past, their answers, and any accompanying information saved in chronological order.

[1311] 9. "Image rendering means" refers to processes and techniques for converting textual and emotional information into image files.

[1312] 10. "Entry saved as an image" means the content of an entry saved in image format.

[1313] 11. "User ID" means identification information that uniquely identifies each user.

[1314] 12. "Visual Verification" means any method or technology that allows a User to view an entry stored as an image via a Terminal.

[1315] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides a function to save the user's emotions in the entry and draw an image that includes those emotions.

[1316] System Overview

[1317] The system includes the following main means:

[1318] 1. User question input and answer acquisition method

[1319] 2. Emotion identification method using an emotion recognition engine

[1320] 3. Entry generation means including timestamp, question, answer, and recognized sentiment

[1321] 4. How to save entries as history

[1322] 5. How to draw the entry content into an image

[1323] 6. How to display entries saved as images

[1324] Hardware and software used

[1325] Generative AI models: For example, open-source language models or cloud-based language model APIs (e.g., GPT-3)

[1326] Emotion recognition engine: For example, natural language processing API (e.g., IBM Watson Natural Language Understanding)

[1327] Image generation software: For example, a Python image processing library (e.g., PIL - Python Imaging Library)

[1328] Database: For example, a relational database (e.g., MySQL, PostgreSQL)

[1329] Specific processing of the program

[1330] The server receives the question entered by the user and sends it to the generative AI model as a prompt. The generative AI model returns an answer to the question, and the answer is received by the server. An example of a prompt is "What is AI?"

[1331] The server then sends the question and answer to an emotion recognition engine to identify the user's emotion. The emotion recognition engine analyzes the text content and the user's input patterns to recognize the emotion, for example, "interest."

[1332] The server adds a timestamp to the question, answer, and recognized emotion, creates a single entry, associates it with the user ID, and stores it in a database.

[1333] The server renders the entry's content (timestamp, question, answer, sentiment) into an image. Specifically, it uses a Python image processing library to convert the text information into an image file. For example, consider the following entry:

[1334] plaintext

[1335] Timestamp: 2023-02-03 12:05:01

[1336] Question: What is AI?

[1337] Answer: AI stands for Artificial Intelligence.

[1338] Emotion: Interest

[1339] The image is saved on the server with a unique name that includes the user ID and a timestamp, for example, "user1234_20230203_120501.png."

[1340] When a user wants to view the history, the server retrieves the image file and displays it on the device, allowing the user to visually confirm the question and the emotional state at that time.

[1341] The above is an embodiment of the present invention. By using this system, it becomes possible to visually and efficiently manage and refer to a user's research history. In particular, by including user emotional information, it is possible to provide a richer history management system.

[1342] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1343] Step 1:

[1344] The user inputs a question through the terminal. The terminal receives this question and temporarily stores the input in memory. This input is used in later processing. For example, the user inputs the question "What is AI?"

[1345] Step 2:

[1346] The device sends a question from the user to the server. The server passes the received question to the generative AI model as a prompt and asks for an answer. The server calls the API of the generative AI model (e.g., GPT-3), sends the question as a prompt, and receives the answer. The input at this time is the user's question, and the output is the answer from the generative AI model.

[1347] Step 3:

[1348] The server receives the answer from the generative AI model and sends it along with the question to the emotion recognition engine. The server calls the API of the emotion recognition engine (e.g., IBM Watson Natural Language Understanding) to analyze the question and answer and identify the user's emotion. The input is the question and answer, and the output is the recognized emotion.

[1349] Step 4:

[1350] The server generates an entry with the question, answer, recognized emotion, and the current date and time. The server obtains the system time and adds it as a timestamp to the entry. This entry contains the question, answer, timestamp, and user emotion. An example of a generated entry is as follows:

[1351] plaintext

[1352] Timestamp: 2023-02-03 12:05:01

[1353] Question: What is AI?

[1354] Answer: AI stands for Artificial Intelligence.

[1355] Emotion: Interest

[1356] Step 5:

[1357] The server saves the generated entries in a database. The database is configured to save entries for each user in chronological order. The input is the generated entries, and the output is the results saved in the database.

[1358] Step 6:

[1359] The server starts the process of rendering the entry content into an image. The server uses image generation software (e.g., PIL) to convert the entry's text information into an image file. The input is the entry, and the output is the image file.

[1360] Step 7:

[1361] The server saves the generated image file with a unique file name that includes the user ID and a timestamp. An example of a saved image file name is "user1234_20230203_120501.png." The input in this case is the image file, and the output is the path to the saved image file.

[1362] Step 8:

[1363] When a user wants to view the history, the device sends a request to the server. The server retrieves the image path of the corresponding entry from the database and sends it to the device. The device receives this and displays it visually to the user. The input in this case is the user request, and the output is the displayed image.

[1364] The above is the specific processing flow of this system, and the data processing and data calculation required at each step will be explained in detail.

[1365] (Application example 2)

[1366] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1367] Providing efficient and effective customer service through proper management of customer interaction history and emotions is a challenge in modern brick-and-mortar stores. Understanding the content of past customer inquiries and their emotions at the time is particularly important for improving customer satisfaction, but many stores lack the means to systematically record and manage this information. Furthermore, appropriate responses are sometimes required even for customers who are not comfortable with direct conversation. Therefore, it is necessary to provide a system that efficiently manages customer interaction history and emotional information and allows for easy visual reference.

[1368] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question and its answer from a user and generating an entry including a timestamp, the question, and the answer, means for saving the generated entry as a history, means for rendering the content of the entry as an image, means for displaying the entry saved as an image, means for identifying the user's emotion, means for adding the identified emotion to the entry, means for saving questions and answers related to specific products, and means for managing the history of customer service in a physical store. This not only makes it possible to effectively manage and visually reference the interaction history and emotion information with customers, but also enables more appropriate and effective customer service by understanding the content of each customer's past inquiries and their emotions at the time.

[1369] "Questions from users" are inquiries provided to the system by users.

[1370] An "answer" is a response provided by the system to a user's question.

[1371] The "timestamp" is information about the date and time when the entry was created.

[1372] An "entry" is a record that combines a question, an answer, a timestamp, and emotional information.

[1373] A "history" is a collection of entries created in the past.

[1374] The "means for rendering to an image" is a method for converting the contents of an entry into an image format that can be visually displayed.

[1375] The "means for displaying" is a method for displaying entries saved in image format so that the user can visually confirm them.

[1376] The "means for identifying user emotions" is a method for inferring emotions from the user's questions and answers.

[1377] The "means for adding identified emotions to an entry" is a method for adding emotion information to an entry.

[1378] The "means for storing questions and answers related to specific products" is a method for recording inquiries about specific products and the answers to those inquiries.

[1379] "Means for managing customer service history in physical stores" refers to a method for systematically storing and managing customer service history in physical stores.

[1380] "Means for adding a user ID" refers to a method for adding a unique ID to an entry to identify a user.

[1381] The system of the present invention provides a method for efficiently managing and visually referencing customer interaction history and emotional information in a physical store. This system is realized by the following steps.

[1382] The system receives questions and answers from users and generates an entry containing a timestamp, question, and answer. It then saves the generated entry as a history and renders the contents of the entry as an image. This rendered image can be used for future reference by customers and staff. It also incorporates a means for identifying emotions from the user's input and adding the identified emotions to the entry. It also saves questions and answers related to specific products, allowing for detailed recording and management of product purchase history and inquiry details. It also manages customer service history in physical stores, making it possible to visually check what inquiries a specific customer has made in the past.

[1383] Hardware and Software Use

[1384] This system is primarily composed of software for the server, user devices, and emotion engine. The server plays the primary role of storing and managing data. The user devices (smartphones and customer service robots) function as interfaces that receive input from users. The emotion engine uses natural language processing libraries (e.g., spaCy and NLTK). The Python Pillow library is used to draw text onto images.

[1385] 1. Ask questions and get answers:

[1386] The user device receives questions posed by customers in the physical store via an input interface, and sends them to the server-side generative AI model to obtain the appropriate answer.

[1387] 2. Identifying emotions:

[1388] The server analyzes the text content of the questions and answers and identifies the user's emotions using an emotion engine, which uses natural language processing techniques to infer emotions from the input text.

[1389] 3. Create and save the entry:

[1390] The server compiles the question, answer, timestamp, and identified emotion into a single entry and stores it in chronological order as a history.

[1391] 4. Image generation and display:

[1392] The server renders the entry contents as an image so that they can be visually confirmed. The Python Pillow library is used for rendering. The generated image is saved with a unique name using the user ID and a timestamp. When the user wants to refer to the history, this image is displayed on the user's device.

[1393] Examples of specific examples and prompts

[1394] For example, if a customer in a clothing store asks, "Can I wash this shirt?" and the robot replies, "Yes, it can be washed," the emotion engine will identify a "feeling of relief." The resulting entry would look like this:

[1395] entry:

[1396] plaintext

[1397] Timestamp: 2023-10-01 12:00:00

[1398] Question: Can this shirt be washed?

[1399] Answer: Yes, it is washable.

[1400] Emotion: A sense of security

[1401] An example prompt is:

[1402] plaintext

[1403] Prompt sentence to be passed to the emotion engine

[1404] Identify the sentiment in the following text:

[1405] Can this shirt be washed?

[1406] A: Yes, it is washable.

[1407] The information recorded in this way is stored visually as an image, providing a detailed customer history including specific questions, answers, and even emotions at the time, enabling more personalized service for each customer and contributing to improved customer satisfaction.

[1408] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1409] Step 1:

[1410] A user inputs a question into a terminal (smartphone or customer service robot) in a physical store. The terminal receives this input and saves it as text data. Specifically, when a user asks, "Can I wash this shirt?", the content is saved in text format on the terminal. Input: User's question (text data), Output: Question saved on the terminal (text data).

[1411] Step 2:

[1412] The device sends a question to the server, which uses the generative AI model to generate an appropriate answer. For example, the device sends the text "Can this shirt be washed?" to the server, and the server uses the generative AI model to obtain the answer "Yes, it can be washed." Input: User's question (text data), Output: Answer from the server (text data).

[1413] Step 3:

[1414] The server passes the captured question and answer text to the emotion engine, which then identifies emotions from this text. Specifically, the emotion engine analyzes the text "Can I wash this shirt?" and "Yes, it can be washed" and identifies the emotion "relief." Input: Question and answer text (text data), Output: Identified emotion (text data).

[1415] Step 4:

[1416] The server compiles the question, answer, timestamp, and identified sentiment into a single entry, for example, the following entry:

[1417] plaintext

[1418] Timestamp: 2023-10-01 12:00:00

[1419] Question: Can this shirt be washed?

[1420] Answer: Yes, it is washable.

[1421] Emotion: A sense of security

[1422] Input: Question, Answer, Timestamp, Sentiment (Text data), Output: Entry (Text data).

[1423] Step 5:

[1424] The server saves the generated entries. The entries are saved in chronological order as history and managed for future reference. For example, the entries are saved in a "history database." Input: entry (text data), output: saved history (database).

[1425] Step 6:

[1426] The server uses the Python Pillow library to render the entry contents into image format. Specifically, it reads the entry's text information, renders it into an image file, and saves it with a unique name. For example, an image file named "user1234_20231001_120000.png" is generated. Input: Entry (text data), Output: Rendered image (image file).

[1427] Step 7:

[1428] When the user wants to view the history, the server displays the saved image on the terminal. This display uses the display function of the terminal. An appropriate user interface is provided so that the user can check the image. For example, the terminal loads the image and displays it on the screen. Input: Saved image (image file), Output: Image displayed on the terminal (visual data).

[1429] In this way, a system is created that manages customer response history and emotional information in physical stores and allows for visual reference.

[1430] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1431] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1432] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1433] [Fourth embodiment]

[1434] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1435] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1436] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1437] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1438] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1441] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1442] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1443] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1444] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1445] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1446] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1447] The present invention relates to a system that receives user questions and their answers, saves them as a history, and allows users to visually review them later. How this system is implemented will be specifically described below.

[1448] System Overview

[1449] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to refer to information in a visually easy-to-recognize format, making it easier to remember.

[1450] Program Operation

[1451] 1. Getting user questions and answers

[1452] The user asks the system a question. The device receives the question and sends it to an appropriate system (e.g., generative AI) to get an answer.

[1453] 2. Creating an entry containing a timestamp, question, and answer

[1454] The device adds a timestamp to the acquired question and answer and records it as an entry, which includes the question, the answer, and the date and time it was recorded.

[1455] 3. Save entries as history

[1456] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[1457] 4. Drawing the entry content onto the image

[1458] A means is used to render the contents of the entry, i.e., the timestamp, question, and answer, as an image, specifically by writing the text information to an image file.

[1459] 5. Save the image

[1460] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[1461] 6. Displaying Images

[1462] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[1463] Specific examples

[1464] For example, if a user asks the following question:

[1465] Question 1: "What is AI?"

[1466] Answer 1: "AI stands for Artificial Intelligence."

[1467] Next, the user asks another question:

[1468] Question 2: "What is Machine Learning?"

[1469] Answer 2: "Machine Learning is a subset of AI."

[1470] In this case, each question and answer is assigned a timestamp and created as an entry. This entry is saved as an image with a file name such as "user1234_20230203_120501.png" or "user1234_20230203_121201.png." Users can later review these images to visually confirm when they asked what question and what answer they received.

[1471] The server manages the storage of these images and provides a means for the user to quickly display them when they want to refer to a particular entry, so that the user can easily recall the content of their research at that time.

[1472] The above is an embodiment of the present invention, which provides a system that allows users to visually and efficiently manage and refer to their investigation history.

[1473] The processing flow will be explained below.

[1474] Step 1:

[1475] The user inputs a question to the system.

[1476] Step 2:

[1477] The device receives questions entered by the user and poses the questions to generative AI or other information provision systems.

[1478] Step 3:

[1479] Generative AI and information provision systems generate answers to questions and return them to the device.

[1480] Step 4:

[1481] The device retrieves the current date and time and formats it as a timestamp.

[1482] python

[1483] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[1484] Step 5:

[1485] The terminal generates an entry that includes a timestamp, a question, and an answer.

[1486] python

[1487] entry = {

[1488] 'timestamp': timestamp,

[1489] 'question': question,

[1490] 'answer': answer

[1491] }

[1492] Step 6:

[1493] The terminal stores the generated entries in its history.

[1494] python

[1495] self.history.append(entry)

[1496] Step 7:

[1497] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[1498] python

[1499] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[1500] draw = ImageDraw.Draw(img)

[1501] font = ImageFont.load_default()

[1502] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[1503] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[1504] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[1505] Step 8:

[1506] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[1507] python

[1508] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[1509] Step 9:

[1510] Saves the image generated by the device to the specified path.

[1511] python

[1512] img.save(img_path)

[1513] Step 10:

[1514] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[1515] Step 11:

[1516] When a user wants to view the history, they send a request from their device to the server. The server searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question and answer they asked.

[1517] Example 1

[1518] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1519] In conventional question-answering systems, there are few ways for users to easily refer to past questions and their answers, making it difficult for users to visually confirm what questions they asked and what the answers were. In addition, storing history in text format lacks visual information, making it difficult to remember and recognize information.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1521] In this invention, the server includes means for receiving questions and answers from users and transmitting prompt sentences to the generative AI model to obtain answers, means for generating entries by adding timestamps to the obtained questions and answers, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for saving the rendered images, and means for displaying the saved images. This allows users to visually check past questions and answers, making it easier for them to remember and recognize information.

[1522] A "user" is an individual or entity that enters a question into the system and obtains an answer.

[1523] A "question" refers to a question or problem that a user inputs into the system.

[1524] "Answer" refers to the response that a generative AI model generates in response to a user's question.

[1525] A "generative AI model" is an artificial intelligence algorithm or system that receives a prompt and generates an answer based on its content.

[1526] A "prompt" is a command or query that a generative AI model uses to generate an appropriate answer.

[1527] A "timestamp" is a digital mark that indicates a specific date and time, and is information that clarifies the point in time when a question or answer was recorded.

[1528] An "entry" refers to a collection of data that includes a question, an answer, and a timestamp.

[1529] "History" refers to a collection of data in which entries created in the past are organized in chronological order.

[1530] "Rendering to image" refers to the act of converting the contents of an entry into a visually identifiable format and saving it as an image file.

[1531] "User identification information" refers to information for uniquely identifying a specific user, and includes, for example, a user ID.

[1532] "Server" means the central computer that manages and operates the entire system and processes requests from users.

[1533] The present invention relates to a system that records questions and answers posed by users along with timestamps, and allows them to be visually saved and referenced. How this system is implemented is described below.

[1534] System configuration

[1535] 1. Hardware configuration:

[1536] Server: The central computer that manages the entire question-answering system. It is responsible for communicating with the database and the generative AI model.

[1537] Terminal: A device (such as a computer or smartphone) that is directly operated by the user and provides functions for inputting questions, displaying them, and saving them.

[1538] 2. Software configuration:

[1539] Generative AI model: A generative AI model uses an artificial intelligence algorithm to generate answers to user questions. As a specific example, we use a model that uses natural language processing technology.

[1540] Pillow Library: Uses Python's Pillow library to draw entries onto the image.

[1541] Overview of program processing

[1542] 1. Getting user questions and answers

[1543] The user inputs a question into the device, which then sends the question to the generative AI model as a prompt. For example, the question "What is AI?" is converted into the prompt "Generate an answer to the user's question: 'What is AI?'"

[1544] The generative AI model receives the prompt sentence, generates a response such as "AI stands for Artificial Intelligence," and returns it to the device.

[1545] 2. Creating an entry containing a timestamp, question, and answer

[1546] The device adds a timestamp indicating the current date and time to the questions and answers it receives, and organizes each piece of information into an entry.

[1547] 3. Save entries as history

[1548] The device stores the generated entries in a database and organizes them in chronological order.

[1549] 4. Drawing the entry content onto the image

[1550] The terminal uses Python's Pillow library to draw the entry contents (timestamp, question, answer) as an image.

[1551] 5. Save the image

[1552] The drawn image is given a unique name using the user's identification information and a timestamp and saved as a file, for example, in the format "user1234_20231003_120000.png".

[1553] 6. Displaying Images

[1554] If the user wants to view the history, the device will display the saved image, allowing the user to visually review the history of questions and answers.

[1555] Specific operation example

[1556] For example, a user asks the following question:

[1557] Question 1: "What is AI?"

[1558] Answer 1: "AI stands for Artificial Intelligence."

[1559] Prompt: "Generate an answer to the user's question: 'What is AI?'"

[1560] Next, the user asks another question:

[1561] Question 2: "What is Machine Learning?"

[1562] Answer 2: "Machine Learning is a subset of AI."

[1563] Prompt: "Generate an answer to the user's question: 'What is Machine Learning?'"

[1564] These questions and answers are each time-stamped and created as entries. These entries are saved in image format with file names such as "user1234_20231003_120000.png" and "user1234_20231003_121000.png." By referring to these images, users can visually confirm when they asked what questions and what answers they received.

[1565] As described above, the present invention provides a specific means for visually and efficiently managing and referencing a user's question and answer history.

[1566] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1567] Step 1:

[1568] The user inputs a question via a terminal.

[1569] Specific behavior:

[1570] The user enters "What is AI?" into the input field on the terminal and presses the send button.

[1571] input:

[1572] A user asks, "What is AI?"

[1573] output:

[1574] The terminal acquires the user's question as text data.

[1575] Step 2:

[1576] The device generates a prompt sentence to send the acquired question to the generative AI model.

[1577] Specific behavior:

[1578] The device converts the question text "What is AI?" into the prompt "Generate an answer to the user's question: 'What is AI?'"

[1579] input:

[1580] The question text obtained from the user: "What is AI?"

[1581] output:

[1582] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[1583] Step 3:

[1584] The device sends the generated prompt to the generative AI model and obtains the answer.

[1585] Specific behavior:

[1586] The device sends the prompt as a request to the generative AI model's API, which then generates the answer "AI stands for Artificial Intelligence." based on the prompt and returns it to the device.

[1587] input:

[1588] The generated prompt reads: "Generate an answer to the user's question: 'What is AI?'"

[1589] output:

[1590] The answer returned by the generative AI model: "AI stands for Artificial Intelligence."

[1591] Step 4:

[1592] The terminal adds a timestamp to the acquired question and answer and creates an entry.

[1593] Specific behavior:

[1594] The device obtains the current date and time and creates entry data containing the question "What is AI?", the answer "AI stands for Artificial Intelligence.", and the timestamp "2023 / 10 / 03 12:00:00".

[1595] input:

[1596] Question: "What is AI?" Answer: "AI stands for Artificial Intelligence." Current date and time: "2023 / 10 / 03 12:00:00".

[1597] output:

[1598] Generated entry data (question, answer, timestamp).

[1599] Step 5:

[1600] The terminal stores the created entries in a database as history.

[1601] Specific behavior:

[1602] Insert the entry data (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Timestamp: "2023 / 10 / 03 12:00:00") into the database.

[1603] input:

[1604] The generated entry data.

[1605] output:

[1606] The history of entries in the database, ordered chronologically.

[1607] Step 6:

[1608] The terminal will render the contents of the entry into an image and save it as a file.

[1609] Specific behavior:

[1610] The device uses Python's Pillow library to convert the entry content (Question: "What is AI?", Answer: "AI stands for Artificial Intelligence.", Time Stamp: "2023 / 10 / 03 12:00:00") into an image and saves it with the file name "user1234_20231003_120000.png".

[1611] input:

[1612] Entry data (question, answer, timestamp).

[1613] output:

[1614] The saved image file is "user1234_20231003_120000.png".

[1615] Step 7:

[1616] If the user wants to view the history, the device displays the saved images.

[1617] Specific behavior:

[1618] When the user performs an operation to open the history display screen, the device loads the corresponding image file "user1234_20231003_120000.png" and displays it on the screen.

[1619] input:

[1620] User reference request, image file "user1234_20231003_120000.png".

[1621] output:

[1622] Image content (question, answer, timestamp) displayed on the device screen.

[1623] (Application example 1)

[1624] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1625] In modern virtual stores, there is a problem in that users have few means to inquire about product information and then visually confirm that information later. Specifically, when a user makes multiple inquiries, it is difficult to centrally manage the details of those inquiries and review them visually. This leads to problems such as users having to ask the same questions repeatedly or being unable to efficiently utilize the information they obtain.

[1626] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1627] In this invention, the server includes means for receiving questions and answers from users and generating entries including a timestamp, the question, and the answer, means for saving the generated entries as history, means for rendering the contents of the entries as images, means for displaying the entries saved as images, means for recording product information inquiries and their answers in a virtual store, means for uniquely naming the entries using a user ID and a timestamp and saving them in cloud storage, and means for displaying a list of image entries in an album view format, thereby enabling users to visually manage their product inquiry history and efficiently review the information.

[1628] A "user question" is a request for information posed by a user to the system.

[1629] An "answer" is information provided by the system in response to an inquiry.

[1630] A "timestamp" is information that indicates the date and time when the entry was created.

[1631] An "entry" is a record that includes a timestamp, question, and answer data.

[1632] A "history" is a collection of entries created in the past, stored in chronological order.

[1633] An "image" is digital data that visually depicts the contents of an entry.

[1634] A "virtual store" is an online sales platform that offers products and services over the Internet.

[1635] A "user ID" is an identifier that uniquely identifies an individual user.

[1636] "Cloud storage" is a service that stores and makes accessible data on remote servers over the Internet.

[1637] The "album view format" is a user interface format for visually displaying a list of image entries.

[1638] "Uniquely named" means adding a unique name to each entry or file so that the same name is never generated twice.

[1639] The present invention is implemented as a system that allows users to manage and visually review questions and answers posed by users in a virtual store. This system effectively manages questions posed by users to search for product information and their answers, and displays them in an album view format.

[1640] System Overview

[1641] This system records user questions and their answers as entries with timestamps, and saves the entries in image format. It also provides a function that allows users to view the saved images like an album. This allows users to visually manage their product inquiry history and easily review the information they have obtained later.

[1642] Hardware and Software Configuration

[1643] This system uses the following hardware and software:

[1644] Hardware: Smartphones, smart glasses

[1645] software:

[1646] Frontend: React Native

[1647] Backend: Python, Flask

[1648] Database: AWS S3, DynamoDB

[1649] AI model: OpenAI GPT-3

[1650] Image processing library: Pillow (Python)

[1651] Details of data processing and data calculation

[1652] Get user questions and answers

[1653] When a user asks the system a question, it is sent to the server via a smartphone or smart glasses, and the server then poses the question to a back-end AI model (GPT-3) to obtain an answer.

[1654] Creating and Saving Entries

[1655] The server generates an entry by adding a timestamp to the question and answer. This entry is saved as an image in an AWS S3 bucket with a unique name and user ID. The image is generated using the Python Pillow library.

[1656] Displaying images

[1657] When a user browses saved entries, the image entries are listed through an album-view style interface on their smartphone or smart glasses. When the user selects a specific entry, its details are displayed on the smart device.

[1658] Specific examples

[1659] For example, if a user asks "What is the battery capacity of this smartphone?" in a virtual store, the system will answer "This smartphone's battery capacity is 4000mAh." This question and answer are generated as an entry with a timestamp and saved in image format. The saved image is uniquely named using the user ID and timestamp and saved in cloud storage.

[1660] Prompt Sentence Examples

[1661] Q: What is the battery capacity of this smartphone?

[1662] A: This smartphone has a battery capacity of 4000mAh.

[1663] This allows users to visually manage product inquiry history and efficiently review information.

[1664] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1665] Step 1:

[1666] The user inputs a question using a smartphone or smart glasses. The input question is sent from the device to the server. Input: User's question. Output: Question data sent to the server.

[1667] Step 2:

[1668] The server inputs the received question into a generative AI model (GPT-3) to generate an answer. Input: User's question. Output: Answer from the generative AI model.

[1669] Step 3:

[1670] When the server receives the question and answer, it adds a timestamp to them and creates an entry that includes the question, answer, and the date and time of recording. Input: Question, answer, timestamp. Output: Entry with timestamp.

[1671] Step 4:

[1672] The server converts the generated entries into an image format. Specifically, it draws each entry into an image using Python's Pillow library. Input: Entry with timestamp. Output: Rendered image.

[1673] Step 5:

[1674] The server assigns a unique file name to the drawn image using the user ID and timestamp and uploads it to an AWS S3 bucket. Input: drawn image, user ID, timestamp. Output: image saved in cloud storage.

[1675] Step 6:

[1676] When a user wants to view a saved entry, the device retrieves the image entry from the server in album view format and displays it. Input: User's view request. Output: Image entry displayed in album view format.

[1677] Step 7:

[1678] When the user selects a particular image entry, the terminal displays the details of that image entry. Input: User's image entry selection. Output: Display of details of the selected image entry.

[1679] Through these processing steps, the system visually manages user inquiries in the virtual store and enables efficient review of information.

[1680] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1681] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining this system with an emotion engine that recognizes the user's emotions, the system also saves the user's emotions in the entry and provides a function to draw an image that includes those emotions.

[1682] System Overview

[1683] This system records user questions and their answers as entries with timestamps, and adds the user's emotions recognized using an emotion engine. These entries are saved in image format, allowing users to view them later visually like an album. In addition, the images also include the user ID and emotion information, making it possible to understand each user's history at a glance.

[1684] Program Operation

[1685] 1. Getting user questions and answers

[1686] The user inputs a question to the system. The device receives the question from the user and sends it to an appropriate system (e.g., generative AI) to obtain an answer.

[1687] 2. Emotion Recognition by Emotion Engine

[1688] The device passes the question and answer to the emotion engine, which then identifies the user's emotion. The emotion engine infers the emotion from the content of the text and the user's input method, and returns the result to the device.

[1689] 3. Generate an entry containing a timestamp, question, answer, and recognized sentiment.

[1690] The device adds a timestamp to the captured question and answer and also generates an entry containing the recognized emotion, including the question, the answer, the recorded date and time, and the user's emotion.

[1691] 4. Saving entries as history

[1692] The device stores the generated entries as a history, which is organized chronologically for the user to refer to later.

[1693] 5. Drawing the entry content onto an image

[1694] The terminal uses a means to render the contents of the entry, i.e., the timestamp, question, answer, and emotion, as an image. Specifically, the terminal writes the text information and emotion information to an image file.

[1695] 6. Save the image

[1696] The device saves the drawn image with a unique name using the user ID and a timestamp, making it easier to identify entries for each user.

[1697] 7. Displaying Images

[1698] When a user wants to refer to the history, the saved images are displayed on the terminal, allowing the user to visually check each entry.

[1699] Specific examples

[1700] For example, if a user asks the following question:

[1701] Question 1: "What is AI?"

[1702] Answer 1: "AI stands for Artificial Intelligence."

[1703] Furthermore, let's say the emotion engine recognizes "interest" in the user's input, and generates the following entry:

[1704] plaintext

[1705] Timestamp: 2023-02-03 12:05:01

[1706] Question: What is AI?

[1707] Answer: AI stands for Artificial Intelligence.

[1708] Emotion: Interest

[1709] Next, the user asks another question:

[1710] Question 2: "What is Machine Learning?"

[1711] Answer 2: "Machine Learning is a subset of AI."

[1712] If the emotion engine recognizes "curiosity" from this input, it will generate an entry like this:

[1713] plaintext

[1714] Timestamp: 2023-02-03 12:12:01

[1715] Question: What is Machine Learning?

[1716] Answer: Machine Learning is a subset of AI.

[1717] Emotion: Curiosity

[1718] These questions and answers are each added with a timestamp and emotional information, and are created as entries. These entries are saved as images with filenames such as "user1234_20230203_120501.png" and "user1234_20230203_121201.png."

[1719] The server manages the storage of these images and provides a means for quickly displaying them when a user wants to refer to a specific entry. By reviewing these images later, users can visually confirm when, what questions were asked, and what emotional state they were in.

[1720] The above is an embodiment of the present invention, which allows management and reference of a user's investigation history to be performed visually and efficiently, and in particular, by including information on the user's emotions, a richer history management system is provided.

[1721] The processing flow will be explained below.

[1722] Step 1:

[1723] The user inputs a question to the system.

[1724] Step 2:

[1725] The device receives questions from the user and poses the questions to the generative AI or information provision system.

[1726] Step 3:

[1727] Generative AI and information provision systems generate answers to questions and return them to the device.

[1728] Step 4:

[1729] The device passes the question and answer to the emotion engine, which identifies the user's emotion. The emotion engine infers the user's emotion from the content of the text and the user's input method, and returns the result to the device.

[1730] Step 5:

[1731] The device retrieves the current date and time and formats it as a timestamp.

[1732] python

[1733] timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

[1734] Step 6:

[1735] The terminal generates an entry that includes a timestamp, a question, an answer, and a recognized emotion.

[1736] python

[1737] entry = {

[1738] 'timestamp': timestamp,

[1739] 'question': question,

[1740] 'answer': answer,

[1741] 'emotion': emotion

[1742] }

[1743] Step 7:

[1744] The terminal stores the generated entries in its history.

[1745] python

[1746] self.history.append(entry)

[1747] Step 8:

[1748] The terminal draws the contents of the entry onto an image. It creates a new image using the PIL library and draws the contents of the entry onto the image.

[1749] python

[1750] img = Image.new('RGB', (800, 600), color=(255, 255, 255))

[1751] draw = ImageDraw.Draw(img)

[1752] font = ImageFont.load_default()

[1753] draw.text((10, 10), f'Timestamp: {timestamp}', fill=(0, 0, 0), font=font)

[1754] draw.text((10, 30), f'Question: {question}', fill=(0, 0, 0), font=font)

[1755] draw.text((10, 50), f'Answer: {answer}', fill=(0, 0, 0), font=font)

[1756] draw.text((10, 70), f'Emotion: {emotion}', fill=(0, 0, 0), font=font)

[1757] Step 9:

[1758] To ensure that the device gives the image file a unique name, the file name is determined based on the user ID and timestamp.

[1759] python

[1760] img_path = f"{self.user_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"

[1761] Step 10:

[1762] Saves the image generated by the device to the specified path.

[1763] python

[1764] img.save(img_path)

[1765] Step 11:

[1766] The server properly classifies and manages image files linked to user IDs, allowing them to be accessed later.

[1767] Step 12:

[1768] When a user wants to view the history, they send a request from their device to the server. The server then searches for the corresponding image file and returns it to the user. By checking the image file, the user can visually recall the question asked, the answer, and their emotional state at the time.

[1769] Example 2

[1770] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1771] Conventional history management systems only record questions and answers in text format, which means they cannot take into account the user's emotional state, making it difficult to fully grasp the meaning and value of the history. Furthermore, limited means for visually checking the history make it difficult for users to quickly and intuitively restore past questions and answers. Furthermore, there is a lack of ingenuity to allow users to understand each user's history at a glance.

[1772] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1773] In this invention, the server includes means for receiving questions and answers from users and inputting them into a generative AI model to obtain answers, means for inputting the questions and answers into an emotion recognition engine to identify the user's emotion, means for generating entries including a timestamp, the question, the answer, and the recognized emotion, means for saving the generated entries as history, means for rendering the contents of the entries as images, and means for displaying the saved entries as images. This allows emotional information to be recorded in the user's question history, making it possible to check the history in a visual format.

[1774] 1. "User" means an entity that uses the system to enter questions and obtain answers.

[1775] 2. A "question" is text that a user enters into a system requesting information.

[1776] 3. "Answer" is the text response generated by a generative AI model in response to a question.

[1777] 4. "Generative AI model" refers to an algorithm or system that generates answers to user questions.

[1778] 5. "Emotion recognition engine" refers to an algorithm or system that analyzes questions and answers to identify a user's emotions.

[1779] 6. "Timestamp" is information that indicates the date and time a particular entry was created.

[1780] 7. "Entry" refers to a collection of data including a question, an answer, a timestamp, and a recognized emotion.

[1781] 8. "History" refers to a record of questions asked by a user in the past, their answers, and any accompanying information saved in chronological order.

[1782] 9. "Image rendering means" refers to processes and techniques for converting textual and emotional information into image files.

[1783] 10. "Entry saved as an image" means the content of an entry saved in image format.

[1784] 11. "User ID" means identification information that uniquely identifies each user.

[1785] 12. "Visual Verification" means any method or technology that allows a User to view an entry stored as an image via a Terminal.

[1786] This invention relates to a system that receives user questions and answers, saves them as a history, and allows users to view them visually later. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides a function to save the user's emotions in the entry and draw an image that includes those emotions.

[1787] System Overview

[1788] The system includes the following main means:

[1789] 1. User question input and answer acquisition method

[1790] 2. Emotion identification method using an emotion recognition engine

[1791] 3. Entry generation means including timestamp, question, answer, and recognized sentiment

[1792] 4. How to save entries as history

[1793] 5. How to draw the entry content into an image

[1794] 6. How to display entries saved as images

[1795] Hardware and software used

[1796] Generative AI models: For example, open-source language models or cloud-based language model APIs (e.g., GPT-3)

[1797] Emotion recognition engine: For example, natural language processing API (e.g., IBM Watson Natural Language Understanding)

[1798] Image generation software: For example, a Python image processing library (e.g., PIL - Python Imaging Library)

[1799] Database: For example, a relational database (e.g., MySQL, PostgreSQL)

[1800] Specific processing of the program

[1801] The server receives the question entered by the user and sends it to the generative AI model as a prompt. The generative AI model returns an answer to the question, and the answer is received by the server. An example of a prompt is "What is AI?"

[1802] The server then sends the question and answer to an emotion recognition engine to identify the user's emotion. The emotion recognition engine analyzes the text content and the user's input patterns to recognize the emotion, for example, "interest."

[1803] The server adds a timestamp to the question, answer, and recognized emotion, creates a single entry, associates it with the user ID, and stores it in a database.

[1804] The server renders the entry's content (timestamp, question, answer, sentiment) into an image. Specifically, it uses a Python image processing library to convert the text information into an image file. For example, consider the following entry:

[1805] plaintext

[1806] Timestamp: 2023-02-03 12:05:01

[1807] Question: What is AI?

[1808] Answer: AI stands for Artificial Intelligence.

[1809] Emotion: Interest

[1810] The image is saved on the server with a unique name that includes the user ID and a timestamp, for example, "user1234_20230203_120501.png."

[1811] When a user wants to view the history, the server retrieves the image file and displays it on the device, allowing the user to visually confirm the question and the emotional state at that time.

[1812] The above is an embodiment of the present invention. By using this system, it becomes possible to visually and efficiently manage and refer to a user's research history. In particular, by including user emotional information, it is possible to provide a richer history management system.

[1813] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1814] Step 1:

[1815] The user inputs a question through the terminal. The terminal receives this question and temporarily stores the input in memory. This input is used in later processing. For example, the user inputs the question "What is AI?"

[1816] Step 2:

[1817] The device sends a question from the user to the server. The server passes the received question to the generative AI model as a prompt and asks for an answer. The server calls the API of the generative AI model (e.g., GPT-3), sends the question as a prompt, and receives the answer. The input at this time is the user's question, and the output is the answer from the generative AI model.

[1818] Step 3:

[1819] The server receives the answer from the generative AI model and sends it along with the question to the emotion recognition engine. The server calls the API of the emotion recognition engine (e.g., IBM Watson Natural Language Understanding) to analyze the question and answer and identify the user's emotion. The input is the question and answer, and the output is the recognized emotion.

[1820] Step 4:

[1821] The server generates an entry with the question, answer, recognized emotion, and the current date and time. The server obtains the system time and adds it as a timestamp to the entry. This entry contains the question, answer, timestamp, and user emotion. An example of a generated entry is as follows:

[1822] plaintext

[1823] Timestamp: 2023-02-03 12:05:01

[1824] Question: What is AI?

[1825] Answer: AI stands for Artificial Intelligence.

[1826] Emotion: Interest

[1827] Step 5:

[1828] The server saves the generated entries in a database. The database is configured to save entries for each user in chronological order. The input is the generated entries, and the output is the results saved in the database.

[1829] Step 6:

[1830] The server starts the process of rendering the entry content into an image. The server uses image generation software (e.g., PIL) to convert the entry's text information into an image file. The input is the entry, and the output is the image file.

[1831] Step 7:

[1832] The server saves the generated image file with a unique file name that includes the user ID and a timestamp. An example of a saved image file name is "user1234_20230203_120501.png." The input in this case is the image file, and the output is the path to the saved image file.

[1833] Step 8:

[1834] When a user wants to view the history, the device sends a request to the server. The server retrieves the image path of the corresponding entry from the database and sends it to the device. The device receives this and displays it visually to the user. The input in this case is the user request, and the output is the displayed image.

[1835] The above is the specific processing flow of this system, and the data processing and data calculation required at each step will be explained in detail.

[1836] (Application example 2)

[1837] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1838] Providing efficient and effective customer service through proper management of customer interaction history and emotions is a challenge in modern brick-and-mortar stores. Understanding the content of past customer inquiries and their emotions at the time is particularly important for improving customer satisfaction, but many stores lack the means to systematically record and manage this information. Furthermore, appropriate responses are sometimes required even for customers who are not comfortable with direct conversation. Therefore, it is necessary to provide a system that efficiently manages customer interaction history and emotional information and allows for easy visual reference.

[1839] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question and its answer from a user and generating an entry including a timestamp, the question, and the answer, means for saving the generated entry as a history, means for rendering the content of the entry as an image, means for displaying the entry saved as an image, means for identifying the user's emotion, means for adding the identified emotion to the entry, means for saving questions and answers related to specific products, and means for managing the history of customer service in a physical store. This not only makes it possible to effectively manage and visually reference the interaction history and emotion information with customers, but also enables more appropriate and effective customer service by understanding the content of each customer's past inquiries and their emotions at the time.

[1840] "Questions from users" are inquiries provided to the system by users.

[1841] An "answer" is a response provided by the system to a user's question.

[1842] The "timestamp" is information about the date and time when the entry was created.

[1843] An "entry" is a record that combines a question, an answer, a timestamp, and emotional information.

[1844] A "history" is a collection of entries created in the past.

[1845] The "means for rendering to an image" is a method for converting the contents of an entry into an image format that can be visually displayed.

[1846] The "means for displaying" is a method for displaying entries saved in image format so that the user can visually confirm them.

[1847] The "means for identifying user emotions" is a method for inferring emotions from the user's questions and answers.

[1848] The "means for adding identified emotions to an entry" is a method for adding emotion information to an entry.

[1849] The "means for storing questions and answers related to specific products" is a method for recording inquiries about specific products and the answers to those inquiries.

[1850] "Means for managing customer service history in physical stores" refers to a method for systematically storing and managing customer service history in physical stores.

[1851] "Means for adding a user ID" refers to a method for adding a unique ID to an entry to identify a user.

[1852] The system of the present invention provides a method for efficiently managing and visually referencing customer interaction history and emotional information in a physical store. This system is realized by the following steps.

[1853] The system receives questions and answers from users and generates an entry containing a timestamp, question, and answer. It then saves the generated entry as a history and renders the contents of the entry as an image. This rendered image can be used for future reference by customers and staff. It also incorporates a means for identifying emotions from the user's input and adding the identified emotions to the entry. It also saves questions and answers related to specific products, allowing for detailed recording and management of product purchase history and inquiry details. It also manages customer service history in physical stores, making it possible to visually check what inquiries a specific customer has made in the past.

[1854] Hardware and Software Use

[1855] This system is primarily composed of software for the server, user devices, and emotion engine. The server plays the primary role of storing and managing data. The user devices (smartphones and customer service robots) function as interfaces that receive input from users. The emotion engine uses natural language processing libraries (e.g., spaCy and NLTK). The Python Pillow library is used to draw text onto images.

[1856] 1. Ask questions and get answers:

[1857] The user device receives questions posed by customers in the physical store via an input interface, and sends them to the server-side generative AI model to obtain the appropriate answer.

[1858] 2. Identifying emotions:

[1859] The server analyzes the text content of the questions and answers and identifies the user's emotions using an emotion engine, which uses natural language processing techniques to infer emotions from the input text.

[1860] 3. Create and save the entry:

[1861] The server compiles the question, answer, timestamp, and identified emotion into a single entry and stores it in chronological order as a history.

[1862] 4. Image generation and display:

[1863] The server renders the entry contents as an image so that they can be visually confirmed. The Python Pillow library is used for rendering. The generated image is saved with a unique name using the user ID and a timestamp. When the user wants to refer to the history, this image is displayed on the user's device.

[1864] Examples of specific examples and prompts

[1865] For example, if a customer in a clothing store asks, "Can I wash this shirt?" and the robot replies, "Yes, it can be washed," the emotion engine will identify a "feeling of relief." The resulting entry would look like this:

[1866] entry:

[1867] plaintext

[1868] Timestamp: 2023-10-01 12:00:00

[1869] Question: Can this shirt be washed?

[1870] Answer: Yes, it is washable.

[1871] Emotion: A sense of security

[1872] An example prompt is:

[1873] plaintext

[1874] Prompt sentence to be passed to the emotion engine

[1875] Identify the sentiment in the following text:

[1876] Can this shirt be washed?

[1877] A: Yes, it is washable.

[1878] The information recorded in this way is stored visually as an image, providing a detailed customer history including specific questions, answers, and even emotions at the time, enabling more personalized service for each customer and contributing to improved customer satisfaction.

[1879] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1880] Step 1:

[1881] A user inputs a question into a terminal (smartphone or customer service robot) in a physical store. The terminal receives this input and saves it as text data. Specifically, when a user asks, "Can I wash this shirt?", the content is saved in text format on the terminal. Input: User's question (text data), Output: Question saved on the terminal (text data).

[1882] Step 2:

[1883] The device sends a question to the server, which uses the generative AI model to generate an appropriate answer. For example, the device sends the text "Can this shirt be washed?" to the server, and the server uses the generative AI model to obtain the answer "Yes, it can be washed." Input: User's question (text data), Output: Answer from the server (text data).

[1884] Step 3:

[1885] The server passes the captured question and answer text to the emotion engine, which then identifies emotions from this text. Specifically, the emotion engine analyzes the text "Can I wash this shirt?" and "Yes, it can be washed" and identifies the emotion "relief." Input: Question and answer text (text data), Output: Identified emotion (text data).

[1886] Step 4:

[1887] The server compiles the question, answer, timestamp, and identified sentiment into a single entry, for example, the following entry:

[1888] plaintext

[1889] Timestamp: 2023-10-01 12:00:00

[1890] Question: Can this shirt be washed?

[1891] Answer: Yes, it is washable.

[1892] Emotion: A sense of security

[1893] Input: Question, Answer, Timestamp, Sentiment (Text data), Output: Entry (Text data).

[1894] Step 5:

[1895] The server saves the generated entries. The entries are saved in chronological order as history and managed for future reference. For example, the entries are saved in a "history database." Input: entry (text data), output: saved history (database).

[1896] Step 6:

[1897] The server uses the Python Pillow library to render the entry contents into image format. Specifically, it reads the entry's text information, renders it into an image file, and saves it with a unique name. For example, an image file named "user1234_20231001_120000.png" is generated. Input: Entry (text data), Output: Rendered image (image file).

[1898] Step 7:

[1899] When the user wants to view the history, the server displays the saved image on the terminal. This display uses the display function of the terminal. An appropriate user interface is provided so that the user can check the image. For example, the terminal loads the image and displays it on the screen. Input: Saved image (image file), Output: Image displayed on the terminal (visual data).

[1900] In this way, a system is created that manages customer response history and emotional information in physical stores and allows for visual reference.

[1901] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1902] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1903] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1904] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1905] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1906] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1907] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1908] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1909] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1910] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1911] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1912] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1913] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1914] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1915] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1916] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1917] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1918] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1919] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1920] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1921] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1922] The following is further disclosed regarding the above embodiment.

[1923] (Claim 1)

[1924] Receive questions and answers from users,

[1925] means for generating an entry including a timestamp, a question, and an answer;

[1926] a means for storing the generated entries as a history;

[1927] means for rendering the contents of said entry into an image;

[1928] means for displaying the saved entries as images;

[1929] A system including:

[1930] (Claim 2)

[1931] 10. The system of claim 1, further comprising means for attaching a user ID to the entry saved as an image.

[1932] (Claim 3)

[1933] 10. The system of claim 1, wherein the information depicted in the image includes a timestamp, a question, and an answer.

[1934] "Example 1"

[1935] (Claim 1)

[1936] A means for receiving a question and its answer from a user and sending a prompt to a generative AI model to obtain an answer;

[1937] a means for adding a timestamp to the acquired question and answer to generate an entry;

[1938] a means for storing the generated entries as a history;

[1939] means for rendering the contents of said entry into an image;

[1940] a means for saving the rendered image;

[1941] means for displaying the stored image;

[1942] A system including:

[1943] (Claim 2)

[1944] 10. The system of claim 1, further comprising means for attaching user identification information to the entry saved as an image.

[1945] (Claim 3)

[1946] 10. The system of claim 1, wherein the information depicted in the image includes a timestamp, a question, and an answer.

[1947] "Application Example 1"

[1948] (Claim 1)

[1949] Receive questions and answers from users,

[1950] means for generating an entry including a timestamp, a question, and an answer;

[1951] a means for storing the generated entries as a history;

[1952] means for rendering the contents of said entry into an image;

[1953] means for displaying the saved entries as images;

[1954] a means for recording inquiries about product information in the virtual store and responses thereto;

[1955] A means for uniquely naming entries using a user ID and a timestamp and storing them in cloud storage;

[1956] a means for displaying a list of image entries in an album view format;

[1957] A system including:

[1958] (Claim 2)

[1959] 10. The system of claim 1, wherein the system adds a user ID to the entry saved as an image.

[1960] (Claim 3)

[1961] 10. The system of claim 1, wherein the information depicted in the image includes a timestamp, a question, and an answer.

[1962] "Example 2: Combining Emotion Engines"

[1963] (Claim 1)

[1964] A means of receiving user questions and their answers and inputting them into a generative AI model to obtain answers;

[1965] means for inputting the question and answer into an emotion recognition engine to identify the emotion of the user;

[1966] means for generating an entry including a timestamp, a question, an answer, and a recognized emotion;

[1967] a means for storing the generated entries as a history;

[1968] means for rendering the contents of said entry into an image;

[1969] means for displaying the saved entries as images;

[1970] A system including:

[1971] (Claim 2)

[1972] 10. The system of claim 1, further comprising means for attaching a user ID to the entry saved as an image.

[1973] (Claim 3)

[1974] 10. The system of claim 1, wherein the information depicted in the image includes a timestamp, a question, an answer, and a recognized emotion.

[1975] "Application example 2 when combining emotion engines"

[1976] (Claim 1)

[1977] Receive questions and answers from users,

[1978] means for generating an entry including a timestamp, a question, and an answer;

[1979] a means for storing the generated entries as a history;

[1980] means for rendering the contents of said entry into an image;

[1981] means for displaying the saved entries as images;

[1982] means for identifying a user's emotion;

[1983] a means for adding the identified emotion to the entry;

[1984] a means of storing questions and answers related to specific products;

[1985] A means to manage customer service history in physical stores,

[1986] A system including:

[1987] (Claim 2)

[1988] 10. The system of claim 1, further comprising means for attaching a user ID to the entry saved as an image.

[1989] (Claim 3)

[1990] 10. The system of claim 1, wherein the information depicted in the image includes emotion information in addition to a timestamp, a question, and an answer. [Explanation of symbols]

[1991] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Receive questions and answers from users, means for generating an entry including a timestamp, a question, and an answer; a means for storing the generated entries as a history; means for rendering the contents of said entry into an image; means for displaying the saved entries as images; A system including:

2. 10. The system of claim 1, further comprising means for attaching a user ID to the entry saved as an image.

3. 10. The system of claim 1, wherein the information depicted in the image includes a timestamp, a question, and an answer.

Images