system

Abstract

We provide the system. [Solution] An input means for the user to input questions about a product they are considering purchasing, An analysis means that receives input from an input means and analyzes the content of the question using natural language processing, A generation means for generating appropriate answers based on the question content analyzed by the analysis means, A means of presenting the generated answer to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional robot system, it is difficult to respond quickly and appropriately to customers' questions, and there are many problems especially when providing detailed product information. Therefore, customers' doubts and anxieties cannot be resolved quickly, resulting in a decrease in customer satisfaction and purchase intention. The present invention aims to solve such problems and realize more effective communication and information provision to customers.

Means for Solving the Problems

[0005] The present invention solves the above problems with a system that includes the following means: an input means for a user to input a question about a product they are considering purchasing; an analysis means that receives the input from the input means and analyzes the content of the question using natural language processing; a generation means for generating an appropriate answer based on the question content analyzed by the analysis means; and a presentation means for presenting the generated answer to the user. The generation means uses a generative AI model that operates on a cloud-based server, and the presentation means uses a speech synthesis engine to present the answer to the user in voice. This enables quick and appropriate responses to user questions, thereby improving customer satisfaction and purchasing intent.

[0006] A "user" is a person or entity that can input questions to the robot about a product they are considering purchasing.

[0007] "Input method" refers to a device or interface used by a user to input questions about a product, and includes methods such as voice input and text input.

[0008] "Analysis means" refers to a device or software that receives input from an input means, analyzes the content of a question using natural language processing, and extracts key keywords and intents.

[0009] "Generation means" refers to a device or software for generating appropriate answers based on the question content analyzed by the analysis means, and this includes generative AI models that operate on cloud-based servers.

[0010] "Presentation means" refers to a device or interface for providing the user with the answer generated by the generation means, and includes speech output using a speech synthesis engine and text display using a display.

[0011] "Natural language processing" refers to all the technologies and methods that enable computers to understand, interpret, and generate human language, including text tokenization and contextual analysis.

[0012] A "generative AI model" is an artificial intelligence (AI) model designed to generate appropriate answers to user questions, and it operates using machine learning and deep learning techniques.

[0013] A "cloud-based server" is a distributed server infrastructure accessible via the internet, used to host and run generative AI models.

[0014] A "speech synthesis engine" is a software or hardware component that converts text data into speech data and has the function of presenting answers to the user in voice. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be described.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a robotic system for providing product information quickly and appropriately according to user needs. This system has a series of processes for receiving user questions, generating appropriate answers, and presenting them. A detailed explanation follows below.

[0037] Program processing

[0038] Enter your question

[0039] Users ask questions to the system verbally or in text. For example, they might type a question like, "What is the battery capacity of the latest smartphones?"

[0040] Receiving and recognizing questions

[0041] The device receives user voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the speech into text. This text-based question is then sent to the next step.

[0042] Question analysis

[0043] The device receives a question in text format and analyzes it using a natural language processing (NLP) engine. The NLP engine tokenizes the question and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[0044] Generating an answer

[0045] The device sends the analyzed question to a cloud-based server. A generative AI model located on the server generates an appropriate answer based on the analysis results. For example, it might generate the answer, "The battery capacity of the latest smartphones is 4000mAh."

[0046] Providing an answer

[0047] The device uses a speech synthesis engine to present the user with the responses received from the generative AI model. The responses are provided to the user in audio format, and also in text format if necessary.

[0048] Specific example

[0049] Scenario 1: When a user asks about the battery capacity of a new smartphone

[0050] 1. User: "What is the battery capacity of the latest smartphones?"

[0051] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0052] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0053] 4. The terminal sends the analysis results to the server.

[0054] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[0055] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0056] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[0057] Scenario 2: When a user asks about the price of a smartphone

[0058] 1. User: "How much does this smartphone cost?"

[0059] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0060] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0061] 4. The terminal sends the analysis results to the server.

[0062] 5. The generative AI model on the server generates the response, "The price of this smartphone is 79,800 yen."

[0063] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0064] 7. Device: Responds to the user with "The price of this smartphone is 79,800 yen."

[0065] As described above, the present invention can provide accurate and immediate answers to inquiries, thereby improving user satisfaction. Furthermore, the use of a cloud-based AI model ensures the flexibility and scalability of the system.

[0066] The following describes the processing flow.

[0067] Program processing steps

[0068] Step 1: Enter the question

[0069] 1. The user enters a question into the system, either verbally or in text.

[0070] Example: "What is the battery capacity of the latest smartphones?"

[0071] Step 2: Receiving and recognizing the question

[0072] 2. The device receives the user's voice input and converts it to text using a speech recognition engine.

[0073] The device uses a microphone to capture the user's voice.

[0074] The device uses speech recognition software to generate text data from speech.

[0075] Example output: The text data "What is the battery capacity of the latest smartphones?" is generated.

[0076] Step 3: Analyzing the Question

[0077] 3. The device analyzes the text-based question using a natural language processing (NLP) engine.

[0078] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[0079] The device then analyzes key keywords and context to determine intent.

[0080] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[0081] Step 4: Submitting the analysis results

[0082] 4. The device converts the analysis results into JSON format and sends them to a cloud-based server.

[0083] The terminal packages the analysis results in JSON format.

[0084] The device sends JSON data to the server via the internet.

[0085] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[0086] Step 5: Generating the answer

[0087] 5. The server uses a generative AI model to generate an appropriate response.

[0088] The server receives the transmitted JSON data and executes a generative AI model based on the analysis results.

[0089] Generative AI models search for appropriate data based on the intent of the question and keywords, and generate answers.

[0090] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[0091] Step 6: Receiving responses and converting them to speech

[0092] 6. The terminal receives the response from the server and converts it into speech using a speech synthesis engine.

[0093] The device receives the response data in JSON format.

[0094] The device uses a speech synthesis engine to convert text data into speech data.

[0095] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[0096] Step 7: Present your answer

[0097] 7. The device provides the user with an answer via voice or text.

[0098] The device plays audio data generated by a speech synthesis engine.

[0099] If necessary, the response will also be displayed in text format on the screen.

[0100] Example: The voice response will say, "The latest smartphones have a battery capacity of 4000mAh."

[0101] Specific example

[0102] Scenario: A user asks a question about their smartphone's battery capacity.

[0103] 1. Step 1:

[0104] User: "What is the battery capacity of the latest smartphones?"

[0105] 2. Step 2:

[0106] The device receives the audio and converts it to text.

[0107] Example output: Text "What is the battery capacity of the latest smartphones?"

[0108] 3. Step 3:

[0109] The device analyzes the text using an NLP engine.

[0110] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[0111] 4. Step 4:

[0112] The terminal converts the analysis results into JSON format and sends them to the server.

[0113] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[0114] 5. Step 5:

[0115] The server generates the answer using a generative AI model.

[0116] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[0117] 6. Step 6:

[0118] The device receives the response and converts it into speech using a speech synthesis engine.

[0119] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[0120] 7. Step 7:

[0121] The device provides the user with the answer via voice.

[0122] Example: "The latest smartphones have a battery capacity of 4000mAh."

[0123] (Example 1)

[0124] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0125] Conventional product inquiry systems have struggled to provide timely and appropriate answers to user questions. This resulted in delays in responses and decreased user satisfaction. Furthermore, insufficient accuracy in speech recognition and natural language processing technologies made it difficult to accurately understand questions and generate appropriate answers.

[0126] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0127] In this invention, the server includes an input means for inputting a question about a product the user is considering purchasing; a conversion means for receiving the input from the input means and converting the speech to text using a speech recognition engine; an analysis means for analyzing the converted text using a natural language processing engine and understanding the content of the question; a generation means for generating an appropriate answer based on the question content analyzed by the analysis means; and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This makes it possible to accurately understand the user's questions in voice or text format and provide a quick and appropriate answer.

[0128] An "input method" is a means by which a user can input questions about a product.

[0129] "Conversion means" refers to means including a speech recognition engine used to convert the user's voice input into text.

[0130] "Analysis means" refers to a method of analyzing text-based questions using a natural language processing engine, understanding the content of the questions, and extracting important keywords and context.

[0131] "Generation means" refers to means including an artificial intelligence model used to generate appropriate answers based on the analyzed question content.

[0132] "Presentation means" refers to means including a speech synthesis engine used to provide the generated response to the user.

[0133] A "cloud-based server" is a server located remotely on the internet that provides services such as computing and data storage.

[0134] A "natural language processing engine" is a software component used to understand and analyze natural language text.

[0135] A "generative artificial intelligence model" is an artificial intelligence algorithm that can generate natural language text based on a given input.

[0136] This invention is a system that quickly and appropriately processes and provides answers to questions from users regarding products they are considering purchasing. Its main components are input means, conversion means, analysis means, generation means, and presentation means. These components and their operation are described in detail below.

[0137] This system begins with the user entering a question about the product via an input device. The user can enter the question in either voice or text format.

[0138] If voice input is selected, the terminal's conversion mechanism uses a speech recognition engine to convert the speech to text. This system uses a common engine widely used as a "speech recognition engine," for example. This conversion process converts the voice signal into a corresponding string of characters.

[0139] Next, the device uses a natural language processing engine as an analysis tool to analyze the converted text. Specifically, a natural language processing engine such as Hugging Face is used to tokenize the text and extract key keywords and context. For example, if the question is "What is the battery capacity of the latest smartphones?", keywords such as "smartphone" and "battery capacity" will be extracted.

[0140] The extracted keywords and contextual information are sent to a cloud-based server. A generative AI model runs on the server; for example, a generative artificial intelligence model such as OpenAI's GPT-3® is used. This model generates an appropriate response based on the analysis results. The generated response is then sent from the server to the terminal.

[0141] The device that receives the response uses a speech synthesis engine as a presentation method to present the response to the user in audio format. By using a speech synthesis engine such as Amazon Polly, the text-based response is converted into audio and provided to the user. The response can also be presented in text format if necessary.

[0142] Specific example

[0143] If a user asks about the battery capacity of a new smartphone, the process would include the following:

[0144] 1. User: "What is the battery capacity of the latest smartphones?"

[0145] 2. The device receives the question and converts it to text using its speech recognition engine.

[0146] 3. The device uses a natural language processing engine to analyze the question and extract key keywords.

[0147] 4. Send the analysis results to a cloud-based server.

[0148] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[0149] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0150] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[0151] Example of a prompt

[0152] A prompt is a text-based question that is input into a generative artificial intelligence model. For example:

[0153] A user is asking, "What is the battery capacity of the latest smartphones?" Please tell me about the battery capacity of the latest smartphones.

[0154] In this way, implementing this system makes it possible to provide quick and accurate answers to user questions. This improves user satisfaction and ensures the system's flexibility and scalability.

[0155] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0156] Step 1:

[0157] Users enter questions about products they are considering purchasing via input methods. These input methods include voice input and text input. For example, if a user asks, "What is the battery capacity of the latest smartphones?", they can enter the question via voice. Input at this stage can be in either voice or text format.

[0158] Step 2:

[0159] The device receives voice input from the user. The input voice data is converted into text using a speech recognition engine. Here, the voice signal is processed as digital data, and the speech recognition engine analyzes that data to generate a corresponding string of characters. This conversion process generates a text-based question such as, "What is the battery capacity of the latest smartphones?"

[0160] Step 3:

[0161] The device passes the converted text-formatted question to a natural language processing (NLP) engine. The NLP engine tokenizes the text and extracts key keywords and context. For example, keywords such as "latest," "smartphone," and "battery capacity" are extracted and processed along with their contextual information. Here, the input is a text-formatted question, and the output is the keywords and contextual information as a result of the analysis.

[0162] Step 4:

[0163] The device sends the analyzed results to a cloud-based server. The data sent to the server at this stage includes extracted keywords and contextual information. For example, keywords such as "smartphone" and "battery capacity" are sent to the server.

[0164] Step 5:

[0165] The server generates an answer using a generative artificial intelligence model based on the received analysis results. Specifically, the server uses a GPT-based generative AI model to generate prompt sentences. For example, a prompt sentence might be generated that says, "The user is asking, 'What is the battery capacity of the latest smartphones?' Please tell me about the battery capacity of the latest smartphones." Based on this prompt sentence, the AI ​​model generates the answer, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is keywords and contextual information as analysis results, and the output is the answer in text format.

[0166] Step 6:

[0167] The device uses a speech synthesis engine to present the response received from the server. Specifically, it converts the text-based response into audio data. The speech synthesis engine analyzes the text data and generates appropriate speech based on its content. Through this process, the user can receive an audio response such as, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is the text-based response, and the output is the audio response.

[0168] Step 7:

[0169] The user reviews the response provided in either audio or text format. The device plays the audio generated by the speech synthesis engine through its speaker. It can also display the response in text format if needed. This allows the user to obtain information quickly and accurately.

[0170] (Application Example 1)

[0171] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0172] Modern brick-and-mortar stores require prompt and accurate provision of a wide range of product information. However, traditional systems often struggled to respond to customer inquiries in a timely manner, leading to decreased customer satisfaction. Furthermore, employees sometimes had to spend time verifying information, resulting in reduced operational efficiency. Against this backdrop, there is a need for a system that can provide customers with product information in real time.

[0173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0174] In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, an analysis means for receiving input from the input means and analyzing the content of the questions using natural language processing, a generation means for generating appropriate answers based on the question content analyzed by the analysis means, a presentation means for presenting the generated answers to the user, a dialogue means for interacting with the user in a physical store, and a guidance means for the dialogue means to provide the user with information about the location and ingredients of the products. As a result, users can obtain quick and accurate product information in physical stores, which is expected to improve customer satisfaction and operational efficiency.

[0175] An "input method" refers to a device or interface for users to input questions about a product they are considering purchasing.

[0176] "Analysis means" refers to a device or software that receives input from an input means and analyzes the content of a question using natural language processing.

[0177] "Generation means" refers to a device or software that generates appropriate answers based on the question content analyzed by the analysis means.

[0178] "Presentation means" refers to a device or software that presents the generated answer to the user.

[0179] "Dialogue means" refers to devices and software used to interact with users within a physical store.

[0180] A "guidance means" refers to a device or software that provides the user with information about the location and ingredients of a product through dialogue means.

[0181] A "cloud-based server" is a remotely located server that can be accessed via the internet.

[0182] A "generative AI model" is an artificial intelligence model that operates on a cloud-based server as a means of generation and generates appropriate answers.

[0183] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[0184] A "speech synthesis engine" is a technology or device that converts text-based responses into speech.

[0185] This invention is a system that allows users to ask questions about products in a physical store and provides them with quick and appropriate answers. The system is configured as follows:

[0186] System Configuration

[0187] Input method:

[0188] This is a method for users to input questions about products they are considering purchasing. It uses an interface that allows for voice or text input.

[0189] Analysis method:

[0190] This system receives input from an input device and analyzes the content of the question using natural language processing. It utilizes natural language processing engines such as Google's Natural Language API.

[0191] Generation means:

[0192] This method generates appropriate answers based on analyzed question content. It utilizes a generative AI model (e.g., OpenAI GPT-4®) installed on a cloud-based server.

[0193] Means of presentation:

[0194] This is a method for presenting generated answers to the user. It uses a speech synthesis engine (e.g., Amazon Polly) to provide the answers to the user in audio or text format.

[0195] Means of communication:

[0196] These are devices and software designed for interacting with users within physical stores. This allows users to obtain real-time information about product locations and ingredients while in the store.

[0197] Means of guidance:

[0198] Interaction means are devices or software that provide users with information about the location and ingredients of a product. This allows users to easily find out where a product is located, its ingredients, and other information.

[0199] Program processing

[0200] The system operates using servers (cloud-based) and terminals (devices in physical stores and users' smartphones). This system functions as follows:

[0201] The server is responsible for question analysis and answer generation. Natural language processing is used for analysis, and the generated answers are produced by a generative AI model.

[0202] The terminal (the user's smartphone or a guidance robot in a physical store) provides the user interface, allowing for question input, presentation of answers, and interaction.

[0203] The speech recognition engine uses the Google Speech-to-Text API to convert user voice input into text.

[0204] The speech synthesis engine uses Amazon Polly to convert the generated text-based responses into speech.

[0205] Specific example

[0206] Example 1: When a user wants to know the battery capacity of a new smartphone

[0207] 1. User: "What is the battery capacity of the latest smartphones?"

[0208] 2. The device receives this question and converts it to text using its speech recognition engine.

[0209] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "smartphone," "battery capacity").

[0210] 4. Based on the analysis results, a generative AI model generates an appropriate answer (e.g., "The latest smartphones have a battery capacity of 4000mAh.").

[0211] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[0212] Example 2: When a user wants to know the location of a specific product

[0213] 1. User: "Where can I find this product?"

[0214] 2. The device receives this question and converts it to text using its speech recognition engine.

[0215] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "product," "location").

[0216] 4. Based on the analysis results, a generative AI model generates an appropriate response (e.g., "This product is on shelf B3 on the second floor.").

[0217] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[0218] Example of a prompt

[0219] "Please tell me the ingredients of the new shampoo."

[0220] "Where can I find this product?"

[0221] "What is the battery capacity of this smartphone?"

[0222] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0223] Step 1:

[0224] Enter questions about the product the user is considering purchasing.

[0225] Specific example: The user uses their smartphone to ask a voice question, "Where can I find this product?"

[0226] Input: Voice input

[0227] Output: Audio data

[0228] Step 2:

[0229] The device receives the user's voice input and uses a speech recognition engine to convert the voice data into text. The Google Speech-to-Text API is used for the speech recognition engine.

[0230] Specific example: Voice input is converted into text data such as "Where can I find this product?".

[0231] Input: Audio data

[0232] Output: Text data

[0233] Step 3:

[0234] The terminal sends text data to the server for analysis.

[0235] Specific example: A text-based question is sent to a server in the cloud.

[0236] Input: Text data

[0237] Output: Text data (sent to the server)

[0238] Step 4:

[0239] The server uses a natural language processing engine to analyze the received questions. The Google Natural Language API is used for this engine to extract key keywords and context.

[0240] Specific example: Extract the main keywords "product" and "location" from the question "Where can I find this product?".

[0241] Input: Text data

[0242] Output: Analysis results (keywords)

[0243] Step 5:

[0244] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate answers based on the analysis results.

[0245] Specific example: The response "This product is located on shelf B3 on the second floor" is generated.

[0246] Input: Analysis results (keywords)

[0247] Output: Generated response (text format)

[0248] Step 6:

[0249] The server sends the generated response to the terminal.

[0250] Specific example: A text response such as "This product is located on shelf B3 on the second floor" is sent to the terminal.

[0251] Input: Generated response (text format)

[0252] Output: Generated response (text format, sent to device)

[0253] Step 7:

[0254] The device converts the received text-based response into speech using a speech synthesis engine (e.g., Amazon Polly).

[0255] Specific example: An audio message is generated saying, "This product is located on shelf B3 on the second floor."

[0256] Input: Generated response (text format)

[0257] Output: Audio data

[0258] Step 8:

[0259] The device presents audio data to the user.

[0260] Specific example: The terminal plays a voice message saying, "This product is located on shelf B3 on the second floor."

[0261] Input: Audio data

[0262] Output: Audio presentation

[0263] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0264] This invention relates to a robotic system that provides product information quickly and appropriately according to user needs. In addition to a series of processes that involve receiving user questions, generating appropriate answers, and presenting them, this system also has the function of recognizing the user's emotions and responding accordingly. A detailed explanation follows below.

[0265] Program processing

[0266] Enter your question

[0267] The user enters a question into the system, either verbally or in text. For example, they might enter a question like, "What is the battery capacity of the latest smartphones?"

[0268] Receiving and recognizing questions

[0269] The device receives the user's voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the voice into text. This text-based question is then sent to the next step.

[0270] Question analysis

[0271] The device analyzes text-based questions using a natural language processing (NLP) engine. The NLP engine tokenizes the questions and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[0272] Recognition of emotions

[0273] The device uses an emotion recognition engine to analyze the user's emotions. The emotion recognition engine extracts emotional data from the user's voice and facial expressions, and identifies emotions such as joy, surprise, and anger.

[0274] Generating an answer

[0275] The terminal sends the question and the sentiment analysis result to a cloud-based server. The generative AI model deployed on the server generates an appropriate answer based on the analysis result. For example, when the user is feeling anxious, an answer that reassures the user is generated. An answer such as "The battery capacity of the latest smartphone is 4000 mAh. Please rest assured, it lasts very long." is generated.

[0276] Presentation of the answer

[0277] The terminal uses a text-to-speech engine to present the answer received from the generative AI model to the user. The answer is provided to the user in audio format and, if necessary, also in text format.

[0278] Specific example

[0279] Scenario 1: When the user asks about the battery capacity of a new smartphone and is feeling anxious

[0280] 1. User: "What is the battery capacity of the latest smartphone?" [[ID=二十]]

[0281] 2. The terminal receives the question and converts it to text format using a speech recognition engine.

[0282] 3. The terminal uses an NLP engine to analyze the question and extract the main keywords.

[0283] 4. The terminal uses a sentiment recognition engine to analyze that the user is feeling anxious.

[0284] 5. The terminal sends the analysis result to the server.

[0285] 6. The generative AI model on the server generates a reassuring answer such as "The battery capacity of the latest smartphone is 4000 mAh. Please rest assured, it lasts very long."

[0286] 7. The terminal receives the answer and converts it into voice using a text-to-speech engine.

[0287] 8. The terminal replies to the user: "The battery capacity of the latest smartphone is 4000 mAh. Please rest assured as it has a very long battery life."

[0288] Scenario 2: When the user asks about the price of a smartphone and is excited

[0289] 1. User: "How much does this smartphone cost?"

[0290] 2. The terminal receives the question and converts it into text format using a speech recognition engine.

[0291] 3. The terminal uses an NLP engine to analyze the question and extract the main keywords.

[0292] 4. The terminal uses an emotion recognition engine to analyze that the user is excited.

[0293] 5. The terminal sends the analysis result to the server.

[0294] 6. The generative AI model on the server generates a response sharing excitement: "The price of this smartphone is 79,800 yen. It's a great price, please check it out!"

[0295] 7. The terminal receives the answer and converts it into voice using a text-to-speech engine.

[0296] 8. The terminal replies to the user: "The price of this smartphone is 79,800 yen. It's a great price, please check it out!"

[0297] As described above, the present invention not only provides accurate and immediate answers to user questions, but also enables responses that take into account the user's emotions. This can further improve user satisfaction and purchasing desire. By using a cloud-based AI model, the flexibility and scalability of the system are also ensured.

[0298] The following describes the processing flow.

[0299] Processing steps of the program

[0300] Step 1: Input of question

[0301] 1. The user inputs a question to the system verbally or in text.

[0302] Example: "What is the battery capacity of the latest smartphone?"

[0303] Step 2: Reception and recognition of question

[0304] 2. The terminal receives the user's voice input and converts it into text using a speech recognition engine.

[0305] The terminal captures the user's voice using a microphone.

[0306] The terminal generates text data from the voice using speech recognition software.

[0307] Output example: Text data "What is the battery capacity of the latest smartphone?" is generated.

[0308] Step 3: Analysis of question

[0309] 3. The terminal analyzes the text-formatted question using a natural language processing (NLP) engine.

[0310] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[0311] The device then analyzes key keywords and context to determine intent.

[0312] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[0313] Step 4: Recognizing Emotions

[0314] 4. The device uses an emotion recognition engine to analyze the user's emotions.

[0315] The device extracts emotional data from the user's voice tone, pace, and facial expressions.

[0316] The device identifies the category of emotion (e.g., joy, surprise, anger, anxiety).

[0317] Example output: The emotion "anxiety" is recognized.

[0318] Step 5: Sending analysis results and sentiment data

[0319] 5. The device converts the question analysis results and sentiment analysis results into JSON format and sends them to a cloud-based server.

[0320] The terminal packages the question analysis and sentiment analysis results into a single JSON file.

[0321] The device sends JSON data to the server via the internet.

[0322] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[0323] Step 6: Generating the answer

[0324] 6. The server uses a generative AI model to generate an appropriate response.

[0325] The server receives the transmitted JSON data and runs a generative AI model based on the question and sentiment analysis results.

[0326] Generative AI models generate responses that match the user's emotions (e.g., reassuring responses for users who are feeling anxious).

[0327] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[0328] Step 7: Receiving responses and converting them to audio.

[0329] 7. The device receives the response from the server and converts it into speech using a speech synthesis engine.

[0330] The device receives the response data in JSON format.

[0331] The device uses a speech synthesis engine to convert text data into speech data.

[0332] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[0333] Step 8: Present your answer

[0334] 8. The device provides the user with an answer via voice or text.

[0335] The device plays audio data generated by a speech synthesis engine.

[0336] If necessary, the response will also be displayed in text format on the screen.

[0337] Example: The voice response might say, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[0338] Specific example

[0339] Scenario: When a user asks about their smartphone's battery capacity and expresses concern.

[0340] Step 1:

[0341] User: "What is the battery capacity of the latest smartphones?"

[0342] Step 2:

[0343] The device receives the audio and converts it to text.

[0344] Example output: Text "What is the battery capacity of the latest smartphones?"

[0345] Step 3:

[0346] The device analyzes the text using an NLP engine.

[0347] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[0348] Step 4:

[0349] The device analyzes the user's emotions using its emotion recognition engine.

[0350] Example output: The emotion "anxiety" is recognized.

[0351] Step 5:

[0352] The device sends the analysis results and sentiment analysis results to the server in JSON format.

[0353] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[0354] Step 6:

[0355] The server generates the answer using a generative AI model.

[0356] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[0357] Step 7:

[0358] The device receives the response and converts it into speech using a speech synthesis engine.

[0359] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[0360] Step 8:

[0361] The device provides the user with the answer via voice.

[0362] Example: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[0363] (Example 2)

[0364] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0365] Modern consumers need to obtain increasingly diverse product information quickly and accurately. Furthermore, traditional product information systems often provide uniform answers without considering user emotions or intentions, which is insufficient for improving user satisfaction. This has led to challenges such as decreased purchase intent and a decline in the quality of the user experience.

[0366] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, a conversion means for receiving input from the input means and converting the voice data into text format using a speech recognition engine, an analysis means for analyzing the converted text data using a natural language processing engine and extracting key keywords, an emotion recognition means for recognizing the user's emotional state, a generation means for generating an appropriate answer based on the data obtained by the analysis means and the emotion recognition means, and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This enables the rapid provision of an appropriate answer that meets the user's needs and also allows for a response that takes the user's emotions into consideration.

[0367] "Input means" refers to a device or method that provides an interface for a user to input a question.

[0368] "Conversion means" refers to a device or method used to convert audio data received from an input means into text format.

[0369] "Analysis means" refers to a device or method that analyzes text data obtained by the conversion means using natural language processing techniques to extract key keywords and context.

[0370] "Emotion recognition means" refers to a device or method that analyzes a user's voice and facial expression data to determine their emotional state.

[0371] "Generation means" refers to a device or method that generates an appropriate response using a generative AI model on the cloud, based on data obtained by analysis means and emotion recognition means.

[0372] "Presentation means" refers to a device or method used to present the generated response to the user, and includes providing the response in voice format using a speech synthesis engine.

[0373] A "generative AI model" is an artificial intelligence model used to generate appropriate text or information from input data.

[0374] A "prompt" is an instruction given to a generative AI model, and it serves as the basis for determining the content of the generated response.

[0375] This invention relates to a system that allows users to input questions about products they are considering purchasing and provides appropriate answers based on those questions. This system has the function of recognizing the user's emotional state and generating corresponding answers. The embodiments of the system are described in detail below.

[0376] System Configuration

[0377] This system consists of the following main components:

[0378] 1. Input Methods: This is the interface for the user to input questions. This includes microphones that support voice input and keyboards that support text input.

[0379] 2. Conversion method: In the case of voice input, a speech recognition engine is used as the conversion method. In this embodiment, the Google Speech-to-Text API is used to convert speech to text.

[0380] 3. Analysis Method: The converted text is analyzed using a natural language processing (NLP) engine. For example, Amazon Comprehend is used to extract the main keywords and intent of the question.

[0381] 4. Emotion Recognition Method: Emotional data is analyzed from the user's voice and facial expressions. The user's emotional state (joy, anxiety, excitement, etc.) is determined using the Microsoft® Azure® Emotion API.

[0382] 5. Generation means: Based on the data obtained by the analysis means and the emotion recognition means, a generative AI model deployed on a cloud-based server is used. In this embodiment, OpenAI's GPT-3 generates the response.

[0383] 6. Presentation Method: A speech synthesis engine is used to present the generated responses to the user. Amazon Polly is used to provide the generated text responses in audio format.

[0384] Implementation method

[0385] The system operates as follows: When a user enters a question, the input device receives it. In the case of voice input, the device converts the speech to text using the Google Speech-to-Text API. The converted text is then analyzed by the analysis device using Amazon Comprehend. Here, key keywords and the intent of the question are extracted. In parallel, the emotion recognition device analyzes the emotional state from the voice data and the user's facial expressions using the Microsoft Azure Emotion API.

[0386] The analyzed data and emotional state are sent to a cloud-based server, where appropriate responses are generated using OpenAI's GPT-3. The generated responses are returned to the device and converted to audio format using Amazon Polly. Finally, the responses are presented to the user in both audio and text formats.

[0387] Specific example

[0388] Scenario 1: When a user asks about the battery capacity of their new smartphone and expresses concern.

[0389] 1. User: "What is the battery capacity of the latest smartphones?"

[0390] 2. The device receives the question and converts it to text format using the Google Speech-to-Text API.

[0391] 3. The device uses Amazon Comprehend to analyze the questions and extract key keywords.

[0392] 4. The device uses the Microsoft Azure Emotion API to analyze if the user is feeling anxious.

[0393] 5. The device sends the analysis results to the cloud server.

[0394] 6. The generative AI model (OpenAI GPT-3) on the server generates the response: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[0395] 7. The device receives the response and converts it to speech using Amazon Polly.

[0396] 8. Device: Responds to the user with, "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[0397] Example of a prompt

[0398] "Please tell me about the battery capacity of this smartphone. Generate an answer that will reassure users who are feeling anxious."

[0399] The system, possessing the above characteristics, can quickly provide appropriate answers tailored to user needs and even respond in a way that considers user emotions. This can improve user satisfaction and purchasing intent.

[0400] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0401] Step 1: Enter the question

[0402] The user enters a question. The question can be entered via voice or text. For example, consider the question, "What is the battery capacity of the latest smartphones?" This input data is obtained as voice data if entered via voice, or directly as text data if entered via text.

[0403] Step 2: Receiving the question and speech recognition

[0404] The device receives user input. In the case of voice input, the Google Speech-to-Text API is used to convert the speech to text. The input is audio data, and the output is a question in text format. Specifically, the device's microphone captures the audio, and the API is called to convert the audio into text data.

[0405] Step 3: Question analysis and tokenization

[0406] The device receives a text-based question and analyzes it using a natural language processing engine such as Amazon Comprehend. The input is a text-based question, and the output is a list of tokenized words and phrases. Specifically, the NLP engine breaks down the text into words and phrases and tokenizes them into categories such as "latest," "smartphone," and "battery capacity."

[0407] Step 4: Keyword Extraction and Intent Interpretation

[0408] The device extracts key keywords from the tokenized data and interprets the intent of the question. The input is the tokenized data obtained in step 3, and the output is the key keywords and their related information. Specifically, the NLP engine scans the tokens and recognizes that "smartphone" is a product category and "battery capacity" is an attribute.

[0409] Step 5: Recognizing and analyzing emotions

[0410] The device uses an emotion recognition engine, such as the Microsoft Azure Emotion API, to analyze the user's emotions. Input is voice or facial expression data, and output is the emotional state (e.g., joy, anxiety, excitement). Specifically, in the case of voice, emotions are analyzed from the tone and speed of the voice, and in the case of facial expressions, emotions are analyzed from images captured by the camera.

[0411] Step 6: Generating and optimizing answers

[0412] The device sends analysis results and sentiment data to a cloud-based server. On the server, generative AI models such as OpenAI's GPT-3 generate responses. The input consists of analyzed keywords and sentiment states, and the output is a response in appropriate text format. Specifically, data is sent to the server, and the generative AI model generates a response based on prompts.

[0413] Step 7: Presentation of answers and speech synthesis

[0414] The device uses a speech synthesis engine such as Amazon Polly to present the user with the response received from the generative AI model. The input is a text-based response, and the output is a voice-based response. Specifically, the speech synthesis engine converts the text response into a voice file, which is then played through the device's speaker, or the text-based response is displayed on the screen.

[0415] (Application Example 2)

[0416] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0417] In modern brick-and-mortar stores, it is crucial to respond quickly and appropriately to customers' questions about products they are considering purchasing. However, traditional systems only provide information in response to each customer's question, failing to address their emotions and needs. As a result, the quality and satisfaction of the information customers receive are limited, making it difficult to increase their purchasing intent. Furthermore, staff shortages and the difficulty of multilingual support among staff are also challenges. To solve these problems, a system is needed that recognizes customer emotions and provides appropriate answers based on those emotions.

[0418] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an input means for inputting a question about a product that the user is considering purchasing, an analysis means for receiving the input from the input means and analyzing the content of the question using natural language processing, a generation means for generating an appropriate answer based on the question content analyzed by the analysis means, an emotion recognition means for recognizing the user's emotions, an answer adjustment means for adjusting the answer generated based on the emotions recognized by the emotion recognition means, and a presentation means for presenting the generated and adjusted answer to the user. This makes it possible to quickly provide appropriate product information that corresponds to the customer's emotions, thereby increasing customer satisfaction and purchasing intent.

[0419] "Input means" refers to a device or function that allows a user to input questions or information into a system.

[0420] "Analysis means" refers to a function that understands and analyzes questions and information received from input means using natural language processing technology.

[0421] "Generation means" refers to a function for creating an appropriate response based on the information analyzed by the analysis means.

[0422] "Emotion recognition means" refers to a function that identifies and analyzes a user's emotions from their voice, facial expressions, tone of voice, etc.

[0423] A "response adjustment mechanism" is a function that adjusts responses generated based on recognized emotions and provides them in an appropriate format.

[0424] "Presentation means" refers to a function for presenting generated and adjusted responses to the user in audio or text format.

[0425] A "generative AI model" is an artificial intelligence model that generates appropriate responses or information based on input data in natural language processing and generative tasks.

[0426] A "cloud-based server" is a system that performs data processing and storage on remote servers accessible via the internet.

[0427] A "speech synthesis engine" is a technology that converts text data into speech data and outputs it as natural-sounding speech.

[0428] This invention is a system that provides product information quickly and appropriately in response to customer inquiries in physical stores. The following describes specific implementations of this system.

[0429] System Configuration

[0430] The system includes the following main components:

[0431] An "input method" for users to enter questions.

[0432] "Analysis means" for analyzing received questions

[0433] A "generation means" for generating answers based on the content of the questions.

[0434] "Means for recognizing user emotions"

[0435] "Response adjustment means" for adjusting responses based on perceived emotions.

[0436] A "presentation method" for presenting adjusted answers to users.

[0437] This section describes the specific functions of each component and the hardware and software that support them.

[0438] Hardware and software details

[0439] 1. Input method:

[0440] This includes a microphone for users to input questions verbally, as well as a touch panel.

[0441] 2. Analysis method:

[0442] A speech recognition engine for converting audio data into text (e.g., Google Cloud Speech-to-Text).

[0443] A natural language processing engine (e.g., Amazon Comprehend) that analyzes the converted text data and extracts keywords and context.

[0444] 3. Generation means:

[0445] A generative AI model (e.g., OpenAI GPT-4) that runs on a cloud-based server and generates appropriate answers based on the question.

[0446] 4. Emotion recognition means:

[0447] An emotion recognition engine (e.g., Microsoft Azure Emotional API) analyzes a user's voice and facial expressions to recognize their emotions.

[0448] 5. Answer adjustment means:

[0449] The generated response is adjusted based on the emotional data obtained by the emotion recognition system.

[0450] 6. Means of presentation:

[0451] A speech synthesis engine (e.g., Amazon Polly) for presenting adjusted responses to users.

[0452] Processing flow and specific examples

[0453] Specific example 1:

[0454] A scenario where a customer comes into the store and asks about the battery life of their new smartphone.

[0455] User: "What is the battery capacity of the latest smartphones?"

[0456] Voice data is input through the microphone, and a speech recognition engine converts that data into text.

[0457] A natural language processing engine analyzes the text and extracts key keywords.

[0458] The system determines that the user's emotion is "anxiety" based on their recognized voice tone and facial expressions.

[0459] The question content and sentiment data are sent to a generative AI model on a cloud server.

[0460] A generative AI model generates responses, such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[0461] The speech synthesis engine converts the generated response into speech and provides it to the user.

[0462] Example of a prompt:

[0463] "User is anxious. Question: What is the battery capacity of the latest smartphones?"

[0464] Benefits of the entire system

[0465] This system will significantly improve customer service in physical stores. In particular, it is expected to increase customer satisfaction and purchasing intent by quickly providing optimal answers tailored to each customer's emotions in response to their questions. The use of a cloud-based AI model ensures flexibility and scalability, making it suitable for use in a variety of physical stores.

[0466] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0467] Step 1:

[0468] The user enters a question. If the user enters the question by voice using a microphone, the voice signal becomes the input data. If the user enters the question by text using a touch panel, the text data becomes the input data. This data is received by the terminal.

[0469] Step 2:

[0470] The device uses a speech recognition engine to convert voice input into text data. The Google Cloud Speech-to-Text engine is used to convert the voice signal into text format. This text data is then input into the next step.

[0471] Step 3:

[0472] The device uses a natural language processing (NLP) engine to analyze text data. Amazon Comprehend is used to tokenize the text and extract key keywords and context. This analysis helps understand the intent of the question, and the extracted keywords are then fed into the next step.

[0473] Step 4:

[0474] The device uses an emotion recognition engine to analyze the user's emotional state. It uses the Microsoft Azure Emotional API to obtain emotional data from the user's voice tone and facial expressions. For example, it analyzes voice intonation and changes in facial expressions to determine if the user is experiencing emotions such as "anxiety," "excitement," or "joy." This emotional data is then input into the next step.

[0475] Step 5:

[0476] The device sends the analyzed question data and sentiment data to a cloud-based server. The server uses a generative AI model (OpenAI GPT-4) to generate appropriate responses based on the input data. For example, if the user is feeling "anxious," a reassuring response will be generated. The text data of the generated response is then input into the next step.

[0477] Step 6:

[0478] The device uses a speech synthesis engine to convert the generated response text into audio data. Amazon Polly is used to convert the text-based response into natural-sounding speech. This audio data is then presented to the user.

[0479] Step 7:

[0480] The user receives voice-generated answers. The answers are delivered in audio format via a speaker, which serves as the presentation method. Furthermore, the answers are also displayed in text format on the touchscreen if necessary. This allows the user to obtain answers to their product-related questions and receive information that is relevant to their emotional state.

[0481] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0482] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0483] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0484] [Second Embodiment]

[0485] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0486] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0487] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0488] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0489] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0490] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0491] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0492] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0493] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0494] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0495] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0496] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0497] This invention is a robotic system for providing product information quickly and appropriately according to user needs. This system has a series of processes for receiving user questions, generating appropriate answers, and presenting them. A detailed explanation follows below.

[0498] Program processing

[0499] Enter your question

[0500] Users ask questions to the system verbally or in text. For example, they might type a question like, "What is the battery capacity of the latest smartphones?"

[0501] Receiving and recognizing questions

[0502] The device receives user voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the speech into text. This text-based question is then sent to the next step.

[0503] Question analysis

[0504] The device receives a question in text format and analyzes it using a natural language processing (NLP) engine. The NLP engine tokenizes the question and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[0505] Generating an answer

[0506] The device sends the analyzed question to a cloud-based server. A generative AI model located on the server generates an appropriate answer based on the analysis results. For example, it might generate the answer, "The battery capacity of the latest smartphones is 4000mAh."

[0507] Providing an answer

[0508] The device uses a speech synthesis engine to present the user with the responses received from the generative AI model. The responses are provided to the user in audio format, and also in text format if necessary.

[0509] Specific example

[0510] Scenario 1: When a user asks about the battery capacity of a new smartphone

[0511] 1. User: "What is the battery capacity of the latest smartphones?"

[0512] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0513] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0514] 4. The terminal sends the analysis results to the server.

[0515] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[0516] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0517] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[0518] Scenario 2: When a user asks about the price of a smartphone

[0519] 1. User: "How much does this smartphone cost?"

[0520] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0521] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0522] 4. The terminal sends the analysis results to the server.

[0523] 5. The generative AI model on the server generates the response, "The price of this smartphone is 79,800 yen."

[0524] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0525] 7. Device: Responds to the user with "The price of this smartphone is 79,800 yen."

[0526] As described above, the present invention can provide accurate and immediate answers to inquiries, thereby improving user satisfaction. Furthermore, the use of a cloud-based AI model ensures the flexibility and scalability of the system.

[0527] The following describes the processing flow.

[0528] Program processing steps

[0529] Step 1: Enter the question

[0530] 1. The user enters a question into the system, either verbally or in text.

[0531] Example: "What is the battery capacity of the latest smartphones?"

[0532] Step 2: Receiving and recognizing the question

[0533] 2. The device receives the user's voice input and converts it to text using a speech recognition engine.

[0534] The device uses a microphone to capture the user's voice.

[0535] The device uses speech recognition software to generate text data from speech.

[0536] Example output: The text data "What is the battery capacity of the latest smartphones?" is generated.

[0537] Step 3: Analyzing the Question

[0538] 3. The device analyzes the text-based question using a natural language processing (NLP) engine.

[0539] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[0540] The device then analyzes key keywords and context to determine intent.

[0541] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[0542] Step 4: Submitting the analysis results

[0543] 4. The device converts the analysis results into JSON format and sends them to a cloud-based server.

[0544] The terminal packages the analysis results in JSON format.

[0545] The device sends JSON data to the server via the internet.

[0546] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[0547] Step 5: Generating the answer

[0548] 5. The server uses a generative AI model to generate an appropriate response.

[0549] The server receives the transmitted JSON data and executes a generative AI model based on the analysis results.

[0550] Generative AI models search for appropriate data based on the intent of the question and keywords, and generate answers.

[0551] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[0552] Step 6: Receiving responses and converting them to speech

[0553] 6. The terminal receives the response from the server and converts it into speech using a speech synthesis engine.

[0554] The device receives the response data in JSON format.

[0555] The device uses a speech synthesis engine to convert text data into speech data.

[0556] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[0557] Step 7: Present your answer

[0558] 7. The device provides the user with an answer via voice or text.

[0559] The device plays audio data generated by a speech synthesis engine.

[0560] If necessary, the response will also be displayed in text format on the screen.

[0561] Example: The voice response will say, "The latest smartphones have a battery capacity of 4000mAh."

[0562] Specific example

[0563] Scenario: A user asks a question about their smartphone's battery capacity.

[0564] 1. Step 1:

[0565] User: "What is the battery capacity of the latest smartphones?"

[0566] 2. Step 2:

[0567] The device receives the audio and converts it to text.

[0568] Example output: Text "What is the battery capacity of the latest smartphones?"

[0569] 3. Step 3:

[0570] The device analyzes the text using an NLP engine.

[0571] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[0572] 4. Step 4:

[0573] The terminal converts the analysis results into JSON format and sends them to the server.

[0574] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[0575] 5. Step 5:

[0576] The server generates the answer using a generative AI model.

[0577] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[0578] 6. Step 6:

[0579] The device receives the response and converts it into speech using a speech synthesis engine.

[0580] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[0581] 7. Step 7:

[0582] The device provides the user with the answer via voice.

[0583] Example: "The latest smartphones have a battery capacity of 4000mAh."

[0584] (Example 1)

[0585] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0586] Conventional product inquiry systems have struggled to provide timely and appropriate answers to user questions. This resulted in delays in responses and decreased user satisfaction. Furthermore, insufficient accuracy in speech recognition and natural language processing technologies made it difficult to accurately understand questions and generate appropriate answers.

[0587] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0588] In this invention, the server includes an input means for inputting a question about a product the user is considering purchasing; a conversion means for receiving the input from the input means and converting the speech to text using a speech recognition engine; an analysis means for analyzing the converted text using a natural language processing engine and understanding the content of the question; a generation means for generating an appropriate answer based on the question content analyzed by the analysis means; and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This makes it possible to accurately understand the user's questions in voice or text format and provide a quick and appropriate answer.

[0589] An "input method" is a means by which a user can input questions about a product.

[0590] "Conversion means" refers to means including a speech recognition engine used to convert the user's voice input into text.

[0591] "Analysis means" refers to a method of analyzing text-based questions using a natural language processing engine, understanding the content of the questions, and extracting important keywords and context.

[0592] "Generation means" refers to means including an artificial intelligence model used to generate appropriate answers based on the analyzed question content.

[0593] "Presentation means" refers to means including a speech synthesis engine used to provide the generated response to the user.

[0594] A "cloud-based server" is a server located remotely on the internet that provides services such as computing and data storage.

[0595] A "natural language processing engine" is a software component used to understand and analyze natural language text.

[0596] A "generative artificial intelligence model" is an artificial intelligence algorithm that can generate natural language text based on a given input.

[0597] This invention is a system that quickly and appropriately processes and provides answers to questions from users regarding products they are considering purchasing. Its main components are input means, conversion means, analysis means, generation means, and presentation means. These components and their operation are described in detail below.

[0598] This system begins with the user entering a question about the product via an input device. The user can enter the question in either voice or text format.

[0599] If voice input is selected, the terminal's conversion mechanism uses a speech recognition engine to convert the speech to text. This system uses a common engine widely used as a "speech recognition engine," for example. This conversion process converts the voice signal into a corresponding string of characters.

[0600] Next, the device uses a natural language processing engine as an analysis tool to analyze the converted text. Specifically, a natural language processing engine such as Hugging Face is used to tokenize the text and extract key keywords and context. For example, if the question is "What is the battery capacity of the latest smartphones?", keywords such as "smartphone" and "battery capacity" will be extracted.

[0601] The extracted keywords and contextual information are sent to a cloud-based server. A generative AI model runs on the server; for example, OpenAI's GPT-3 is used. This model generates an appropriate response based on the analysis results. The generated response is then sent from the server to the terminal.

[0602] The device that receives the response uses a speech synthesis engine as a presentation method to present the response to the user in audio format. By using a speech synthesis engine such as Amazon Polly, the text-based response is converted into audio and provided to the user. The response can also be presented in text format if necessary.

[0603] Specific example

[0604] If a user asks about the battery capacity of a new smartphone, the process would include the following:

[0605] 1. User: "What is the battery capacity of the latest smartphones?"

[0606] 2. The device receives the question and converts it to text using its speech recognition engine.

[0607] 3. The device uses a natural language processing engine to analyze the question and extract key keywords.

[0608] 4. Send the analysis results to a cloud-based server.

[0609] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[0610] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0611] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[0612] Example of a prompt

[0613] A prompt is a text-based question that is input into a generative artificial intelligence model. For example:

[0614] A user is asking, "What is the battery capacity of the latest smartphones?" Please tell me about the battery capacity of the latest smartphones.

[0615] In this way, implementing this system makes it possible to provide quick and accurate answers to user questions. This improves user satisfaction and ensures the system's flexibility and scalability.

[0616] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0617] Step 1:

[0618] Users enter questions about products they are considering purchasing via input methods. These input methods include voice input and text input. For example, if a user asks, "What is the battery capacity of the latest smartphones?", they can enter the question via voice. Input at this stage can be in either voice or text format.

[0619] Step 2:

[0620] The device receives voice input from the user. The input voice data is converted into text using a speech recognition engine. Here, the voice signal is processed as digital data, and the speech recognition engine analyzes that data to generate a corresponding string of characters. This conversion process generates a text-based question such as, "What is the battery capacity of the latest smartphones?"

[0621] Step 3:

[0622] The device passes the converted text-formatted question to a natural language processing (NLP) engine. The NLP engine tokenizes the text and extracts key keywords and context. For example, keywords such as "latest," "smartphone," and "battery capacity" are extracted and processed along with their contextual information. Here, the input is a text-formatted question, and the output is the keywords and contextual information as a result of the analysis.

[0623] Step 4:

[0624] The device sends the analyzed results to a cloud-based server. The data sent to the server at this stage includes extracted keywords and contextual information. For example, keywords such as "smartphone" and "battery capacity" are sent to the server.

[0625] Step 5:

[0626] The server generates an answer using a generative artificial intelligence model based on the received analysis results. Specifically, the server uses a GPT-based generative AI model to generate prompt sentences. For example, a prompt sentence might be generated that says, "The user is asking, 'What is the battery capacity of the latest smartphones?' Please tell me about the battery capacity of the latest smartphones." Based on this prompt sentence, the AI ​​model generates the answer, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is keywords and contextual information as analysis results, and the output is the answer in text format.

[0627] Step 6:

[0628] The device uses a speech synthesis engine to present the response received from the server. Specifically, it converts the text-based response into audio data. The speech synthesis engine analyzes the text data and generates appropriate speech based on its content. Through this process, the user can receive an audio response such as, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is the text-based response, and the output is the audio response.

[0629] Step 7:

[0630] The user reviews the response provided in either audio or text format. The device plays the audio generated by the speech synthesis engine through its speaker. It can also display the response in text format if needed. This allows the user to obtain information quickly and accurately.

[0631] (Application Example 1)

[0632] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0633] Modern brick-and-mortar stores require prompt and accurate provision of a wide range of product information. However, traditional systems often struggled to respond to customer inquiries in a timely manner, leading to decreased customer satisfaction. Furthermore, employees sometimes had to spend time verifying information, resulting in reduced operational efficiency. Against this backdrop, there is a need for a system that can provide customers with product information in real time.

[0634] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0635] In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, an analysis means for receiving input from the input means and analyzing the content of the questions using natural language processing, a generation means for generating appropriate answers based on the question content analyzed by the analysis means, a presentation means for presenting the generated answers to the user, a dialogue means for interacting with the user in a physical store, and a guidance means for the dialogue means to provide the user with information about the location and ingredients of the products. As a result, users can obtain quick and accurate product information in physical stores, which is expected to improve customer satisfaction and operational efficiency.

[0636] An "input method" refers to a device or interface for users to input questions about a product they are considering purchasing.

[0637] "Analysis means" refers to a device or software that receives input from an input means and analyzes the content of a question using natural language processing.

[0638] "Generation means" refers to a device or software that generates appropriate answers based on the question content analyzed by the analysis means.

[0639] "Presentation means" refers to a device or software that presents the generated answer to the user.

[0640] "Dialogue means" refers to devices and software used to interact with users within a physical store.

[0641] A "guidance means" refers to a device or software that provides the user with information about the location and ingredients of a product through dialogue means.

[0642] A "cloud-based server" is a remotely located server that can be accessed via the internet.

[0643] A "generative AI model" is an artificial intelligence model that operates on a cloud-based server as a means of generation and generates appropriate answers.

[0644] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[0645] A "speech synthesis engine" is a technology or device that converts text-based responses into speech.

[0646] This invention is a system that allows users to ask questions about products in a physical store and provides them with quick and appropriate answers. The system is configured as follows:

[0647] System Configuration

[0648] Input method:

[0649] This is a method for users to input questions about products they are considering purchasing. It uses an interface that allows for voice or text input.

[0650] Analysis method:

[0651] This system receives input from an input device and analyzes the content of the question using natural language processing. It utilizes natural language processing engines such as the Google Natural Language API.

[0652] Generation means:

[0653] This method generates appropriate answers based on analyzed question content. It utilizes a generative AI model (e.g., OpenAI GPT-4) installed on a cloud-based server.

[0654] Means of presentation:

[0655] This is a method for presenting generated answers to the user. It uses a speech synthesis engine (e.g., Amazon Polly) to provide the answers to the user in audio or text format.

[0656] Means of communication:

[0657] These are devices and software designed for interacting with users within physical stores. This allows users to obtain real-time information about product locations and ingredients while in the store.

[0658] Means of guidance:

[0659] Interaction means are devices or software that provide users with information about the location and ingredients of a product. This allows users to easily find out where a product is located, its ingredients, and other information.

[0660] Program processing

[0661] The system operates using servers (cloud-based) and terminals (devices in physical stores and users' smartphones). This system functions as follows:

[0662] The server is responsible for question analysis and answer generation. Natural language processing is used for analysis, and the generated answers are produced by a generative AI model.

[0663] The terminal (the user's smartphone or a guidance robot in a physical store) provides the user interface, allowing for question input, presentation of answers, and interaction.

[0664] The speech recognition engine uses the Google Speech-to-Text API to convert user voice input into text.

[0665] The speech synthesis engine uses Amazon Polly to convert the generated text-based responses into speech.

[0666] Specific example

[0667] Example 1: When a user wants to know the battery capacity of a new smartphone

[0668] 1. User: "What is the battery capacity of the latest smartphones?"

[0669] 2. The device receives this question and converts it to text using its speech recognition engine.

[0670] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "smartphone," "battery capacity").

[0671] 4. Based on the analysis results, a generative AI model generates an appropriate answer (e.g., "The latest smartphones have a battery capacity of 4000mAh.").

[0672] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[0673] Example 2: When a user wants to know the location of a specific product

[0674] 1. User: "Where can I find this product?"

[0675] 2. The device receives this question and converts it to text using its speech recognition engine.

[0676] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "product," "location").

[0677] 4. Based on the analysis results, a generative AI model generates an appropriate response (e.g., "This product is on shelf B3 on the second floor.").

[0678] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[0679] Example of a prompt

[0680] "Please tell me the ingredients of the new shampoo."

[0681] "Where can I find this product?"

[0682] "What is the battery capacity of this smartphone?"

[0683] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0684] Step 1:

[0685] Enter questions about the product the user is considering purchasing.

[0686] Specific example: The user uses their smartphone to ask a voice question, "Where can I find this product?"

[0687] Input: Voice input

[0688] Output: Audio data

[0689] Step 2:

[0690] The device receives the user's voice input and uses a speech recognition engine to convert the voice data into text. The Google Speech-to-Text API is used for the speech recognition engine.

[0691] Specific example: Voice input is converted into text data such as "Where can I find this product?".

[0692] Input: Audio data

[0693] Output: Text data

[0694] Step 3:

[0695] The terminal sends text data to the server for analysis.

[0696] Specific example: A text-based question is sent to a server in the cloud.

[0697] Input: Text data

[0698] Output: Text data (sent to the server)

[0699] Step 4:

[0700] The server uses a natural language processing engine to analyze the received questions. The Google Natural Language API is used for this engine to extract key keywords and context.

[0701] Specific example: Extract the main keywords "product" and "location" from the question "Where can I find this product?".

[0702] Input: Text data

[0703] Output: Analysis results (keywords)

[0704] Step 5:

[0705] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate answers based on the analysis results.

[0706] Specific example: The response "This product is located on shelf B3 on the second floor" is generated.

[0707] Input: Analysis results (keywords)

[0708] Output: Generated response (text format)

[0709] Step 6:

[0710] The server sends the generated response to the terminal.

[0711] Specific example: A text response such as "This product is located on shelf B3 on the second floor" is sent to the terminal.

[0712] Input: Generated response (text format)

[0713] Output: Generated response (text format, sent to device)

[0714] Step 7:

[0715] The device converts the received text-based response into speech using a speech synthesis engine (e.g., Amazon Polly).

[0716] Specific example: An audio message is generated saying, "This product is located on shelf B3 on the second floor."

[0717] Input: Generated response (text format)

[0718] Output: Audio data

[0719] Step 8:

[0720] The device presents audio data to the user.

[0721] Specific example: The terminal plays a voice message saying, "This product is located on shelf B3 on the second floor."

[0722] Input: Audio data

[0723] Output: Audio presentation

[0724] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0725] This invention relates to a robotic system that provides product information quickly and appropriately according to user needs. In addition to a series of processes that involve receiving user questions, generating appropriate answers, and presenting them, this system also has the function of recognizing the user's emotions and responding accordingly. A detailed explanation follows below.

[0726] Program processing

[0727] Enter your question

[0728] The user enters a question into the system, either verbally or in text. For example, they might enter a question like, "What is the battery capacity of the latest smartphones?"

[0729] Receiving and recognizing questions

[0730] The device receives the user's voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the voice into text. This text-based question is then sent to the next step.

[0731] Question analysis

[0732] The device analyzes text-based questions using a natural language processing (NLP) engine. The NLP engine tokenizes the questions and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[0733] Recognition of emotions

[0734] The device uses an emotion recognition engine to analyze the user's emotions. The emotion recognition engine extracts emotional data from the user's voice and facial expressions, and identifies emotions such as joy, surprise, and anger.

[0735] Generating an answer

[0736] The device sends the question and sentiment analysis results to a cloud-based server. A generative AI model located on the server generates an appropriate response based on the analysis results. For example, if the user is feeling anxious, it will generate a reassuring response. A response such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time," might be generated.

[0737] Providing an answer

[0738] The device uses a speech synthesis engine to present the user with the responses received from the generative AI model. The responses are provided to the user in audio format, and also in text format if necessary.

[0739] Specific example

[0740] Scenario 1: When a user asks about the battery capacity of their new smartphone and expresses concern.

[0741] 1. User: "What is the battery capacity of the latest smartphones?"

[0742] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0743] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0744] 4. The device uses an emotion recognition engine to analyze if the user is feeling anxious.

[0745] 5. The terminal sends the analysis results to the server.

[0746] 6. The generative AI model on the server generates a reassuring response such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[0747] 7. The device receives the response and converts it into speech using a speech synthesis engine.

[0748] 8. Device: Responds to the user with, "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[0749] Scenario 2: When the user asks about the price of a smartphone and gets excited

[0750] 1. User: "How much does this smartphone cost?"

[0751] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0752] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0753] 4. The device uses an emotion recognition engine to analyze whether the user is excited.

[0754] 5. The terminal sends the analysis results to the server.

[0755] 6. A generative AI model on the server generates an exciting response: "This smartphone costs 79,800 yen. It's a great deal, so be sure to check it out!"

[0756] 7. The device receives the response and converts it into speech using a speech synthesis engine.

[0757] 8. Device: Responds to the user with "This smartphone is priced at 79,800 yen. It's a great price, so be sure to check it out!"

[0758] As described above, the present invention not only provides accurate and immediate answers to user questions, but also enables responses that take user emotions into consideration. This can further improve user satisfaction and purchasing intent. By using a cloud-based AI model, the system's flexibility and scalability are also ensured.

[0759] The following describes the processing flow.

[0760] Program processing steps

[0761] Step 1: Enter the question

[0762] 1. The user enters a question into the system, either verbally or in text.

[0763] Example: "What is the battery capacity of the latest smartphones?"

[0764] Step 2: Receiving and recognizing the question

[0765] 2. The device receives the user's voice input and converts it to text using a speech recognition engine.

[0766] The device uses a microphone to capture the user's voice.

[0767] The device uses speech recognition software to generate text data from speech.

[0768] Example output: The text data "What is the battery capacity of the latest smartphones?" is generated.

[0769] Step 3: Analyzing the Question

[0770] 3. The device analyzes the text-based question using a natural language processing (NLP) engine.

[0771] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[0772] The device then analyzes key keywords and context to determine intent.

[0773] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[0774] Step 4: Recognizing Emotions

[0775] 4. The device uses an emotion recognition engine to analyze the user's emotions.

[0776] The device extracts emotional data from the user's voice tone, pace, and facial expressions.

[0777] The device identifies the category of emotion (e.g., joy, surprise, anger, anxiety).

[0778] Example output: The emotion "anxiety" is recognized.

[0779] Step 5: Sending analysis results and sentiment data

[0780] 5. The device converts the question analysis results and sentiment analysis results into JSON format and sends them to a cloud-based server.

[0781] The terminal packages the question analysis and sentiment analysis results into a single JSON file.

[0782] The device sends JSON data to the server via the internet.

[0783] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[0784] Step 6: Generating the answer

[0785] 6. The server uses a generative AI model to generate an appropriate response.

[0786] The server receives the transmitted JSON data and runs a generative AI model based on the question and sentiment analysis results.

[0787] Generative AI models generate responses that match the user's emotions (e.g., reassuring responses for users who are feeling anxious).

[0788] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[0789] Step 7: Receiving responses and converting them to audio.

[0790] 7. The device receives the response from the server and converts it into speech using a speech synthesis engine.

[0791] The device receives the response data in JSON format.

[0792] The device uses a speech synthesis engine to convert text data into speech data.

[0793] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[0794] Step 8: Present your answer

[0795] 8. The device provides the user with an answer via voice or text.

[0796] The device plays audio data generated by a speech synthesis engine.

[0797] If necessary, the response will also be displayed in text format on the screen.

[0798] Example: The voice response might say, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[0799] Specific example

[0800] Scenario: When a user asks about their smartphone's battery capacity and expresses concern.

[0801] Step 1:

[0802] User: "What is the battery capacity of the latest smartphones?"

[0803] Step 2:

[0804] The device receives the audio and converts it to text.

[0805] Example output: Text "What is the battery capacity of the latest smartphones?"

[0806] Step 3:

[0807] The device analyzes the text using an NLP engine.

[0808] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[0809] Step 4:

[0810] The device analyzes the user's emotions using its emotion recognition engine.

[0811] Example output: The emotion "anxiety" is recognized.

[0812] Step 5:

[0813] The device sends the analysis results and sentiment analysis results to the server in JSON format.

[0814] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[0815] Step 6:

[0816] The server generates the answer using a generative AI model.

[0817] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[0818] Step 7:

[0819] The device receives the response and converts it into speech using a speech synthesis engine.

[0820] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[0821] Step 8:

[0822] The device provides the user with the answer via voice.

[0823] Example: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[0824] (Example 2)

[0825] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0826] Modern consumers need to obtain increasingly diverse product information quickly and accurately. Furthermore, traditional product information systems often provide uniform answers without considering user emotions or intentions, which is insufficient for improving user satisfaction. This has led to challenges such as decreased purchase intent and a decline in the quality of the user experience.

[0827] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, a conversion means for receiving input from the input means and converting the voice data into text format using a speech recognition engine, an analysis means for analyzing the converted text data using a natural language processing engine and extracting key keywords, an emotion recognition means for recognizing the user's emotional state, a generation means for generating an appropriate answer based on the data obtained by the analysis means and the emotion recognition means, and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This enables the rapid provision of an appropriate answer that meets the user's needs and also allows for a response that takes the user's emotions into consideration.

[0828] "Input means" refers to a device or method that provides an interface for a user to input a question.

[0829] "Conversion means" refers to a device or method used to convert audio data received from an input means into text format.

[0830] "Analysis means" refers to a device or method that analyzes text data obtained by the conversion means using natural language processing techniques to extract key keywords and context.

[0831] "Emotion recognition means" refers to a device or method that analyzes a user's voice and facial expression data to determine their emotional state.

[0832] "Generation means" refers to a device or method that generates an appropriate response using a generative AI model on the cloud, based on data obtained by analysis means and emotion recognition means.

[0833] "Presentation means" refers to a device or method used to present the generated response to the user, and includes providing the response in voice format using a speech synthesis engine.

[0834] A "generative AI model" is an artificial intelligence model used to generate appropriate text or information from input data.

[0835] A "prompt" is an instruction given to a generative AI model, and it serves as the basis for determining the content of the generated response.

[0836] This invention relates to a system that allows users to input questions about products they are considering purchasing and provides appropriate answers based on those questions. This system has the function of recognizing the user's emotional state and generating corresponding answers. The embodiments of the system are described in detail below.

[0837] System Configuration

[0838] This system consists of the following main components:

[0839] 1. Input Methods: This is the interface for the user to input questions. This includes microphones that support voice input and keyboards that support text input.

[0840] 2. Conversion method: In the case of voice input, a speech recognition engine is used as the conversion method. In this embodiment, the Google Speech-to-Text API is used to convert speech to text.

[0841] 3. Analysis Method: The converted text is analyzed using a natural language processing (NLP) engine. For example, Amazon Comprehend is used to extract the main keywords and intent of the question.

[0842] 4. Emotion Recognition Method: Emotional data is analyzed from the user's voice and facial expressions. The Microsoft Azure Emotion API is used to determine the user's emotional state (joy, anxiety, excitement, etc.).

[0843] 5. Generation means: Based on the data obtained by the analysis means and the emotion recognition means, a generative AI model deployed on a cloud-based server is used. In this embodiment, OpenAI's GPT-3 generates the response.

[0844] 6. Presentation Method: A speech synthesis engine is used to present the generated responses to the user. Amazon Polly is used to provide the generated text responses in audio format.

[0845] Implementation method

[0846] The system operates as follows: When a user enters a question, the input device receives it. In the case of voice input, the device converts the speech to text using the Google Speech-to-Text API. The converted text is then analyzed by the analysis device using Amazon Comprehend. Here, key keywords and the intent of the question are extracted. In parallel, the emotion recognition device analyzes the emotional state from the voice data and the user's facial expressions using the Microsoft Azure Emotion API.

[0847] The analyzed data and emotional state are sent to a cloud-based server, where appropriate responses are generated using OpenAI's GPT-3. The generated responses are returned to the device and converted to audio format using Amazon Polly. Finally, the responses are presented to the user in both audio and text formats.

[0848] Specific example

[0849] Scenario 1: When a user asks about the battery capacity of their new smartphone and expresses concern.

[0850] 1. User: "What is the battery capacity of the latest smartphones?"

[0851] 2. The device receives the question and converts it to text format using the Google Speech-to-Text API.

[0852] 3. The device uses Amazon Comprehend to analyze the questions and extract key keywords.

[0853] 4. The device uses the Microsoft Azure Emotion API to analyze if the user is feeling anxious.

[0854] 5. The device sends the analysis results to the cloud server.

[0855] 6. The generative AI model (OpenAI GPT-3) on the server generates the response: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[0856] 7. The device receives the response and converts it to speech using Amazon Polly.

[0857] 8. Device: Responds to the user with, "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[0858] Example of a prompt

[0859] "Please tell me about the battery capacity of this smartphone. Generate an answer that will reassure users who are feeling anxious."

[0860] The system, possessing the above characteristics, can quickly provide appropriate answers tailored to user needs and even respond in a way that considers user emotions. This, in turn, can improve user satisfaction and purchasing intent.

[0861] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0862] Step 1: Enter your question

[0863] The user enters a question. The question can be entered via voice or text. For example, consider the question, "What is the battery capacity of the latest smartphones?" This input data is captured as voice data if entered via voice, or directly as text data if entered via text.

[0864] Step 2: Receiving the question and speech recognition

[0865] The device receives user input. In the case of voice input, the Google Speech-to-Text API is used to convert the speech to text. The input is audio data, and the output is a question in text format. Specifically, the device's microphone captures the audio, and the API is called to convert the audio into text data.

[0866] Step 3: Question analysis and tokenization

[0867] The device receives a text-based question and analyzes it using a natural language processing engine such as Amazon Comprehend. The input is a text-based question, and the output is a list of tokenized words and phrases. Specifically, the NLP engine breaks down the text into words and phrases and tokenizes them into categories such as "latest," "smartphone," and "battery capacity."

[0868] Step 4: Keyword Extraction and Intent Interpretation

[0869] The device extracts key keywords from the tokenized data and interprets the intent of the question. The input is the tokenized data obtained in step 3, and the output is the key keywords and their related information. Specifically, the NLP engine scans the tokens and recognizes that "smartphone" is a product category and "battery capacity" is an attribute.

[0870] Step 5: Recognizing and analyzing emotions

[0871] The device uses an emotion recognition engine, such as the Microsoft Azure Emotion API, to analyze the user's emotions. Input is voice or facial expression data, and output is the emotional state (e.g., joy, anxiety, excitement). Specifically, in the case of voice, emotions are analyzed from the tone and speed of the voice, and in the case of facial expressions, emotions are analyzed from images captured by the camera.

[0872] Step 6: Generating and optimizing answers

[0873] The device sends analysis results and sentiment data to a cloud-based server. On the server, generative AI models such as OpenAI's GPT-3 generate responses. The input consists of analyzed keywords and sentiment states, and the output is a response in appropriate text format. Specifically, data is sent to the server, and the generative AI model generates a response based on prompts.

[0874] Step 7: Presentation of answers and speech synthesis

[0875] The device uses a speech synthesis engine such as Amazon Polly to present the user with the response received from the generative AI model. The input is a text-based response, and the output is a voice-based response. Specifically, the speech synthesis engine converts the text response into a voice file, which is then played through the device's speaker, or the text-based response is displayed on the screen.

[0876] (Application Example 2)

[0877] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0878] In modern brick-and-mortar stores, it is crucial to respond quickly and appropriately to customers' questions about products they are considering purchasing. However, traditional systems only provide information in response to each customer's question, failing to address their emotions and needs. As a result, the quality and satisfaction of the information customers receive are limited, making it difficult to increase their purchasing intent. Furthermore, staff shortages and the difficulty of multilingual support among staff are also challenges. To solve these problems, a system is needed that recognizes customer emotions and provides appropriate answers based on those emotions.

[0879] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an input means for inputting a question about a product that the user is considering purchasing, an analysis means for receiving the input from the input means and analyzing the content of the question using natural language processing, a generation means for generating an appropriate answer based on the question content analyzed by the analysis means, an emotion recognition means for recognizing the user's emotions, an answer adjustment means for adjusting the answer generated based on the emotions recognized by the emotion recognition means, and a presentation means for presenting the generated and adjusted answer to the user. This makes it possible to quickly provide appropriate product information that corresponds to the customer's emotions, thereby increasing customer satisfaction and purchasing intent.

[0880] "Input means" refers to a device or function that allows a user to input questions or information into a system.

[0881] "Analysis means" refers to a function that understands and analyzes questions and information received from input means using natural language processing technology.

[0882] A "generation means" is a function for creating an appropriate answer based on the information analyzed by the analysis means.

[0883] "Emotion recognition means" refers to a function that identifies and analyzes a user's emotions from their voice, facial expressions, tone of voice, etc.

[0884] A "response adjustment mechanism" is a function that adjusts responses generated based on recognized emotions and provides them in an appropriate format.

[0885] "Presentation means" refers to a function for presenting generated and adjusted responses to the user in audio or text format.

[0886] A "generative AI model" is an artificial intelligence model that generates appropriate responses or information based on input data in natural language processing and generative tasks.

[0887] A "cloud-based server" is a system that performs data processing and storage on remote servers accessible via the internet.

[0888] A "speech synthesis engine" is a technology that converts text data into speech data and outputs it as natural-sounding speech.

[0889] This invention is a system that provides product information quickly and appropriately in response to customer inquiries in physical stores. The following describes specific implementations of this system.

[0890] System Configuration

[0891] The system includes the following main components:

[0892] An "input method" for users to enter questions.

[0893] "Analysis means" for analyzing received questions

[0894] A "generation means" for generating answers based on the content of the questions.

[0895] "Means for recognizing user emotions"

[0896] "Response adjustment means" for adjusting responses based on perceived emotions.

[0897] A "presentation method" for presenting adjusted answers to users.

[0898] This section describes the specific functions of each component and the hardware and software that support them.

[0899] Hardware and software details

[0900] 1. Input method:

[0901] This includes a microphone for users to input questions verbally, as well as a touch panel.

[0902] 2. Analysis method:

[0903] A speech recognition engine for converting audio data into text (e.g., Google Cloud Speech-to-Text).

[0904] A natural language processing engine (e.g., Amazon Comprehend) that analyzes the converted text data and extracts keywords and context.

[0905] 3. Generation means:

[0906] A generative AI model (e.g., OpenAI GPT-4) that runs on a cloud-based server and generates appropriate answers based on the question.

[0907] 4. Emotion recognition means:

[0908] An emotion recognition engine (e.g., Microsoft Azure Emotional API) analyzes a user's voice and facial expressions to recognize their emotions.

[0909] 5. Answer adjustment means:

[0910] The generated response is adjusted based on the emotional data obtained by the emotion recognition system.

[0911] 6. Means of presentation:

[0912] A speech synthesis engine (e.g., Amazon Polly) for presenting adjusted responses to users.

[0913] Processing flow and specific examples

[0914] Specific example 1:

[0915] A scenario where a customer comes into the store and asks about the battery life of their new smartphone.

[0916] User: "What is the battery capacity of the latest smartphones?"

[0917] Voice data is input through the microphone, and a speech recognition engine converts that data into text.

[0918] A natural language processing engine analyzes the text and extracts key keywords.

[0919] The system determines that the user's emotion is "anxiety" based on their recognized voice tone and facial expressions.

[0920] The question content and sentiment data are sent to a generative AI model on a cloud server.

[0921] A generative AI model generates responses, such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[0922] The speech synthesis engine converts the generated response into speech and provides it to the user.

[0923] Example of a prompt:

[0924] "User is anxious. Question: What is the battery capacity of the latest smartphones?"

[0925] Benefits of the entire system

[0926] This system will significantly improve customer service in physical stores. In particular, it is expected to increase customer satisfaction and purchasing intent by quickly providing optimal answers tailored to each customer's emotions in response to their questions. The use of a cloud-based AI model ensures flexibility and scalability, making it suitable for use in a variety of physical stores.

[0927] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0928] Step 1:

[0929] The user enters a question. If the user enters the question by voice using a microphone, the voice signal becomes the input data. If the user enters the question by text using a touch panel, the text data becomes the input data. This data is received by the terminal.

[0930] Step 2:

[0931] The device uses a speech recognition engine to convert voice input into text data. The Google Cloud Speech-to-Text engine is used to convert the voice signal into text format. This text data is then input into the next step.

[0932] Step 3:

[0933] The device uses a natural language processing (NLP) engine to analyze text data. Amazon Comprehend is used to tokenize the text and extract key keywords and context. This analysis helps understand the intent of the question, and the extracted keywords are then fed into the next step.

[0934] Step 4:

[0935] The device uses an emotion recognition engine to analyze the user's emotional state. It uses the Microsoft Azure Emotional API to obtain emotional data from the user's voice tone and facial expressions. For example, it analyzes voice intonation and changes in facial expressions to determine if the user is experiencing emotions such as "anxiety," "excitement," or "joy." This emotional data is then input into the next step.

[0936] Step 5:

[0937] The device sends the analyzed question data and sentiment data to a cloud-based server. The server uses a generative AI model (OpenAI GPT-4) to generate appropriate responses based on the input data. For example, if the user is feeling "anxious," a reassuring response will be generated. The text data of the generated response is then input into the next step.

[0938] Step 6:

[0939] The device uses a speech synthesis engine to convert the generated response text into audio data. Amazon Polly is used to convert the text-based response into natural-sounding speech. This audio data is then presented to the user.

[0940] Step 7:

[0941] The user receives voice-generated answers. The answers are delivered in audio format via a speaker, which serves as the presentation method. Furthermore, the answers are also displayed in text format on the touchscreen if necessary. This allows the user to obtain answers to their product-related questions and receive information that is relevant to their emotional state.

[0942] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0943] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0944] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0945] [Third Embodiment]

[0946] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0947] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0948] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0949] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0950] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0951] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0952] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0953] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0954] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0955] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0956] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0957] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0958] This invention is a robotic system for providing product information quickly and appropriately according to user needs. This system has a series of processes for receiving user questions, generating appropriate answers, and presenting them. A detailed explanation follows below.

[0959] Program processing

[0960] Enter your question

[0961] Users ask questions to the system verbally or in text. For example, they might type a question like, "What is the battery capacity of the latest smartphones?"

[0962] Receiving and recognizing questions

[0963] The device receives user voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the speech into text. This text-based question is then sent to the next step.

[0964] Question analysis

[0965] The device receives a question in text format and analyzes it using a natural language processing (NLP) engine. The NLP engine tokenizes the question and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[0966] Generating an answer

[0967] The device sends the analyzed question to a cloud-based server. A generative AI model located on the server generates an appropriate answer based on the analysis results. For example, it might generate the answer, "The battery capacity of the latest smartphones is 4000mAh."

[0968] Providing an answer

[0969] The device uses a speech synthesis engine to present the user with the responses received from the generative AI model. The responses are provided to the user in audio format, and also in text format if necessary.

[0970] Specific example

[0971] Scenario 1: When a user asks about the battery capacity of a new smartphone

[0972] 1. User: "What is the battery capacity of the latest smartphones?"

[0973] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0974] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0975] 4. The terminal sends the analysis results to the server.

[0976] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[0977] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0978] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[0979] Scenario 2: When a user asks about the price of a smartphone

[0980] 1. User: "How much does this smartphone cost?"

[0981] 2. The device receives the question and converts it into text format using its speech recognition engine.

[0982] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[0983] 4. The terminal sends the analysis results to the server.

[0984] 5. The generative AI model on the server generates the response, "The price of this smartphone is 79,800 yen."

[0985] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[0986] 7. Device: Responds to the user with "The price of this smartphone is 79,800 yen."

[0987] As described above, the present invention can provide accurate and immediate answers to inquiries, thereby improving user satisfaction. Furthermore, the use of a cloud-based AI model ensures the flexibility and scalability of the system.

[0988] The following describes the processing flow.

[0989] Program processing steps

[0990] Step 1: Enter the question

[0991] 1. The user enters a question into the system, either verbally or in text.

[0992] Example: "What is the battery capacity of the latest smartphones?"

[0993] Step 2: Receiving and recognizing the question

[0994] 2. The device receives the user's voice input and converts it to text using a speech recognition engine.

[0995] The device uses a microphone to capture the user's voice.

[0996] The device uses speech recognition software to generate text data from speech.

[0997] Example output: The text data "What is the battery capacity of the latest smartphones?" is generated.

[0998] Step 3: Analyzing the Question

[0999] 3. The device analyzes the text-based question using a natural language processing (NLP) engine.

[1000] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[1001] The device then analyzes key keywords and context to determine intent.

[1002] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[1003] Step 4: Submitting the analysis results

[1004] 4. The device converts the analysis results into JSON format and sends them to a cloud-based server.

[1005] The terminal packages the analysis results in JSON format.

[1006] The device sends JSON data to the server via the internet.

[1007] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[1008] Step 5: Generating the answer

[1009] 5. The server uses a generative AI model to generate an appropriate response.

[1010] The server receives the transmitted JSON data and executes a generative AI model based on the analysis results.

[1011] Generative AI models search for appropriate data based on the intent of the question and keywords, and generate answers.

[1012] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[1013] Step 6: Receiving responses and converting them to speech

[1014] 6. The terminal receives the response from the server and converts it into speech using a speech synthesis engine.

[1015] The device receives the response data in JSON format.

[1016] The device uses a speech synthesis engine to convert text data into speech data.

[1017] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[1018] Step 7: Present your answer

[1019] 7. The device provides the user with an answer via voice or text.

[1020] The device plays audio data generated by a speech synthesis engine.

[1021] If necessary, the response will also be displayed in text format on the screen.

[1022] Example: The voice response will say, "The latest smartphones have a battery capacity of 4000mAh."

[1023] Specific example

[1024] Scenario: A user asks a question about their smartphone's battery capacity.

[1025] 1. Step 1:

[1026] User: "What is the battery capacity of the latest smartphones?"

[1027] 2. Step 2:

[1028] The device receives the audio and converts it to text.

[1029] Example output: Text "What is the battery capacity of the latest smartphones?"

[1030] 3. Step 3:

[1031] The device analyzes the text using an NLP engine.

[1032] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[1033] 4. Step 4:

[1034] The terminal converts the analysis results into JSON format and sends them to the server.

[1035] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[1036] 5. Step 5:

[1037] The server generates the answer using a generative AI model.

[1038] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[1039] 6. Step 6:

[1040] The device receives the response and converts it into speech using a speech synthesis engine.

[1041] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[1042] 7. Step 7:

[1043] The device provides the user with the answer via voice.

[1044] Example: "The latest smartphones have a battery capacity of 4000mAh."

[1045] (Example 1)

[1046] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1047] Conventional product inquiry systems have struggled to provide timely and appropriate answers to user questions. This resulted in delays in responses and decreased user satisfaction. Furthermore, insufficient accuracy in speech recognition and natural language processing technologies made it difficult to accurately understand questions and generate appropriate answers.

[1048] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1049] In this invention, the server includes an input means for inputting a question about a product the user is considering purchasing; a conversion means for receiving the input from the input means and converting the speech to text using a speech recognition engine; an analysis means for analyzing the converted text using a natural language processing engine and understanding the content of the question; a generation means for generating an appropriate answer based on the question content analyzed by the analysis means; and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This makes it possible to accurately understand the user's questions in voice or text format and provide a quick and appropriate answer.

[1050] An "input method" is a means by which a user can input questions about a product.

[1051] "Conversion means" refers to means including a speech recognition engine used to convert the user's voice input into text.

[1052] "Analysis means" refers to a method of analyzing text-based questions using a natural language processing engine, understanding the content of the questions, and extracting important keywords and context.

[1053] "Generation means" refers to means including an artificial intelligence model used to generate appropriate answers based on the analyzed question content.

[1054] "Presentation means" refers to means including a speech synthesis engine used to provide the generated response to the user.

[1055] A "cloud-based server" is a server located remotely on the internet that provides services such as computing and data storage.

[1056] A "natural language processing engine" is a software component used to understand and analyze natural language text.

[1057] A "generative artificial intelligence model" is an artificial intelligence algorithm that can generate natural language text based on a given input.

[1058] This invention is a system that quickly and appropriately processes and provides answers to questions from users regarding products they are considering purchasing. Its main components are input means, conversion means, analysis means, generation means, and presentation means. These components and their operation are described in detail below.

[1059] This system begins with the user entering a question about the product via an input device. The user can enter the question in either voice or text format.

[1060] If voice input is selected, the terminal's conversion mechanism uses a speech recognition engine to convert the speech to text. This system uses a common engine widely used as a "speech recognition engine," for example. This conversion process converts the voice signal into a corresponding string of characters.

[1061] Next, the device uses a natural language processing engine as an analysis tool to analyze the converted text. Specifically, a natural language processing engine such as Hugging Face is used to tokenize the text and extract key keywords and context. For example, if the question is "What is the battery capacity of the latest smartphones?", keywords such as "smartphone" and "battery capacity" will be extracted.

[1062] The extracted keywords and contextual information are sent to a cloud-based server. A generative AI model runs on the server; for example, OpenAI's GPT-3 is used. This model generates an appropriate response based on the analysis results. The generated response is then sent from the server to the terminal.

[1063] The device that receives the response uses a speech synthesis engine as a presentation method to present the response to the user in audio format. By using a speech synthesis engine such as Amazon Polly, the text-based response is converted into audio and provided to the user. The response can also be presented in text format if necessary.

[1064] Specific example

[1065] If a user asks about the battery capacity of a new smartphone, the process would include the following:

[1066] 1. User: "What is the battery capacity of the latest smartphones?"

[1067] 2. The device receives the question and converts it to text using its speech recognition engine.

[1068] 3. The device uses a natural language processing engine to analyze the question and extract key keywords.

[1069] 4. Send the analysis results to a cloud-based server.

[1070] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[1071] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[1072] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[1073] Example of a prompt

[1074] A prompt is a text-based question that is input into a generative artificial intelligence model. For example:

[1075] A user is asking, "What is the battery capacity of the latest smartphones?" Please tell me about the battery capacity of the latest smartphones.

[1076] In this way, implementing this system makes it possible to provide quick and accurate answers to user questions. This improves user satisfaction and ensures the system's flexibility and scalability.

[1077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1078] Step 1:

[1079] Users enter questions about products they are considering purchasing via input methods. These input methods include voice input and text input. For example, if a user asks, "What is the battery capacity of the latest smartphones?", they can enter the question via voice. Input at this stage can be in either voice or text format.

[1080] Step 2:

[1081] The device receives voice input from the user. The input voice data is converted into text using a speech recognition engine. Here, the voice signal is processed as digital data, and the speech recognition engine analyzes that data to generate a corresponding string of characters. This conversion process generates a text-based question such as, "What is the battery capacity of the latest smartphones?"

[1082] Step 3:

[1083] The device passes the converted text-formatted question to a natural language processing (NLP) engine. The NLP engine tokenizes the text and extracts key keywords and context. For example, keywords such as "latest," "smartphone," and "battery capacity" are extracted and processed along with their contextual information. Here, the input is a text-formatted question, and the output is the keywords and contextual information as a result of the analysis.

[1084] Step 4:

[1085] The device sends the analyzed results to a cloud-based server. The data sent to the server at this stage includes extracted keywords and contextual information. For example, keywords such as "smartphone" and "battery capacity" are sent to the server.

[1086] Step 5:

[1087] The server generates an answer using a generative artificial intelligence model based on the received analysis results. Specifically, the server uses a GPT-based generative AI model to generate prompt sentences. For example, a prompt sentence might be generated that says, "The user is asking, 'What is the battery capacity of the latest smartphones?' Please tell me about the battery capacity of the latest smartphones." Based on this prompt sentence, the AI ​​model generates the answer, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is keywords and contextual information as analysis results, and the output is the answer in text format.

[1088] Step 6:

[1089] The device uses a speech synthesis engine to present the response received from the server. Specifically, it converts the text-based response into audio data. The speech synthesis engine analyzes the text data and generates appropriate speech based on its content. Through this process, the user can receive an audio response such as, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is the text-based response, and the output is the audio response.

[1090] Step 7:

[1091] The user reviews the response provided in either audio or text format. The device plays the audio generated by the speech synthesis engine through its speaker. It can also display the response in text format if needed. This allows the user to obtain information quickly and accurately.

[1092] (Application Example 1)

[1093] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1094] Modern brick-and-mortar stores require prompt and accurate provision of a wide range of product information. However, traditional systems often struggled to respond to customer inquiries in a timely manner, leading to decreased customer satisfaction. Furthermore, employees sometimes had to spend time verifying information, resulting in reduced operational efficiency. Against this backdrop, there is a need for a system that can provide customers with product information in real time.

[1095] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1096] In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, an analysis means for receiving input from the input means and analyzing the content of the questions using natural language processing, a generation means for generating appropriate answers based on the question content analyzed by the analysis means, a presentation means for presenting the generated answers to the user, a dialogue means for interacting with the user in a physical store, and a guidance means for the dialogue means to provide the user with information about the location and ingredients of the products. As a result, users can obtain quick and accurate product information in physical stores, which is expected to improve customer satisfaction and operational efficiency.

[1097] An "input method" refers to a device or interface for users to input questions about a product they are considering purchasing.

[1098] "Analysis means" refers to a device or software that receives input from an input means and analyzes the content of a question using natural language processing.

[1099] "Generation means" refers to a device or software that generates appropriate answers based on the question content analyzed by the analysis means.

[1100] "Presentation means" refers to a device or software that presents the generated answer to the user.

[1101] "Dialogue means" refers to devices and software used to interact with users within a physical store.

[1102] A "guidance means" refers to a device or software that provides the user with information about the location and ingredients of a product through dialogue means.

[1103] A "cloud-based server" is a remotely located server that can be accessed via the internet.

[1104] A "generative AI model" is an artificial intelligence model that operates on a cloud-based server as a means of generation and generates appropriate answers.

[1105] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[1106] A "speech synthesis engine" is a technology or device that converts text-based responses into speech.

[1107] This invention is a system that allows users to ask questions about products in a physical store and provides them with quick and appropriate answers. The system is configured as follows:

[1108] System Configuration

[1109] Input method:

[1110] This is a method for users to input questions about products they are considering purchasing. It uses an interface that allows for voice or text input.

[1111] Analysis method:

[1112] This system receives input from an input device and analyzes the content of the question using natural language processing. It utilizes natural language processing engines such as the Google Natural Language API.

[1113] Generation means:

[1114] This method generates appropriate answers based on analyzed question content. It utilizes a generative AI model (e.g., OpenAI GPT-4) installed on a cloud-based server.

[1115] Means of presentation:

[1116] This is a method for presenting generated answers to the user. It uses a speech synthesis engine (e.g., Amazon Polly) to provide the answers to the user in audio or text format.

[1117] Means of communication:

[1118] These are devices and software designed for interacting with users within physical stores. This allows users to obtain real-time information about product locations and ingredients while in the store.

[1119] Means of guidance:

[1120] Interaction means are devices or software that provide users with information about the location and ingredients of a product. This allows users to easily find out where a product is located, its ingredients, and other information.

[1121] Program processing

[1122] The system operates using servers (cloud-based) and terminals (devices in physical stores and users' smartphones). This system functions as follows:

[1123] The server is responsible for question analysis and answer generation. Natural language processing is used for analysis, and the generated answers are produced by a generative AI model.

[1124] The terminal (the user's smartphone or a guidance robot in a physical store) provides the user interface, allowing for question input, presentation of answers, and interaction.

[1125] The speech recognition engine uses the Google Speech-to-Text API to convert user voice input into text.

[1126] The speech synthesis engine uses Amazon Polly to convert the generated text-based responses into speech.

[1127] Specific example

[1128] Example 1: When a user wants to know the battery capacity of a new smartphone

[1129] 1. User: "What is the battery capacity of the latest smartphones?"

[1130] 2. The device receives this question and converts it to text using its speech recognition engine.

[1131] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "smartphone," "battery capacity").

[1132] 4. Based on the analysis results, a generative AI model generates an appropriate answer (e.g., "The latest smartphones have a battery capacity of 4000mAh.").

[1133] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[1134] Example 2: When a user wants to know the location of a specific product

[1135] 1. User: "Where can I find this product?"

[1136] 2. The device receives this question and converts it to text using its speech recognition engine.

[1137] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "product," "location").

[1138] 4. Based on the analysis results, a generative AI model generates an appropriate response (e.g., "This product is on shelf B3 on the second floor.").

[1139] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[1140] Example of a prompt

[1141] "Please tell me the ingredients of the new shampoo."

[1142] "Where can I find this product?"

[1143] "What is the battery capacity of this smartphone?"

[1144] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1145] Step 1:

[1146] Enter questions about the product the user is considering purchasing.

[1147] Specific example: The user uses their smartphone to ask a voice question, "Where can I find this product?"

[1148] Input: Voice input

[1149] Output: Audio data

[1150] Step 2:

[1151] The device receives the user's voice input and uses a speech recognition engine to convert the voice data into text. The Google Speech-to-Text API is used for the speech recognition engine.

[1152] Specific example: Voice input is converted into text data such as "Where can I find this product?".

[1153] Input: Audio data

[1154] Output: Text data

[1155] Step 3:

[1156] The terminal sends text data to the server for analysis.

[1157] Specific example: A text-based question is sent to a server in the cloud.

[1158] Input: Text data

[1159] Output: Text data (sent to the server)

[1160] Step 4:

[1161] The server uses a natural language processing engine to analyze the received questions. The Google Natural Language API is used for this engine to extract key keywords and context.

[1162] Specific example: Extract the main keywords "product" and "location" from the question "Where can I find this product?".

[1163] Input: Text data

[1164] Output: Analysis results (keywords)

[1165] Step 5:

[1166] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate answers based on the analysis results.

[1167] Specific example: The response "This product is located on shelf B3 on the second floor" is generated.

[1168] Input: Analysis results (keywords)

[1169] Output: Generated response (text format)

[1170] Step 6:

[1171] The server sends the generated response to the terminal.

[1172] Specific example: A text response such as "This product is located on shelf B3 on the second floor" is sent to the terminal.

[1173] Input: Generated response (text format)

[1174] Output: Generated response (text format, sent to device)

[1175] Step 7:

[1176] The device converts the received text-based response into speech using a speech synthesis engine (e.g., Amazon Polly).

[1177] Specific example: An audio message is generated saying, "This product is located on shelf B3 on the second floor."

[1178] Input: Generated response (text format)

[1179] Output: Audio data

[1180] Step 8:

[1181] The device presents audio data to the user.

[1182] Specific example: The terminal plays a voice message saying, "This product is located on shelf B3 on the second floor."

[1183] Input: Audio data

[1184] Output: Audio presentation

[1185] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1186] This invention relates to a robotic system that provides product information quickly and appropriately according to user needs. In addition to a series of processes that involve receiving user questions, generating appropriate answers, and presenting them, this system also has the function of recognizing the user's emotions and responding accordingly. A detailed explanation follows below.

[1187] Program processing

[1188] Enter your question

[1189] The user enters a question into the system, either verbally or in text. For example, they might enter a question like, "What is the battery capacity of the latest smartphones?"

[1190] Receiving and recognizing questions

[1191] The device receives the user's voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the voice into text. This text-based question is then sent to the next step.

[1192] Question analysis

[1193] The device analyzes text-based questions using a natural language processing (NLP) engine. The NLP engine tokenizes the questions and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[1194] Recognition of emotions

[1195] The device uses an emotion recognition engine to analyze the user's emotions. The emotion recognition engine extracts emotional data from the user's voice and facial expressions, and identifies emotions such as joy, surprise, and anger.

[1196] Generating an answer

[1197] The device sends the question and sentiment analysis results to a cloud-based server. A generative AI model located on the server generates an appropriate response based on the analysis results. For example, if the user is feeling anxious, it will generate a reassuring response. A response such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time," might be generated.

[1198] Providing an answer

[1199] The device uses a speech synthesis engine to present the user with the responses received from the generative AI model. The responses are provided to the user in audio format, and also in text format if necessary.

[1200] Specific example

[1201] Scenario 1: When a user asks about the battery capacity of their new smartphone and expresses concern.

[1202] 1. User: "What is the battery capacity of the latest smartphones?"

[1203] 2. The device receives the question and converts it into text format using its speech recognition engine.

[1204] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[1205] 4. The device uses an emotion recognition engine to analyze if the user is feeling anxious.

[1206] 5. The terminal sends the analysis results to the server.

[1207] 6. The generative AI model on the server generates a reassuring response such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[1208] 7. The device receives the response and converts it into speech using a speech synthesis engine.

[1209] 8. Device: Responds to the user with, "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1210] Scenario 2: When the user asks about the price of a smartphone and gets excited

[1211] 1. User: "How much does this smartphone cost?"

[1212] 2. The device receives the question and converts it into text format using its speech recognition engine.

[1213] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[1214] 4. The device uses an emotion recognition engine to analyze whether the user is excited.

[1215] 5. The terminal sends the analysis results to the server.

[1216] 6. A generative AI model on the server generates an exciting response: "This smartphone costs 79,800 yen. It's a great deal, so be sure to check it out!"

[1217] 7. The device receives the response and converts it into speech using a speech synthesis engine.

[1218] 8. Device: Responds to the user with "This smartphone is priced at 79,800 yen. It's a great price, so be sure to check it out!"

[1219] As described above, the present invention not only provides accurate and immediate answers to user questions, but also enables responses that take user emotions into consideration. This can further improve user satisfaction and purchasing intent. By using a cloud-based AI model, the system's flexibility and scalability are also ensured.

[1220] The following describes the processing flow.

[1221] Program processing steps

[1222] Step 1: Enter the question

[1223] 1. The user enters a question into the system, either verbally or in text.

[1224] Example: "What is the battery capacity of the latest smartphones?"

[1225] Step 2: Receiving and recognizing the question

[1226] 2. The device receives the user's voice input and converts it to text using a speech recognition engine.

[1227] The device uses a microphone to capture the user's voice.

[1228] The device uses speech recognition software to generate text data from speech.

[1229] Example output: The text data "What is the battery capacity of the latest smartphones?" is generated.

[1230] Step 3: Analyzing the Question

[1231] 3. The device analyzes the text-based question using a natural language processing (NLP) engine.

[1232] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[1233] The device then analyzes key keywords and context to determine intent.

[1234] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[1235] Step 4: Recognizing Emotions

[1236] 4. The device uses an emotion recognition engine to analyze the user's emotions.

[1237] The device extracts emotional data from the user's voice tone, pace, and facial expressions.

[1238] The device identifies the category of emotion (e.g., joy, surprise, anger, anxiety).

[1239] Example output: The emotion "anxiety" is recognized.

[1240] Step 5: Sending analysis results and sentiment data

[1241] 5. The device converts the question analysis results and sentiment analysis results into JSON format and sends them to a cloud-based server.

[1242] The terminal packages the question analysis and sentiment analysis results into a single JSON file.

[1243] The device sends JSON data to the server via the internet.

[1244] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[1245] Step 6: Generating the answer

[1246] 6. The server uses a generative AI model to generate an appropriate response.

[1247] The server receives the transmitted JSON data and runs a generative AI model based on the question and sentiment analysis results.

[1248] Generative AI models generate responses that match the user's emotions (e.g., reassuring responses for users who are feeling anxious).

[1249] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[1250] Step 7: Receiving responses and converting them to audio.

[1251] 7. The device receives the response from the server and converts it into speech using a speech synthesis engine.

[1252] The device receives the response data in JSON format.

[1253] The device uses a speech synthesis engine to convert text data into speech data.

[1254] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1255] Step 8: Present your answer

[1256] 8. The device provides the user with an answer via voice or text.

[1257] The device plays audio data generated by a speech synthesis engine.

[1258] If necessary, the response will also be displayed in text format on the screen.

[1259] Example: The voice response might say, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[1260] Specific example

[1261] Scenario: When a user asks about their smartphone's battery capacity and expresses concern.

[1262] Step 1:

[1263] User: "What is the battery capacity of the latest smartphones?"

[1264] Step 2:

[1265] The device receives the audio and converts it to text.

[1266] Example output: Text "What is the battery capacity of the latest smartphones?"

[1267] Step 3:

[1268] The device analyzes the text using an NLP engine.

[1269] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[1270] Step 4:

[1271] The device analyzes the user's emotions using its emotion recognition engine.

[1272] Example output: The emotion "anxiety" is recognized.

[1273] Step 5:

[1274] The device sends the analysis results and sentiment analysis results to the server in JSON format.

[1275] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[1276] Step 6:

[1277] The server generates the answer using a generative AI model.

[1278] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[1279] Step 7:

[1280] The device receives the response and converts it into speech using a speech synthesis engine.

[1281] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1282] Step 8:

[1283] The device provides the user with the answer via voice.

[1284] Example: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[1285] (Example 2)

[1286] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1287] Modern consumers need to obtain increasingly diverse product information quickly and accurately. Furthermore, traditional product information systems often provide uniform answers without considering user emotions or intentions, which is insufficient for improving user satisfaction. This has led to challenges such as decreased purchase intent and a decline in the quality of the user experience.

[1288] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, a conversion means for receiving input from the input means and converting the voice data into text format using a speech recognition engine, an analysis means for analyzing the converted text data using a natural language processing engine and extracting key keywords, an emotion recognition means for recognizing the user's emotional state, a generation means for generating an appropriate answer based on the data obtained by the analysis means and the emotion recognition means, and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This enables the rapid provision of an appropriate answer that meets the user's needs and also allows for a response that takes the user's emotions into consideration.

[1289] "Input means" refers to a device or method that provides an interface for a user to input a question.

[1290] "Conversion means" refers to a device or method used to convert audio data received from an input means into text format.

[1291] "Analysis means" refers to a device or method that analyzes text data obtained by the conversion means using natural language processing techniques to extract key keywords and context.

[1292] "Emotion recognition means" refers to a device or method that analyzes a user's voice and facial expression data to determine their emotional state.

[1293] "Generation means" refers to a device or method that generates an appropriate response using a generative AI model on the cloud, based on data obtained by analysis means and emotion recognition means.

[1294] "Presentation means" refers to a device or method used to present the generated response to the user, and includes providing the response in voice format using a speech synthesis engine.

[1295] A "generative AI model" is an artificial intelligence model used to generate appropriate text or information from input data.

[1296] A "prompt" is an instruction given to a generative AI model, and it serves as the basis for determining the content of the generated response.

[1297] This invention relates to a system that allows users to input questions about products they are considering purchasing and provides appropriate answers based on those questions. This system has the function of recognizing the user's emotional state and generating corresponding answers. The embodiments of the system are described in detail below.

[1298] System Configuration

[1299] This system consists of the following main components:

[1300] 1. Input Methods: This is the interface for the user to input questions. This includes microphones that support voice input and keyboards that support text input.

[1301] 2. Conversion method: In the case of voice input, a speech recognition engine is used as the conversion method. In this embodiment, the Google Speech-to-Text API is used to convert speech to text.

[1302] 3. Analysis Method: The converted text is analyzed using a natural language processing (NLP) engine. For example, Amazon Comprehend is used to extract the main keywords and intent of the question.

[1303] 4. Emotion Recognition Method: Emotional data is analyzed from the user's voice and facial expressions. The Microsoft Azure Emotion API is used to determine the user's emotional state (joy, anxiety, excitement, etc.).

[1304] 5. Generation means: Based on the data obtained by the analysis means and the emotion recognition means, a generative AI model deployed on a cloud-based server is used. In this embodiment, OpenAI's GPT-3 generates the response.

[1305] 6. Presentation Method: A speech synthesis engine is used to present the generated responses to the user. Amazon Polly is used to provide the generated text responses in audio format.

[1306] Implementation method

[1307] The system operates as follows: When a user enters a question, the input device receives it. In the case of voice input, the device converts the speech to text using the Google Speech-to-Text API. The converted text is then analyzed by the analysis device using Amazon Comprehend. Here, key keywords and the intent of the question are extracted. In parallel, the emotion recognition device analyzes the emotional state from the voice data and the user's facial expressions using the Microsoft Azure Emotion API.

[1308] The analyzed data and emotional state are sent to a cloud-based server, where appropriate responses are generated using OpenAI's GPT-3. The generated responses are returned to the device and converted to audio format using Amazon Polly. Finally, the responses are presented to the user in both audio and text formats.

[1309] Specific example

[1310] Scenario 1: When a user asks about the battery capacity of their new smartphone and expresses concern.

[1311] 1. User: "What is the battery capacity of the latest smartphones?"

[1312] 2. The device receives the question and converts it to text format using the Google Speech-to-Text API.

[1313] 3. The device uses Amazon Comprehend to analyze the questions and extract key keywords.

[1314] 4. The device uses the Microsoft Azure Emotion API to analyze if the user is feeling anxious.

[1315] 5. The device sends the analysis results to the cloud server.

[1316] 6. The generative AI model (OpenAI GPT-3) on the server generates the response: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[1317] 7. The device receives the response and converts it to speech using Amazon Polly.

[1318] 8. Device: Responds to the user with, "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1319] Example of a prompt

[1320] "Please tell me about the battery capacity of this smartphone. Generate an answer that will reassure users who are feeling anxious."

[1321] The system, possessing the above characteristics, can quickly provide appropriate answers tailored to user needs and even respond in a way that considers user emotions. This, in turn, can improve user satisfaction and purchasing intent.

[1322] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1323] Step 1: Enter your question

[1324] The user enters a question. The question can be entered via voice or text. For example, consider the question, "What is the battery capacity of the latest smartphones?" This input data is captured as voice data if entered via voice, or directly as text data if entered via text.

[1325] Step 2: Receiving the question and speech recognition

[1326] The device receives user input. In the case of voice input, the Google Speech-to-Text API is used to convert the speech to text. The input is audio data, and the output is a question in text format. Specifically, the device's microphone captures the audio, and the API is called to convert the audio into text data.

[1327] Step 3: Question analysis and tokenization

[1328] The device receives a text-based question and analyzes it using a natural language processing engine such as Amazon Comprehend. The input is a text-based question, and the output is a list of tokenized words and phrases. Specifically, the NLP engine breaks down the text into words and phrases and tokenizes them into categories such as "latest," "smartphone," and "battery capacity."

[1329] Step 4: Keyword Extraction and Intent Interpretation

[1330] The device extracts key keywords from the tokenized data and interprets the intent of the question. The input is the tokenized data obtained in step 3, and the output is the key keywords and their related information. Specifically, the NLP engine scans the tokens and recognizes that "smartphone" is a product category and "battery capacity" is an attribute.

[1331] Step 5: Recognizing and analyzing emotions

[1332] The device uses an emotion recognition engine, such as the Microsoft Azure Emotion API, to analyze the user's emotions. Input is voice or facial expression data, and output is the emotional state (e.g., joy, anxiety, excitement). Specifically, in the case of voice, emotions are analyzed from the tone and speed of the voice, and in the case of facial expressions, emotions are analyzed from images captured by the camera.

[1333] Step 6: Generating and optimizing answers

[1334] The device sends analysis results and sentiment data to a cloud-based server. On the server, generative AI models such as OpenAI's GPT-3 generate responses. The input consists of analyzed keywords and sentiment states, and the output is a response in appropriate text format. Specifically, data is sent to the server, and the generative AI model generates a response based on prompts.

[1335] Step 7: Presentation of answers and speech synthesis

[1336] The device uses a speech synthesis engine such as Amazon Polly to present the user with the response received from the generative AI model. The input is a text-based response, and the output is a voice-based response. Specifically, the speech synthesis engine converts the text response into a voice file, which is then played through the device's speaker, or the text-based response is displayed on the screen.

[1337] (Application Example 2)

[1338] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1339] In modern brick-and-mortar stores, it is crucial to respond quickly and appropriately to customers' questions about products they are considering purchasing. However, traditional systems only provide information in response to each customer's question, failing to address their emotions and needs. As a result, the quality and satisfaction of the information customers receive are limited, making it difficult to increase their purchasing intent. Furthermore, staff shortages and the difficulty of multilingual support among staff are also challenges. To solve these problems, a system is needed that recognizes customer emotions and provides appropriate answers based on those emotions.

[1340] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an input means for inputting a question about a product that the user is considering purchasing, an analysis means for receiving the input from the input means and analyzing the content of the question using natural language processing, a generation means for generating an appropriate answer based on the question content analyzed by the analysis means, an emotion recognition means for recognizing the user's emotions, an answer adjustment means for adjusting the answer generated based on the emotions recognized by the emotion recognition means, and a presentation means for presenting the generated and adjusted answer to the user. This makes it possible to quickly provide appropriate product information that corresponds to the customer's emotions, thereby increasing customer satisfaction and purchasing intent.

[1341] "Input means" refers to a device or function that allows a user to input questions or information into a system.

[1342] "Analysis means" refers to a function that understands and analyzes questions and information received from input means using natural language processing technology.

[1343] A "generation means" is a function for creating an appropriate answer based on the information analyzed by the analysis means.

[1344] "Emotion recognition means" refers to a function that identifies and analyzes a user's emotions from their voice, facial expressions, tone of voice, etc.

[1345] A "response adjustment mechanism" is a function that adjusts responses generated based on recognized emotions and provides them in an appropriate format.

[1346] "Presentation means" refers to a function for presenting generated and adjusted responses to the user in audio or text format.

[1347] A "generative AI model" is an artificial intelligence model that generates appropriate responses or information based on input data in natural language processing and generative tasks.

[1348] A "cloud-based server" is a system that performs data processing and storage on remote servers accessible via the internet.

[1349] A "speech synthesis engine" is a technology that converts text data into speech data and outputs it as natural-sounding speech.

[1350] This invention is a system that provides product information quickly and appropriately in response to customer inquiries in physical stores. The following describes specific implementations of this system.

[1351] System Configuration

[1352] The system includes the following main components:

[1353] An "input method" for users to enter questions.

[1354] "Analysis means" for analyzing received questions

[1355] A "generation means" for generating answers based on the content of the questions.

[1356] "Means for recognizing user emotions"

[1357] "Response adjustment means" for adjusting responses based on perceived emotions.

[1358] A "presentation method" for presenting adjusted answers to users.

[1359] This section describes the specific functions of each component and the hardware and software that support them.

[1360] Hardware and software details

[1361] 1. Input method:

[1362] This includes a microphone for users to input questions verbally, as well as a touch panel.

[1363] 2. Analysis method:

[1364] A speech recognition engine for converting audio data into text (e.g., Google Cloud Speech-to-Text).

[1365] A natural language processing engine (e.g., Amazon Comprehend) that analyzes the converted text data and extracts keywords and context.

[1366] 3. Generation means:

[1367] A generative AI model (e.g., OpenAI GPT-4) that runs on a cloud-based server and generates appropriate answers based on the question.

[1368] 4. Emotion recognition means:

[1369] An emotion recognition engine (e.g., Microsoft Azure Emotional API) analyzes a user's voice and facial expressions to recognize their emotions.

[1370] 5. Answer adjustment means:

[1371] The generated response is adjusted based on the emotional data obtained by the emotion recognition system.

[1372] 6. Means of presentation:

[1373] A speech synthesis engine (e.g., Amazon Polly) for presenting adjusted responses to users.

[1374] Processing flow and specific examples

[1375] Specific example 1:

[1376] A scenario where a customer comes into the store and asks about the battery life of their new smartphone.

[1377] User: "What is the battery capacity of the latest smartphones?"

[1378] Voice data is input through the microphone, and a speech recognition engine converts that data into text.

[1379] A natural language processing engine analyzes the text and extracts key keywords.

[1380] The system determines that the user's emotion is "anxiety" based on their recognized voice tone and facial expressions.

[1381] The question content and sentiment data are sent to a generative AI model on a cloud server.

[1382] A generative AI model generates responses, such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[1383] The speech synthesis engine converts the generated response into speech and provides it to the user.

[1384] Example of a prompt:

[1385] "User is anxious. Question: What is the battery capacity of the latest smartphones?"

[1386] Benefits of the entire system

[1387] This system will significantly improve customer service in physical stores. In particular, it is expected to increase customer satisfaction and purchasing intent by quickly providing optimal answers tailored to each customer's emotions in response to their questions. The use of a cloud-based AI model ensures flexibility and scalability, making it suitable for use in a variety of physical stores.

[1388] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1389] Step 1:

[1390] The user enters a question. If the user enters the question by voice using a microphone, the voice signal becomes the input data. If the user enters the question by text using a touch panel, the text data becomes the input data. This data is received by the terminal.

[1391] Step 2:

[1392] The device uses a speech recognition engine to convert voice input into text data. The Google Cloud Speech-to-Text engine is used to convert the voice signal into text format. This text data is then input into the next step.

[1393] Step 3:

[1394] The device uses a natural language processing (NLP) engine to analyze text data. Amazon Comprehend is used to tokenize the text and extract key keywords and context. This analysis helps understand the intent of the question, and the extracted keywords are then fed into the next step.

[1395] Step 4:

[1396] The device uses an emotion recognition engine to analyze the user's emotional state. It uses the Microsoft Azure Emotional API to obtain emotional data from the user's voice tone and facial expressions. For example, it analyzes voice intonation and changes in facial expressions to determine if the user is experiencing emotions such as "anxiety," "excitement," or "joy." This emotional data is then input into the next step.

[1397] Step 5:

[1398] The device sends the analyzed question data and sentiment data to a cloud-based server. The server uses a generative AI model (OpenAI GPT-4) to generate appropriate responses based on the input data. For example, if the user is feeling "anxious," a reassuring response will be generated. The text data of the generated response is then input into the next step.

[1399] Step 6:

[1400] The device uses a speech synthesis engine to convert the generated response text into audio data. Amazon Polly is used to convert the text-based response into natural-sounding speech. This audio data is then presented to the user.

[1401] Step 7:

[1402] The user receives voice-generated answers. The answers are delivered in audio format via a speaker, which serves as the presentation method. Furthermore, the answers are also displayed in text format on the touchscreen if necessary. This allows the user to obtain answers to their product-related questions and receive information that is relevant to their emotional state.

[1403] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1404] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1405] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1406] [Fourth Embodiment]

[1407] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1408] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1409] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1410] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1411] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1412] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1413] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1414] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1415] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1416] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1417] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1418] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1419] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1420] This invention is a robotic system for providing product information quickly and appropriately according to user needs. This system has a series of processes for receiving user questions, generating appropriate answers, and presenting them. A detailed explanation follows below.

[1421] Program processing

[1422] Enter your question

[1423] Users ask questions to the system verbally or in text. For example, they might type a question like, "What is the battery capacity of the latest smartphones?"

[1424] Receiving and recognizing questions

[1425] The device receives user voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the speech into text. This text-based question is then sent to the next step.

[1426] Question analysis

[1427] The device receives a question in text format and analyzes it using a natural language processing (NLP) engine. The NLP engine tokenizes the question and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[1428] Generating an answer

[1429] The device sends the analyzed question to a cloud-based server. A generative AI model located on the server generates an appropriate answer based on the analysis results. For example, it might generate the answer, "The battery capacity of the latest smartphones is 4000mAh."

[1430] Providing an answer

[1431] The device uses a speech synthesis engine to present the user with the responses received from the generative AI model. The responses are provided to the user in audio format, and also in text format if necessary.

[1432] Specific example

[1433] Scenario 1: When a user asks about the battery capacity of a new smartphone

[1434] 1. User: "What is the battery capacity of the latest smartphones?"

[1435] 2. The device receives the question and converts it into text format using its speech recognition engine.

[1436] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[1437] 4. The terminal sends the analysis results to the server.

[1438] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[1439] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[1440] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[1441] Scenario 2: When a user asks about the price of a smartphone

[1442] 1. User: "How much does this smartphone cost?"

[1443] 2. The device receives the question and converts it into text format using its speech recognition engine.

[1444] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[1445] 4. The terminal sends the analysis results to the server.

[1446] 5. The generative AI model on the server generates the response, "The price of this smartphone is 79,800 yen."

[1447] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[1448] 7. Device: Responds to the user with "The price of this smartphone is 79,800 yen."

[1449] As described above, the present invention can provide accurate and immediate answers to inquiries, thereby improving user satisfaction. Furthermore, the use of a cloud-based AI model ensures the flexibility and scalability of the system.

[1450] The following describes the processing flow.

[1451] Program processing steps

[1452] Step 1: Enter the question

[1453] 1. The user enters a question into the system, either verbally or in text.

[1454] Example: "What is the battery capacity of the latest smartphones?"

[1455] Step 2: Receiving and recognizing the question

[1456] 2. The device receives the user's voice input and converts it to text using a speech recognition engine.

[1457] The device uses a microphone to capture the user's voice.

[1458] The device uses speech recognition software to generate text data from speech.

[1459] Example output: The text data "What is the battery capacity of the latest smartphones?" is generated.

[1460] Step 3: Analyzing the Question

[1461] 3. The device analyzes the text-based question using a natural language processing (NLP) engine.

[1462] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[1463] The device then analyzes key keywords and context to determine intent.

[1464] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[1465] Step 4: Submitting the analysis results

[1466] 4. The device converts the analysis results into JSON format and sends them to a cloud-based server.

[1467] The terminal packages the analysis results in JSON format.

[1468] The device sends JSON data to the server via the internet.

[1469] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[1470] Step 5: Generating the answer

[1471] 5. The server uses a generative AI model to generate an appropriate response.

[1472] The server receives the transmitted JSON data and executes a generative AI model based on the analysis results.

[1473] Generative AI models search for appropriate data based on the intent of the question and keywords, and generate answers.

[1474] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[1475] Step 6: Receiving responses and converting them to speech

[1476] 6. The terminal receives the response from the server and converts it into speech using a speech synthesis engine.

[1477] The device receives the response data in JSON format.

[1478] The device uses a speech synthesis engine to convert text data into speech data.

[1479] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[1480] Step 7: Present your answer

[1481] 7. The device provides the user with an answer via voice or text.

[1482] The device plays audio data generated by a speech synthesis engine.

[1483] If necessary, the response will also be displayed in text format on the screen.

[1484] Example: The voice response will say, "The latest smartphones have a battery capacity of 4000mAh."

[1485] Specific example

[1486] Scenario: A user asks a question about their smartphone's battery capacity.

[1487] 1. Step 1:

[1488] User: "What is the battery capacity of the latest smartphones?"

[1489] 2. Step 2:

[1490] The device receives the audio and converts it to text.

[1491] Example output: Text "What is the battery capacity of the latest smartphones?"

[1492] 3. Step 3:

[1493] The device analyzes the text using an NLP engine.

[1494] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[1495] 4. Step 4:

[1496] The terminal converts the analysis results into JSON format and sends them to the server.

[1497] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"]}

[1498] 5. Step 5:

[1499] The server generates the answer using a generative AI model.

[1500] Example output: {"response": "The battery capacity of the latest smartphones is 4000mAh."}

[1501] 6. Step 6:

[1502] The device receives the response and converts it into speech using a speech synthesis engine.

[1503] Example output: "The battery capacity of the latest smartphones is 4000mAh."

[1504] 7. Step 7:

[1505] The device provides the user with the answer via voice.

[1506] Example: "The latest smartphones have a battery capacity of 4000mAh."

[1507] (Example 1)

[1508] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1509] Conventional product inquiry systems have struggled to provide timely and appropriate answers to user questions. This resulted in delays in responses and decreased user satisfaction. Furthermore, insufficient accuracy in speech recognition and natural language processing technologies made it difficult to accurately understand questions and generate appropriate answers.

[1510] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1511] In this invention, the server includes an input means for inputting a question about a product the user is considering purchasing; a conversion means for receiving the input from the input means and converting the speech to text using a speech recognition engine; an analysis means for analyzing the converted text using a natural language processing engine and understanding the content of the question; a generation means for generating an appropriate answer based on the question content analyzed by the analysis means; and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This makes it possible to accurately understand the user's questions in voice or text format and provide a quick and appropriate answer.

[1512] An "input method" is a means by which a user can input questions about a product.

[1513] "Conversion means" refers to means including a speech recognition engine used to convert the user's voice input into text.

[1514] "Analysis means" refers to a method of analyzing text-based questions using a natural language processing engine, understanding the content of the questions, and extracting important keywords and context.

[1515] "Generation means" refers to means including an artificial intelligence model used to generate appropriate answers based on the analyzed question content.

[1516] "Presentation means" refers to means including a speech synthesis engine used to provide the generated response to the user.

[1517] A "cloud-based server" is a server located remotely on the internet that provides services such as computing and data storage.

[1518] A "natural language processing engine" is a software component used to understand and analyze natural language text.

[1519] A "generative artificial intelligence model" is an artificial intelligence algorithm that can generate natural language text based on a given input.

[1520] This invention is a system that quickly and appropriately processes and provides answers to questions from users regarding products they are considering purchasing. Its main components are input means, conversion means, analysis means, generation means, and presentation means. These components and their operation are described in detail below.

[1521] This system begins with the user entering a question about the product via an input device. The user can enter the question in either voice or text format.

[1522] If voice input is selected, the terminal's conversion mechanism uses a speech recognition engine to convert the speech to text. This system uses a common engine widely used as a "speech recognition engine," for example. This conversion process converts the voice signal into a corresponding string of characters.

[1523] Next, the device uses a natural language processing engine as an analysis tool to analyze the converted text. Specifically, a natural language processing engine such as Hugging Face is used to tokenize the text and extract key keywords and context. For example, if the question is "What is the battery capacity of the latest smartphones?", keywords such as "smartphone" and "battery capacity" will be extracted.

[1524] The extracted keywords and contextual information are sent to a cloud-based server. A generative AI model runs on the server; for example, OpenAI's GPT-3 is used. This model generates an appropriate response based on the analysis results. The generated response is then sent from the server to the terminal.

[1525] The device that receives the response uses a speech synthesis engine as a presentation method to present the response to the user in audio format. By using a speech synthesis engine such as Amazon Polly, the text-based response is converted into audio and provided to the user. The response can also be presented in text format if necessary.

[1526] Specific example

[1527] If a user asks about the battery capacity of a new smartphone, the process would include the following:

[1528] 1. User: "What is the battery capacity of the latest smartphones?"

[1529] 2. The device receives the question and converts it to text using its speech recognition engine.

[1530] 3. The device uses a natural language processing engine to analyze the question and extract key keywords.

[1531] 4. Send the analysis results to a cloud-based server.

[1532] 5. The generative AI model on the server generates the answer "The latest smartphones have a battery capacity of 4000mAh."

[1533] 6. The device receives the response and converts it into speech using a speech synthesis engine.

[1534] 7. Device: Responds to the user with "The latest smartphones have a battery capacity of 4000mAh."

[1535] Example of a prompt

[1536] A prompt is a text-based question that is input into a generative artificial intelligence model. For example:

[1537] A user is asking, "What is the battery capacity of the latest smartphones?" Please tell me about the battery capacity of the latest smartphones.

[1538] In this way, implementing this system makes it possible to provide quick and accurate answers to user questions. This improves user satisfaction and ensures the system's flexibility and scalability.

[1539] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1540] Step 1:

[1541] Users enter questions about products they are considering purchasing via input methods. These input methods include voice input and text input. For example, if a user asks, "What is the battery capacity of the latest smartphones?", they can enter the question via voice. Input at this stage can be in either voice or text format.

[1542] Step 2:

[1543] The device receives voice input from the user. The input voice data is converted into text using a speech recognition engine. Here, the voice signal is processed as digital data, and the speech recognition engine analyzes that data to generate a corresponding string of characters. This conversion process generates a text-based question such as, "What is the battery capacity of the latest smartphones?"

[1544] Step 3:

[1545] The device passes the converted text-formatted question to a natural language processing (NLP) engine. The NLP engine tokenizes the text and extracts key keywords and context. For example, keywords such as "latest," "smartphone," and "battery capacity" are extracted and processed along with their contextual information. Here, the input is a text-formatted question, and the output is the keywords and contextual information as a result of the analysis.

[1546] Step 4:

[1547] The device sends the analyzed results to a cloud-based server. The data sent to the server at this stage includes extracted keywords and contextual information. For example, keywords such as "smartphone" and "battery capacity" are sent to the server.

[1548] Step 5:

[1549] The server generates an answer using a generative artificial intelligence model based on the received analysis results. Specifically, the server uses a GPT-based generative AI model to generate prompt sentences. For example, a prompt sentence might be generated that says, "The user is asking, 'What is the battery capacity of the latest smartphones?' Please tell me about the battery capacity of the latest smartphones." Based on this prompt sentence, the AI ​​model generates the answer, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is keywords and contextual information as analysis results, and the output is the answer in text format.

[1550] Step 6:

[1551] The device uses a speech synthesis engine to present the response received from the server. Specifically, it converts the text-based response into audio data. The speech synthesis engine analyzes the text data and generates appropriate speech based on its content. Through this process, the user can receive an audio response such as, "The battery capacity of the latest smartphones is 4000mAh." Here, the input is the text-based response, and the output is the audio response.

[1552] Step 7:

[1553] The user reviews the response provided in either audio or text format. The device plays the audio generated by the speech synthesis engine through its speaker. It can also display the response in text format if needed. This allows the user to obtain information quickly and accurately.

[1554] (Application Example 1)

[1555] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1556] Modern brick-and-mortar stores require prompt and accurate provision of a wide range of product information. However, traditional systems often struggled to respond to customer inquiries in a timely manner, leading to decreased customer satisfaction. Furthermore, employees sometimes had to spend time verifying information, resulting in reduced operational efficiency. Against this backdrop, there is a need for a system that can provide customers with product information in real time.

[1557] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1558] In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, an analysis means for receiving input from the input means and analyzing the content of the questions using natural language processing, a generation means for generating appropriate answers based on the question content analyzed by the analysis means, a presentation means for presenting the generated answers to the user, a dialogue means for interacting with the user in a physical store, and a guidance means for the dialogue means to provide the user with information about the location and ingredients of the products. As a result, users can obtain quick and accurate product information in physical stores, which is expected to improve customer satisfaction and operational efficiency.

[1559] An "input method" refers to a device or interface for users to input questions about a product they are considering purchasing.

[1560] "Analysis means" refers to a device or software that receives input from an input means and analyzes the content of a question using natural language processing.

[1561] "Generation means" refers to a device or software that generates appropriate answers based on the question content analyzed by the analysis means.

[1562] "Presentation means" refers to a device or software that presents the generated answer to the user.

[1563] "Dialogue means" refers to devices and software used to interact with users within a physical store.

[1564] A "guidance means" refers to a device or software that provides the user with information about the location and ingredients of a product through dialogue means.

[1565] A "cloud-based server" is a remotely located server that can be accessed via the internet.

[1566] A "generative AI model" is an artificial intelligence model that operates on a cloud-based server as a means of generation and generates appropriate answers.

[1567] "Natural language processing" is a technology that enables computers to understand and analyze human language.

[1568] A "speech synthesis engine" is a technology or device that converts text-based responses into speech.

[1569] This invention is a system that allows users to ask questions about products in a physical store and provides them with quick and appropriate answers. The system is configured as follows:

[1570] System Configuration

[1571] Input method:

[1572] This is a method for users to input questions about products they are considering purchasing. It uses an interface that allows for voice or text input.

[1573] Analysis method:

[1574] This system receives input from an input device and analyzes the content of the question using natural language processing. It utilizes natural language processing engines such as the Google Natural Language API.

[1575] Generation means:

[1576] This method generates appropriate answers based on analyzed question content. It utilizes a generative AI model (e.g., OpenAI GPT-4) installed on a cloud-based server.

[1577] Means of presentation:

[1578] This is a method for presenting generated answers to the user. It uses a speech synthesis engine (e.g., Amazon Polly) to provide the answers to the user in audio or text format.

[1579] Means of communication:

[1580] These are devices and software designed for interacting with users within physical stores. This allows users to obtain real-time information about product locations and ingredients while in the store.

[1581] Means of guidance:

[1582] Interaction means are devices or software that provide users with information about the location and ingredients of a product. This allows users to easily find out where a product is located, its ingredients, and other information.

[1583] Program processing

[1584] The system operates using servers (cloud-based) and terminals (devices in physical stores and users' smartphones). This system functions as follows:

[1585] The server is responsible for question analysis and answer generation. Natural language processing is used for analysis, and the generated answers are produced by a generative AI model.

[1586] The terminal (the user's smartphone or a guidance robot in a physical store) provides the user interface, allowing for question input, presentation of answers, and interaction.

[1587] The speech recognition engine uses the Google Speech-to-Text API to convert user voice input into text.

[1588] The speech synthesis engine uses Amazon Polly to convert the generated text-based responses into speech.

[1589] Specific example

[1590] Example 1: When a user wants to know the battery capacity of a new smartphone

[1591] 1. User: "What is the battery capacity of the latest smartphones?"

[1592] 2. The device receives this question and converts it to text using its speech recognition engine.

[1593] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "smartphone," "battery capacity").

[1594] 4. Based on the analysis results, a generative AI model generates an appropriate answer (e.g., "The latest smartphones have a battery capacity of 4000mAh.").

[1595] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[1596] Example 2: When a user wants to know the location of a specific product

[1597] 1. User: "Where can I find this product?"

[1598] 2. The device receives this question and converts it to text using its speech recognition engine.

[1599] 3. The server receives the question in text format and analyzes it using a natural language processing engine. It extracts key keywords (e.g., "product," "location").

[1600] 4. Based on the analysis results, a generative AI model generates an appropriate response (e.g., "This product is on shelf B3 on the second floor.").

[1601] 5. The generated response is converted into speech using a speech synthesis engine and provided to the user via the device.

[1602] Example of a prompt

[1603] "Please tell me the ingredients of the new shampoo."

[1604] "Where can I find this product?"

[1605] "What is the battery capacity of this smartphone?"

[1606] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1607] Step 1:

[1608] Enter questions about the product the user is considering purchasing.

[1609] Specific example: The user uses their smartphone to ask a voice question, "Where can I find this product?"

[1610] Input: Voice input

[1611] Output: Audio data

[1612] Step 2:

[1613] The device receives the user's voice input and uses a speech recognition engine to convert the voice data into text. The Google Speech-to-Text API is used for the speech recognition engine.

[1614] Specific example: Voice input is converted into text data such as "Where can I find this product?".

[1615] Input: Audio data

[1616] Output: Text data

[1617] Step 3:

[1618] The terminal sends text data to the server for analysis.

[1619] Specific example: A text-based question is sent to a server in the cloud.

[1620] Input: Text data

[1621] Output: Text data (sent to the server)

[1622] Step 4:

[1623] The server uses a natural language processing engine to analyze the received questions. The Google Natural Language API is used for this engine to extract key keywords and context.

[1624] Specific example: Extract the main keywords "product" and "location" from the question "Where can I find this product?".

[1625] Input: Text data

[1626] Output: Analysis results (keywords)

[1627] Step 5:

[1628] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate answers based on the analysis results.

[1629] Specific example: The response "This product is located on shelf B3 on the second floor" is generated.

[1630] Input: Analysis results (keywords)

[1631] Output: Generated response (text format)

[1632] Step 6:

[1633] The server sends the generated response to the terminal.

[1634] Specific example: A text response such as "This product is located on shelf B3 on the second floor" is sent to the terminal.

[1635] Input: Generated response (text format)

[1636] Output: Generated response (text format, sent to device)

[1637] Step 7:

[1638] The device converts the received text-based response into speech using a speech synthesis engine (e.g., Amazon Polly).

[1639] Specific example: An audio message is generated saying, "This product is located on shelf B3 on the second floor."

[1640] Input: Generated response (text format)

[1641] Output: Audio data

[1642] Step 8:

[1643] The device presents audio data to the user.

[1644] Specific example: The terminal plays a voice message saying, "This product is located on shelf B3 on the second floor."

[1645] Input: Audio data

[1646] Output: Audio presentation

[1647] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1648] This invention relates to a robotic system that provides product information quickly and appropriately according to user needs. In addition to a series of processes that involve receiving user questions, generating appropriate answers, and presenting them, this system also has the function of recognizing the user's emotions and responding accordingly. A detailed explanation follows below.

[1649] Program processing

[1650] Enter your question

[1651] The user enters a question into the system, either verbally or in text. For example, they might enter a question like, "What is the battery capacity of the latest smartphones?"

[1652] Receiving and recognizing questions

[1653] The device receives the user's voice or text input. If voice input is provided, the device uses a speech recognition engine to convert the voice into text. This text-based question is then sent to the next step.

[1654] Question analysis

[1655] The device analyzes text-based questions using a natural language processing (NLP) engine. The NLP engine tokenizes the questions and extracts important keywords and context. For example, key keywords such as "smartphone" and "battery capacity" are extracted, and the intent of the question is analyzed.

[1656] Recognition of emotions

[1657] The device uses an emotion recognition engine to analyze the user's emotions. The emotion recognition engine extracts emotional data from the user's voice and facial expressions, and identifies emotions such as joy, surprise, and anger.

[1658] Generating an answer

[1659] The device sends the question and sentiment analysis results to a cloud-based server. A generative AI model located on the server generates an appropriate response based on the analysis results. For example, if the user is feeling anxious, it will generate a reassuring response. A response such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time," might be generated.

[1660] Providing an answer

[1661] The device uses a speech synthesis engine to present the user with the responses received from the generative AI model. The responses are provided to the user in audio format, and also in text format if necessary.

[1662] Specific example

[1663] Scenario 1: When a user asks about the battery capacity of their new smartphone and expresses concern.

[1664] 1. User: "What is the battery capacity of the latest smartphones?"

[1665] 2. The device receives the question and converts it into text format using its speech recognition engine.

[1666] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[1667] 4. The device uses an emotion recognition engine to analyze if the user is feeling anxious.

[1668] 5. The terminal sends the analysis results to the server.

[1669] 6. The generative AI model on the server generates a reassuring response such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[1670] 7. The device receives the response and converts it into speech using a speech synthesis engine.

[1671] 8. Device: Responds to the user with, "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1672] Scenario 2: When the user asks about the price of a smartphone and gets excited

[1673] 1. User: "How much does this smartphone cost?"

[1674] 2. The device receives the question and converts it into text format using its speech recognition engine.

[1675] 3. The device uses an NLP engine to analyze the question and extract key keywords.

[1676] 4. The device uses an emotion recognition engine to analyze whether the user is excited.

[1677] 5. The terminal sends the analysis results to the server.

[1678] 6. A generative AI model on the server generates an exciting response: "This smartphone costs 79,800 yen. It's a great deal, so be sure to check it out!"

[1679] 7. The device receives the response and converts it into speech using a speech synthesis engine.

[1680] 8. Device: Responds to the user with "This smartphone is priced at 79,800 yen. It's a great price, so be sure to check it out!"

[1681] As described above, the present invention not only provides accurate and immediate answers to user questions, but also enables responses that take user emotions into consideration. This can further improve user satisfaction and purchasing intent. By using a cloud-based AI model, the system's flexibility and scalability are also ensured.

[1682] The following describes the processing flow.

[1683] Program processing steps

[1684] Step 1: Enter the question

[1685] 1. The user enters a question into the system, either verbally or in text.

[1686] Example: "What is the battery capacity of the latest smartphones?"

[1687] Step 2: Receiving and recognizing the question

[1688] 2. The device receives the user's voice input and converts it to text using a speech recognition engine.

[1689] The device uses a microphone to capture the user's voice.

[1690] The device uses speech recognition software to generate text data from speech.

[1691] Example output: The text data "What is the battery capacity of the latest smartphones?" is generated.

[1692] Step 3: Analyzing the Question

[1693] 3. The device analyzes the text-based question using a natural language processing (NLP) engine.

[1694] The device uses an NLP engine to tokenize the question (e.g., "latest", "smartphone", "battery capacity").

[1695] The device then analyzes key keywords and context to determine intent.

[1696] Example output: The keywords "smartphone" and "battery capacity" are extracted, along with the intent "request for battery capacity information."

[1697] Step 4: Recognizing Emotions

[1698] 4. The device uses an emotion recognition engine to analyze the user's emotions.

[1699] The device extracts emotional data from the user's voice tone, pace, and facial expressions.

[1700] The device identifies the category of emotion (e.g., joy, surprise, anger, anxiety).

[1701] Example output: The emotion "anxiety" is recognized.

[1702] Step 5: Sending analysis results and sentiment data

[1703] 5. The device converts the question analysis results and sentiment analysis results into JSON format and sends them to a cloud-based server.

[1704] The terminal packages the question analysis and sentiment analysis results into a single JSON file.

[1705] The device sends JSON data to the server via the internet.

[1706] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[1707] Step 6: Generating the answer

[1708] 6. The server uses a generative AI model to generate an appropriate response.

[1709] The server receives the transmitted JSON data and runs a generative AI model based on the question and sentiment analysis results.

[1710] Generative AI models generate responses that match the user's emotions (e.g., reassuring responses for users who are feeling anxious).

[1711] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[1712] Step 7: Receiving responses and converting them to audio.

[1713] 7. The device receives the response from the server and converts it into speech using a speech synthesis engine.

[1714] The device receives the response data in JSON format.

[1715] The device uses a speech synthesis engine to convert text data into speech data.

[1716] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1717] Step 8: Present your answer

[1718] 8. The device provides the user with an answer via voice or text.

[1719] The device plays audio data generated by a speech synthesis engine.

[1720] If necessary, the response will also be displayed in text format on the screen.

[1721] Example: The voice response might say, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[1722] Specific example

[1723] Scenario: When a user asks about their smartphone's battery capacity and expresses concern.

[1724] Step 1:

[1725] User: "What is the battery capacity of the latest smartphones?"

[1726] Step 2:

[1727] The device receives the audio and converts it to text.

[1728] Example output: Text "What is the battery capacity of the latest smartphones?"

[1729] Step 3:

[1730] The device analyzes the text using an NLP engine.

[1731] The keywords "smartphone" and "battery capacity," and the intention "request for battery capacity information" are extracted.

[1732] Step 4:

[1733] The device analyzes the user's emotions using its emotion recognition engine.

[1734] Example output: The emotion "anxiety" is recognized.

[1735] Step 5:

[1736] The device sends the analysis results and sentiment analysis results to the server in JSON format.

[1737] Example output: {"intent": "Request for battery capacity information", "keywords": ["smartphone", "battery capacity"], "emotion": "anxiety"}

[1738] Step 6:

[1739] The server generates the answer using a generative AI model.

[1740] Example output: {"response": "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it lasts a very long time."}

[1741] Step 7:

[1742] The device receives the response and converts it into speech using a speech synthesis engine.

[1743] Example output: "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1744] Step 8:

[1745] The device provides the user with the answer via voice.

[1746] Example: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, it lasts a very long time."

[1747] (Example 2)

[1748] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1749] Modern consumers need to obtain increasingly diverse product information quickly and accurately. Furthermore, traditional product information systems often provide uniform answers without considering user emotions or intentions, which is insufficient for improving user satisfaction. This has led to challenges such as decreased purchase intent and a decline in the quality of the user experience.

[1750] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting questions about a product the user is considering purchasing, a conversion means for receiving input from the input means and converting the voice data into text format using a speech recognition engine, an analysis means for analyzing the converted text data using a natural language processing engine and extracting key keywords, an emotion recognition means for recognizing the user's emotional state, a generation means for generating an appropriate answer based on the data obtained by the analysis means and the emotion recognition means, and a presentation means including a speech synthesis engine for presenting the generated answer to the user. This enables the rapid provision of an appropriate answer that meets the user's needs and also allows for a response that takes the user's emotions into consideration.

[1751] "Input means" refers to a device or method that provides an interface for a user to input a question.

[1752] "Conversion means" refers to a device or method used to convert audio data received from an input means into text format.

[1753] "Analysis means" refers to a device or method that analyzes text data obtained by the conversion means using natural language processing techniques to extract key keywords and context.

[1754] "Emotion recognition means" refers to a device or method that analyzes a user's voice and facial expression data to determine their emotional state.

[1755] "Generation means" refers to a device or method that generates an appropriate response using a generative AI model on the cloud, based on data obtained by analysis means and emotion recognition means.

[1756] "Presentation means" refers to a device or method used to present the generated response to the user, and includes providing the response in voice format using a speech synthesis engine.

[1757] A "generative AI model" is an artificial intelligence model used to generate appropriate text or information from input data.

[1758] A "prompt" is an instruction given to a generative AI model, and it serves as the basis for determining the content of the generated response.

[1759] This invention relates to a system that allows users to input questions about products they are considering purchasing and provides appropriate answers based on those questions. This system has the function of recognizing the user's emotional state and generating corresponding answers. The embodiments of the system are described in detail below.

[1760] System Configuration

[1761] This system consists of the following main components:

[1762] 1. Input Methods: This is the interface for the user to input questions. This includes microphones that support voice input and keyboards that support text input.

[1763] 2. Conversion method: In the case of voice input, a speech recognition engine is used as the conversion method. In this embodiment, the Google Speech-to-Text API is used to convert speech to text.

[1764] 3. Analysis Method: The converted text is analyzed using a natural language processing (NLP) engine. For example, Amazon Comprehend is used to extract the main keywords and intent of the question.

[1765] 4. Emotion Recognition Method: Emotional data is analyzed from the user's voice and facial expressions. The Microsoft Azure Emotion API is used to determine the user's emotional state (joy, anxiety, excitement, etc.).

[1766] 5. Generation means: Based on the data obtained by the analysis means and the emotion recognition means, a generative AI model deployed on a cloud-based server is used. In this embodiment, OpenAI's GPT-3 generates the response.

[1767] 6. Presentation Method: A speech synthesis engine is used to present the generated responses to the user. Amazon Polly is used to provide the generated text responses in audio format.

[1768] Implementation method

[1769] The system operates as follows: When a user enters a question, the input device receives it. In the case of voice input, the device converts the speech to text using the Google Speech-to-Text API. The converted text is then analyzed by the analysis device using Amazon Comprehend. Here, key keywords and the intent of the question are extracted. In parallel, the emotion recognition device analyzes the emotional state from the voice data and the user's facial expressions using the Microsoft Azure Emotion API.

[1770] The analyzed data and emotional state are sent to a cloud-based server, where appropriate responses are generated using OpenAI's GPT-3. The generated responses are returned to the device and converted to audio format using Amazon Polly. Finally, the responses are presented to the user in both audio and text formats.

[1771] Specific example

[1772] Scenario 1: When a user asks about the battery capacity of their new smartphone and expresses concern.

[1773] 1. User: "What is the battery capacity of the latest smartphones?"

[1774] 2. The device receives the question and converts it to text format using the Google Speech-to-Text API.

[1775] 3. The device uses Amazon Comprehend to analyze the questions and extract key keywords.

[1776] 4. The device uses the Microsoft Azure Emotion API to analyze if the user is feeling anxious.

[1777] 5. The device sends the analysis results to the cloud server.

[1778] 6. The generative AI model (OpenAI GPT-3) on the server generates the response: "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[1779] 7. The device receives the response and converts it to speech using Amazon Polly.

[1780] 8. Device: Responds to the user with, "The latest smartphones have a battery capacity of 4000mAh. Rest assured, it will last a very long time."

[1781] Example of a prompt

[1782] "Please tell me about the battery capacity of this smartphone. Generate an answer that will reassure users who are feeling anxious."

[1783] The system, possessing the above characteristics, can quickly provide appropriate answers tailored to user needs and even respond in a way that considers user emotions. This, in turn, can improve user satisfaction and purchasing intent.

[1784] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1785] Step 1: Enter your question

[1786] The user enters a question. The question can be entered via voice or text. For example, consider the question, "What is the battery capacity of the latest smartphones?" This input data is captured as voice data if entered via voice, or directly as text data if entered via text.

[1787] Step 2: Receiving the question and speech recognition

[1788] The device receives user input. In the case of voice input, the Google Speech-to-Text API is used to convert the speech to text. The input is audio data, and the output is a question in text format. Specifically, the device's microphone captures the audio, and the API is called to convert the audio into text data.

[1789] Step 3: Question analysis and tokenization

[1790] The device receives a text-based question and analyzes it using a natural language processing engine such as Amazon Comprehend. The input is a text-based question, and the output is a list of tokenized words and phrases. Specifically, the NLP engine breaks down the text into words and phrases and tokenizes them into categories such as "latest," "smartphone," and "battery capacity."

[1791] Step 4: Keyword Extraction and Intent Interpretation

[1792] The device extracts key keywords from the tokenized data and interprets the intent of the question. The input is the tokenized data obtained in step 3, and the output is the key keywords and their related information. Specifically, the NLP engine scans the tokens and recognizes that "smartphone" is a product category and "battery capacity" is an attribute.

[1793] Step 5: Recognizing and analyzing emotions

[1794] The device uses an emotion recognition engine, such as the Microsoft Azure Emotion API, to analyze the user's emotions. Input is voice or facial expression data, and output is the emotional state (e.g., joy, anxiety, excitement). Specifically, in the case of voice, emotions are analyzed from the tone and speed of the voice, and in the case of facial expressions, emotions are analyzed from images captured by the camera.

[1795] Step 6: Generating and optimizing answers

[1796] The device sends analysis results and sentiment data to a cloud-based server. On the server, generative AI models such as OpenAI's GPT-3 generate responses. The input consists of analyzed keywords and sentiment states, and the output is a response in appropriate text format. Specifically, data is sent to the server, and the generative AI model generates a response based on prompts.

[1797] Step 7: Presentation of answers and speech synthesis

[1798] The device uses a speech synthesis engine such as Amazon Polly to present the user with the response received from the generative AI model. The input is a text-based response, and the output is a voice-based response. Specifically, the speech synthesis engine converts the text response into a voice file, which is then played through the device's speaker, or the text-based response is displayed on the screen.

[1799] (Application Example 2)

[1800] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1801] In modern brick-and-mortar stores, it is crucial to respond quickly and appropriately to customers' questions about products they are considering purchasing. However, traditional systems only provide information in response to each customer's question, failing to address their emotions and needs. As a result, the quality and satisfaction of the information customers receive are limited, making it difficult to increase their purchasing intent. Furthermore, staff shortages and the difficulty of multilingual support among staff are also challenges. To solve these problems, a system is needed that recognizes customer emotions and provides appropriate answers based on those emotions.

[1802] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an input means for inputting a question about a product that the user is considering purchasing, an analysis means for receiving the input from the input means and analyzing the content of the question using natural language processing, a generation means for generating an appropriate answer based on the question content analyzed by the analysis means, an emotion recognition means for recognizing the user's emotions, an answer adjustment means for adjusting the answer generated based on the emotions recognized by the emotion recognition means, and a presentation means for presenting the generated and adjusted answer to the user. This makes it possible to quickly provide appropriate product information that corresponds to the customer's emotions, thereby increasing customer satisfaction and purchasing intent.

[1803] "Input means" refers to a device or function that allows a user to input questions or information into a system.

[1804] "Analysis means" refers to a function that understands and analyzes questions and information received from input means using natural language processing technology.

[1805] A "generation means" is a function for creating an appropriate answer based on the information analyzed by the analysis means.

[1806] "Emotion recognition means" refers to a function that identifies and analyzes a user's emotions from their voice, facial expressions, tone of voice, etc.

[1807] A "response adjustment mechanism" is a function that adjusts responses generated based on recognized emotions and provides them in an appropriate format.

[1808] "Presentation means" refers to a function for presenting generated and adjusted responses to the user in audio or text format.

[1809] A "generative AI model" is an artificial intelligence model that generates appropriate responses or information based on input data in natural language processing and generative tasks.

[1810] A "cloud-based server" is a system that performs data processing and storage on remote servers accessible via the internet.

[1811] A "speech synthesis engine" is a technology that converts text data into speech data and outputs it as natural-sounding speech.

[1812] This invention is a system that provides product information quickly and appropriately in response to customer inquiries in physical stores. The following describes specific implementations of this system.

[1813] System Configuration

[1814] The system includes the following main components:

[1815] An "input method" for users to enter questions.

[1816] "Analysis means" for analyzing received questions

[1817] A "generation means" for generating answers based on the content of the questions.

[1818] "Means for recognizing user emotions"

[1819] "Response adjustment means" for adjusting responses based on perceived emotions.

[1820] A "presentation method" for presenting adjusted answers to users.

[1821] This section describes the specific functions of each component and the hardware and software that support them.

[1822] Hardware and software details

[1823] 1. Input method:

[1824] This includes a microphone for users to input questions verbally, as well as a touch panel.

[1825] 2. Analysis method:

[1826] A speech recognition engine for converting audio data into text (e.g., Google Cloud Speech-to-Text).

[1827] A natural language processing engine (e.g., Amazon Comprehend) that analyzes the converted text data and extracts keywords and context.

[1828] 3. Generation means:

[1829] A generative AI model (e.g., OpenAI GPT-4) that runs on a cloud-based server and generates appropriate answers based on the question.

[1830] 4. Emotion recognition means:

[1831] An emotion recognition engine (e.g., Microsoft Azure Emotional API) analyzes a user's voice and facial expressions to recognize their emotions.

[1832] 5. Answer adjustment means:

[1833] The generated response is adjusted based on the emotional data obtained by the emotion recognition system.

[1834] 6. Means of presentation:

[1835] A speech synthesis engine (e.g., Amazon Polly) for presenting adjusted responses to users.

[1836] Processing flow and specific examples

[1837] Specific example 1:

[1838] A scenario where a customer comes into the store and asks about the battery life of their new smartphone.

[1839] User: "What is the battery capacity of the latest smartphones?"

[1840] Voice data is input through the microphone, and a speech recognition engine converts that data into text.

[1841] A natural language processing engine analyzes the text and extracts key keywords.

[1842] The system determines that the user's emotion is "anxiety" based on their recognized voice tone and facial expressions.

[1843] The question content and sentiment data are sent to a generative AI model on a cloud server.

[1844] A generative AI model generates responses, such as, "The latest smartphones have a battery capacity of 4000mAh. Don't worry, they last a very long time."

[1845] The speech synthesis engine converts the generated response into speech and provides it to the user.

[1846] Example of a prompt:

[1847] "User is anxious. Question: What is the battery capacity of the latest smartphones?"

[1848] Benefits of the entire system

[1849] This system will significantly improve customer service in physical stores. In particular, it is expected to increase customer satisfaction and purchasing intent by quickly providing optimal answers tailored to each customer's emotions in response to their questions. The use of a cloud-based AI model ensures flexibility and scalability, making it suitable for use in a variety of physical stores.

[1850] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1851] Step 1:

[1852] The user enters a question. If the user enters the question by voice using a microphone, the voice signal becomes the input data. If the user enters the question by text using a touch panel, the text data becomes the input data. This data is received by the terminal.

[1853] Step 2:

[1854] The device uses a speech recognition engine to convert voice input into text data. The Google Cloud Speech-to-Text engine is used to convert the voice signal into text format. This text data is then input into the next step.

[1855] Step 3:

[1856] The device uses a natural language processing (NLP) engine to analyze text data. Amazon Comprehend is used to tokenize the text and extract key keywords and context. This analysis helps understand the intent of the question, and the extracted keywords are then fed into the next step.

[1857] Step 4:

[1858] The device uses an emotion recognition engine to analyze the user's emotional state. It uses the Microsoft Azure Emotional API to obtain emotional data from the user's voice tone and facial expressions. For example, it analyzes voice intonation and changes in facial expressions to determine if the user is experiencing emotions such as "anxiety," "excitement," or "joy." This emotional data is then input into the next step.

[1859] Step 5:

[1860] The device sends the analyzed question data and sentiment data to a cloud-based server. The server uses a generative AI model (OpenAI GPT-4) to generate appropriate responses based on the input data. For example, if the user is feeling "anxious," a reassuring response will be generated. The text data of the generated response is then input into the next step.

[1861] Step 6:

[1862] The device uses a speech synthesis engine to convert the generated response text into audio data. Amazon Polly is used to convert the text-based response into natural-sounding speech. This audio data is then presented to the user.

[1863] Step 7:

[1864] The user receives voice-generated answers. The answers are delivered in audio format via a speaker, which serves as the presentation method. Furthermore, the answers are also displayed in text format on the touchscreen if necessary. This allows the user to obtain answers to their product-related questions and receive information that is relevant to their emotional state.

[1865] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1866] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1867] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1868] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1869] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1870] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1871] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1872] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1873] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1874] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1875] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1876] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1877] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1878] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1879] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1880] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1881] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1882] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1883] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1884] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1885] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1886] The following is further disclosed regarding the embodiments described above.

[1887] (Claim 1)

[1888] An input method for users to enter questions about products they are considering purchasing,

[1889] An analysis means that receives input from an input means and analyzes the content of the question using natural language processing,

[1890] A generation means for generating appropriate answers based on the question content analyzed by the analysis means,

[1891] A means of presenting the generated answer to the user,

[1892] A system that includes this.

[1893] (Claim 2)

[1894] The system according to claim 1, characterized in that the generation means uses a generative AI model that operates on a cloud-based server.

[1895] (Claim 3)

[1896] The system according to claim 1, characterized in that the presentation means presents the answer to the user in voice using a speech synthesis engine.

[1897] "Example 1"

[1898] (Claim 1)

[1899] An input method for users to enter questions about products they are considering purchasing,

[1900] A conversion means that receives input from an input means and converts the speech into text using a speech recognition engine,

[1901] An analysis method that uses a natural language processing engine to analyze the converted text and understand the content of the question,

[1902] A generation means for generating appropriate answers based on the question content analyzed by the analysis means,

[1903] A presentation means including a speech synthesis engine for presenting the generated answer to the user,

[1904] A system that includes this.

[1905] (Claim 2)

[1906] The system according to claim 1, characterized in that the generation means uses a generative artificial intelligence model that operates on a cloud-based server.

[1907] (Claim 3)

[1908] The system according to claim 1, characterized in that the analysis means tokenizes the question using a natural language processing engine and extracts important keywords and context.

[1909] "Application Example 1"

[1910] (Claim 1)

[1911] An input method for users to enter questions about products they are considering purchasing,

[1912] An analysis means that receives input from an input means and analyzes the content of the question using natural language processing,

[1913] A generation means for generating appropriate answers based on the question content analyzed by the analysis means,

[1914] A means of presenting the generated answer to the user,

[1915] A means of communication for interacting with users in a physical store,

[1916] The dialogue means is a guidance means for providing the user with information about the location and ingredients of the product,

[1917] A system that includes this.

[1918] (Claim 2)

[1919] The system according to claim 1, characterized in that the generation means uses a generative AI model that operates on a cloud-based server.

[1920] (Claim 3)

[1921] The system according to claim 1, characterized in that the presentation means presents the answer to the user in voice using a speech synthesis engine.

[1922] "Example 2 of combining an emotion engine"

[1923] (Claim 1)

[1924] An input method for users to enter questions about products they are considering purchasing,

[1925] A conversion means that receives input from an input means and converts the audio data into text format using a speech recognition engine,

[1926] An analysis method that analyzes the converted text data using a natural language processing engine and extracts key keywords,

[1927] An emotion recognition means for recognizing the user's emotional state,

[1928] A generation means for generating an appropriate response based on data obtained by an analysis means and an emotion recognition means,

[1929] A presentation means including a speech synthesis engine for presenting the generated answer to the user,

[1930] A system that includes this.

[1931] (Claim 2)

[1932] The system according to claim 1, characterized in that the generation means uses a generative AI model that operates on a cloud-based server.

[1933] (Claim 3)

[1934] The system according to claim 1, characterized in that the emotion recognition means analyzes emotions from the user's voice or facial expression data.

[1935] "Application example 2 when combining with an emotional engine"

[1936] (Claim 1)

[1937] An input method for users to enter questions about products they are considering purchasing,

[1938] An analysis means that receives input from an input means and analyzes the content of the question using natural language processing,

[1939] A generation means for generating appropriate answers based on the question content analyzed by the analysis means,

[1940] A means of recognizing user emotions,

[1941] A response adjustment means that adjusts the generated response based on the emotion recognized by the emotion recognition means,

[1942] A presentation means for presenting generated and adjusted answers to the user,

[1943] A system that includes this.

[1944] (Claim 2)

[1945] The system according to claim 1, characterized in that the generation means and the response adjustment means use a generative AI model that operates on a cloud-based server.

[1946] (Claim 3)

[1947] The system according to claim 1, characterized in that the presentation means presents the answer to the user in voice using a speech synthesis engine. [Explanation of symbols]

[1948] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An input method for users to enter questions about products they are considering purchasing, An analysis means that receives input from an input means and analyzes the content of the question using natural language processing, A generation means for generating appropriate answers based on the question content analyzed by the analysis means, A means of presenting the generated answer to the user, A system that includes this.

2. The system according to claim 1, characterized in that the generation means uses a generative AI model that operates on a cloud-based server.

3. The system according to claim 1, characterized in that the presentation means presents the answer to the user in voice using a speech synthesis engine.

Images