system

Abstract

We provide the system. [Solution] A voice recognition means that receives voice input from the customer and analyzes its contents, A product search method that searches for relevant product information from a product database based on the analyzed content, An information presentation method that displays searched product information and explains it to the customer using voice and screen display, A payment method that guides customers through the purchase process for the products they have selected and allows them to make electronic payments. A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional large-scale household appliance retailers, when customers select products, they often feel that the explanations and solicitations by human salespersons are persistent. Also, there are situations where it is difficult for customers to collect and compare product information at their own pace. Therefore, there is a demand for constructing an environment where customers can select products without stress. Furthermore, means for performing the purchase process quickly and safely are also necessary. Also, it is a problem that there is a lack of a method for efficiently utilizing customers' past purchase histories and preferences to recommend optimal products.

Means for Solving the Problems

[0005] This invention relates to a system that includes a speech recognition means for receiving and analyzing customer voice input, a product search means for searching for relevant product information from a product database based on the analyzed content, an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display, and a payment means for guiding the customer through the purchase procedure of the product selected by the customer and performing electronic payment. It also includes a recommendation means for recommending the most suitable product based on the customer's past purchase history and preferences. Furthermore, by using a server for managing product information and training an AI model, the system can provide the customer with optimal product information and complete the purchase process smoothly. This system allows customers to obtain product information without stress, select products at their own pace, and complete purchases quickly and safely.

[0006] "Voice recognition means" refers to a device or program that analyzes voice input from a customer and converts it into text data.

[0007] "Product search means" refers to a device or program for searching a product database for relevant product information based on text data obtained by speech recognition means.

[0008] A "product database" is a data storage system that accumulates and manages information about products within large consumer electronics retail stores.

[0009] "Information presentation means" refers to a device or program for providing searched product information to customers, and presents the information in the form of audio and screen display.

[0010] A "payment method" is a device or program that allows a customer to complete the purchase process for selected goods and finalize the electronic payment.

[0011] A "recommendation tool" is a device or program that recommends the most suitable products based on a customer's past purchase history and preferences.

[0012] A "server" is a computer system used for managing product information and training AI models. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, when an emotion engine is combined. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiment for Implementing the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention relates to a robotic system that communicates with customers in large consumer electronics stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[0035] server

[0036] Product Database Management

[0037] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies that there are no omissions or duplicates.

[0038] Training an AI model

[0039] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[0040] Payment processing management

[0041] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[0042] Terminal (robot)

[0043] Product Information

[0044] The terminal greets customers verbally upon entering the store and receives customer requests using voice recognition. Based on product information retrieved from the server, the terminal provides customers with detailed information via voice and screen display.

[0045] Product comparison and recommendations

[0046] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality."

[0047] Electronic payment support

[0048] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process. After the customer selects a payment method, the terminal sends the payment information to the server and completes the process. Upon successful payment completion, the terminal notifies the customer with the message, "Payment complete. Thank you."

[0049] User (customer)

[0050] Product search and inquiry

[0051] Users can request the products they want from the device using voice commands and receive detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[0052] Product Comparison Selection

[0053] If a user wants to compare multiple products, they can get detailed information by giving instructions to the device. For example, if a user asks, "Which one is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0054] Purchase procedure

[0055] Once the user selects an item and indicates their intention to purchase, the terminal guides them through the electronic payment method and completes the payment using a credit or debit card. The user simply says, "I'll buy this item," and the terminal prompts them to "Please insert your credit card," completing the payment.

[0056] Thus, the system according to the present invention helps customers collect product information, compare and consider options, and ultimately complete the purchase quickly and safely, without stress. This significantly improves the in-store shopping experience.

[0057] The following describes the processing flow.

[0058] Step 1:

[0059] Initial contact (device)

[0060] When a customer enters the store, the terminal uses voice recognition to greet them with, "Welcome. What can I help you with?" Once the customer responds, the system automatically converts the voice into text.

[0061] Step 2:

[0062] Request acceptance (terminal)

[0063] When a customer states that they want to know about a specific product, the terminal analyzes their voice and converts it into text data. This text data is then sent to a server to request a search for product information.

[0064] Step 3:

[0065] Product information search (server)

[0066] Based on the received request, the server searches its product database for relevant product information. For example, in response to a request for "new 4K TV," it extracts information such as specifications, price, availability, and user reviews.

[0067] Step 4:

[0068] Information provision (terminal)

[0069] The terminal provides customers with product information retrieved from the server via voice and on-screen display. For example, it might explain, "This 4K TV is 55 inches and priced at $800. It is currently in stock."

[0070] Step 5:

[0071] Product comparison (devices)

[0072] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality."

[0073] Step 6:

[0074] Confirmation of purchase intent (device)

[0075] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[0076] Step 7:

[0077] Electronic payment instructions (terminal)

[0078] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[0079] Step 8:

[0080] Sending payment information (terminal)

[0081] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[0082] Step 9:

[0083] Payment processing and confirmation (server)

[0084] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[0085] Step 10:

[0086] Payment completion notification (device)

[0087] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[0088] This series of steps allows customers to easily search for, compare, and complete product information within the store. This enables customers to enjoy a stress-free and efficient shopping experience.

[0089] (Example 1)

[0090] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] Traditional in-store purchasing processes were cumbersome, involving the collection, comparison, and purchase of product information, increasing customer stress and inconvenience. Furthermore, the need for store staff to provide information and payment support led to challenges such as staff shortages and inconsistent skill levels, resulting in decreased service quality. Concerns also existed regarding information discrepancies between online and in-store purchases, as well as payment security.

[0092] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0093] In this invention, the server includes a voice recognition means for receiving and analyzing customer voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for explaining detailed product information obtained from a management server to the customer via voice and screen display; a product comparison means for comparing multiple products and explaining their features; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables customers to quickly and easily obtain and compare product information and safely complete the purchase procedure. The server also includes a recommendation means for recommending the most suitable product based on the customer's past purchase history and preferences; a data management means for verifying the integrity of product information and storing it in a database; and a security management means for executing a fraud detection algorithm to ensure the security of payments. This improves the consistency and security of the service and enhances customer satisfaction.

[0094] "Voice recognition means" refers to technology or equipment for receiving customer voice input and analyzing its content.

[0095] "Product search means" refers to a technology or device for searching for relevant product information from a product database based on the analyzed content.

[0096] "Information presentation means" refers to technology or equipment for providing customers with detailed product information obtained from a management server via audio and screen display.

[0097] A "product comparison method" is a technology or device for comparing multiple products and explaining their characteristics.

[0098] "Payment method" refers to the technology or device that guides customers through the purchase process of selected goods and enables electronic payment.

[0099] "Recommendation methods" refer to technologies or devices for recommending the most suitable products based on a customer's past purchase history and preferences.

[0100] "Data management means" refers to technology or equipment for verifying the integrity of product information and storing it in a database.

[0101] "Security management measures" refer to technologies or devices that execute fraud detection algorithms and ensure the security of payments.

[0102] This invention relates to a robotic system that communicates with customers in large stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[0103] server

[0104] Product Database Management

[0105] The server manages the product database for large stores. Using Python libraries such as Beautiful Soup and Scrapy, the server periodically collects product specifications, prices, stock availability, and review information, and checks data integrity. This ensures that detailed product data is stored in the database. For example, by using scraping techniques to collect information from product web pages, it's possible to update the data in real time.

[0106] Training an AI model

[0107] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. Neural network models are built and trained using TENSORFLOW® and PyTorch. For example, the AI ​​model is trained epoch-by-epoch to solve classification and regression tasks. Specifically, the model is evaluated using prompts such as, "Please recommend a model based on reviews of the latest refrigerators."

[0108] Payment processing management

[0109] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. The server ensures secure data communication using the SSL / TLS protocol and manages consistent transactions by building a payment management system with the Django framework. It also uses machine learning to train anomaly detection algorithms based on historical transaction data to evaluate new transactions.

[0110] Terminal (robot)

[0111] Product Information

[0112] The terminal greets customers with a voice message upon entering the store and uses voice recognition technology to receive requests from customers. It analyzes customer voices using Microsoft® Azure® Cognitive Services and Google® Cloud's Speech-to-Text API. Based on product information obtained from the server, it uses Google Cloud's Text-to-Speech API to synthesize speech and provides detailed information through voice and screen display. For example, if a customer says, "I'm looking for a new refrigerator," the terminal will respond, "Here are our latest models."

[0113] Product comparison and recommendations

[0114] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. If a customer says, "Please compare the TVs," the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." An example of a specific prompt would be, "Which refrigerator is the most energy-efficient?"

[0115] Electronic payment support

[0116] After the customer decides to purchase, the terminal guides them through the electronic payment process using voice and on-screen displays. The customer's payment information is sent to a server for processing. Once the server completes the payment processing, the terminal notifies the customer that "payment is complete." Specifically, when the customer says, "I'll buy this item," the terminal prompts them to "insert your credit card," and the payment is completed.

[0117] User (customer)

[0118] Product search and inquiry

[0119] Users can request desired products from the device using voice commands and obtain detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[0120] Product Comparison Selection

[0121] Users can request product comparisons from the device using voice commands and obtain detailed information. For example, if asked, "Which refrigerator is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity." A specific example of a prompt would be, "Please tell me the difference between these two refrigerators."

[0122] Purchase procedure

[0123] Users indicate their intention to purchase by voice and make payments using credit or debit cards according to the terminal's instructions. For example, if a user says, "I want to buy this item," the terminal will prompt them to "Please insert your credit card" and assist them through the entire payment process.

[0124] Thus, the system of the present invention provides a comprehensive flow that allows customers to smoothly obtain and compare product information that suits their needs and to make purchases securely, significantly improving the in-store shopping experience. Furthermore, the consistency and security of the service are guaranteed through server data management and security measures.

[0125] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0126] Step 1:

[0127] The server collects product data. The server uses Python libraries such as Beautiful Soup and Scrapy to perform web scraping and collect product specifications, prices, stock availability, and review information from large retail stores.

[0128] Input: Product webpage URL

[0129] Data processing: HTML parsing, extraction of necessary data.

[0130] Output: Product data list (specifications, price, stock status, review information)

[0131] Step 2:

[0132] The server checks for data integrity. The server verifies the collected product data for duplicates or missing data. If duplicates are found, the data is updated with the latest data; if missing data is found, it is collected again.

[0133] Input: Product data list

[0134] Data processing: Duplicate checking, data uniqueness verification, recollection of missing data.

[0135] Output: Consistent product database

[0136] Step 3:

[0137] The server stores product data in the database. An ORM such as SQLAlchemy is used to ensure consistent product data is stored in the database.

[0138] Input: Consistent product database

[0139] Data processing: Saving operations to a database

[0140] Output: Updated product database

[0141] Step 4:

[0142] The server trains the AI ​​model. The server uses product data, user reviews, and past purchase history to train the generative AI model. It builds and trains the neural network using TensorFlow or PyTorch.

[0143] Input: Product data, user reviews, past purchase history

[0144] Data processing: Training and evaluation of AI models

[0145] Output: Trained generative AI model

[0146] Step 5:

[0147] The device receives the customer's voice request. The device performs speech recognition using Microsoft Azure's Cognitive Services or Google Cloud's Speech-to-Text API and retrieves the customer's request in text format.

[0148] Input: Customer voice request

[0149] Data processing: Speech-to-text conversion

[0150] Output: Request content in text format

[0151] Step 6:

[0152] The device retrieves product information. The device sends a text request to the server and retrieves the corresponding product information.

[0153] Input: Request content in text format

[0154] Data processing: Sending requests to the server, retrieving product information.

[0155] Output: Product information (specifications, price, stock status, review information)

[0156] Step 7:

[0157] The terminal presents product information to the customer. The terminal uses Google Cloud's Text-to-Speech API to synthesize speech and also displays the product information on the screen.

[0158] Input: Product Information

[0159] Data processing: Voice generation, screen display updates.

[0160] Output: Providing product information via audio and screen display.

[0161] Step 8:

[0162] The terminal receives a customer comparison request. The terminal uses speech recognition technology to obtain the customer comparison request in text format.

[0163] Input: Customer voice request

[0164] Data processing: Speech-to-text conversion

[0165] Output: Comparison request in text format

[0166] Step 9:

[0167] The device retrieves comparison information. The device sends a comparison request to the server and retrieves comparison information for multiple products.

[0168] Input: Text format comparison request

[0169] Data processing: Sending requests to the server, obtaining comparison information.

[0170] Output: Comparison information (features, price, availability, etc.)

[0171] Step 10:

[0172] The terminal presents the customer with product comparison information. The terminal explains the acquired comparison information to the customer through voice and screen display.

[0173] Input: Comparison Information

[0174] Data processing: Voice generation, screen display updates.

[0175] Output: Provision of comparative information via audio and screen display.

[0176] Step 11:

[0177] The terminal receives the customer's purchase intention. When the customer indicates their intention to purchase a specific product, speech recognition technology is used to obtain that intention in text format.

[0178] Input: Customer's voice expression of purchase intent

[0179] Data processing: Speech-to-text conversion

[0180] Output: Purchase intention in text format

[0181] Step 12:

[0182] The terminal guides the customer through the payment process. The terminal uses voice and on-screen displays to instruct the customer to complete the payment using a credit or debit card.

[0183] Input: Customer's purchase intention

[0184] Data processing: Voice generation for payment instructions, updating of screen display.

[0185] Output: Payment instructions

[0186] Step 13:

[0187] The terminal sends payment information to the server. When a customer enters payment information or uses a card, the terminal sends that information to the server and requests processing.

[0188] Input: Customer payment information

[0189] Data processing: Sending payment information to the server

[0190] Output: Payment request to the server

[0191] Step 14:

[0192] The server processes the payment. The server uses the SSL / TLS protocol to perform secure payment processing, transaction management, and fraud detection.

[0193] Input: Customer payment information

[0194] Data processing: Payment processing, fraud detection, transaction management

[0195] Output: Payment processing result

[0196] Step 15:

[0197] The terminal notifies the customer that the payment is complete. Upon receiving the successful payment processing result from the server, the terminal notifies the customer via voice and on-screen display that "Payment complete."

[0198] Input: Payment processing result from the server

[0199] Data processing: Generate audio for notifications, update screen display.

[0200] Output: Payment completion notification to customer

[0201] (Application Example 1)

[0202] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0203] Traditional systems made it difficult to provide a smooth purchasing experience for customers when searching for and comparing products online, as they had to find the necessary information. Furthermore, product descriptions were uniform, making it difficult to address the individual needs of customers. In addition, electronic payments require security and smoothness in the transaction process.

[0204] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0205] In this invention, the server includes a speech recognition means for receiving customer voice input and analyzing its content; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment; and a product description generation means for generating a description of product features using a generation AI model. This makes it possible to provide product information quickly and accurately based on customer voice input and generate product descriptions tailored to individual needs. It also enables safe and smooth electronic payment.

[0206] "Voice recognition means" refers to a device or software that has the function of receiving voice input from a customer and analyzing its content.

[0207] A "product search tool" is a device or software that has the function of searching for relevant product information from a product database based on the analyzed content.

[0208] "Information presentation means" refers to a device or software that has the function of presenting searched product information and explaining it to the customer using voice and screen display.

[0209] A "payment method" is a device or software that guides customers through the purchase process for selected products and has the function of processing electronic payments.

[0210] A "generative AI model" is an algorithm or software that learns from a large amount of data and performs natural language generation.

[0211] "Product description generation means" refers to a device or software that has the function of generating a description of product features using a generation AI model.

[0212] A "product database" is a database that stores data including product specifications, prices, stock availability, and review information.

[0213] A "server" is a computer that manages the product database, trains AI models, and processes payments.

[0214] This invention provides a system that allows customers to make voice requests, which can be used in both in-store and online environments to quickly and accurately provide product information and enable secure electronic payments. The specific system configuration and operation are described below.

[0215] server

[0216] Product Database Management

[0217] The server manages the product database. The product database stores data including product specifications, prices, inventory status, and review information. The server periodically collects this data and checks its consistency. Once collected, the data is stored in the database and checked for any missing or duplicate entries.

[0218] Training an AI model

[0219] The server uses the collected data to train a generative AI model. The AI ​​model learns based on product features, user reviews, and past purchase history, and by performing speech recognition and natural language generation, it can provide users with appropriate product information.

[0220] Payment processing management

[0221] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and fast payment processing.

[0222] Device (smartphone)

[0223] Voice recognition and product search

[0224] The terminal uses voice recognition to receive customer voice input and analyzes its content. For example, if a customer requests by voice, "I'm looking for a new refrigerator," the voice recognition technology analyzes this and the product search mechanism returns the relevant search results.

[0225] Product information presentation and product description

[0226] The terminal uses information presentation tools to explain product information to the customer via voice and screen display based on the product search results. In particular, it uses a generative AI model to generate expert and easy-to-understand descriptions of product features. For this purpose, a prompt such as "Please describe the following product: Latest refrigerator model" is used.

[0227] Comparison function and recommendations

[0228] When a customer wishes to compare multiple products, the terminal provides explanations based on product information through voice and screen displays. It also recommends the most suitable product based on past purchase history and preferences.

[0229] Electronic payment

[0230] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process, sends the payment information to the server, and completes the transaction. This ensures a secure payment, and the customer receives a notification stating, "Payment complete. Thank you."

[0231] User (customer)

[0232] Voice request

[0233] Users request the products they want by voice to the device. For example, if they say, "I'm looking for the latest TV," the device will respond, "This is the latest model. Its features are..."

[0234] Check and compare product descriptions

[0235] Users can view detailed product descriptions generated by the AI ​​model and request comparisons between multiple products. For example, if asked "Which is better?", the device might respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0236] Secure purchase process

[0237] When a user expresses their intention to purchase, the terminal guides them through the electronic payment method and securely completes the payment process. For example, simply saying "I want to buy this item" will prompt the user to "Please insert your credit card."

[0238] Thus, the system according to the present invention supports customers in searching for products by voice, obtaining detailed information, and completing the purchase process safely. As a result, users can enjoy a smooth and stress-free purchasing experience.

[0239] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0240] Step 1:

[0241] Receiving voice input

[0242] The terminal receives customer voice input and converts it into text data using speech recognition. The input is the customer's voice, and the output is the parsed text data.

[0243] Step 2:

[0244] Voice analysis and search query generation

[0245] The terminal analyzes the text data obtained in step 1 and generates a search query using a product search mechanism. The input is the analyzed text data, and the output is a search query to the product database.

[0246] Step 3:

[0247] Product Database Search

[0248] The terminal sends a search query to the server and retrieves the corresponding product information from the product database. The server returns the matching product information from the product database. The input is a product search query, and the output is product information.

[0249] Step 4:

[0250] Product information presentation and explanation generation using an AI model.

[0251] The terminal uses information presentation tools to explain product information to the customer via screen display and audio, based on the acquired product information. It also uses a generative AI model to generate a description that includes detailed product features. A prompt phrase such as "Please describe the following product: ○○○" is used. The input is product information and the prompt phrase, and the output is the generated product description.

[0252] Step 5:

[0253] Product comparison and recommendations

[0254] When a customer wishes to compare multiple products, the terminal displays the comparison results via voice and on the screen, based on the product information. Furthermore, the server uses past purchase history and preference data to recommend the most suitable product. Inputs are product information and the customer's past purchase history and preference data, while outputs are comparison results and recommended products.

[0255] Step 6:

[0256] Confirmation of purchase intent and guidance on electronic payment.

[0257] The terminal confirms the customer's purchase intention and guides them through the electronic payment process using their chosen payment method. Once the customer selects a payment method, the terminal sends the payment information to the server and completes the process. The input is the customer's purchase intention and payment information, and the output is a payment completion notification.

[0258] Step 7:

[0259] Customer notification

[0260] The terminal notifies the customer that the payment has been successfully completed. The input is payment completion information, and the output is the notification to the customer.

[0261] In each step, data processing and calculations are as follows: speech recognition involves converting voice input into text data, and product search involves narrowing down products based on the text data. The generative AI model generates descriptions in natural language based on prompts and product information, and electronic payment procedures involve data management based on security protocols.

[0262] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0263] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functions. Furthermore, by combining this with an emotion engine that recognizes user emotions and adapts its response accordingly, it provides a more personalized purchasing experience.

[0264] server

[0265] Product Database Management

[0266] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies for errors and duplicates.

[0267] Training an AI model

[0268] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[0269] Training an emotion recognition model

[0270] The server uses audio and image data to train an emotion recognition model. Specifically, it analyzes factors such as voice tone, speaking speed, and facial expression data to infer the customer's emotional state.

[0271] Payment processing management

[0272] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[0273] Terminal (robot)

[0274] Product Information

[0275] The terminal greets customers verbally upon entering the store and accepts customer requests using voice recognition. Based on product information retrieved from the server, it provides detailed information through voice and screen displays.

[0276] Product comparison and recommendations

[0277] If a customer wishes to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." It also uses recommendation tools to suggest the most suitable product based on the customer's past purchase history and preferences.

[0278] Emotion recognition and response

[0279] The device is equipped with emotion recognition technology that identifies the customer's emotions from their voice and facial expressions. For example, if the customer appears confused, the device will follow up with a message such as, "You seem to be having trouble. Do you have any questions?" If the customer is happy, the device will offer additional product recommendations, adapting its response to the customer's emotions.

[0280] Electronic payment support

[0281] Once the customer decides to purchase, the terminal guides them through the electronic payment process. If the customer selects a credit card, the terminal reads the card information and securely transmits it to the server.

[0282] User (customer)

[0283] Product search and inquiry

[0284] Users make voice requests to the device to obtain product information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[0285] Selection of product comparison

[0286] If the user wishes to compare multiple products, they can give an instruction to the terminal to obtain detailed information. For example, if the user asks "Which one is better?", the terminal will answer "This refrigerator is energy-efficient, but that model has a larger capacity", etc.

[0287] Purchase procedure and emotional follow-up

[0288] When the user selects a product and indicates their intention to purchase, the terminal guides them on the method of electronic payment and takes the most appropriate communication according to the customer's emotional state. For example, if the user seems anxious, the terminal will follow up with "This payment method is safe, so please rest assured".

[0289] In this way, the system according to the present invention helps the customer to collect product information without stress, compare and consider while empathizing with their emotions, and complete the purchase procedure quickly and safely. Thereby, a more personalized and highly satisfactory purchase experience can be provided.

[0290] The following describes the processing flow.

[0291] Step 1:

[0292] Initial contact (terminal)

[0293] [[ID=3�]]When the customer enters the store, the terminal uses the voice recognition function to greet with "Welcome. What can I help you find?" When the customer replies, the voice is automatically converted into text data.

[0294] Step 2:

[0295] Receiving the request (terminal)

[0296] When a customer states that they want to know about a specific product, the terminal analyzes the voice and converts it into text data. This text data is sent to the server to request a search for product information.

[0297] Step 3:

[0298] Search for Product Information (Server)

[0299] Based on the received request, the server searches for the corresponding product information from the product database. For example, for a request of "new 4K TV", information such as specifications, price, inventory status, user reviews, etc. are extracted.

[0300] Step 4:

[0301] Provision of Information (Terminal)

[0302] The terminal provides the customer with the product information obtained from the server in voice and on the screen display. For example, it explains "This 4K TV is 55 inches and the price is $800. There is currently inventory available."

[0303] Step 5:

[0304] Emotion Recognition (Terminal)

[0305] The terminal analyzes the emotion from the customer's expression and voice tone. For example, when the customer is confused, it detects facial tension and voice uneasiness and saves them as emotion recognition data.

[0306] Step 6:

[0307] Response According to Emotion (Terminal)

[0308] Based on the emotion recognition data, the terminal makes a response suitable for the customer's emotion. For example, when the customer is confused, it follows up with "You seem to be having trouble. What kind of information are you looking for?"

[0309] Step 7:

[0310] Product comparison and recommendations (devices)

[0311] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality." It also recommends the most suitable product based on the customer's past purchase history and preferences.

[0312] Step 8:

[0313] Confirmation of purchase intent (device)

[0314] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[0315] Step 9:

[0316] Electronic payment instructions (terminal)

[0317] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[0318] Step 10:

[0319] Sending payment information (terminal)

[0320] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[0321] Step 11:

[0322] Payment processing and confirmation (server)

[0323] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[0324] Step 12:

[0325] Payment completion notification (device)

[0326] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[0327] This series of steps allows customers to easily search for and compare product information in-store and complete the purchase process while receiving emotionally tailored service. This enables customers to enjoy a stress-free, efficient, and personalized shopping experience.

[0328] (Example 2)

[0329] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0330] Large consumer electronics retailers attract many customers, and it is essential to provide services tailored to each individual's purchasing needs and emotional state. However, conventional systems lack the ability to recognize and respond to customer emotions, making it difficult to provide a personalized shopping experience. Furthermore, simply presenting product information or recommending products based on purchase history may not allow customers to make the best choice. To solve these problems, there is a need to develop a robotic system that adapts its response based on the customer's emotional state.

[0331] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0332] In this invention, the server includes a voice recognition means for receiving and analyzing the customer's voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; an emotion recognition means for recognizing the customer's emotions and responding accordingly; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables the customer to obtain appropriate product information and proceed with the purchase procedure with peace of mind while receiving personalized responses that respond to their emotions.

[0333] "Voice recognition means" refers to a device or technology for receiving customer voice input and analyzing its content.

[0334] "Product search means" refers to a device or technology for searching for relevant product information from a product database based on the analyzed audio content.

[0335] "Information presentation means" refers to a device or technology for presenting searched product information and explaining it to customers using audio and screen displays.

[0336] "Emotion recognition means" refers to a device or technology for recognizing a customer's emotions and responding accordingly.

[0337] "Payment method" refers to a device or technology that guides customers through the purchase process of selected goods and enables electronic payment.

[0338] "Recommendation method" refers to a device or technology for recommending the most suitable product based on a customer's past purchase history and preferences.

[0339] A "server" is a computer system used for managing product information and training AI models that generate it.

[0340] A "product database" is a data storage system used to store and manage details such as product specifications, prices, inventory status, and review information.

[0341] A "generative AI model" is an artificial intelligence model trained using machine learning techniques based on collected data.

[0342] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functionality. Furthermore, by combining this with an emotion engine that recognizes customer emotions and adapts its response accordingly, the system provides a more personalized purchasing experience.

[0343] Server-based processing

[0344] 1. Product database management:

[0345] The server accesses the product database and periodically collects product specifications, prices, stock availability, and review information. The collected data is temporarily stored, checked for duplicates and errors, and then saved to the database. For example, a Python script can be used to retrieve data from an API and store it in a MySQL® database.

[0346] 2. Training the Generative AI Model:

[0347] The server uses the collected data to train a generative AI model. This training uses data such as past purchase history, user reviews, and product features. Machine learning frameworks like TensorFlow and PyTorch are used for data preprocessing and model training. For example, a model is built to predict recommended products based on a user's purchase history.

[0348] 3. Training the emotion recognition model:

[0349] The server collects audio and image data, analyzes speech tone, speech rate, and facial expression data, and trains an emotion recognition model. Librosa is used for preprocessing audio data, and OpenCV is used for preprocessing image data. The emotion recognition model is built using a framework such as Keras.

[0350] 4. Payment processing management:

[0351] The server implements the necessary security protocols for electronic payments (e.g., SSL / TLS) and runs transaction management and fraud detection algorithms. It uses the Django framework and payment APIs (e.g., Stripe) to perform secure payment processing.

[0352] Processing by terminal (robot)

[0353] 1. Providing product information:

[0354] The terminal greets customers with a voice message upon entering the store and accepts requests using a voice recognition system (e.g., Google Speech-to-Text). It retrieves product information from the server and explains it to the customer using text-to-speech and on-screen displays.

[0355] Specific example: When a customer requests, "Tell me about the latest refrigerator," the terminal responds, "This is our latest refrigerator. It is energy-efficient and has a large capacity."

[0356] 2. Product comparison and recommendations:

[0357] The terminal retrieves information on multiple products from the server in response to the customer's request. Based on the customer's past purchase history and preferences, it recommends suitable products.

[0358] Specific example: When a customer asks, "What's the difference between this refrigerator and that refrigerator?", the terminal replies, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0359] 3. Emotion Recognition and Response:

[0360] The device captures the customer's voice and facial expressions using a camera and microphone, and analyzes them in real time. If the customer appears confused, it follows up with "Are you having any trouble?", and if they are pleased, it responds with "Do you like this as well?".

[0361] Specific example: The device's camera captures facial expressions and records audio. The analysis engine uses this information to determine the emotional state and take appropriate action.

[0362] 4. Support for electronic payments:

[0363] The terminal displays a user interface for entering payment information, and once the customer enters their credit card information, it is encrypted and sent to the server.

[0364] Specific example: If a customer says, "I'll buy this refrigerator," the terminal will ask, "Do you have a credit card?" and read and process the card information.

[0365] User (customer) operation

[0366] 1. Product search and inquiry:

[0367] The user requests by voice, "Show me the latest refrigerators." The terminal queries the server for product information and provides the retrieved information via voice and display.

[0368] For example, if you say, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[0369] 2. Product comparison and selection:

[0370] The user asks the terminal, "What's the difference between this refrigerator and that refrigerator?" The terminal retrieves comparison data from the server and explains it to the user.

[0371] For example, when asked "Which is better?", the device replies, "This refrigerator is more energy-efficient, while that model has a larger capacity."

[0372] 3. Purchase process and emotional follow-up:

[0373] When a user says, "I'll buy this refrigerator," the terminal guides them through the payment process. If the user appears unsure, the terminal reassures them by saying, "This payment method is secure, so please rest assured."

[0374] Specific example: The user enters payment information following the voice guidance, and the terminal displays an appropriate follow-up message.

[0375] Thus, the system according to the present invention helps customers gather product information without stress, compare and consider products while being sensitive to their emotions, and complete the purchase process quickly and safely. This makes it possible to provide a more personalized and satisfying purchasing experience.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1: Managing the product database

[0378] The server collects product data from online APIs and internal systems at regular intervals (e.g., daily). The input is JSON data from the API. The server uses a Python script to temporarily store the data in memory. Next, the script detects and corrects duplicate and incorrectly formatted data. The output is the product data with verified integrity, which is then stored in a MySQL database.

[0379] Step 2: Training the Generative AI Model

[0380] The server takes a CSV file containing text data such as user reviews and purchase history as input. It uses machine learning frameworks like TensorFlow and PyTorch to preprocess the data, specifically performing data cleaning and normalization. Next, it trains a generative model and evaluates its accuracy. The output of this step is the trained generative AI model, which is saved to disk.

[0381] Step 3: Training the emotion recognition model

[0382] The server takes audio and image data as input and preprocesses the audio for tone, speech rate, and facial expression data. Specifically, it uses Librosa for audio data preprocessing and OpenCV for image data preprocessing. It builds and trains an emotion recognition model using a framework such as Keras. The output is the trained emotion recognition model, which is saved to disk.

[0383] Step 4: Managing Payment Processing

[0384] The server receives payment requests from customers as input and protects the data using SSL / TLS encrypted communication protocols. It runs fraud detection algorithms to detect unusual transactions in real time. It securely stores payment information and completes transactions. The output is the completed payment transaction.

[0385] Step 5: Provide product information

[0386] The terminal takes the customer's voice request as input and converts the content into text using a speech recognition system (Google Speech-to-Text). It then sends a query to the server to retrieve product information. The retrieved product information is provided to the customer via text-to-speech and on the display. The output consists of product information presented via voice and on the display.

[0387] Step 6: Product Comparison and Recommendation

[0388] The terminal takes a customer's comparison request as input and retrieves information on multiple products from the server. Based on the customer's past purchase history and preferences, it suggests recommended products. Specifically, it describes the features of the multiple products in both text and audio. The output includes information on the compared products and recommended products.

[0389] Step 7: Recognizing and responding to emotions

[0390] The device acquires customer voice and facial expressions as input via camera and microphone. It uses an emotion recognition model in real time to analyze the customer's emotional state. For example, it captures camera and audio data using Python and inputs it into the model. If the customer is confused, it follows up with "Are you having trouble?", and if they are happy, it responds with "Did you like it?". The output is a follow-up message tailored to the customer's emotion.

[0391] Step 8: Support for electronic payments

[0392] The terminal takes the customer's purchase intention as input and displays a UI for electronic payment. Once the customer enters their credit card information, the data is encrypted and sent to the server. Specifically, SSL / TLS-protected communication is used to securely process payment information. The output is the completed payment procedure and transaction confirmation information.

[0393] Step 9: Product Search and Inquiry

[0394] The user enters a voice request saying, "Show me the latest refrigerators." The device uses Google Speech-to-Text to convert the speech to text and sends the query to the server. The data retrieved from the server is provided to the user via speech synthesis and on the display. The output is the searched product information.

[0395] Step 10: Compare and select products

[0396] The user enters a question such as, "What is the difference between this refrigerator and that refrigerator?" The terminal sends a query to the server, retrieves the necessary comparison information, and displays it. Specifically, it explains the features of each product in text and audio. The output is the compared product information.

[0397] Step 11: Purchase Procedure and Emotional Follow-up

[0398] When a user enters "I will buy this refrigerator," the terminal guides them through the payment process. To alleviate customer concerns, a reassuring follow-up message is added. Specifically, a combination of speech recognition and text generation is used to explain the secure payment process. The output includes the completed purchase and the follow-up message.

[0399] (Application Example 2)

[0400] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0401] The process by which customers compare and purchase products in physical stores is often cumbersome, leading to decreased customer satisfaction. Furthermore, current customer service systems struggle to provide personalized responses that take into account individual customer emotional states, resulting in a standardized customer experience. Moreover, few such systems offer secure and rapid electronic payment capabilities. As a result, the overall in-store purchasing experience is unsatisfactory.

[0402] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means that receives voice input from a customer and analyzes its contents; a product search means that searches for relevant product information from a product database based on the analyzed contents; an information presentation means that presents the searched product information and explains it to the customer using voice and screen display; a payment means that guides the customer through the purchase procedure of the product selected by the customer and performs electronic payment; an emotion recognition means that analyzes the customer's emotional state from voice and facial expressions; and a response means that adapts the response based on the analyzed emotional state. As a result, customers can collect and compare product information and make safe and quick electronic payments while receiving personalized responses tailored to their individual emotional state in a physical store.

[0403] A "voice recognition system" is a system that receives voice input from a customer and analyzes its content.

[0404] "Product search method" refers to a function that searches for relevant product information from a product database based on the analyzed content.

[0405] "Information presentation means" refers to a function that presents searched product information and explains it to the customer via voice and screen.

[0406] "Payment method" refers to the function that guides customers through the purchase process for the products they have selected and allows them to make electronic payments.

[0407] An "emotion recognition method" is a system that analyzes a customer's emotional state from their voice and facial expressions.

[0408] "Response methods" refer to the function of performing optimal communication based on the analyzed emotional state.

[0409] "Recommendation methods" refer to functions that recommend the most suitable products based on the customer's past purchase history and preferences.

[0410] A "generative AI model" is an artificial intelligence model trained based on collected data.

[0411] A "server" is a computer system that manages product information and trains AI models for generating it.

[0412] "Smart glasses" are glasses-type devices equipped with a display and audio output functions.

[0413] This system uses smart glasses in physical stores to provide customers with optimal product information and to realize a personalized purchasing experience through emotion recognition. The elements that make up this system and their operation are described below.

[0414] server

[0415] The server manages and operates the entire system using the following methods.

[0416] Speech recognition means

[0417] The server receives customer voice input and analyzes its content. Using speech recognition technologies such as the Google Cloud Speech-to-Text API, the voice data is converted into text data. When a customer enters a voice command such as "I'm looking for a new refrigerator," the content is first converted into text data by the speech recognition system.

[0418] Product search method

[0419] The server analyzes the text data obtained by the speech recognition system and searches the product database for relevant product information. For example, if the keyword "refrigerator" is extracted, it retrieves detailed information about products that match that condition from the database. High-speed product information retrieval is possible by using SQL or NoSQL databases (e.g., MySQL, MongoDB).

[0420] Information presentation means

[0421] The acquired product information is provided to the customer via both audio and screen display. Specifically, the product name, detailed information, price, etc., are displayed on the smart glasses' screen, along with voice guidance. This allows customers to obtain the necessary information in real time, both visually and audibly. By using speech synthesis services such as Amazon Web Services' Polly, natural-sounding voice information is provided.

[0422] Payment methods

[0423] The customer is guided through the purchase process for their selected items via smart glasses, and electronic payment is processed. Secure and fast payment processing is achieved by using payment gateways that support credit card payments and digital wallets (e.g., Stripe, Square). Once the customer confirms their purchase, the payment process is completed through the smart glasses interface, and a payment completion notification is displayed.

[0424] emotion recognition means

[0425] To analyze customer emotions from their voice and facial expressions, emotion recognition technology using machine learning models (e.g., TensorFlow, PyTorch) is employed. By analyzing voice tone and facial expression data, the customer's emotional state (e.g., confusion, joy, anxiety) is identified. If the customer appears confused, follow-up questions such as, "You seem to be having trouble. Do you have any questions?" are asked.

[0426] Countermeasures

[0427] Based on the analyzed emotional state, optimal communication is provided. If the customer is happy, appropriate responses are given, such as suggesting additional products. This improves customer satisfaction.

[0428] Device (smart glasses)

[0429] Product Information

[0430] The smart glasses are equipped with a display and audio output function, allowing customers to easily obtain product information while walking around the store. For example, if a customer asks, "What are the features of this refrigerator?", the product's specifications and price will be displayed on the screen, and a detailed explanation will be provided simultaneously via audio.

[0431] Emotion recognition and response

[0432] The smart glasses are equipped with a camera and microphone, which are used to capture the customer's voice and facial expressions and send them to a server. Emotion recognition technology identifies the customer's emotions, and the appropriate response is provided. For example, if a customer appears anxious, they might be reassured with a message such as, "Please rest assured, this payment method is very secure."

[0433] User (customer)

[0434] Product search and inquiry

[0435] Users request product information via voice commands through smart glasses. Using simple voice commands, they can obtain detailed information about specific products or compare multiple products.

[0436] Product Comparison Selection

[0437] For example, if a user asks, "What's the difference between this refrigerator and that refrigerator?", the smart glasses display will show comparison information, along with an audio explanation.

[0438] Purchase process and emotional follow-up

[0439] Once a user selects a product and indicates their intention to purchase, the smart glasses guide them through the electronic payment method. Furthermore, by providing follow-up support tailored to the customer's emotional state, they ensure a secure and comfortable purchasing experience. For example, if a customer appears anxious, encouraging messages such as, "This payment method is very secure. Please confirm," will be displayed.

[0440] Example of a prompt

[0441] For example, the following prompt statement is used:

[0442] "I'm looking for a new refrigerator."

[0443] "Please tell me more about this model."

[0444] "Which refrigerator is the most energy-efficient?"

[0445] Through these features, customers can gather product information without stress, compare and consider options in an emotionally engaging way, and complete the purchase process safely and quickly. This results in a more personalized and satisfying shopping experience.

[0446] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0447] Step 1:

[0448] The server receives the customer's voice input through the smart glasses' microphone. The voice data is sent from the smart glasses to the server and converted into text data by the Google Cloud Speech-to-Text API. For example, a voice command such as "I'm looking for a new refrigerator" would be converted into text data.

[0449] Step 2:

[0450] The server analyzes the speech data transcribed into text by a speech recognition device and uses a product search device to retrieve relevant product information from the product database. The text data received as input, "I'm looking for a new refrigerator," is analyzed, and information about refrigerators is extracted from the product database. At this point, a database search is performed using SQL queries or NoSQL queries.

[0451] Step 3:

[0452] The server sends product information obtained through the product search method back to the smart glasses. The search results are displayed on the smart glasses' screen in the format of product name, details, price, etc. At this time, a voice description is also provided using Amazon Web Services' Polly, etc.

[0453] Step 4:

[0454] The device (smart glasses) uses an information display mechanism to show product information obtained from the server, along with audio. For example, the smart glasses' display might show "New Refrigerator X, price 50,000 yen, energy efficiency A class," while simultaneously providing an audio explanation such as "This refrigerator is very energy efficient and has a large capacity."

[0455] Step 5:

[0456] The device captures the customer's reactions and sends them back to the server. It collects audio and facial expression data using the camera and microphone until the customer utters the next command. For example, if the customer shows a confused expression, that data is sent to the server.

[0457] Step 6:

[0458] The server analyzes the collected audio and facial expression data using emotion recognition tools. A machine learning model using TensorFlow or PyTorch identifies the emotional state (e.g., confusion, joy, anxiety). The audio and facial expression data received as input are analyzed, and the emotion "confusion" is identified.

[0459] Step 7:

[0460] The server uses response tools to generate the most appropriate response for the customer based on the results of analysis by emotion recognition tools. For example, a prompt such as "You seem to be having trouble. Do you have any questions?" is generated by the generation AI model and sent to the smart glasses.

[0461] Step 8:

[0462] The terminal displays a response message received from the server to the customer. Using the smart glasses' display and audio output, it provides guidance via both text and voice, such as, "It seems you're having trouble. Do you have any questions?"

[0463] Step 9:

[0464] When a customer indicates their intention to purchase a product, the terminal sends a purchase request to the server and activates the payment method. Voice commands such as "I will purchase this product" are entered through the smart glasses interface and sent to the server.

[0465] Step 10:

[0466] The server processes electronic payments using the payment method. Payment information is received as input, and the transaction is executed through a secure payment gateway (e.g., Stripe, Square). If the payment is successful, a confirmation message is generated.

[0467] Step 11:

[0468] The device displays and provides audio confirmation messages after payment is complete. The smart glasses' display shows "Your purchase is complete. Thank you for using our service," and the same message is also provided verbally.

[0469] Example of a prompt

[0470] The following prompts are used:

[0471] "I'm looking for a new refrigerator."

[0472] "Please tell me more about this model."

[0473] "Which refrigerator is the most energy-efficient?"

[0474] This allows customers to gather product information without stress, receive appropriate emotional support, and complete the purchase process safely and quickly.

[0475] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0476] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0477] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0478] [Second Embodiment]

[0479] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0480] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0481] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0482] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0483] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0484] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0485] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0486] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0487] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0488] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0489] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0490] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0491] This invention relates to a robotic system that communicates with customers in large consumer electronics stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[0492] server

[0493] Product Database Management

[0494] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies that there are no omissions or duplicates.

[0495] Training an AI model

[0496] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[0497] Payment processing management

[0498] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[0499] Terminal (robot)

[0500] Product Information

[0501] The terminal greets customers verbally upon entering the store and receives customer requests using voice recognition. Based on product information retrieved from the server, the terminal provides customers with detailed information via voice and screen display.

[0502] Product comparison and recommendations

[0503] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality."

[0504] Electronic payment support

[0505] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process. After the customer selects a payment method, the terminal sends the payment information to the server and completes the process. Upon successful payment completion, the terminal notifies the customer with the message, "Payment complete. Thank you."

[0506] User (customer)

[0507] Product search and inquiry

[0508] Users can request the products they want from the device using voice commands and receive detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[0509] Product Comparison Selection

[0510] If a user wants to compare multiple products, they can get detailed information by giving instructions to the device. For example, if a user asks, "Which one is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0511] Purchase procedure

[0512] Once the user selects an item and indicates their intention to purchase, the terminal guides them through the electronic payment method and completes the payment using a credit or debit card. The user simply says, "I'll buy this item," and the terminal prompts them to "Please insert your credit card," completing the payment.

[0513] Thus, the system according to the present invention helps customers collect product information, compare and consider options, and ultimately complete the purchase quickly and safely, without stress. This significantly improves the in-store shopping experience.

[0514] The following describes the processing flow.

[0515] Step 1:

[0516] Initial contact (device)

[0517] When a customer enters the store, the terminal uses voice recognition to greet them with, "Welcome. What can I help you with?" Once the customer responds, the system automatically converts the voice into text.

[0518] Step 2:

[0519] Request acceptance (terminal)

[0520] When a customer states that they want to know about a specific product, the terminal analyzes their voice and converts it into text data. This text data is then sent to a server to request a search for product information.

[0521] Step 3:

[0522] Product information search (server)

[0523] Based on the received request, the server searches its product database for relevant product information. For example, in response to a request for "new 4K TV," it extracts information such as specifications, price, availability, and user reviews.

[0524] Step 4:

[0525] Information provision (terminal)

[0526] The terminal provides customers with product information retrieved from the server via voice and on-screen display. For example, it might explain, "This 4K TV is 55 inches and priced at $800. It is currently in stock."

[0527] Step 5:

[0528] Product comparison (devices)

[0529] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality."

[0530] Step 6:

[0531] Confirmation of purchase intent (device)

[0532] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[0533] Step 7:

[0534] Electronic payment instructions (terminal)

[0535] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[0536] Step 8:

[0537] Sending payment information (terminal)

[0538] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[0539] Step 9:

[0540] Payment processing and confirmation (server)

[0541] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[0542] Step 10:

[0543] Payment completion notification (device)

[0544] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[0545] This series of steps allows customers to easily search for, compare, and complete product information within the store. This enables customers to enjoy a stress-free and efficient shopping experience.

[0546] (Example 1)

[0547] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0548] Traditional in-store purchasing processes were cumbersome, involving the collection, comparison, and purchase of product information, increasing customer stress and inconvenience. Furthermore, the need for store staff to provide information and payment support led to challenges such as staff shortages and inconsistent skill levels, resulting in decreased service quality. Concerns also existed regarding information discrepancies between online and in-store purchases, as well as payment security.

[0549] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0550] In this invention, the server includes a voice recognition means for receiving and analyzing customer voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for explaining detailed product information obtained from a management server to the customer via voice and screen display; a product comparison means for comparing multiple products and explaining their features; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables customers to quickly and easily obtain and compare product information and safely complete the purchase procedure. The server also includes a recommendation means for recommending the most suitable product based on the customer's past purchase history and preferences; a data management means for verifying the integrity of product information and storing it in a database; and a security management means for executing a fraud detection algorithm to ensure the security of payments. This improves the consistency and security of the service and enhances customer satisfaction.

[0551] "Voice recognition means" refers to technology or equipment for receiving customer voice input and analyzing its content.

[0552] "Product search means" refers to a technology or device for searching for relevant product information from a product database based on the analyzed content.

[0553] "Information presentation means" refers to technology or equipment for providing customers with detailed product information obtained from a management server via audio and screen display.

[0554] A "product comparison method" is a technology or device for comparing multiple products and explaining their characteristics.

[0555] "Payment method" refers to the technology or device that guides customers through the purchase process of selected goods and enables electronic payment.

[0556] "Recommendation methods" refer to technologies or devices for recommending the most suitable products based on a customer's past purchase history and preferences.

[0557] "Data management means" refers to technology or equipment for verifying the integrity of product information and storing it in a database.

[0558] "Security management measures" refer to technologies or devices that execute fraud detection algorithms and ensure the security of payments.

[0559] This invention relates to a robotic system that communicates with customers in large stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[0560] server

[0561] Product Database Management

[0562] The server manages the product database for large stores. Using Python libraries such as Beautiful Soup and Scrapy, the server periodically collects product specifications, prices, stock availability, and review information, and checks data integrity. This ensures that detailed product data is stored in the database. For example, by using scraping techniques to collect information from product web pages, it's possible to update the data in real time.

[0563] Training an AI model

[0564] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. It uses TensorFlow or PyTorch to build and train a neural network model. For example, the AI ​​model is trained epoch-by-epoch to solve classification and regression tasks. Specifically, the model is evaluated using prompts such as, "Please recommend a model based on reviews of the latest refrigerators."

[0565] Payment processing management

[0566] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. The server ensures secure data communication using the SSL / TLS protocol and manages consistent transactions by building a payment management system with the Django framework. It also uses machine learning to train anomaly detection algorithms based on historical transaction data to evaluate new transactions.

[0567] Terminal (robot)

[0568] Product Information

[0569] The terminal greets customers with a voice message upon entering the store and uses speech recognition technology to receive requests from customers. It analyzes customer voices using Microsoft Azure Cognitive Services and Google Cloud's Speech-to-Text API. Based on product information obtained from the server, it uses Google Cloud's Text-to-Speech API to synthesize speech and provides detailed information through both voice and screen display. For example, if a customer says, "I'm looking for a new refrigerator," the terminal will respond, "Here are our latest models."

[0570] Product comparison and recommendations

[0571] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. If a customer says, "Please compare the TVs," the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." An example of a specific prompt would be, "Which refrigerator is the most energy-efficient?"

[0572] Electronic payment support

[0573] After the customer decides to purchase, the terminal guides them through the electronic payment process using voice and on-screen displays. The customer's payment information is sent to a server for processing. Once the server completes the payment processing, the terminal notifies the customer that "payment is complete." Specifically, when the customer says, "I'll buy this item," the terminal prompts them to "insert your credit card," and the payment is completed.

[0574] User (customer)

[0575] Product search and inquiry

[0576] Users can request desired products from the device using voice commands and obtain detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[0577] Product Comparison Selection

[0578] Users can request product comparisons from the device using voice commands and obtain detailed information. For example, if asked, "Which refrigerator is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity." A specific example of a prompt would be, "Please tell me the difference between these two refrigerators."

[0579] Purchase procedure

[0580] Users indicate their intention to purchase by voice and make payments using credit or debit cards according to the terminal's instructions. For example, if a user says, "I want to buy this item," the terminal will prompt them to "Please insert your credit card" and assist them through the entire payment process.

[0581] Thus, the system of the present invention provides a comprehensive flow that allows customers to smoothly obtain and compare product information that suits their needs and to make purchases securely, significantly improving the in-store shopping experience. Furthermore, the consistency and security of the service are guaranteed through server data management and security measures.

[0582] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0583] Step 1:

[0584] The server collects product data. The server uses Python libraries such as Beautiful Soup and Scrapy to perform web scraping and collect product specifications, prices, stock availability, and review information from large retail stores.

[0585] Input: Product webpage URL

[0586] Data processing: HTML parsing, extraction of necessary data.

[0587] Output: Product data list (specifications, price, stock status, review information)

[0588] Step 2:

[0589] The server checks for data integrity. The server verifies the collected product data for duplicates or missing data. If duplicates are found, the data is updated with the latest data; if missing data is found, it is collected again.

[0590] Input: Product data list

[0591] Data processing: Duplicate checking, data uniqueness verification, recollection of missing data.

[0592] Output: Consistent product database

[0593] Step 3:

[0594] The server stores product data in the database. An ORM such as SQLAlchemy is used to ensure consistent product data is stored in the database.

[0595] Input: Consistent product database

[0596] Data processing: Saving operations to a database

[0597] Output: Updated product database

[0598] Step 4:

[0599] The server trains the AI ​​model. The server uses product data, user reviews, and past purchase history to train the generative AI model. It builds and trains the neural network using TensorFlow or PyTorch.

[0600] Input: Product data, user reviews, past purchase history

[0601] Data processing: Training and evaluation of AI models

[0602] Output: Trained generative AI model

[0603] Step 5:

[0604] The device receives the customer's voice request. The device performs speech recognition using Microsoft Azure's Cognitive Services or Google Cloud's Speech-to-Text API and retrieves the customer's request in text format.

[0605] Input: Customer voice request

[0606] Data processing: Speech-to-text conversion

[0607] Output: Request content in text format

[0608] Step 6:

[0609] The device retrieves product information. The device sends a text request to the server and retrieves the corresponding product information.

[0610] Input: Request content in text format

[0611] Data processing: Sending requests to the server, retrieving product information.

[0612] Output: Product information (specifications, price, stock status, review information)

[0613] Step 7:

[0614] The terminal presents product information to the customer. The terminal uses Google Cloud's Text-to-Speech API to synthesize speech and also displays the product information on the screen.

[0615] Input: Product Information

[0616] Data processing: Voice generation, screen display updates.

[0617] Output: Providing product information via audio and screen display.

[0618] Step 8:

[0619] The terminal receives a customer comparison request. The terminal uses speech recognition technology to obtain the customer comparison request in text format.

[0620] Input: Customer voice request

[0621] Data processing: Speech-to-text conversion

[0622] Output: Comparison request in text format

[0623] Step 9:

[0624] The device retrieves comparison information. The device sends a comparison request to the server and retrieves comparison information for multiple products.

[0625] Input: Text format comparison request

[0626] Data processing: Sending requests to the server, obtaining comparison information.

[0627] Output: Comparison information (features, price, availability, etc.)

[0628] Step 10:

[0629] The terminal presents the customer with product comparison information. The terminal explains the acquired comparison information to the customer through voice and screen display.

[0630] Input: Comparison Information

[0631] Data processing: Voice generation, screen display updates.

[0632] Output: Provision of comparative information via audio and screen display.

[0633] Step 11:

[0634] The terminal receives the customer's purchase intention. When the customer indicates their intention to purchase a specific product, speech recognition technology is used to obtain that intention in text format.

[0635] Input: Customer's voice expression of purchase intent

[0636] Data processing: Speech-to-text conversion

[0637] Output: Purchase intention in text format

[0638] Step 12:

[0639] The terminal guides the customer through the payment process. The terminal uses voice and on-screen displays to instruct the customer to complete the payment using a credit or debit card.

[0640] Input: Customer's purchase intention

[0641] Data processing: Voice generation for payment instructions, updating of screen display.

[0642] Output: Payment instructions

[0643] Step 13:

[0644] The terminal sends payment information to the server. When a customer enters payment information or uses a card, the terminal sends that information to the server and requests processing.

[0645] Input: Customer payment information

[0646] Data processing: Sending payment information to the server

[0647] Output: Payment request to the server

[0648] Step 14:

[0649] The server processes the payment. The server uses the SSL / TLS protocol to perform secure payment processing, transaction management, and fraud detection.

[0650] Input: Customer payment information

[0651] Data processing: Payment processing, fraud detection, transaction management

[0652] Output: Payment processing result

[0653] Step 15:

[0654] The terminal notifies the customer that the payment is complete. Upon receiving the successful payment processing result from the server, the terminal notifies the customer via voice and on-screen display that "Payment complete."

[0655] Input: Payment processing result from the server

[0656] Data processing: Generate audio for notifications, update screen display.

[0657] Output: Payment completion notification to customer

[0658] (Application Example 1)

[0659] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0660] Traditional systems made it difficult to provide a smooth purchasing experience for customers when searching for and comparing products online, as they had to find the necessary information. Furthermore, product descriptions were uniform, making it difficult to address the individual needs of customers. In addition, electronic payments require security and smoothness in the transaction process.

[0661] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0662] In this invention, the server includes a speech recognition means for receiving customer voice input and analyzing its content; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment; and a product description generation means for generating a description of product features using a generation AI model. This makes it possible to provide product information quickly and accurately based on customer voice input and generate product descriptions tailored to individual needs. It also enables safe and smooth electronic payment.

[0663] "Voice recognition means" refers to a device or software that has the function of receiving voice input from a customer and analyzing its content.

[0664] A "product search tool" is a device or software that has the function of searching for relevant product information from a product database based on the analyzed content.

[0665] "Information presentation means" refers to a device or software that has the function of presenting searched product information and explaining it to the customer using voice and screen display.

[0666] A "payment method" is a device or software that guides customers through the purchase process for selected products and has the function of processing electronic payments.

[0667] A "generative AI model" is an algorithm or software that learns from a large amount of data and performs natural language generation.

[0668] "Product description generation means" refers to a device or software that has the function of generating a description of product features using a generation AI model.

[0669] A "product database" is a database that stores data including product specifications, prices, stock availability, and review information.

[0670] A "server" is a computer that manages the product database, trains AI models, and processes payments.

[0671] This invention provides a system that allows customers to make voice requests, which can be used in both in-store and online environments to quickly and accurately provide product information and enable secure electronic payments. The specific system configuration and operation are described below.

[0672] server

[0673] Product Database Management

[0674] The server manages the product database. The product database stores data including product specifications, prices, inventory status, and review information. The server periodically collects this data and checks its consistency. Once collected, the data is stored in the database and checked for any missing or duplicate entries.

[0675] Training an AI model

[0676] The server uses the collected data to train a generative AI model. The AI ​​model learns based on product features, user reviews, and past purchase history, and by performing speech recognition and natural language generation, it can provide users with appropriate product information.

[0677] Payment processing management

[0678] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and fast payment processing.

[0679] Device (smartphone)

[0680] Voice recognition and product search

[0681] The terminal uses voice recognition to receive customer voice input and analyzes its content. For example, if a customer requests by voice, "I'm looking for a new refrigerator," the voice recognition technology analyzes this and the product search mechanism returns the relevant search results.

[0682] Product information presentation and product description

[0683] The terminal uses information presentation tools to explain product information to the customer via voice and screen display based on the product search results. In particular, it uses a generative AI model to generate expert and easy-to-understand descriptions of product features. For this purpose, a prompt such as "Please describe the following product: Latest refrigerator model" is used.

[0684] Comparison function and recommendations

[0685] When a customer wishes to compare multiple products, the terminal provides explanations based on product information through voice and screen displays. It also recommends the most suitable product based on past purchase history and preferences.

[0686] Electronic payment

[0687] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process, sends the payment information to the server, and completes the transaction. This ensures a secure payment, and the customer receives a notification stating, "Payment complete. Thank you."

[0688] User (customer)

[0689] Voice request

[0690] Users request the products they want by voice to the device. For example, if they say, "I'm looking for the latest TV," the device will respond, "This is the latest model. Its features are..."

[0691] Check and compare product descriptions

[0692] Users can view detailed product descriptions generated by the AI ​​model and request comparisons between multiple products. For example, if asked "Which is better?", the device might respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0693] Secure purchase process

[0694] When a user expresses their intention to purchase, the terminal guides them through the electronic payment method and securely completes the payment process. For example, simply saying "I want to buy this item" will prompt the user to "Please insert your credit card."

[0695] Thus, the system according to the present invention supports customers in searching for products by voice, obtaining detailed information, and completing the purchase process safely. As a result, users can enjoy a smooth and stress-free purchasing experience.

[0696] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0697] Step 1:

[0698] Receiving voice input

[0699] The terminal receives customer voice input and converts it into text data using speech recognition. The input is the customer's voice, and the output is the parsed text data.

[0700] Step 2:

[0701] Voice analysis and search query generation

[0702] The terminal analyzes the text data obtained in step 1 and generates a search query using a product search mechanism. The input is the analyzed text data, and the output is a search query to the product database.

[0703] Step 3:

[0704] Product Database Search

[0705] The terminal sends a search query to the server and retrieves the corresponding product information from the product database. The server returns the matching product information from the product database. The input is a product search query, and the output is product information.

[0706] Step 4:

[0707] Product information presentation and explanation generation using an AI model.

[0708] The terminal uses information presentation tools to explain product information to the customer via screen display and audio, based on the acquired product information. It also uses a generative AI model to generate a description that includes detailed product features. A prompt phrase such as "Please describe the following product: ○○○" is used. The input is product information and the prompt phrase, and the output is the generated product description.

[0709] Step 5:

[0710] Product comparison and recommendations

[0711] When a customer wishes to compare multiple products, the terminal displays the comparison results via voice and on the screen, based on the product information. Furthermore, the server uses past purchase history and preference data to recommend the most suitable product. Inputs are product information and the customer's past purchase history and preference data, while outputs are comparison results and recommended products.

[0712] Step 6:

[0713] Confirmation of purchase intent and guidance on electronic payment.

[0714] The terminal confirms the customer's purchase intention and guides them through the electronic payment process using their chosen payment method. Once the customer selects a payment method, the terminal sends the payment information to the server and completes the process. The input is the customer's purchase intention and payment information, and the output is a payment completion notification.

[0715] Step 7:

[0716] Customer notification

[0717] The terminal notifies the customer that the payment has been successfully completed. The input is payment completion information, and the output is the notification to the customer.

[0718] In each step, data processing and calculations are as follows: speech recognition involves converting voice input into text data, and product search involves narrowing down products based on the text data. The generative AI model generates descriptions in natural language based on prompts and product information, and electronic payment procedures involve data management based on security protocols.

[0719] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0720] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functions. Furthermore, by combining this with an emotion engine that recognizes user emotions and adapts its response accordingly, it provides a more personalized purchasing experience.

[0721] server

[0722] Product Database Management

[0723] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies for errors and duplicates.

[0724] Training an AI model

[0725] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[0726] Training an emotion recognition model

[0727] The server uses audio and image data to train an emotion recognition model. Specifically, it analyzes factors such as voice tone, speaking speed, and facial expression data to infer the customer's emotional state.

[0728] Payment processing management

[0729] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[0730] Terminal (robot)

[0731] Product Information

[0732] The terminal greets customers verbally upon entering the store and accepts customer requests using voice recognition. Based on product information retrieved from the server, it provides detailed information through voice and screen displays.

[0733] Product comparison and recommendations

[0734] If a customer wishes to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." It also uses recommendation tools to suggest the most suitable product based on the customer's past purchase history and preferences.

[0735] Emotion recognition and response

[0736] The device is equipped with emotion recognition technology that identifies the customer's emotions from their voice and facial expressions. For example, if the customer appears confused, the device will follow up with a message such as, "You seem to be having trouble. Do you have any questions?" If the customer is happy, the device will offer additional product recommendations, adapting its response to the customer's emotions.

[0737] Electronic payment support

[0738] Once the customer decides to purchase, the terminal guides them through the electronic payment process. If the customer selects a credit card, the terminal reads the card information and securely transmits it to the server.

[0739] User (customer)

[0740] Product search and inquiry

[0741] Users make voice requests to the device to obtain product information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[0742] Product Comparison Selection

[0743] If a user wants to compare multiple products, they can give instructions to the device to obtain detailed information. For example, if a user asks, "Which one is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0744] Purchase process and emotional follow-up

[0745] Once a user selects a product and indicates their intention to purchase, the terminal guides them through the electronic payment method and communicates appropriately according to the customer's emotional state. For example, if the user appears anxious, it will reassure them by saying, "This payment method is secure, so please rest assured."

[0746] Thus, the system according to the present invention helps customers gather product information without stress, compare and consider products while being sensitive to their emotions, and complete the purchase process quickly and safely. This enables a more personalized and satisfying purchasing experience.

[0747] The following describes the processing flow.

[0748] Step 1:

[0749] Initial contact (device)

[0750] When a customer enters the store, the terminal uses voice recognition to greet them with, "Welcome. What can I help you with?" Once the customer responds, the system automatically converts the voice into text.

[0751] Step 2:

[0752] Request acceptance (terminal)

[0753] When a customer states that they want to know about a specific product, the terminal analyzes their voice and converts it into text data. This text data is then sent to a server to request a search for product information.

[0754] Step 3:

[0755] Product information search (server)

[0756] Based on the received request, the server searches its product database for relevant product information. For example, in response to a request for "new 4K TV," it extracts information such as specifications, price, availability, and user reviews.

[0757] Step 4:

[0758] Information provision (terminal)

[0759] The terminal provides customers with product information retrieved from the server via voice and on-screen display. For example, it might explain, "This 4K TV is 55 inches and priced at $800. It is currently in stock."

[0760] Step 5:

[0761] Emotion recognition (device)

[0762] The device analyzes the customer's emotions from their facial expressions and voice tone. For example, if a customer is confused, it detects facial tension and voice anxiety and stores this as emotion recognition data.

[0763] Step 6:

[0764] Emotion-based response (device)

[0765] Based on emotion recognition data, the device responds in a way that is appropriate to the customer's emotions. For example, if the customer is confused, it will follow up with, "You seem to be having trouble. What kind of information are you looking for?"

[0766] Step 7:

[0767] Product comparison and recommendations (devices)

[0768] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality." It also recommends the most suitable product based on the customer's past purchase history and preferences.

[0769] Step 8:

[0770] Confirmation of purchase intent (device)

[0771] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[0772] Step 9:

[0773] Electronic payment instructions (terminal)

[0774] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[0775] Step 10:

[0776] Sending payment information (terminal)

[0777] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[0778] Step 11:

[0779] Payment processing and confirmation (server)

[0780] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[0781] Step 12:

[0782] Payment completion notification (device)

[0783] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[0784] This series of steps allows customers to easily search for and compare product information in-store and complete the purchase process while receiving emotionally tailored service. This enables customers to enjoy a stress-free, efficient, and personalized shopping experience.

[0785] (Example 2)

[0786] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0787] Large consumer electronics retailers attract many customers, and it is essential to provide services tailored to each individual's purchasing needs and emotional state. However, conventional systems lack the ability to recognize and respond to customer emotions, making it difficult to provide a personalized shopping experience. Furthermore, simply presenting product information or recommending products based on purchase history may not allow customers to make the best choice. To solve these problems, there is a need to develop a robotic system that adapts its response based on the customer's emotional state.

[0788] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0789] In this invention, the server includes a voice recognition means for receiving and analyzing the customer's voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; an emotion recognition means for recognizing the customer's emotions and responding accordingly; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables the customer to obtain appropriate product information and proceed with the purchase procedure with peace of mind while receiving personalized responses that respond to their emotions.

[0790] "Voice recognition means" refers to a device or technology for receiving customer voice input and analyzing its content.

[0791] "Product search means" refers to a device or technology for searching for relevant product information from a product database based on the analyzed audio content.

[0792] "Information presentation means" refers to a device or technology for presenting searched product information and explaining it to customers using audio and screen displays.

[0793] "Emotion recognition means" refers to a device or technology for recognizing a customer's emotions and responding accordingly.

[0794] "Payment method" refers to a device or technology that guides customers through the purchase process of selected goods and enables electronic payment.

[0795] "Recommendation method" refers to a device or technology for recommending the most suitable product based on a customer's past purchase history and preferences.

[0796] A "server" is a computer system used for managing product information and training AI models that generate it.

[0797] A "product database" is a data storage system used to store and manage details such as product specifications, prices, inventory status, and review information.

[0798] A "generative AI model" is an artificial intelligence model trained using machine learning techniques based on collected data.

[0799] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functionality. Furthermore, by combining this with an emotion engine that recognizes customer emotions and adapts its response accordingly, the system provides a more personalized purchasing experience.

[0800] Server-based processing

[0801] 1. Product database management:

[0802] The server accesses the product database and periodically collects product specifications, prices, stock availability, and review information. The collected data is temporarily stored, checked for duplicates and errors, and then saved to the database. For example, a Python script could be used to retrieve data from an API and store it in a MySQL database.

[0803] 2. Training the Generative AI Model:

[0804] The server uses the collected data to train a generative AI model. This training uses data such as past purchase history, user reviews, and product features. Machine learning frameworks like TensorFlow and PyTorch are used for data preprocessing and model training. For example, a model is built to predict recommended products based on a user's purchase history.

[0805] 3. Training the emotion recognition model:

[0806] The server collects audio and image data, analyzes speech tone, speech rate, and facial expression data, and trains an emotion recognition model. Librosa is used for preprocessing audio data, and OpenCV is used for preprocessing image data. The emotion recognition model is built using a framework such as Keras.

[0807] 4. Payment processing management:

[0808] The server implements the necessary security protocols for electronic payments (e.g., SSL / TLS) and runs transaction management and fraud detection algorithms. It uses the Django framework and payment APIs (e.g., Stripe) to perform secure payment processing.

[0809] Processing by terminal (robot)

[0810] 1. Providing product information:

[0811] The terminal greets customers with a voice message upon entering the store and accepts requests using a voice recognition system (e.g., Google Speech-to-Text). It retrieves product information from the server and explains it to the customer using text-to-speech and on-screen displays.

[0812] Specific example: When a customer requests, "Tell me about the latest refrigerator," the terminal responds, "This is our latest refrigerator. It is energy-efficient and has a large capacity."

[0813] 2. Product comparison and recommendations:

[0814] The terminal retrieves information on multiple products from the server in response to the customer's request. Based on the customer's past purchase history and preferences, it recommends suitable products.

[0815] Specific example: When a customer asks, "What's the difference between this refrigerator and that refrigerator?", the terminal replies, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0816] 3. Emotion Recognition and Response:

[0817] The device captures the customer's voice and facial expressions using a camera and microphone, and analyzes them in real time. If the customer appears confused, it follows up with "Are you having any trouble?", and if they are pleased, it responds with "Do you like this as well?".

[0818] Specific example: The device's camera captures facial expressions and records audio. The analysis engine uses this information to determine the emotional state and take appropriate action.

[0819] 4. Support for electronic payments:

[0820] The terminal displays a user interface for entering payment information, and once the customer enters their credit card information, it is encrypted and sent to the server.

[0821] Specific example: If a customer says, "I'll buy this refrigerator," the terminal will ask, "Do you have a credit card?" and read and process the card information.

[0822] User (customer) operation

[0823] 1. Product search and inquiry:

[0824] The user requests by voice, "Show me the latest refrigerators." The terminal queries the server for product information and provides the retrieved information via voice and display.

[0825] For example, if you say, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[0826] 2. Product comparison and selection:

[0827] The user asks the terminal, "What's the difference between this refrigerator and that refrigerator?" The terminal retrieves comparison data from the server and explains it to the user.

[0828] For example, when asked "Which is better?", the device replies, "This refrigerator is more energy-efficient, while that model has a larger capacity."

[0829] 3. Purchase process and emotional follow-up:

[0830] When a user says, "I'll buy this refrigerator," the terminal guides them through the payment process. If the user appears unsure, the terminal reassures them by saying, "This payment method is secure, so please rest assured."

[0831] Specific example: The user enters payment information following the voice guidance, and the terminal displays an appropriate follow-up message.

[0832] Thus, the system according to the present invention helps customers gather product information without stress, compare and consider products while being sensitive to their emotions, and complete the purchase process quickly and safely. This makes it possible to provide a more personalized and satisfying purchasing experience.

[0833] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0834] Step 1: Managing the product database

[0835] The server collects product data from online APIs and internal systems at regular intervals (e.g., daily). The input is JSON data from the API. The server uses a Python script to temporarily store the data in memory. Next, the script detects and corrects duplicate and incorrectly formatted data. The output is the product data with verified integrity, which is then stored in a MySQL database.

[0836] Step 2: Training the Generative AI Model

[0837] The server takes a CSV file containing text data such as user reviews and purchase history as input. It uses machine learning frameworks like TensorFlow and PyTorch to preprocess the data, specifically performing data cleaning and normalization. Next, it trains a generative model and evaluates its accuracy. The output of this step is the trained generative AI model, which is saved to disk.

[0838] Step 3: Training the emotion recognition model

[0839] The server takes audio and image data as input and preprocesses the audio for tone, speech rate, and facial expression data. Specifically, it uses Librosa for audio data preprocessing and OpenCV for image data preprocessing. It builds and trains an emotion recognition model using a framework such as Keras. The output is the trained emotion recognition model, which is saved to disk.

[0840] Step 4: Managing Payment Processing

[0841] The server receives payment requests from customers as input and protects the data using SSL / TLS encrypted communication protocols. It runs fraud detection algorithms to detect unusual transactions in real time. It securely stores payment information and completes transactions. The output is the completed payment transaction.

[0842] Step 5: Provide product information

[0843] The terminal takes the customer's voice request as input and converts the content into text using a speech recognition system (Google Speech-to-Text). It then sends a query to the server to retrieve product information. The retrieved product information is provided to the customer via text-to-speech and on the display. The output consists of product information presented via voice and on the display.

[0844] Step 6: Product Comparison and Recommendation

[0845] The terminal takes a customer's comparison request as input and retrieves information on multiple products from the server. Based on the customer's past purchase history and preferences, it suggests recommended products. Specifically, it describes the features of the multiple products in both text and audio. The output includes information on the compared products and recommended products.

[0846] Step 7: Recognizing and responding to emotions

[0847] The device acquires customer voice and facial expressions as input via camera and microphone. It uses an emotion recognition model in real time to analyze the customer's emotional state. For example, it captures camera and audio data using Python and inputs it into the model. If the customer is confused, it follows up with "Are you having trouble?", and if they are happy, it responds with "Did you like it?". The output is a follow-up message tailored to the customer's emotion.

[0848] Step 8: Support for electronic payments

[0849] The terminal takes the customer's purchase intention as input and displays a UI for electronic payment. Once the customer enters their credit card information, the data is encrypted and sent to the server. Specifically, SSL / TLS-protected communication is used to securely process payment information. The output is the completed payment procedure and transaction confirmation information.

[0850] Step 9: Product Search and Inquiry

[0851] The user enters a voice request saying, "Show me the latest refrigerators." The device uses Google Speech-to-Text to convert the speech to text and sends the query to the server. The data retrieved from the server is provided to the user via speech synthesis and on the display. The output is the searched product information.

[0852] Step 10: Compare and select products

[0853] The user enters a question such as, "What is the difference between this refrigerator and that refrigerator?" The terminal sends a query to the server, retrieves the necessary comparison information, and displays it. Specifically, it explains the features of each product in text and audio. The output is the compared product information.

[0854] Step 11: Purchase Procedure and Emotional Follow-up

[0855] When a user enters "I will buy this refrigerator," the terminal guides them through the payment process. To alleviate customer concerns, a reassuring follow-up message is added. Specifically, a combination of speech recognition and text generation is used to explain the secure payment process. The output includes the completed purchase and the follow-up message.

[0856] (Application Example 2)

[0857] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0858] The process by which customers compare and purchase products in physical stores is often cumbersome, leading to decreased customer satisfaction. Furthermore, current customer service systems struggle to provide personalized responses that take into account individual customer emotional states, resulting in a standardized customer experience. Moreover, few such systems offer secure and rapid electronic payment capabilities. As a result, the overall in-store purchasing experience is unsatisfactory.

[0859] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means that receives voice input from a customer and analyzes its contents; a product search means that searches for relevant product information from a product database based on the analyzed contents; an information presentation means that presents the searched product information and explains it to the customer using voice and screen display; a payment means that guides the customer through the purchase procedure of the product selected by the customer and performs electronic payment; an emotion recognition means that analyzes the customer's emotional state from voice and facial expressions; and a response means that adapts the response based on the analyzed emotional state. As a result, customers can collect and compare product information and make safe and quick electronic payments while receiving personalized responses tailored to their individual emotional state in a physical store.

[0860] A "voice recognition system" is a system that receives voice input from a customer and analyzes its content.

[0861] "Product search method" refers to a function that searches for relevant product information from a product database based on the analyzed content.

[0862] "Information presentation means" refers to a function that presents searched product information and explains it to the customer via voice and screen.

[0863] "Payment method" refers to the function that guides customers through the purchase process for the products they have selected and allows them to make electronic payments.

[0864] An "emotion recognition method" is a system that analyzes a customer's emotional state from their voice and facial expressions.

[0865] "Response methods" refer to the function of performing optimal communication based on the analyzed emotional state.

[0866] "Recommendation methods" refer to functions that recommend the most suitable products based on the customer's past purchase history and preferences.

[0867] A "generative AI model" is an artificial intelligence model trained based on collected data.

[0868] A "server" is a computer system that manages product information and trains AI models for generating it.

[0869] "Smart glasses" are glasses-type devices equipped with a display and audio output functions.

[0870] This system uses smart glasses in physical stores to provide customers with optimal product information and to realize a personalized purchasing experience through emotion recognition. The elements that make up this system and their operation are described below.

[0871] server

[0872] The server manages and operates the entire system using the following methods.

[0873] Speech recognition means

[0874] The server receives customer voice input and analyzes its content. Using speech recognition technologies such as the Google Cloud Speech-to-Text API, the voice data is converted into text data. When a customer enters a voice command such as "I'm looking for a new refrigerator," the content is first converted into text data by the speech recognition system.

[0875] Product search method

[0876] The server analyzes the text data obtained by the speech recognition system and searches the product database for relevant product information. For example, if the keyword "refrigerator" is extracted, it retrieves detailed information about products that match that condition from the database. High-speed product information retrieval is possible by using SQL or NoSQL databases (e.g., MySQL, MongoDB).

[0877] Information presentation means

[0878] The acquired product information is provided to the customer via both audio and screen display. Specifically, the product name, detailed information, price, etc., are displayed on the smart glasses' screen, along with voice guidance. This allows customers to obtain the necessary information in real time, both visually and audibly. By using speech synthesis services such as Amazon Web Services' Polly, natural-sounding voice information is provided.

[0879] Payment methods

[0880] The customer is guided through the purchase process for their selected items via smart glasses, and electronic payment is processed. Secure and fast payment processing is achieved by using payment gateways that support credit card payments and digital wallets (e.g., Stripe, Square). Once the customer confirms their purchase, the payment process is completed through the smart glasses interface, and a payment completion notification is displayed.

[0881] emotion recognition means

[0882] To analyze customer emotions from their voice and facial expressions, emotion recognition technology using machine learning models (e.g., TensorFlow, PyTorch) is employed. By analyzing voice tone and facial expression data, the customer's emotional state (e.g., confusion, joy, anxiety) is identified. If the customer appears confused, follow-up questions such as, "You seem to be having trouble. Do you have any questions?" are asked.

[0883] Countermeasures

[0884] Based on the analyzed emotional state, optimal communication is provided. If the customer is happy, appropriate responses are given, such as suggesting additional products. This improves customer satisfaction.

[0885] Device (smart glasses)

[0886] Product Information

[0887] The smart glasses are equipped with a display and audio output function, allowing customers to easily obtain product information while walking around the store. For example, if a customer asks, "What are the features of this refrigerator?", the product's specifications and price will be displayed on the screen, and a detailed explanation will be provided simultaneously via audio.

[0888] Emotion recognition and response

[0889] The smart glasses are equipped with a camera and microphone, which are used to capture the customer's voice and facial expressions and send them to a server. Emotion recognition technology identifies the customer's emotions, and the appropriate response is provided. For example, if a customer appears anxious, they might be reassured with a message such as, "Please rest assured, this payment method is very secure."

[0890] User (customer)

[0891] Product search and inquiry

[0892] Users request product information via voice commands through smart glasses. Using simple voice commands, they can obtain detailed information about specific products or compare multiple products.

[0893] Product Comparison Selection

[0894] For example, if a user asks, "What's the difference between this refrigerator and that refrigerator?", the smart glasses display will show comparison information, along with an audio explanation.

[0895] Purchase process and emotional follow-up

[0896] Once a user selects a product and indicates their intention to purchase, the smart glasses guide them through the electronic payment method. Furthermore, by providing follow-up support tailored to the customer's emotional state, they ensure a secure and comfortable purchasing experience. For example, if a customer appears anxious, encouraging messages such as, "This payment method is very secure. Please confirm," will be displayed.

[0897] Example of a prompt

[0898] For example, the following prompt statement is used:

[0899] "I'm looking for a new refrigerator."

[0900] "Please tell me more about this model."

[0901] "Which refrigerator is the most energy-efficient?"

[0902] Through these features, customers can gather product information without stress, compare and consider options in an emotionally engaging way, and complete the purchase process safely and quickly. This results in a more personalized and satisfying shopping experience.

[0903] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0904] Step 1:

[0905] The server receives the customer's voice input through the smart glasses' microphone. The voice data is sent from the smart glasses to the server and converted into text data by the Google Cloud Speech-to-Text API. For example, a voice command such as "I'm looking for a new refrigerator" would be converted into text data.

[0906] Step 2:

[0907] The server analyzes the speech data transcribed into text by a speech recognition device and uses a product search device to retrieve relevant product information from the product database. The text data received as input, "I'm looking for a new refrigerator," is analyzed, and information about refrigerators is extracted from the product database. At this point, a database search is performed using SQL queries or NoSQL queries.

[0908] Step 3:

[0909] The server sends product information obtained through the product search method back to the smart glasses. The search results are displayed on the smart glasses' screen in the format of product name, details, price, etc. At this time, a voice description is also provided using Amazon Web Services' Polly, etc.

[0910] Step 4:

[0911] The device (smart glasses) uses an information display mechanism to show product information obtained from the server, along with audio. For example, the smart glasses' display might show "New Refrigerator X, price 50,000 yen, energy efficiency A class," while simultaneously providing an audio explanation such as "This refrigerator is very energy efficient and has a large capacity."

[0912] Step 5:

[0913] The device captures the customer's reactions and sends them back to the server. It collects audio and facial expression data using the camera and microphone until the customer utters the next command. For example, if the customer shows a confused expression, that data is sent to the server.

[0914] Step 6:

[0915] The server analyzes the collected audio and facial expression data using emotion recognition tools. A machine learning model using TensorFlow or PyTorch identifies the emotional state (e.g., confusion, joy, anxiety). The audio and facial expression data received as input are analyzed, and the emotion "confusion" is identified.

[0916] Step 7:

[0917] The server uses response tools to generate the most appropriate response for the customer based on the results of analysis by emotion recognition tools. For example, a prompt such as "You seem to be having trouble. Do you have any questions?" is generated by the generation AI model and sent to the smart glasses.

[0918] Step 8:

[0919] The terminal displays a response message received from the server to the customer. Using the smart glasses' display and audio output, it provides guidance via both text and voice, such as, "It seems you're having trouble. Do you have any questions?"

[0920] Step 9:

[0921] When a customer indicates their intention to purchase a product, the terminal sends a purchase request to the server and activates the payment method. Voice commands such as "I will purchase this product" are entered through the smart glasses interface and sent to the server.

[0922] Step 10:

[0923] The server processes electronic payments using the payment method. Payment information is received as input, and the transaction is executed through a secure payment gateway (e.g., Stripe, Square). If the payment is successful, a confirmation message is generated.

[0924] Step 11:

[0925] The device displays and provides audio confirmation messages after payment is complete. The smart glasses' display shows "Your purchase is complete. Thank you for using our service," and the same message is also provided verbally.

[0926] Example of a prompt

[0927] The following prompts are used:

[0928] "I'm looking for a new refrigerator."

[0929] "Please tell me more about this model."

[0930] "Which refrigerator is the most energy-efficient?"

[0931] This allows customers to gather product information without stress, receive appropriate emotional support, and complete the purchase process safely and quickly.

[0932] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0933] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0934] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0935] [Third Embodiment]

[0936] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0937] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0938] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0939] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0940] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0941] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0942] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0943] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0944] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0945] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0946] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0947] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0948] This invention relates to a robotic system that communicates with customers in large consumer electronics stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[0949] server

[0950] Product Database Management

[0951] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies that there are no omissions or duplicates.

[0952] Training an AI model

[0953] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[0954] Payment processing management

[0955] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[0956] Terminal (robot)

[0957] Product Information

[0958] The terminal greets customers verbally upon entering the store and receives customer requests using voice recognition. Based on product information retrieved from the server, the terminal provides customers with detailed information via voice and screen display.

[0959] Product comparison and recommendations

[0960] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality."

[0961] Electronic payment support

[0962] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process. After the customer selects a payment method, the terminal sends the payment information to the server and completes the process. Upon successful payment completion, the terminal notifies the customer with the message, "Payment complete. Thank you."

[0963] User (customer)

[0964] Product search and inquiry

[0965] Users can request the products they want from the device using voice commands and receive detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[0966] Product Comparison Selection

[0967] If a user wants to compare multiple products, they can get detailed information by giving instructions to the device. For example, if a user asks, "Which one is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[0968] Purchase procedure

[0969] Once the user selects an item and indicates their intention to purchase, the terminal guides them through the electronic payment method and completes the payment using a credit or debit card. The user simply says, "I'll buy this item," and the terminal prompts them to "Please insert your credit card," completing the payment.

[0970] Thus, the system according to the present invention helps customers collect product information, compare and consider options, and ultimately complete the purchase quickly and safely, without stress. This significantly improves the in-store shopping experience.

[0971] The following describes the processing flow.

[0972] Step 1:

[0973] Initial contact (device)

[0974] When a customer enters the store, the terminal uses voice recognition to greet them with, "Welcome. What can I help you with?" Once the customer responds, the system automatically converts the voice into text.

[0975] Step 2:

[0976] Request acceptance (terminal)

[0977] When a customer states that they want to know about a specific product, the terminal analyzes their voice and converts it into text data. This text data is then sent to a server to request a search for product information.

[0978] Step 3:

[0979] Product information search (server)

[0980] Based on the received request, the server searches its product database for relevant product information. For example, in response to a request for "new 4K TV," it extracts information such as specifications, price, availability, and user reviews.

[0981] Step 4:

[0982] Information provision (terminal)

[0983] The terminal provides customers with product information retrieved from the server via voice and on-screen display. For example, it might explain, "This 4K TV is 55 inches and priced at $800. It is currently in stock."

[0984] Step 5:

[0985] Product comparison (devices)

[0986] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality."

[0987] Step 6:

[0988] Confirmation of purchase intent (device)

[0989] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[0990] Step 7:

[0991] Electronic payment instructions (terminal)

[0992] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[0993] Step 8:

[0994] Sending payment information (terminal)

[0995] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[0996] Step 9:

[0997] Payment processing and confirmation (server)

[0998] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[0999] Step 10:

[1000] Payment completion notification (device)

[1001] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[1002] This series of steps allows customers to easily search for, compare, and complete product information within the store. This enables customers to enjoy a stress-free and efficient shopping experience.

[1003] (Example 1)

[1004] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1005] Traditional in-store purchasing processes were cumbersome, involving the collection, comparison, and purchase of product information, increasing customer stress and inconvenience. Furthermore, the need for store staff to provide information and payment support led to challenges such as staff shortages and inconsistent skill levels, resulting in decreased service quality. Concerns also existed regarding information discrepancies between online and in-store purchases, as well as payment security.

[1006] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1007] In this invention, the server includes a voice recognition means for receiving and analyzing customer voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for explaining detailed product information obtained from a management server to the customer via voice and screen display; a product comparison means for comparing multiple products and explaining their features; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables customers to quickly and easily obtain and compare product information and safely complete the purchase procedure. The server also includes a recommendation means for recommending the most suitable product based on the customer's past purchase history and preferences; a data management means for verifying the integrity of product information and storing it in a database; and a security management means for executing a fraud detection algorithm to ensure the security of payments. This improves the consistency and security of the service and enhances customer satisfaction.

[1008] "Voice recognition means" refers to technology or equipment for receiving customer voice input and analyzing its content.

[1009] "Product search means" refers to a technology or device for searching for relevant product information from a product database based on the analyzed content.

[1010] "Information presentation means" refers to technology or equipment for providing customers with detailed product information obtained from a management server via audio and screen display.

[1011] A "product comparison method" is a technology or device for comparing multiple products and explaining their characteristics.

[1012] "Payment method" refers to the technology or device that guides customers through the purchase process of selected goods and enables electronic payment.

[1013] "Recommendation methods" refer to technologies or devices for recommending the most suitable products based on a customer's past purchase history and preferences.

[1014] "Data management means" refers to technology or equipment for verifying the integrity of product information and storing it in a database.

[1015] "Security management measures" refer to technologies or devices that execute fraud detection algorithms and ensure the security of payments.

[1016] This invention relates to a robotic system that communicates with customers in large stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[1017] server

[1018] Product Database Management

[1019] The server manages the product database for large stores. Using Python libraries such as Beautiful Soup and Scrapy, the server periodically collects product specifications, prices, stock availability, and review information, and checks data integrity. This ensures that detailed product data is stored in the database. For example, by using scraping techniques to collect information from product web pages, it's possible to update the data in real time.

[1020] Training an AI model

[1021] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. It uses TensorFlow or PyTorch to build and train a neural network model. For example, the AI ​​model is trained epoch-by-epoch to solve classification and regression tasks. Specifically, the model is evaluated using prompts such as, "Please recommend a model based on reviews of the latest refrigerators."

[1022] Payment processing management

[1023] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. The server ensures secure data communication using the SSL / TLS protocol and manages consistent transactions by building a payment management system with the Django framework. It also uses machine learning to train anomaly detection algorithms based on historical transaction data to evaluate new transactions.

[1024] Terminal (robot)

[1025] Product Information

[1026] The terminal greets customers with a voice message upon entering the store and uses speech recognition technology to receive requests from customers. It analyzes customer voices using Microsoft Azure Cognitive Services and Google Cloud's Speech-to-Text API. Based on product information obtained from the server, it uses Google Cloud's Text-to-Speech API to synthesize speech and provides detailed information through both voice and screen display. For example, if a customer says, "I'm looking for a new refrigerator," the terminal will respond, "Here are our latest models."

[1027] Product comparison and recommendations

[1028] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. If a customer says, "Please compare the TVs," the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." An example of a specific prompt would be, "Which refrigerator is the most energy-efficient?"

[1029] Electronic payment support

[1030] After the customer decides to purchase, the terminal guides them through the electronic payment process using voice and on-screen displays. The customer's payment information is sent to a server for processing. Once the server completes the payment processing, the terminal notifies the customer that "payment is complete." Specifically, when the customer says, "I'll buy this item," the terminal prompts them to "insert your credit card," and the payment is completed.

[1031] User (customer)

[1032] Product search and inquiry

[1033] Users can request desired products from the device using voice commands and obtain detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[1034] Product Comparison Selection

[1035] Users can request product comparisons from the device using voice commands and obtain detailed information. For example, if asked, "Which refrigerator is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity." A specific example of a prompt would be, "Please tell me the difference between these two refrigerators."

[1036] Purchase procedure

[1037] Users indicate their intention to purchase by voice and make payments using credit or debit cards according to the terminal's instructions. For example, if a user says, "I want to buy this item," the terminal will prompt them to "Please insert your credit card" and assist them through the entire payment process.

[1038] Thus, the system of the present invention provides a comprehensive flow that allows customers to smoothly obtain and compare product information that suits their needs and to make purchases securely, significantly improving the in-store shopping experience. Furthermore, the consistency and security of the service are guaranteed through server data management and security measures.

[1039] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1040] Step 1:

[1041] The server collects product data. The server uses Python libraries such as Beautiful Soup and Scrapy to perform web scraping and collect product specifications, prices, stock availability, and review information from large retail stores.

[1042] Input: Product webpage URL

[1043] Data processing: HTML parsing, extraction of necessary data.

[1044] Output: Product data list (specifications, price, stock status, review information)

[1045] Step 2:

[1046] The server checks for data integrity. The server verifies the collected product data for duplicates or missing data. If duplicates are found, the data is updated with the latest data; if missing data is found, it is collected again.

[1047] Input: Product data list

[1048] Data processing: Duplicate checking, data uniqueness verification, recollection of missing data.

[1049] Output: Consistent product database

[1050] Step 3:

[1051] The server stores product data in the database. An ORM such as SQLAlchemy is used to ensure consistent product data is stored in the database.

[1052] Input: Consistent product database

[1053] Data processing: Saving operations to a database

[1054] Output: Updated product database

[1055] Step 4:

[1056] The server trains the AI ​​model. The server uses product data, user reviews, and past purchase history to train the generative AI model. It builds and trains the neural network using TensorFlow or PyTorch.

[1057] Input: Product data, user reviews, past purchase history

[1058] Data processing: Training and evaluation of AI models

[1059] Output: Trained generative AI model

[1060] Step 5:

[1061] The device receives the customer's voice request. The device performs speech recognition using Microsoft Azure's Cognitive Services or Google Cloud's Speech-to-Text API and retrieves the customer's request in text format.

[1062] Input: Customer voice request

[1063] Data processing: Speech-to-text conversion

[1064] Output: Request content in text format

[1065] Step 6:

[1066] The device retrieves product information. The device sends a text request to the server and retrieves the corresponding product information.

[1067] Input: Request content in text format

[1068] Data processing: Sending requests to the server, retrieving product information.

[1069] Output: Product information (specifications, price, stock status, review information)

[1070] Step 7:

[1071] The terminal presents product information to the customer. The terminal uses Google Cloud's Text-to-Speech API to synthesize speech and also displays the product information on the screen.

[1072] Input: Product Information

[1073] Data processing: Voice generation, screen display updates.

[1074] Output: Providing product information via audio and screen display.

[1075] Step 8:

[1076] The terminal receives a customer comparison request. The terminal uses speech recognition technology to obtain the customer comparison request in text format.

[1077] Input: Customer voice request

[1078] Data processing: Speech-to-text conversion

[1079] Output: Comparison request in text format

[1080] Step 9:

[1081] The device retrieves comparison information. The device sends a comparison request to the server and retrieves comparison information for multiple products.

[1082] Input: Text format comparison request

[1083] Data processing: Sending requests to the server, obtaining comparison information.

[1084] Output: Comparison information (features, price, availability, etc.)

[1085] Step 10:

[1086] The terminal presents the customer with product comparison information. The terminal explains the acquired comparison information to the customer through voice and screen display.

[1087] Input: Comparison Information

[1088] Data processing: Voice generation, screen display updates.

[1089] Output: Provision of comparative information via audio and screen display.

[1090] Step 11:

[1091] The terminal receives the customer's purchase intention. When the customer indicates their intention to purchase a specific product, speech recognition technology is used to obtain that intention in text format.

[1092] Input: Customer's voice expression of purchase intent

[1093] Data processing: Speech-to-text conversion

[1094] Output: Purchase intention in text format

[1095] Step 12:

[1096] The terminal guides the customer through the payment process. The terminal uses voice and on-screen displays to instruct the customer to complete the payment using a credit or debit card.

[1097] Input: Customer's purchase intention

[1098] Data processing: Voice generation for payment instructions, updating of screen display.

[1099] Output: Payment instructions

[1100] Step 13:

[1101] The terminal sends payment information to the server. When a customer enters payment information or uses a card, the terminal sends that information to the server and requests processing.

[1102] Input: Customer payment information

[1103] Data processing: Sending payment information to the server

[1104] Output: Payment request to the server

[1105] Step 14:

[1106] The server processes the payment. The server uses the SSL / TLS protocol to perform secure payment processing, transaction management, and fraud detection.

[1107] Input: Customer payment information

[1108] Data processing: Payment processing, fraud detection, transaction management

[1109] Output: Payment processing result

[1110] Step 15:

[1111] The terminal notifies the customer that the payment is complete. Upon receiving the successful payment processing result from the server, the terminal notifies the customer via voice and on-screen display that "Payment complete."

[1112] Input: Payment processing result from the server

[1113] Data processing: Generate audio for notifications, update screen display.

[1114] Output: Payment completion notification to customer

[1115] (Application Example 1)

[1116] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1117] Traditional systems made it difficult to provide a smooth purchasing experience for customers when searching for and comparing products online, as they had to find the necessary information. Furthermore, product descriptions were uniform, making it difficult to address the individual needs of customers. In addition, electronic payments require security and smoothness in the transaction process.

[1118] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1119] In this invention, the server includes a speech recognition means for receiving customer voice input and analyzing its content; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment; and a product description generation means for generating a description of product features using a generation AI model. This makes it possible to provide product information quickly and accurately based on customer voice input and generate product descriptions tailored to individual needs. It also enables safe and smooth electronic payment.

[1120] "Voice recognition means" refers to a device or software that has the function of receiving voice input from a customer and analyzing its content.

[1121] A "product search tool" is a device or software that has the function of searching for relevant product information from a product database based on the analyzed content.

[1122] "Information presentation means" refers to a device or software that has the function of presenting searched product information and explaining it to the customer using voice and screen display.

[1123] A "payment method" is a device or software that guides customers through the purchase process for selected products and has the function of processing electronic payments.

[1124] A "generative AI model" is an algorithm or software that learns from a large amount of data and performs natural language generation.

[1125] "Product description generation means" refers to a device or software that has the function of generating a description of product features using a generation AI model.

[1126] A "product database" is a database that stores data including product specifications, prices, stock availability, and review information.

[1127] A "server" is a computer that manages the product database, trains AI models, and processes payments.

[1128] This invention provides a system that allows customers to make voice requests, which can be used in both in-store and online environments to quickly and accurately provide product information and enable secure electronic payments. The specific system configuration and operation are described below.

[1129] server

[1130] Product Database Management

[1131] The server manages the product database. The product database stores data including product specifications, prices, inventory status, and review information. The server periodically collects this data and checks its consistency. Once collected, the data is stored in the database and checked for any missing or duplicate entries.

[1132] Training an AI model

[1133] The server uses the collected data to train a generative AI model. The AI ​​model learns based on product features, user reviews, and past purchase history, and by performing speech recognition and natural language generation, it can provide users with appropriate product information.

[1134] Payment processing management

[1135] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and fast payment processing.

[1136] Device (smartphone)

[1137] Voice recognition and product search

[1138] The terminal uses voice recognition to receive customer voice input and analyzes its content. For example, if a customer requests by voice, "I'm looking for a new refrigerator," the voice recognition technology analyzes this and the product search mechanism returns the relevant search results.

[1139] Product information presentation and product description

[1140] The terminal uses information presentation tools to explain product information to the customer via voice and screen display based on the product search results. In particular, it uses a generative AI model to generate expert and easy-to-understand descriptions of product features. For this purpose, a prompt such as "Please describe the following product: Latest refrigerator model" is used.

[1141] Comparison function and recommendations

[1142] When a customer wishes to compare multiple products, the terminal provides explanations based on product information through voice and screen displays. It also recommends the most suitable product based on past purchase history and preferences.

[1143] Electronic payment

[1144] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process, sends the payment information to the server, and completes the transaction. This ensures a secure payment, and the customer receives a notification stating, "Payment complete. Thank you."

[1145] User (customer)

[1146] Voice request

[1147] Users request the products they want by voice to the device. For example, if they say, "I'm looking for the latest TV," the device will respond, "This is the latest model. Its features are..."

[1148] Check and compare product descriptions

[1149] Users can view detailed product descriptions generated by the AI ​​model and request comparisons between multiple products. For example, if asked "Which is better?", the device might respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[1150] Secure purchase process

[1151] When a user expresses their intention to purchase, the terminal guides them through the electronic payment method and securely completes the payment process. For example, simply saying "I want to buy this item" will prompt the user to "Please insert your credit card."

[1152] Thus, the system according to the present invention supports customers in searching for products by voice, obtaining detailed information, and completing the purchase process safely. As a result, users can enjoy a smooth and stress-free purchasing experience.

[1153] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1154] Step 1:

[1155] Receiving voice input

[1156] The terminal receives customer voice input and converts it into text data using speech recognition. The input is the customer's voice, and the output is the parsed text data.

[1157] Step 2:

[1158] Voice analysis and search query generation

[1159] The terminal analyzes the text data obtained in step 1 and generates a search query using a product search mechanism. The input is the analyzed text data, and the output is a search query to the product database.

[1160] Step 3:

[1161] Product Database Search

[1162] The terminal sends a search query to the server and retrieves the corresponding product information from the product database. The server returns the matching product information from the product database. The input is a product search query, and the output is product information.

[1163] Step 4:

[1164] Product information presentation and explanation generation using an AI model.

[1165] The terminal uses information presentation tools to explain product information to the customer via screen display and audio, based on the acquired product information. It also uses a generative AI model to generate a description that includes detailed product features. A prompt phrase such as "Please describe the following product: ○○○" is used. The input is product information and the prompt phrase, and the output is the generated product description.

[1166] Step 5:

[1167] Product comparison and recommendations

[1168] When a customer wishes to compare multiple products, the terminal displays the comparison results via voice and on the screen, based on the product information. Furthermore, the server uses past purchase history and preference data to recommend the most suitable product. Inputs are product information and the customer's past purchase history and preference data, while outputs are comparison results and recommended products.

[1169] Step 6:

[1170] Confirmation of purchase intent and guidance on electronic payment.

[1171] The terminal confirms the customer's purchase intention and guides them through the electronic payment process using their chosen payment method. Once the customer selects a payment method, the terminal sends the payment information to the server and completes the process. The input is the customer's purchase intention and payment information, and the output is a payment completion notification.

[1172] Step 7:

[1173] Customer notification

[1174] The terminal notifies the customer that the payment has been successfully completed. The input is payment completion information, and the output is the notification to the customer.

[1175] In each step, data processing and calculations are as follows: speech recognition involves converting voice input into text data, and product search involves narrowing down products based on the text data. The generative AI model generates descriptions in natural language based on prompts and product information, and electronic payment procedures involve data management based on security protocols.

[1176] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1177] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functions. Furthermore, by combining this with an emotion engine that recognizes user emotions and adapts its response accordingly, it provides a more personalized purchasing experience.

[1178] server

[1179] Product Database Management

[1180] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies for errors and duplicates.

[1181] Training an AI model

[1182] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[1183] Training an emotion recognition model

[1184] The server uses audio and image data to train an emotion recognition model. Specifically, it analyzes factors such as voice tone, speaking speed, and facial expression data to infer the customer's emotional state.

[1185] Payment processing management

[1186] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[1187] Terminal (robot)

[1188] Product Information

[1189] The terminal greets customers verbally upon entering the store and accepts customer requests using voice recognition. Based on product information retrieved from the server, it provides detailed information through voice and screen displays.

[1190] Product comparison and recommendations

[1191] If a customer wishes to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." It also uses recommendation tools to suggest the most suitable product based on the customer's past purchase history and preferences.

[1192] Emotion recognition and response

[1193] The device is equipped with emotion recognition technology that identifies the customer's emotions from their voice and facial expressions. For example, if the customer appears confused, the device will follow up with a message such as, "You seem to be having trouble. Do you have any questions?" If the customer is happy, the device will offer additional product recommendations, adapting its response to the customer's emotions.

[1194] Electronic payment support

[1195] Once the customer decides to purchase, the terminal guides them through the electronic payment process. If the customer selects a credit card, the terminal reads the card information and securely transmits it to the server.

[1196] User (customer)

[1197] Product search and inquiry

[1198] Users make voice requests to the device to obtain product information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[1199] Product Comparison Selection

[1200] If a user wants to compare multiple products, they can give instructions to the device to obtain detailed information. For example, if a user asks, "Which one is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[1201] Purchase process and emotional follow-up

[1202] Once a user selects a product and indicates their intention to purchase, the terminal guides them through the electronic payment method and communicates appropriately according to the customer's emotional state. For example, if the user appears anxious, it will reassure them by saying, "This payment method is secure, so please rest assured."

[1203] Thus, the system according to the present invention helps customers gather product information without stress, compare and consider products while being sensitive to their emotions, and complete the purchase process quickly and safely. This enables a more personalized and satisfying purchasing experience.

[1204] The following describes the processing flow.

[1205] Step 1:

[1206] Initial contact (device)

[1207] When a customer enters the store, the terminal uses voice recognition to greet them with, "Welcome. What can I help you with?" Once the customer responds, the system automatically converts the voice into text.

[1208] Step 2:

[1209] Request acceptance (terminal)

[1210] When a customer states that they want to know about a specific product, the terminal analyzes their voice and converts it into text data. This text data is then sent to a server to request a search for product information.

[1211] Step 3:

[1212] Product information search (server)

[1213] Based on the received request, the server searches its product database for relevant product information. For example, in response to a request for "new 4K TV," it extracts information such as specifications, price, availability, and user reviews.

[1214] Step 4:

[1215] Information provision (terminal)

[1216] The terminal provides customers with product information retrieved from the server via voice and on-screen display. For example, it might explain, "This 4K TV is 55 inches and priced at $800. It is currently in stock."

[1217] Step 5:

[1218] Emotion recognition (device)

[1219] The device analyzes the customer's emotions from their facial expressions and voice tone. For example, if a customer is confused, it detects facial tension and voice anxiety and stores this as emotion recognition data.

[1220] Step 6:

[1221] Emotion-based response (device)

[1222] Based on emotion recognition data, the device responds in a way that is appropriate to the customer's emotions. For example, if the customer is confused, it will follow up with, "You seem to be having trouble. What kind of information are you looking for?"

[1223] Step 7:

[1224] Product comparison and recommendations (devices)

[1225] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality." It also recommends the most suitable product based on the customer's past purchase history and preferences.

[1226] Step 8:

[1227] Confirmation of purchase intent (device)

[1228] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[1229] Step 9:

[1230] Electronic payment instructions (terminal)

[1231] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[1232] Step 10:

[1233] Sending payment information (terminal)

[1234] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[1235] Step 11:

[1236] Payment processing and confirmation (server)

[1237] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[1238] Step 12:

[1239] Payment completion notification (device)

[1240] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[1241] This series of steps allows customers to easily search for and compare product information in-store and complete the purchase process while receiving emotionally tailored service. This enables customers to enjoy a stress-free, efficient, and personalized shopping experience.

[1242] (Example 2)

[1243] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1244] Large consumer electronics retailers attract many customers, and it is essential to provide services tailored to each individual's purchasing needs and emotional state. However, conventional systems lack the ability to recognize and respond to customer emotions, making it difficult to provide a personalized shopping experience. Furthermore, simply presenting product information or recommending products based on purchase history may not allow customers to make the best choice. To solve these problems, there is a need to develop a robotic system that adapts its response based on the customer's emotional state.

[1245] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1246] In this invention, the server includes a voice recognition means for receiving and analyzing the customer's voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; an emotion recognition means for recognizing the customer's emotions and responding accordingly; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables the customer to obtain appropriate product information and proceed with the purchase procedure with peace of mind while receiving personalized responses that respond to their emotions.

[1247] "Voice recognition means" refers to a device or technology for receiving customer voice input and analyzing its content.

[1248] "Product search means" refers to a device or technology for searching for relevant product information from a product database based on the analyzed audio content.

[1249] "Information presentation means" refers to a device or technology for presenting searched product information and explaining it to customers using audio and screen displays.

[1250] "Emotion recognition means" refers to a device or technology for recognizing a customer's emotions and responding accordingly.

[1251] "Payment method" refers to a device or technology that guides customers through the purchase process of selected goods and enables electronic payment.

[1252] "Recommendation method" refers to a device or technology for recommending the most suitable product based on a customer's past purchase history and preferences.

[1253] A "server" is a computer system used for managing product information and training AI models that generate it.

[1254] A "product database" is a data storage system used to store and manage details such as product specifications, prices, inventory status, and review information.

[1255] A "generative AI model" is an artificial intelligence model trained using machine learning techniques based on collected data.

[1256] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functionality. Furthermore, by combining this with an emotion engine that recognizes customer emotions and adapts its response accordingly, the system provides a more personalized purchasing experience.

[1257] Server-based processing

[1258] 1. Product database management:

[1259] The server accesses the product database and periodically collects product specifications, prices, stock availability, and review information. The collected data is temporarily stored, checked for duplicates and errors, and then saved to the database. For example, a Python script could be used to retrieve data from an API and store it in a MySQL database.

[1260] 2. Training the Generative AI Model:

[1261] The server uses the collected data to train a generative AI model. This training uses data such as past purchase history, user reviews, and product features. Machine learning frameworks like TensorFlow and PyTorch are used for data preprocessing and model training. For example, a model is built to predict recommended products based on a user's purchase history.

[1262] 3. Training the emotion recognition model:

[1263] The server collects audio and image data, analyzes speech tone, speech rate, and facial expression data, and trains an emotion recognition model. Librosa is used for preprocessing audio data, and OpenCV is used for preprocessing image data. The emotion recognition model is built using a framework such as Keras.

[1264] 4. Payment processing management:

[1265] The server implements the necessary security protocols for electronic payments (e.g., SSL / TLS) and runs transaction management and fraud detection algorithms. It uses the Django framework and payment APIs (e.g., Stripe) to perform secure payment processing.

[1266] Processing by terminal (robot)

[1267] 1. Providing product information:

[1268] The terminal greets customers with a voice message upon entering the store and accepts requests using a voice recognition system (e.g., Google Speech-to-Text). It retrieves product information from the server and explains it to the customer using text-to-speech and on-screen displays.

[1269] Specific example: When a customer requests, "Tell me about the latest refrigerator," the terminal responds, "This is our latest refrigerator. It is energy-efficient and has a large capacity."

[1270] 2. Product comparison and recommendations:

[1271] The terminal retrieves information on multiple products from the server in response to the customer's request. Based on the customer's past purchase history and preferences, it recommends suitable products.

[1272] Specific example: When a customer asks, "What's the difference between this refrigerator and that refrigerator?", the terminal replies, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[1273] 3. Emotion Recognition and Response:

[1274] The device captures the customer's voice and facial expressions using a camera and microphone, and analyzes them in real time. If the customer appears confused, it follows up with "Are you having any trouble?", and if they are pleased, it responds with "Do you like this as well?".

[1275] Specific example: The device's camera captures facial expressions and records audio. The analysis engine uses this information to determine the emotional state and take appropriate action.

[1276] 4. Support for electronic payments:

[1277] The terminal displays a user interface for entering payment information, and once the customer enters their credit card information, it is encrypted and sent to the server.

[1278] Specific example: If a customer says, "I'll buy this refrigerator," the terminal will ask, "Do you have a credit card?" and read and process the card information.

[1279] User (customer) operation

[1280] 1. Product search and inquiry:

[1281] The user requests by voice, "Show me the latest refrigerators." The terminal queries the server for product information and provides the retrieved information via voice and display.

[1282] For example, if you say, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[1283] 2. Product comparison and selection:

[1284] The user asks the terminal, "What's the difference between this refrigerator and that refrigerator?" The terminal retrieves comparison data from the server and explains it to the user.

[1285] For example, when asked "Which is better?", the device replies, "This refrigerator is more energy-efficient, while that model has a larger capacity."

[1286] 3. Purchase process and emotional follow-up:

[1287] When a user says, "I'll buy this refrigerator," the terminal guides them through the payment process. If the user appears unsure, the terminal reassures them by saying, "This payment method is secure, so please rest assured."

[1288] Specific example: The user enters payment information following the voice guidance, and the terminal displays an appropriate follow-up message.

[1289] Thus, the system according to the present invention helps customers gather product information without stress, compare and consider products while being sensitive to their emotions, and complete the purchase process quickly and safely. This makes it possible to provide a more personalized and satisfying purchasing experience.

[1290] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1291] Step 1: Managing the product database

[1292] The server collects product data from online APIs and internal systems at regular intervals (e.g., daily). The input is JSON data from the API. The server uses a Python script to temporarily store the data in memory. Next, the script detects and corrects duplicate and incorrectly formatted data. The output is the product data with verified integrity, which is then stored in a MySQL database.

[1293] Step 2: Training the Generative AI Model

[1294] The server takes a CSV file containing text data such as user reviews and purchase history as input. It uses machine learning frameworks like TensorFlow and PyTorch to preprocess the data, specifically performing data cleaning and normalization. Next, it trains a generative model and evaluates its accuracy. The output of this step is the trained generative AI model, which is saved to disk.

[1295] Step 3: Training the emotion recognition model

[1296] The server takes audio and image data as input and preprocesses the audio for tone, speech rate, and facial expression data. Specifically, it uses Librosa for audio data preprocessing and OpenCV for image data preprocessing. It builds and trains an emotion recognition model using a framework such as Keras. The output is the trained emotion recognition model, which is saved to disk.

[1297] Step 4: Managing Payment Processing

[1298] The server receives payment requests from customers as input and protects the data using SSL / TLS encrypted communication protocols. It runs fraud detection algorithms to detect unusual transactions in real time. It securely stores payment information and completes transactions. The output is the completed payment transaction.

[1299] Step 5: Provide product information

[1300] The terminal takes the customer's voice request as input and converts the content into text using a speech recognition system (Google Speech-to-Text). It then sends a query to the server to retrieve product information. The retrieved product information is provided to the customer via text-to-speech and on the display. The output consists of product information presented via voice and on the display.

[1301] Step 6: Product Comparison and Recommendation

[1302] The terminal takes a customer's comparison request as input and retrieves information on multiple products from the server. Based on the customer's past purchase history and preferences, it suggests recommended products. Specifically, it describes the features of the multiple products in both text and audio. The output includes information on the compared products and recommended products.

[1303] Step 7: Recognizing and responding to emotions

[1304] The device acquires customer voice and facial expressions as input via camera and microphone. It uses an emotion recognition model in real time to analyze the customer's emotional state. For example, it captures camera and audio data using Python and inputs it into the model. If the customer is confused, it follows up with "Are you having trouble?", and if they are happy, it responds with "Did you like it?". The output is a follow-up message tailored to the customer's emotion.

[1305] Step 8: Support for electronic payments

[1306] The terminal takes the customer's purchase intention as input and displays a UI for electronic payment. Once the customer enters their credit card information, the data is encrypted and sent to the server. Specifically, SSL / TLS-protected communication is used to securely process payment information. The output is the completed payment procedure and transaction confirmation information.

[1307] Step 9: Product Search and Inquiry

[1308] The user enters a voice request saying, "Show me the latest refrigerators." The device uses Google Speech-to-Text to convert the speech to text and sends the query to the server. The data retrieved from the server is provided to the user via speech synthesis and on the display. The output is the searched product information.

[1309] Step 10: Compare and select products

[1310] The user enters a question such as, "What is the difference between this refrigerator and that refrigerator?" The terminal sends a query to the server, retrieves the necessary comparison information, and displays it. Specifically, it explains the features of each product in text and audio. The output is the compared product information.

[1311] Step 11: Purchase Procedure and Emotional Follow-up

[1312] When a user enters "I will buy this refrigerator," the terminal guides them through the payment process. To alleviate customer concerns, a reassuring follow-up message is added. Specifically, a combination of speech recognition and text generation is used to explain the secure payment process. The output includes the completed purchase and the follow-up message.

[1313] (Application Example 2)

[1314] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1315] The process by which customers compare and purchase products in physical stores is often cumbersome, leading to decreased customer satisfaction. Furthermore, current customer service systems struggle to provide personalized responses that take into account individual customer emotional states, resulting in a standardized customer experience. Moreover, few such systems offer secure and rapid electronic payment capabilities. As a result, the overall in-store purchasing experience is unsatisfactory.

[1316] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means that receives voice input from a customer and analyzes its contents; a product search means that searches for relevant product information from a product database based on the analyzed contents; an information presentation means that presents the searched product information and explains it to the customer using voice and screen display; a payment means that guides the customer through the purchase procedure of the product selected by the customer and performs electronic payment; an emotion recognition means that analyzes the customer's emotional state from voice and facial expressions; and a response means that adapts the response based on the analyzed emotional state. As a result, customers can collect and compare product information and make safe and quick electronic payments while receiving personalized responses tailored to their individual emotional state in a physical store.

[1317] A "voice recognition system" is a system that receives voice input from a customer and analyzes its content.

[1318] "Product search method" refers to a function that searches for relevant product information from a product database based on the analyzed content.

[1319] "Information presentation means" refers to a function that presents searched product information and explains it to the customer via voice and screen.

[1320] "Payment method" refers to the function that guides customers through the purchase process for the products they have selected and allows them to make electronic payments.

[1321] An "emotion recognition method" is a system that analyzes a customer's emotional state from their voice and facial expressions.

[1322] "Response methods" refer to the function of performing optimal communication based on the analyzed emotional state.

[1323] "Recommendation methods" refer to functions that recommend the most suitable products based on the customer's past purchase history and preferences.

[1324] A "generative AI model" is an artificial intelligence model trained based on collected data.

[1325] A "server" is a computer system that manages product information and trains AI models for generating it.

[1326] "Smart glasses" are glasses-type devices equipped with a display and audio output functions.

[1327] This system uses smart glasses in physical stores to provide customers with optimal product information and to realize a personalized purchasing experience through emotion recognition. The elements that make up this system and their operation are described below.

[1328] server

[1329] The server manages and operates the entire system using the following methods.

[1330] Speech recognition means

[1331] The server receives customer voice input and analyzes its content. Using speech recognition technologies such as the Google Cloud Speech-to-Text API, the voice data is converted into text data. When a customer enters a voice command such as "I'm looking for a new refrigerator," the content is first converted into text data by the speech recognition system.

[1332] Product search method

[1333] The server analyzes the text data obtained by the speech recognition system and searches the product database for relevant product information. For example, if the keyword "refrigerator" is extracted, it retrieves detailed information about products that match that condition from the database. High-speed product information retrieval is possible by using SQL or NoSQL databases (e.g., MySQL, MongoDB).

[1334] Information presentation means

[1335] The acquired product information is provided to the customer via both audio and screen display. Specifically, the product name, detailed information, price, etc., are displayed on the smart glasses' screen, along with voice guidance. This allows customers to obtain the necessary information in real time, both visually and audibly. By using speech synthesis services such as Amazon Web Services' Polly, natural-sounding voice information is provided.

[1336] Payment methods

[1337] The customer is guided through the purchase process for their selected items via smart glasses, and electronic payment is processed. Secure and fast payment processing is achieved by using payment gateways that support credit card payments and digital wallets (e.g., Stripe, Square). Once the customer confirms their purchase, the payment process is completed through the smart glasses interface, and a payment completion notification is displayed.

[1338] emotion recognition means

[1339] To analyze customer emotions from their voice and facial expressions, emotion recognition technology using machine learning models (e.g., TensorFlow, PyTorch) is employed. By analyzing voice tone and facial expression data, the customer's emotional state (e.g., confusion, joy, anxiety) is identified. If the customer appears confused, follow-up questions such as, "You seem to be having trouble. Do you have any questions?" are asked.

[1340] Countermeasures

[1341] Based on the analyzed emotional state, optimal communication is provided. If the customer is happy, appropriate responses are given, such as suggesting additional products. This improves customer satisfaction.

[1342] Device (smart glasses)

[1343] Product Information

[1344] The smart glasses are equipped with a display and audio output function, allowing customers to easily obtain product information while walking around the store. For example, if a customer asks, "What are the features of this refrigerator?", the product's specifications and price will be displayed on the screen, and a detailed explanation will be provided simultaneously via audio.

[1345] Emotion recognition and response

[1346] The smart glasses are equipped with a camera and microphone, which are used to capture the customer's voice and facial expressions and send them to a server. Emotion recognition technology identifies the customer's emotions, and the appropriate response is provided. For example, if a customer appears anxious, they might be reassured with a message such as, "Please rest assured, this payment method is very secure."

[1347] User (customer)

[1348] Product search and inquiry

[1349] Users request product information via voice commands through smart glasses. Using simple voice commands, they can obtain detailed information about specific products or compare multiple products.

[1350] Product Comparison Selection

[1351] For example, if a user asks, "What's the difference between this refrigerator and that refrigerator?", the smart glasses display will show comparison information, along with an audio explanation.

[1352] Purchase process and emotional follow-up

[1353] Once a user selects a product and indicates their intention to purchase, the smart glasses guide them through the electronic payment method. Furthermore, by providing follow-up support tailored to the customer's emotional state, they ensure a secure and comfortable purchasing experience. For example, if a customer appears anxious, encouraging messages such as, "This payment method is very secure. Please confirm," will be displayed.

[1354] Example of a prompt

[1355] For example, the following prompt statement is used:

[1356] "I'm looking for a new refrigerator."

[1357] "Please tell me more about this model."

[1358] "Which refrigerator is the most energy-efficient?"

[1359] Through these features, customers can gather product information without stress, compare and consider options in an emotionally engaging way, and complete the purchase process safely and quickly. This results in a more personalized and satisfying shopping experience.

[1360] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1361] Step 1:

[1362] The server receives the customer's voice input through the smart glasses' microphone. The voice data is sent from the smart glasses to the server and converted into text data by the Google Cloud Speech-to-Text API. For example, a voice command such as "I'm looking for a new refrigerator" would be converted into text data.

[1363] Step 2:

[1364] The server analyzes the speech data transcribed into text by a speech recognition device and uses a product search device to retrieve relevant product information from the product database. The text data received as input, "I'm looking for a new refrigerator," is analyzed, and information about refrigerators is extracted from the product database. At this point, a database search is performed using SQL queries or NoSQL queries.

[1365] Step 3:

[1366] The server sends product information obtained through the product search method back to the smart glasses. The search results are displayed on the smart glasses' screen in the format of product name, details, price, etc. At this time, a voice description is also provided using Amazon Web Services' Polly, etc.

[1367] Step 4:

[1368] The device (smart glasses) uses an information display mechanism to show product information obtained from the server, along with audio. For example, the smart glasses' display might show "New Refrigerator X, price 50,000 yen, energy efficiency A class," while simultaneously providing an audio explanation such as "This refrigerator is very energy efficient and has a large capacity."

[1369] Step 5:

[1370] The device captures the customer's reactions and sends them back to the server. It collects audio and facial expression data using the camera and microphone until the customer utters the next command. For example, if the customer shows a confused expression, that data is sent to the server.

[1371] Step 6:

[1372] The server analyzes the collected audio and facial expression data using emotion recognition tools. A machine learning model using TensorFlow or PyTorch identifies the emotional state (e.g., confusion, joy, anxiety). The audio and facial expression data received as input are analyzed, and the emotion "confusion" is identified.

[1373] Step 7:

[1374] The server uses response tools to generate the most appropriate response for the customer based on the results of analysis by emotion recognition tools. For example, a prompt such as "You seem to be having trouble. Do you have any questions?" is generated by the generation AI model and sent to the smart glasses.

[1375] Step 8:

[1376] The terminal displays a response message received from the server to the customer. Using the smart glasses' display and audio output, it provides guidance via both text and voice, such as, "It seems you're having trouble. Do you have any questions?"

[1377] Step 9:

[1378] When a customer indicates their intention to purchase a product, the terminal sends a purchase request to the server and activates the payment method. Voice commands such as "I will purchase this product" are entered through the smart glasses interface and sent to the server.

[1379] Step 10:

[1380] The server processes electronic payments using the payment method. Payment information is received as input, and the transaction is executed through a secure payment gateway (e.g., Stripe, Square). If the payment is successful, a confirmation message is generated.

[1381] Step 11:

[1382] The device displays and provides audio confirmation messages after payment is complete. The smart glasses' display shows "Your purchase is complete. Thank you for using our service," and the same message is also provided verbally.

[1383] Example of a prompt

[1384] The following prompts are used:

[1385] "I'm looking for a new refrigerator."

[1386] "Please tell me more about this model."

[1387] "Which refrigerator is the most energy-efficient?"

[1388] This allows customers to gather product information without stress, receive appropriate emotional support, and complete the purchase process safely and quickly.

[1389] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1390] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1391] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1392] [Fourth Embodiment]

[1393] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1394] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1395] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1396] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1397] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1398] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1399] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1400] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1401] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1402] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1403] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1404] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1405] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1406] This invention relates to a robotic system that communicates with customers in large consumer electronics stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[1407] server

[1408] Product Database Management

[1409] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies that there are no omissions or duplicates.

[1410] Training an AI model

[1411] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[1412] Payment processing management

[1413] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[1414] Terminal (robot)

[1415] Product Information

[1416] The terminal greets customers verbally upon entering the store and receives customer requests using voice recognition. Based on product information retrieved from the server, the terminal provides customers with detailed information via voice and screen display.

[1417] Product comparison and recommendations

[1418] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality."

[1419] Electronic payment support

[1420] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process. After the customer selects a payment method, the terminal sends the payment information to the server and completes the process. Upon successful payment completion, the terminal notifies the customer with the message, "Payment complete. Thank you."

[1421] User (customer)

[1422] Product search and inquiry

[1423] Users can request the products they want from the device using voice commands and receive detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[1424] Product Comparison Selection

[1425] If a user wants to compare multiple products, they can get detailed information by giving instructions to the device. For example, if a user asks, "Which one is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[1426] Purchase procedure

[1427] Once the user selects an item and indicates their intention to purchase, the terminal guides them through the electronic payment method and completes the payment using a credit or debit card. The user simply says, "I'll buy this item," and the terminal prompts them to "Please insert your credit card," completing the payment.

[1428] Thus, the system according to the present invention helps customers collect product information, compare and consider options, and ultimately complete the purchase quickly and safely, without stress. This significantly improves the in-store shopping experience.

[1429] The following describes the processing flow.

[1430] Step 1:

[1431] Initial contact (device)

[1432] When a customer enters the store, the terminal uses voice recognition to greet them with, "Welcome. What can I help you with?" Once the customer responds, the system automatically converts the voice into text.

[1433] Step 2:

[1434] Request acceptance (terminal)

[1435] When a customer states that they want to know about a specific product, the terminal analyzes their voice and converts it into text data. This text data is then sent to a server to request a search for product information.

[1436] Step 3:

[1437] Product information search (server)

[1438] Based on the received request, the server searches its product database for relevant product information. For example, in response to a request for "new 4K TV," it extracts information such as specifications, price, availability, and user reviews.

[1439] Step 4:

[1440] Information provision (terminal)

[1441] The terminal provides customers with product information retrieved from the server via voice and on-screen display. For example, it might explain, "This 4K TV is 55 inches and priced at $800. It is currently in stock."

[1442] Step 5:

[1443] Product comparison (devices)

[1444] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality."

[1445] Step 6:

[1446] Confirmation of purchase intent (device)

[1447] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[1448] Step 7:

[1449] Electronic payment instructions (terminal)

[1450] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[1451] Step 8:

[1452] Sending payment information (terminal)

[1453] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[1454] Step 9:

[1455] Payment processing and confirmation (server)

[1456] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[1457] Step 10:

[1458] Payment completion notification (device)

[1459] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[1460] This series of steps allows customers to easily search for, compare, and complete product information within the store. This enables customers to enjoy a stress-free and efficient shopping experience.

[1461] (Example 1)

[1462] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1463] Traditional in-store purchasing processes were cumbersome, involving the collection, comparison, and purchase of product information, increasing customer stress and inconvenience. Furthermore, the need for store staff to provide information and payment support led to challenges such as staff shortages and inconsistent skill levels, resulting in decreased service quality. Concerns also existed regarding information discrepancies between online and in-store purchases, as well as payment security.

[1464] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1465] In this invention, the server includes a voice recognition means for receiving and analyzing customer voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for explaining detailed product information obtained from a management server to the customer via voice and screen display; a product comparison means for comparing multiple products and explaining their features; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables customers to quickly and easily obtain and compare product information and safely complete the purchase procedure. The server also includes a recommendation means for recommending the most suitable product based on the customer's past purchase history and preferences; a data management means for verifying the integrity of product information and storing it in a database; and a security management means for executing a fraud detection algorithm to ensure the security of payments. This improves the consistency and security of the service and enhances customer satisfaction.

[1466] "Voice recognition means" refers to technology or equipment for receiving customer voice input and analyzing its content.

[1467] "Product search means" refers to a technology or device for searching for relevant product information from a product database based on the analyzed content.

[1468] "Information presentation means" refers to technology or equipment for providing customers with detailed product information obtained from a management server via audio and screen display.

[1469] A "product comparison method" is a technology or device for comparing multiple products and explaining their characteristics.

[1470] "Payment method" refers to the technology or device that guides customers through the purchase process of selected goods and enables electronic payment.

[1471] "Recommendation methods" refer to technologies or devices for recommending the most suitable products based on a customer's past purchase history and preferences.

[1472] "Data management means" refers to technology or equipment for verifying the integrity of product information and storing it in a database.

[1473] "Security management measures" refer to technologies or devices that execute fraud detection algorithms and ensure the security of payments.

[1474] This invention relates to a robotic system that communicates with customers in large stores, provides product comparison explanations, and completes the payment process using electronic payment functionality. The specific operation of the system is described below.

[1475] server

[1476] Product Database Management

[1477] The server manages the product database for large stores. Using Python libraries such as Beautiful Soup and Scrapy, the server periodically collects product specifications, prices, stock availability, and review information, and checks data integrity. This ensures that detailed product data is stored in the database. For example, by using scraping techniques to collect information from product web pages, it's possible to update the data in real time.

[1478] Training an AI model

[1479] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. It uses TensorFlow or PyTorch to build and train a neural network model. For example, the AI ​​model is trained epoch-by-epoch to solve classification and regression tasks. Specifically, the model is evaluated using prompts such as, "Please recommend a model based on reviews of the latest refrigerators."

[1480] Payment processing management

[1481] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. The server ensures secure data communication using the SSL / TLS protocol and manages consistent transactions by building a payment management system with the Django framework. It also uses machine learning to train anomaly detection algorithms based on historical transaction data to evaluate new transactions.

[1482] Terminal (robot)

[1483] Product Information

[1484] The terminal greets customers with a voice message upon entering the store and uses speech recognition technology to receive requests from customers. It analyzes customer voices using Microsoft Azure Cognitive Services and Google Cloud's Speech-to-Text API. Based on product information obtained from the server, it uses Google Cloud's Text-to-Speech API to synthesize speech and provides detailed information through both voice and screen display. For example, if a customer says, "I'm looking for a new refrigerator," the terminal will respond, "Here are our latest models."

[1485] Product comparison and recommendations

[1486] If a customer wants to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. If a customer says, "Please compare the TVs," the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." An example of a specific prompt would be, "Which refrigerator is the most energy-efficient?"

[1487] Electronic payment support

[1488] After the customer decides to purchase, the terminal guides them through the electronic payment process using voice and on-screen displays. The customer's payment information is sent to a server for processing. Once the server completes the payment processing, the terminal notifies the customer that "payment is complete." Specifically, when the customer says, "I'll buy this item," the terminal prompts them to "insert your credit card," and the payment is completed.

[1489] User (customer)

[1490] Product search and inquiry

[1491] Users can request desired products from the device using voice commands and obtain detailed information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[1492] Product Comparison Selection

[1493] Users can request product comparisons from the device using voice commands and obtain detailed information. For example, if asked, "Which refrigerator is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity." A specific example of a prompt would be, "Please tell me the difference between these two refrigerators."

[1494] Purchase procedure

[1495] Users indicate their intention to purchase by voice and make payments using credit or debit cards according to the terminal's instructions. For example, if a user says, "I want to buy this item," the terminal will prompt them to "Please insert your credit card" and assist them through the entire payment process.

[1496] Thus, the system of the present invention provides a comprehensive flow that allows customers to smoothly obtain and compare product information that suits their needs and to make purchases securely, significantly improving the in-store shopping experience. Furthermore, the consistency and security of the service are guaranteed through server data management and security measures.

[1497] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1498] Step 1:

[1499] The server collects product data. The server uses Python libraries such as Beautiful Soup and Scrapy to perform web scraping and collect product specifications, prices, stock availability, and review information from large retail stores.

[1500] Input: Product webpage URL

[1501] Data processing: HTML parsing, extraction of necessary data.

[1502] Output: Product data list (specifications, price, stock status, review information)

[1503] Step 2:

[1504] The server checks for data integrity. The server verifies the collected product data for duplicates or missing data. If duplicates are found, the data is updated with the latest data; if missing data is found, it is collected again.

[1505] Input: Product data list

[1506] Data processing: Duplicate checking, data uniqueness verification, recollection of missing data.

[1507] Output: Consistent product database

[1508] Step 3:

[1509] The server stores product data in the database. An ORM such as SQLAlchemy is used to ensure consistent product data is stored in the database.

[1510] Input: Consistent product database

[1511] Data processing: Saving operations to a database

[1512] Output: Updated product database

[1513] Step 4:

[1514] The server trains the AI ​​model. The server uses product data, user reviews, and past purchase history to train the generative AI model. It builds and trains the neural network using TensorFlow or PyTorch.

[1515] Input: Product data, user reviews, past purchase history

[1516] Data processing: Training and evaluation of AI models

[1517] Output: Trained generative AI model

[1518] Step 5:

[1519] The device receives the customer's voice request. The device performs speech recognition using Microsoft Azure's Cognitive Services or Google Cloud's Speech-to-Text API and retrieves the customer's request in text format.

[1520] Input: Customer voice request

[1521] Data processing: Speech-to-text conversion

[1522] Output: Request content in text format

[1523] Step 6:

[1524] The device retrieves product information. The device sends a text request to the server and retrieves the corresponding product information.

[1525] Input: Request content in text format

[1526] Data processing: Sending requests to the server, retrieving product information.

[1527] Output: Product information (specifications, price, stock status, review information)

[1528] Step 7:

[1529] The terminal presents product information to the customer. The terminal uses Google Cloud's Text-to-Speech API to synthesize speech and also displays the product information on the screen.

[1530] Input: Product Information

[1531] Data processing: Voice generation, screen display updates.

[1532] Output: Providing product information via audio and screen display.

[1533] Step 8:

[1534] The terminal receives a customer comparison request. The terminal uses speech recognition technology to obtain the customer comparison request in text format.

[1535] Input: Customer voice request

[1536] Data processing: Speech-to-text conversion

[1537] Output: Comparison request in text format

[1538] Step 9:

[1539] The device retrieves comparison information. The device sends a comparison request to the server and retrieves comparison information for multiple products.

[1540] Input: Text format comparison request

[1541] Data processing: Sending requests to the server, obtaining comparison information.

[1542] Output: Comparison information (features, price, availability, etc.)

[1543] Step 10:

[1544] The terminal presents the customer with product comparison information. The terminal explains the acquired comparison information to the customer through voice and screen display.

[1545] Input: Comparison Information

[1546] Data processing: Voice generation, screen display updates.

[1547] Output: Provision of comparative information via audio and screen display.

[1548] Step 11:

[1549] The terminal receives the customer's purchase intention. When the customer indicates their intention to purchase a specific product, speech recognition technology is used to obtain that intention in text format.

[1550] Input: Customer's voice expression of purchase intent

[1551] Data processing: Speech-to-text conversion

[1552] Output: Purchase intention in text format

[1553] Step 12:

[1554] The terminal guides the customer through the payment process. The terminal uses voice and on-screen displays to instruct the customer to complete the payment using a credit or debit card.

[1555] Input: Customer's purchase intention

[1556] Data processing: Voice generation for payment instructions, updating of screen display.

[1557] Output: Payment instructions

[1558] Step 13:

[1559] The terminal sends payment information to the server. When a customer enters payment information or uses a card, the terminal sends that information to the server and requests processing.

[1560] Input: Customer payment information

[1561] Data processing: Sending payment information to the server

[1562] Output: Payment request to the server

[1563] Step 14:

[1564] The server processes the payment. The server uses the SSL / TLS protocol to perform secure payment processing, transaction management, and fraud detection.

[1565] Input: Customer payment information

[1566] Data processing: Payment processing, fraud detection, transaction management

[1567] Output: Payment processing result

[1568] Step 15:

[1569] The terminal notifies the customer that the payment is complete. Upon receiving the successful payment processing result from the server, the terminal notifies the customer via voice and on-screen display that "Payment complete."

[1570] Input: Payment processing result from the server

[1571] Data processing: Generate audio for notifications, update screen display.

[1572] Output: Payment completion notification to customer

[1573] (Application Example 1)

[1574] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1575] Traditional systems made it difficult to provide a smooth purchasing experience for customers when searching for and comparing products online, as they had to find the necessary information. Furthermore, product descriptions were uniform, making it difficult to address the individual needs of customers. In addition, electronic payments require security and smoothness in the transaction process.

[1576] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1577] In this invention, the server includes a speech recognition means for receiving customer voice input and analyzing its content; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment; and a product description generation means for generating a description of product features using a generation AI model. This makes it possible to provide product information quickly and accurately based on customer voice input and generate product descriptions tailored to individual needs. It also enables safe and smooth electronic payment.

[1578] "Voice recognition means" refers to a device or software that has the function of receiving voice input from a customer and analyzing its content.

[1579] A "product search tool" is a device or software that has the function of searching for relevant product information from a product database based on the analyzed content.

[1580] "Information presentation means" refers to a device or software that has the function of presenting searched product information and explaining it to the customer using voice and screen display.

[1581] A "payment method" is a device or software that guides customers through the purchase process for selected products and has the function of processing electronic payments.

[1582] A "generative AI model" is an algorithm or software that learns from a large amount of data and performs natural language generation.

[1583] "Product description generation means" refers to a device or software that has the function of generating a description of product features using a generation AI model.

[1584] A "product database" is a database that stores data including product specifications, prices, stock availability, and review information.

[1585] A "server" is a computer that manages the product database, trains AI models, and processes payments.

[1586] This invention provides a system that allows customers to make voice requests, which can be used in both in-store and online environments to quickly and accurately provide product information and enable secure electronic payments. The specific system configuration and operation are described below.

[1587] server

[1588] Product Database Management

[1589] The server manages the product database. The product database stores data including product specifications, prices, inventory status, and review information. The server periodically collects this data and checks its consistency. Once collected, the data is stored in the database and checked for any missing or duplicate entries.

[1590] Training an AI model

[1591] The server uses the collected data to train a generative AI model. The AI ​​model learns based on product features, user reviews, and past purchase history, and by performing speech recognition and natural language generation, it can provide users with appropriate product information.

[1592] Payment processing management

[1593] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and fast payment processing.

[1594] Device (smartphone)

[1595] Voice recognition and product search

[1596] The terminal uses voice recognition to receive customer voice input and analyzes its content. For example, if a customer requests by voice, "I'm looking for a new refrigerator," the voice recognition technology analyzes this and the product search mechanism returns the relevant search results.

[1597] Product information presentation and product description

[1598] The terminal uses information presentation tools to explain product information to the customer via voice and screen display based on the product search results. In particular, it uses a generative AI model to generate expert and easy-to-understand descriptions of product features. For this purpose, a prompt such as "Please describe the following product: Latest refrigerator model" is used.

[1599] Comparison function and recommendations

[1600] When a customer wishes to compare multiple products, the terminal provides explanations based on product information through voice and screen displays. It also recommends the most suitable product based on past purchase history and preferences.

[1601] Electronic payment

[1602] Once the customer decides to make a purchase, the terminal guides them through the electronic payment process, sends the payment information to the server, and completes the transaction. This ensures a secure payment, and the customer receives a notification stating, "Payment complete. Thank you."

[1603] User (customer)

[1604] Voice request

[1605] Users request the products they want by voice to the device. For example, if they say, "I'm looking for the latest TV," the device will respond, "This is the latest model. Its features are..."

[1606] Check and compare product descriptions

[1607] Users can view detailed product descriptions generated by the AI ​​model and request comparisons between multiple products. For example, if asked "Which is better?", the device might respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[1608] Secure purchase process

[1609] When a user expresses their intention to purchase, the terminal guides them through the electronic payment method and securely completes the payment process. For example, simply saying "I want to buy this item" will prompt the user to "Please insert your credit card."

[1610] Thus, the system according to the present invention supports customers in searching for products by voice, obtaining detailed information, and completing the purchase process safely. As a result, users can enjoy a smooth and stress-free purchasing experience.

[1611] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1612] Step 1:

[1613] Receiving voice input

[1614] The terminal receives customer voice input and converts it into text data using speech recognition. The input is the customer's voice, and the output is the parsed text data.

[1615] Step 2:

[1616] Voice analysis and search query generation

[1617] The terminal analyzes the text data obtained in step 1 and generates a search query using a product search mechanism. The input is the analyzed text data, and the output is a search query to the product database.

[1618] Step 3:

[1619] Product Database Search

[1620] The terminal sends a search query to the server and retrieves the corresponding product information from the product database. The server returns the matching product information from the product database. The input is a product search query, and the output is product information.

[1621] Step 4:

[1622] Product information presentation and explanation generation using an AI model.

[1623] The terminal uses information presentation tools to explain product information to the customer via screen display and audio, based on the acquired product information. It also uses a generative AI model to generate a description that includes detailed product features. A prompt phrase such as "Please describe the following product: ○○○" is used. The input is product information and the prompt phrase, and the output is the generated product description.

[1624] Step 5:

[1625] Product comparison and recommendations

[1626] When a customer wishes to compare multiple products, the terminal displays the comparison results via voice and on the screen, based on the product information. Furthermore, the server uses past purchase history and preference data to recommend the most suitable product. Inputs are product information and the customer's past purchase history and preference data, while outputs are comparison results and recommended products.

[1627] Step 6:

[1628] Confirmation of purchase intent and guidance on electronic payment.

[1629] The terminal confirms the customer's purchase intention and guides them through the electronic payment process using their chosen payment method. Once the customer selects a payment method, the terminal sends the payment information to the server and completes the process. The input is the customer's purchase intention and payment information, and the output is a payment completion notification.

[1630] Step 7:

[1631] Customer notification

[1632] The terminal notifies the customer that the payment has been successfully completed. The input is payment completion information, and the output is the notification to the customer.

[1633] In each step, data processing and calculations are as follows: speech recognition involves converting voice input into text data, and product search involves narrowing down products based on the text data. The generative AI model generates descriptions in natural language based on prompts and product information, and electronic payment procedures involve data management based on security protocols.

[1634] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1635] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functions. Furthermore, by combining this with an emotion engine that recognizes user emotions and adapts its response accordingly, it provides a more personalized purchasing experience.

[1636] server

[1637] Product Database Management

[1638] The server manages the product database for large electronics retailers. The server periodically collects data such as product specifications, prices, inventory status, and review information, and checks for consistency. Once product information is collected, the server stores it in the database and verifies for errors and duplicates.

[1639] Training an AI model

[1640] The server uses the collected data to train a generative AI model. This training data includes product features, user reviews, and past purchase history. The server then uses this data to improve speech recognition and natural language processing technologies.

[1641] Training an emotion recognition model

[1642] The server uses audio and image data to train an emotion recognition model. Specifically, it analyzes factors such as voice tone, speaking speed, and facial expression data to infer the customer's emotional state.

[1643] Payment processing management

[1644] The server implements the security protocols necessary for electronic payments and runs transaction management and fraud detection algorithms. This enables secure and rapid payment processing.

[1645] Terminal (robot)

[1646] Product Information

[1647] The terminal greets customers verbally upon entering the store and accepts customer requests using voice recognition. Based on product information retrieved from the server, it provides detailed information through voice and screen displays.

[1648] Product comparison and recommendations

[1649] If a customer wishes to compare specific products, the terminal will select several products and explain their features using voice and on-screen displays. For example, if a customer asks about televisions, the terminal will explain, "This model is energy-efficient, and this one has superior sound quality." It also uses recommendation tools to suggest the most suitable product based on the customer's past purchase history and preferences.

[1650] Emotion recognition and response

[1651] The device is equipped with emotion recognition technology that identifies the customer's emotions from their voice and facial expressions. For example, if the customer appears confused, the device will follow up with a message such as, "You seem to be having trouble. Do you have any questions?" If the customer is happy, the device will offer additional product recommendations, adapting its response to the customer's emotions.

[1652] Electronic payment support

[1653] Once the customer decides to purchase, the terminal guides them through the electronic payment process. If the customer selects a credit card, the terminal reads the card information and securely transmits it to the server.

[1654] User (customer)

[1655] Product search and inquiry

[1656] Users make voice requests to the device to obtain product information. For example, if a user says, "I'm looking for a new refrigerator," the device will respond, "Here's the latest model. Its features are..."

[1657] Product Comparison Selection

[1658] If a user wants to compare multiple products, they can give instructions to the device to obtain detailed information. For example, if a user asks, "Which one is better?", the device will respond, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[1659] Purchase process and emotional follow-up

[1660] Once a user selects a product and indicates their intention to purchase, the terminal guides them through the electronic payment method and communicates appropriately according to the customer's emotional state. For example, if the user appears anxious, it will reassure them by saying, "This payment method is secure, so please rest assured."

[1661] Thus, the system according to the present invention helps customers gather product information without stress, compare and consider products while being sensitive to their emotions, and complete the purchase process quickly and safely. This enables a more personalized and satisfying purchasing experience.

[1662] The following describes the processing flow.

[1663] Step 1:

[1664] Initial contact (device)

[1665] When a customer enters the store, the terminal uses voice recognition to greet them with, "Welcome. What can I help you with?" Once the customer responds, the system automatically converts the voice into text.

[1666] Step 2:

[1667] Request acceptance (terminal)

[1668] When a customer states that they want to know about a specific product, the terminal analyzes their voice and converts it into text data. This text data is then sent to a server to request a search for product information.

[1669] Step 3:

[1670] Product information search (server)

[1671] Based on the received request, the server searches its product database for relevant product information. For example, in response to a request for "new 4K TV," it extracts information such as specifications, price, availability, and user reviews.

[1672] Step 4:

[1673] Information provision (terminal)

[1674] The terminal provides customers with product information retrieved from the server via voice and on-screen display. For example, it might explain, "This 4K TV is 55 inches and priced at $800. It is currently in stock."

[1675] Step 5:

[1676] Emotion recognition (device)

[1677] The device analyzes the customer's emotions from their facial expressions and voice tone. For example, if a customer is confused, it detects facial tension and voice anxiety and stores this as emotion recognition data.

[1678] Step 6:

[1679] Emotion-based response (device)

[1680] Based on emotion recognition data, the device responds in a way that is appropriate to the customer's emotions. For example, if the customer is confused, it will follow up with, "You seem to be having trouble. What kind of information are you looking for?"

[1681] Step 7:

[1682] Product comparison and recommendations (devices)

[1683] When a customer requests to compare multiple products, the terminal sends another request to the server to search for the products to compare. Based on the search results, the terminal provides a comparative explanation, such as, "This model is energy-efficient, but this one has superior sound quality." It also recommends the most suitable product based on the customer's past purchase history and preferences.

[1684] Step 8:

[1685] Confirmation of purchase intent (device)

[1686] When the customer selects a specific product, the terminal will ask, "Do you want to purchase this product?" If the customer answers "Yes," the terminal proceeds to the next step.

[1687] Step 9:

[1688] Electronic payment instructions (terminal)

[1689] If the customer indicates their intention to purchase, the terminal guides them through the electronic payment process. It asks questions such as, "Please select your payment method. Credit card, debit card, or cash?" and waits for the customer's selection.

[1690] Step 10:

[1691] Sending payment information (terminal)

[1692] When a customer selects a credit card, the terminal reads the credit card information, encrypts it, and then sends it to the server.

[1693] Step 11:

[1694] Payment processing and confirmation (server)

[1695] The server processes the transaction based on the received payment information. If the payment is successful, the server sends a success message to the terminal.

[1696] Step 12:

[1697] Payment completion notification (device)

[1698] The terminal will announce "Payment complete. Thank you" via voice message and issue a paper or electronic receipt. It will also provide the customer with information about the purchased items.

[1699] This series of steps allows customers to easily search for and compare product information in-store and complete the purchase process while receiving emotionally tailored service. This enables customers to enjoy a stress-free, efficient, and personalized shopping experience.

[1700] (Example 2)

[1701] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1702] Large consumer electronics retailers attract many customers, and it is essential to provide services tailored to each individual's purchasing needs and emotional state. However, conventional systems lack the ability to recognize and respond to customer emotions, making it difficult to provide a personalized shopping experience. Furthermore, simply presenting product information or recommending products based on purchase history may not allow customers to make the best choice. To solve these problems, there is a need to develop a robotic system that adapts its response based on the customer's emotional state.

[1703] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1704] In this invention, the server includes a voice recognition means for receiving and analyzing the customer's voice input; a product search means for searching for relevant product information from a product database based on the analyzed content; an information presentation means for presenting the searched product information and explaining it to the customer via voice and screen display; an emotion recognition means for recognizing the customer's emotions and responding accordingly; and a payment means for guiding the customer through the purchase procedure for the selected product and performing electronic payment. This enables the customer to obtain appropriate product information and proceed with the purchase procedure with peace of mind while receiving personalized responses that respond to their emotions.

[1705] "Voice recognition means" refers to a device or technology for receiving customer voice input and analyzing its content.

[1706] "Product search means" refers to a device or technology for searching for relevant product information from a product database based on the analyzed audio content.

[1707] "Information presentation means" refers to a device or technology for presenting searched product information and explaining it to customers using audio and screen displays.

[1708] "Emotion recognition means" refers to a device or technology for recognizing a customer's emotions and responding accordingly.

[1709] "Payment method" refers to a device or technology that guides customers through the purchase process of selected goods and enables electronic payment.

[1710] "Recommendation method" refers to a device or technology for recommending the most suitable product based on a customer's past purchase history and preferences.

[1711] A "server" is a computer system used for managing product information and training AI models that generate it.

[1712] A "product database" is a data storage system used to store and manage details such as product specifications, prices, inventory status, and review information.

[1713] A "generative AI model" is an artificial intelligence model trained using machine learning techniques based on collected data.

[1714] This invention relates to a robot system for use in large consumer electronics stores that communicates with customers, provides product comparison explanations, and completes payment using electronic payment functionality. Furthermore, by combining this with an emotion engine that recognizes customer emotions and adapts its response accordingly, the system provides a more personalized purchasing experience.

[1715] Server-based processing

[1716] 1. Product database management:

[1717] The server accesses the product database and periodically collects product specifications, prices, stock availability, and review information. The collected data is temporarily stored, checked for duplicates and errors, and then saved to the database. For example, a Python script could be used to retrieve data from an API and store it in a MySQL database.

[1718] 2. Training the Generative AI Model:

[1719] The server uses the collected data to train a generative AI model. This training uses data such as past purchase history, user reviews, and product features. Machine learning frameworks like TensorFlow and PyTorch are used for data preprocessing and model training. For example, a model is built to predict recommended products based on a user's purchase history.

[1720] 3. Training the emotion recognition model:

[1721] The server collects audio and image data, analyzes speech tone, speech rate, and facial expression data, and trains an emotion recognition model. Librosa is used for preprocessing audio data, and OpenCV is used for preprocessing image data. The emotion recognition model is built using a framework such as Keras.

[1722] 4. Payment processing management:

[1723] The server implements the necessary security protocols for electronic payments (e.g., SSL / TLS) and runs transaction management and fraud detection algorithms. It uses the Django framework and payment APIs (e.g., Stripe) to perform secure payment processing.

[1724] Processing by terminal (robot)

[1725] 1. Providing product information:

[1726] The terminal greets customers with a voice message upon entering the store and accepts requests using a voice recognition system (e.g., Google Speech-to-Text). It retrieves product information from the server and explains it to the customer using text-to-speech and on-screen displays.

[1727] Specific example: When a customer requests, "Tell me about the latest refrigerator," the terminal responds, "This is our latest refrigerator. It is energy-efficient and has a large capacity."

[1728] 2. Product comparison and recommendations:

[1729] The terminal retrieves information on multiple products from the server in response to the customer's request. Based on the customer's past purchase history and preferences, it recommends suitable products.

[1730] Specific example: When a customer asks, "What's the difference between this refrigerator and that refrigerator?", the terminal replies, "This refrigerator is more energy-efficient, but that model has a larger capacity."

[1731] 3. Emotion Recognition and Response:

[1732] The device captures the customer's voice and facial expressions using a camera and microphone, and analyzes them in real time. If the customer appears confused, it follows up with "Are you having any trouble?", and if they are pleased, it responds with "Do you like this as well?".

[1733] Specific example: The device's camera captures facial expressions and records audio. The analysis engine uses this information to determine the emotional state and take appropriate action.

[1734] 4. Support for electronic payments:

[1735] The terminal displays a user interface for entering payment information, and once the customer enters their credit card information, it is encrypted and sent to the server.

[1736] Specific example: If a customer says, "I'll buy this refrigerator," the terminal will ask, "Do you have a credit card?" and read and process the card information.

[1737] User (customer) operation

[1738] 1. Product search and inquiry:

[1739] The user requests by voice, "Show me the latest refrigerators." The terminal queries the server for product information and provides the retrieved information via voice and display.

[1740] For example, if you say, "I'm looking for a new refrigerator," the device will respond, "Here's our latest model. Its features are..."

[1741] 2. Product comparison and selection:

[1742] The user asks the terminal, "What's the difference between this refrigerator and that refrigerator?" The terminal retrieves comparison data from the server and explains it to the user.

[1743] For example, when asked "Which is better?", the device replies, "This refrigerator is more energy-efficient, while that model has a larger capacity."

[1744] 3. Purchase process and emotional follow-up:

[1745] When a user says, "I'll buy this refrigerator," the terminal guides them through the payment process. If the user appears unsure, the terminal reassures them by saying, "This payment method is secure, so please rest assured."

[1746] Specific example: The user enters payment information following the voice guidance, and the terminal displays an appropriate follow-up message.

[1747] Thus, the system according to the present invention helps customers gather product information without stress, compare and consider products while being sensitive to their emotions, and complete the purchase process quickly and safely. This makes it possible to provide a more personalized and satisfying purchasing experience.

[1748] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1749] Step 1: Managing the product database

[1750] The server collects product data from online APIs and internal systems at regular intervals (e.g., daily). The input is JSON data from the API. The server uses a Python script to temporarily store the data in memory. Next, the script detects and corrects duplicate and incorrectly formatted data. The output is the product data with verified integrity, which is then stored in a MySQL database.

[1751] Step 2: Training the Generative AI Model

[1752] The server takes a CSV file containing text data such as user reviews and purchase history as input. It uses machine learning frameworks like TensorFlow and PyTorch to preprocess the data, specifically performing data cleaning and normalization. Next, it trains a generative model and evaluates its accuracy. The output of this step is the trained generative AI model, which is saved to disk.

[1753] Step 3: Training the emotion recognition model

[1754] The server takes audio and image data as input and preprocesses the audio for tone, speech rate, and facial expression data. Specifically, it uses Librosa for audio data preprocessing and OpenCV for image data preprocessing. It builds and trains an emotion recognition model using a framework such as Keras. The output is the trained emotion recognition model, which is saved to disk.

[1755] Step 4: Managing Payment Processing

[1756] The server receives payment requests from customers as input and protects the data using SSL / TLS encrypted communication protocols. It runs fraud detection algorithms to detect unusual transactions in real time. It securely stores payment information and completes transactions. The output is the completed payment transaction.

[1757] Step 5: Provide product information

[1758] The terminal takes the customer's voice request as input and converts the content into text using a speech recognition system (Google Speech-to-Text). It then sends a query to the server to retrieve product information. The retrieved product information is provided to the customer via text-to-speech and on the display. The output consists of product information presented via voice and on the display.

[1759] Step 6: Product Comparison and Recommendation

[1760] The terminal takes a customer's comparison request as input and retrieves information on multiple products from the server. Based on the customer's past purchase history and preferences, it suggests recommended products. Specifically, it describes the features of the multiple products in both text and audio. The output includes information on the compared products and recommended products.

[1761] Step 7: Recognizing and responding to emotions

[1762] The device acquires customer voice and facial expressions as input via camera and microphone. It uses an emotion recognition model in real time to analyze the customer's emotional state. For example, it captures camera and audio data using Python and inputs it into the model. If the customer is confused, it follows up with "Are you having trouble?", and if they are happy, it responds with "Did you like it?". The output is a follow-up message tailored to the customer's emotion.

[1763] Step 8: Support for electronic payments

[1764] The terminal takes the customer's purchase intention as input and displays a UI for electronic payment. Once the customer enters their credit card information, the data is encrypted and sent to the server. Specifically, SSL / TLS-protected communication is used to securely process payment information. The output is the completed payment procedure and transaction confirmation information.

[1765] Step 9: Product Search and Inquiry

[1766] The user enters a voice request saying, "Show me the latest refrigerators." The device uses Google Speech-to-Text to convert the speech to text and sends the query to the server. The data retrieved from the server is provided to the user via speech synthesis and on the display. The output is the searched product information.

[1767] Step 10: Compare and select products

[1768] The user enters a question such as, "What is the difference between this refrigerator and that refrigerator?" The terminal sends a query to the server, retrieves the necessary comparison information, and displays it. Specifically, it explains the features of each product in text and audio. The output is the compared product information.

[1769] Step 11: Purchase Procedure and Emotional Follow-up

[1770] When a user enters "I will buy this refrigerator," the terminal guides them through the payment process. To alleviate customer concerns, a reassuring follow-up message is added. Specifically, a combination of speech recognition and text generation is used to explain the secure payment process. The output includes the completed purchase and the follow-up message.

[1771] (Application Example 2)

[1772] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1773] The process by which customers compare and purchase products in physical stores is often cumbersome, leading to decreased customer satisfaction. Furthermore, current customer service systems struggle to provide personalized responses that take into account individual customer emotional states, resulting in a standardized customer experience. Moreover, few such systems offer secure and rapid electronic payment capabilities. As a result, the overall in-store purchasing experience is unsatisfactory.

[1774] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means that receives voice input from a customer and analyzes its contents; a product search means that searches for relevant product information from a product database based on the analyzed contents; an information presentation means that presents the searched product information and explains it to the customer using voice and screen display; a payment means that guides the customer through the purchase procedure of the product selected by the customer and performs electronic payment; an emotion recognition means that analyzes the customer's emotional state from voice and facial expressions; and a response means that adapts the response based on the analyzed emotional state. As a result, customers can collect and compare product information and make safe and quick electronic payments while receiving personalized responses tailored to their individual emotional state in a physical store.

[1775] A "voice recognition system" is a system that receives voice input from a customer and analyzes its content.

[1776] "Product search method" refers to a function that searches for relevant product information from a product database based on the analyzed content.

[1777] "Information presentation means" refers to a function that presents searched product information and explains it to the customer via voice and screen.

[1778] "Payment method" refers to the function that guides customers through the purchase process for the products they have selected and allows them to make electronic payments.

[1779] An "emotion recognition method" is a system that analyzes a customer's emotional state from their voice and facial expressions.

[1780] "Response methods" refer to the function of performing optimal communication based on the analyzed emotional state.

[1781] "Recommendation methods" refer to functions that recommend the most suitable products based on the customer's past purchase history and preferences.

[1782] A "generative AI model" is an artificial intelligence model trained based on collected data.

[1783] A "server" is a computer system that manages product information and trains AI models for generating it.

[1784] "Smart glasses" are glasses-type devices equipped with a display and audio output functions.

[1785] This system uses smart glasses in physical stores to provide customers with optimal product information and to realize a personalized purchasing experience through emotion recognition. The elements that make up this system and their operation are described below.

[1786] server

[1787] The server manages and operates the entire system using the following methods.

[1788] Speech recognition means

[1789] The server receives customer voice input and analyzes its content. Using speech recognition technologies such as the Google Cloud Speech-to-Text API, the voice data is converted into text data. When a customer enters a voice command such as "I'm looking for a new refrigerator," the content is first converted into text data by the speech recognition system.

[1790] Product search method

[1791] The server analyzes the text data obtained by the speech recognition system and searches the product database for relevant product information. For example, if the keyword "refrigerator" is extracted, it retrieves detailed information about products that match that condition from the database. High-speed product information retrieval is possible by using SQL or NoSQL databases (e.g., MySQL, MongoDB).

[1792] Information presentation means

[1793] The acquired product information is provided to the customer via both audio and screen display. Specifically, the product name, detailed information, price, etc., are displayed on the smart glasses' screen, along with voice guidance. This allows customers to obtain the necessary information in real time, both visually and audibly. By using speech synthesis services such as Amazon Web Services' Polly, natural-sounding voice information is provided.

[1794] Payment methods

[1795] The customer is guided through the purchase process for their selected items via smart glasses, and electronic payment is processed. Secure and fast payment processing is achieved by using payment gateways that support credit card payments and digital wallets (e.g., Stripe, Square). Once the customer confirms their purchase, the payment process is completed through the smart glasses interface, and a payment completion notification is displayed.

[1796] emotion recognition means

[1797] To analyze customer emotions from their voice and facial expressions, emotion recognition technology using machine learning models (e.g., TensorFlow, PyTorch) is employed. By analyzing voice tone and facial expression data, the customer's emotional state (e.g., confusion, joy, anxiety) is identified. If the customer appears confused, follow-up questions such as, "You seem to be having trouble. Do you have any questions?" are asked.

[1798] Countermeasures

[1799] Based on the analyzed emotional state, optimal communication is provided. If the customer is happy, appropriate responses are given, such as suggesting additional products. This improves customer satisfaction.

[1800] Device (smart glasses)

[1801] Product Information

[1802] The smart glasses are equipped with a display and audio output function, allowing customers to easily obtain product information while walking around the store. For example, if a customer asks, "What are the features of this refrigerator?", the product's specifications and price will be displayed on the screen, and a detailed explanation will be provided simultaneously via audio.

[1803] Emotion recognition and response

[1804] The smart glasses are equipped with a camera and microphone, which are used to capture the customer's voice and facial expressions and send them to a server. Emotion recognition technology identifies the customer's emotions, and the appropriate response is provided. For example, if a customer appears anxious, they might be reassured with a message such as, "Please rest assured, this payment method is very secure."

[1805] User (customer)

[1806] Product search and inquiry

[1807] Users request product information via voice commands through smart glasses. Using simple voice commands, they can obtain detailed information about specific products or compare multiple products.

[1808] Product Comparison Selection

[1809] For example, if a user asks, "What's the difference between this refrigerator and that refrigerator?", the smart...

Claims

1. A voice recognition system that receives customer voice input and analyzes its content, A product search method that searches for relevant product information from a product database based on the analyzed content, An information presentation method that displays searched product information and explains it to the customer using voice and screen display, A payment method that guides customers through the purchase process for the products they have selected and allows them to make electronic payments. A system that includes this.

2. The system according to claim 1, further comprising a recommendation means for recommending the most suitable product based on the customer's past purchase history and preferences.

3. The system according to claim 1, which uses a server for managing product information and training an AI model.

Images