system

Abstract

We provide the system. [Solution] A means for detecting a user's visit, A means of interacting with users using generative artificial intelligence, A means of checking product inventory in response to user requests, A means of processing payment for the product selected by the user, A method of having a robot deliver the goods after payment is completed, Methods for managing store inventory, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention relates to reducing labor costs and improving efficiency in the operation of unmanned stores. In conventional unmanned stores, many tasks such as product sales, inventory management, and customer service were performed manually, resulting in high labor costs being a major cost factor. In addition, this led to a decrease in operational efficiency, and there were many situations where it was difficult to provide optimal services. Furthermore, incentives through agents were also a major burden. It is required to solve these problems.

Means for Solving the Problems

[0005] The present invention solves the above problems by the following means. Specifically, it provides a system that includes means for detecting a user's arrival, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for managing the store's inventory. This system automates the entire process from the user's arrival to product purchase, delivery, and inventory management, thereby reducing labor costs and improving operational efficiency.

[0006] "User" refers to a customer who uses an unmanned shop.

[0007] "Means of detecting visits" refers to devices such as sensors and cameras used to detect when a user enters a store.

[0008] "Generative artificial intelligence" refers to artificial intelligence programs that can interact with users using natural language processing.

[0009] "Means of checking inventory" refers to a database system used to manage and check the inventory status of products.

[0010] "Means of processing payment" refers to systems such as terminal devices and credit card readers that users use to pay for goods.

[0011] "Means of delivering goods by robot" refers to robots used to transport and provide goods to users after payment has been completed.

[0012] "Means of inventory management" refers to an artificial intelligence program that monitors and updates the inventory status within a store in real time and takes appropriate action when supplies run low. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, a labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The following describes the specific operating procedures and program processing of this system in natural language.

[0035] System Overview

[0036] This system detects user visits, uses generative artificial intelligence to engage in dialogue, and automates product search and selection, payment, delivery, and inventory management. The detailed operation of each element is described below.

[0037] User visit detection

[0038] When a user arrives at the unmanned shop, a sensor installed at the entrance detects the user. The information from this sensor acts as a trigger to activate the artificial intelligence customer service system, which will be described next.

[0039] AI-powered customer service

[0040] When the sensor detects a user, the server activates generative artificial intelligence and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, if the user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?"

[0041] Product search and selection

[0042] When a user selects a smartphone based on the suggestions of the generative artificial intelligence, the server accesses the inventory database to check the stock status of that product. If the product is in stock, the server notifies the generative artificial intelligence, which then informs the user.

[0043] Payment processing

[0044] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code (registered trademark) or credit card on the terminal. After payment is complete, the server receives the information and begins the next step in the product delivery process.

[0045] Product handover

[0046] Once payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[0047] Store inventory management

[0048] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[0049] Specific example

[0050] For example, suppose a user visits an unmanned shop called "TECH STORE" and wants to buy a new smartphone. In this case, the process would be as follows:

[0051] 1. When a user reaches the entrance, the terminal (sensor) detects the user.

[0052] 2. The server activates the generative artificial intelligence and speaks to the user.

[0053] 3. The generative artificial intelligence asks, "Welcome. What kind of product are you looking for?"

[0054] 4. The user responds, "I want a new smartphone."

[0055] 5. The generative AI checks the inventory via the server and notifies the user, "We have the latest Model X. There are 5 in stock. Would you like to purchase it?"

[0056] 6. The user responds, "Yes, I will purchase it."

[0057] 7. The server displays the payment screen on the payment terminal, and the user completes the payment using the QR code.

[0058] 8. The server confirms that the payment is complete and sends instructions to the robot to pick up the goods.

[0059] 9. The robot hands the product to the user, and the user receives the product.

[0060] 10. The terminal (store management AI) recognizes the decrease in inventory and sends a replenishment instruction to the server.

[0061] In this way, this system enables the operation of automated, unmanned shops, providing reduced labor costs and improved operational efficiency.

[0062] The following describes the processing flow.

[0063] Step 1:

[0064] The user reaches the entrance of the unmanned shop. A terminal (a sensor installed at the entrance) detects the user's movement and sends that information to the server.

[0065] Step 2:

[0066] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI automatically displays a greeting message to the user.

[0067] Step 3:

[0068] A user who sees a message displayed by a generative artificial intelligence asks questions about the product they want.

[0069] Generative artificial intelligence analyzes user statements and suggests the most suitable products. If the user says, "I want a new smartphone," it will suggest, "How about the latest smartphone model X?"

[0070] Step 4:

[0071] Upon receiving a user's request, the server accesses the inventory database to check the stock status of the requested product. The server then sends the inventory information to a generative artificial intelligence (AI), which notifies the user. For example, it might display a message such as, "There are 5 items left in stock."

[0072] Step 5:

[0073] The user selects an item and indicates their intention to purchase it. The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the user's payment information.

[0074] Step 6:

[0075] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[0076] Step 7:

[0077] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the goods from the warehouse and prepares them for delivery to the user.

[0078] Step 8:

[0079] A robot delivers the product to a designated pick-up station and notifies the user that the product is ready. The user then picks up the product from the station.

[0080] Step 9:

[0081] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[0082] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, it significantly reduces labor costs and improves operational efficiency.

[0083] (Example 1)

[0084] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0085] Conventional unmanned stores often rely on manual or semi-automatic processes for tasks such as detecting user arrival, selecting products, processing payments, handing over products, and managing inventory, resulting in reduced overall operational efficiency. High operating costs and frequent stockouts were also problematic. Furthermore, the mechanical nature of user interaction made it difficult to provide a smooth purchasing experience. This invention aims to solve these problems, streamline the operation of unmanned stores, and reduce costs.

[0086] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0087] In this invention, the server includes means for detecting a user's arrival, means for transmitting the detected information to the server, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for monitoring inventory status for store operations and automatically generating replenishment instructions. This automates the entire process from the user's arrival to the completion of purchase, significantly improving operational efficiency and enabling cost reduction.

[0088] "Means of detecting user visits" refers to a system that uses devices such as sensors and cameras to detect when a user visits a store and collects that information.

[0089] "Means for transmitting detected information to a server" refers to a system that includes communication devices and protocols for transmitting user visitor information collected by sensors, cameras, etc., to a server.

[0090] "Means of engaging in dialogue with users using generative artificial intelligence" refers to a system that utilizes the latest generative artificial intelligence technology to enable natural dialogue with users through voice or text.

[0091] "A means of checking product inventory in response to user requests" refers to a system that queries a database for inventory information of products selected or requested by the user and retrieves the results.

[0092] "A means of processing payment for products selected by the user" refers to a system that processes the payment for products that the user has decided to purchase using payment methods such as QR codes or credit cards.

[0093] "A method of handing over goods by robot after payment is completed" refers to a system in which, after payment is completed, a robot picks up the designated goods and either hands them directly to the user or places them at a handover station.

[0094] "A means of monitoring inventory status and automatically generating replenishment instructions for store operations" refers to a system that constantly monitors inventory within a store, and when inventory levels become low, the server automatically generates replenishment instructions, thereby streamlining store operations.

[0095] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The system detects user arrivals, engages in dialogue using generative artificial intelligence, and automates product search and selection, payment, delivery, and inventory management. The following describes the specific operating procedures of this system.

[0096] The system includes the following main hardware and software components:

[0097] 1. Sensors: Installed at the entrance to detect user arrivals. For example, infrared sensors or cameras are used.

[0098] 2. Server: Receives information from sensors and activates generative artificial intelligence. It also accesses the inventory database, processes payments, and sends instructions to robots.

[0099] 3. Generative Artificial Intelligence: Enables voice or text-based interaction with the user. This artificial intelligence performs tasks such as answering questions and suggesting products.

[0100] 4. Inventory Database: A database that manages the inventory status of products in real time.

[0101] 5. Payment terminal: A terminal that includes a QR code reader and credit card reader to process payments for the items selected by the user.

[0102] 6. Robot: After payment is completed, the robot picks up the goods from the warehouse and either hands them to the user or places them at a designated handover station.

[0103] 7. Store Management AI: An artificial intelligence program that constantly monitors inventory levels and automatically generates replenishment instructions.

[0104] As a concrete example, the process for a user visiting an unmanned shop and wanting to purchase a new smartphone would be as follows:

[0105] 1. When a user reaches the store entrance, a sensor installed on the terminal detects the user and sends that information to the server.

[0106] 2. The server receives the information, activates the generative artificial intelligence, and speaks to the user, saying, "Welcome. What kind of product are you looking for?"

[0107] 3. When a user says, "I want a new smartphone," the generative artificial intelligence suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0108] 4. Once the user decides to make a purchase, the server displays the payment screen on the payment terminal.

[0109] 5. Once the user scans the QR code and completes the payment, the server sends instructions to the robot to pick up the items.

[0110] 6. The robot retrieves a smartphone from the warehouse and hands it to the user.

[0111] 7. The store management AI checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment order.

[0112] This system automates the entire process, significantly improving operational efficiency and reducing costs. Furthermore, users can enjoy a smooth and intuitive purchasing experience.

[0113] The above describes embodiments for carrying out the present invention.

[0114] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0115] Step 1:

[0116] When a user reaches the store entrance, a sensor installed on the terminal detects the user. The sensor captures the user's movements and transmits the data to a server. The sensor's input is the detection of the user's movement or presence, and its output is a signal containing the detection information. This signal notifies the server that the user has entered the store. Infrared sensors and cameras are used specifically for this purpose.

[0117] Step 2:

[0118] When the server receives information from the sensor that a user has arrived, it activates the generative artificial intelligence (AI). The server's input is the detection information from the sensor, and its output is the command to activate the generative AI. The generative AI then begins a voice or text-based conversation with the user. For example, the server might send the command "A user has arrived. Please begin the conversation" to the generative AI.

[0119] Step 3:

[0120] The generative artificial intelligence (AI) initiates a conversation with the user and listens to their requests. The input to the AI ​​is the startup command from the server and the user's statements, and the output is the response. If the AI ​​greets the user with "Welcome. What product are you looking for?" and the user replies "I want a new smartphone," the AI ​​proceeds to the next step based on that request.

[0121] Step 4:

[0122] The server receives a user request, accesses the inventory database, and checks the product's stock status. The server's input is the user's request information from the generative artificial intelligence, and its output is the inventory check result. Specifically, the server sends a query to the database saying, "Please check the inventory of Model X smartphones," and the database responds, "There are 5 in stock."

[0123] Step 5:

[0124] The generative artificial intelligence provides product information to the user based on inventory check results from the server. The input to the generative AI is the inventory check results, and the output is a notification to the user. Specifically, it notifies the user, "We have the latest smartphone model X. There are 5 in stock. Would you like to purchase it?"

[0125] Step 6:

[0126] Once the user selects an item and decides to purchase it, the server displays a payment screen on the payment terminal. The server's input is the user's purchase decision and selected item information, and its output is instructions for the payment terminal. Specifically, the server sends the message "Process payment for Model X smartphone" to the payment terminal.

[0127] Step 7:

[0128] Users pay using a QR code or credit card at a payment terminal. The terminal receives payment instructions from the server and the user's payment information, and the output is a confirmation that the payment is complete. Specifically, the user scans a QR code with their smartphone, and the payment amount is displayed on the terminal.

[0129] Step 8:

[0130] Once the server confirms payment is complete, it sends an instruction to the robot to pick up the item. The server's input is payment completion information, and its output is an instruction to the robot. Specifically, it commands the robot to "Pick up the Model X smartphone and deliver it to the user."

[0131] Step 9:

[0132] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The robot receives instructions from a server as input, and its output is the completion of product delivery. For example, the robot might take a smartphone from a shelf and hand it to the user.

[0133] Step 10:

[0134] The store management AI on the terminal checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment instruction. The input to the store management AI is inventory data, and the output is a replenishment instruction to the server. For example, if the inventory falls to two or fewer units, it automatically notifies the server with "Inventory replenishment is needed."

[0135] (Application Example 1)

[0136] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0137] In operating unmanned stores, there is a need for specific methods and systems to achieve efficiency and cost reduction by fully automating everything from user arrival to product purchase, handover, and inventory management. In this process, rapid and accurate product recommendations and payment processing are required to enhance the user experience.

[0138] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0139] In this invention, the server includes means for detecting a user's arrival, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, means for managing the store's inventory, means for detecting the user using the user's smart device and beacon device, means for generative artificial intelligence to suggest products based on the user's statements, and means for processing payment via QR code or electronic payment service. This enables improved user experience, increased operational efficiency, and cost reduction in unmanned stores.

[0140] A "user" is a customer who visits an unmanned store and intends to purchase goods.

[0141] "Means of detecting visits" refer to devices such as sensors and beacon devices that detect a user's presence when they access a store.

[0142] "Generative artificial intelligence" refers to artificial intelligence programs that generate appropriate responses and suggestions based on user statements and requests.

[0143] "Means for checking product inventory" refers to a database system that manages the inventory status of products and provides that information to users upon request.

[0144] "Means of processing payment" refers to terminal devices and software used to process payments for products selected by the user through QR codes or electronic payment services.

[0145] "A means of delivering goods by robot" refers to a robotic system that picks up goods from a warehouse after payment is completed and delivers them to the user.

[0146] "A means of managing store inventory" refers to an artificial intelligence program that constantly monitors the inventory of products in a store and automatically instructs replenishment as needed.

[0147] A "smart device" is a portable electronic device that can connect to the internet, such as a smartphone or tablet.

[0148] A "beacon device" is a small device that provides location information via wireless communication and interacts with nearby smart devices.

[0149] A "QR code" is a type of two-dimensional barcode, a pattern used to efficiently transmit information.

[0150] "Electronic payment services" are services that allow payments to be completed electronically in an online or mobile environment (e.g., credit cards, various payment apps).

[0151] A "prompt" is an initial word or phrase that a generative artificial intelligence uses to initiate a conversation with a user.

[0152] System Overview

[0153] The unmanned store operation system accurately detects user arrivals, interacts with users using generative artificial intelligence, and automates product search and selection, payment, handover, and inventory management. The system of the present invention is composed of the following hardware and software.

[0154] User visit detection

[0155] First, when a user arrives at the unmanned store, their smart device is detected by a beacon device installed at the entrance. This allows the system to recognize the user's arrival as a trigger.

[0156] AI-powered customer service

[0157] When a user's arrival is detected by a beacon, the server activates a generative artificial intelligence (for example, OpenAI®'s GPT-4® model) and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, the following prompts are used.

[0158] Prompt message: "Welcome. What kind of product are you looking for?"

[0159] Product search and selection

[0160] When a user selects a product based on the suggestions of a generative artificial intelligence, the server accesses the store's inventory database to check the product's availability. If the product is in stock, the generative AI notifies the user and confirms their intention to purchase.

[0161] Payment processing

[0162] Once a user selects an item and indicates their intention to purchase, the server displays a payment screen on the payment terminal. The user pays using a QR code or an electronic payment service (e.g., Stripe API or Square API) on the terminal. After the payment is complete, the information is sent to the server.

[0163] Product handover

[0164] After payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[0165] Store inventory management

[0166] The server constantly monitors the store's inventory levels and automatically detects when replenishment is needed if inventory levels drop. This prevents stockouts and improves the efficiency of store operations.

[0167] Specific example

[0168] Suppose a user visits an unmanned store and wants to purchase the latest smartphone. In this case, the process would proceed as follows:

[0169] 1. When a user arrives at the store entrance, a beacon device detects the user's smart device, and the arrival is recognized as a trigger.

[0170] 2. The server starts the generative artificial intelligence and sends the following prompt message to the user.

[0171] "Welcome. What kind of product are you looking for?"

[0172] 3. When a user responds with "I want a new smartphone," the generative artificial intelligence retrieves appropriate product information from the store's inventory database and suggests it to the user.

[0173] "We have the latest Model X in stock. We have 5 units available. Would you like to purchase one?"

[0174] 4. When the user responds with "Yes, I will purchase," the payment screen is displayed.

[0175] 5. The user completes the payment using a QR code or electronic payment service, and that information is sent to the server.

[0176] 6. The server instructs the delivery robot to pick up the product, and the robot delivers the product to the user.

[0177] 7. The server updates the store's inventory and automatically issues instructions if replenishment is needed.

[0178] The system described above streamlines the operation of unmanned stores, improves the user shopping experience, and reduces costs.

[0179] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0180] Step 1:

[0181] When a user arrives at an unmanned store, a beacon device installed at the entrance detects the user's smart device. The input is the ID signal of the smart device connected to the beacon device, and the output is a visitor notification sent to the server. Upon receiving this notification, the server recognizes the user's arrival and initiates the next processing step.

[0182] Step 2:

[0183] The server activates the generative artificial intelligence and begins interacting with the user. The input is a visitor notification, and the output is the initial prompt message sent to the user. Specifically, the generative AI generates the prompt message "Welcome. What products are you looking for?" and displays it on the smart device.

[0184] Step 3:

[0185] When a user responds via a smart device with "I want a new smartphone," the server sends that input to a generative artificial intelligence (AI). The input is the user's statement, and the output is the AI's response to it. The AI ​​accesses an inventory database, checks the stock status of the latest smartphone model, and then generates a response such as, "We have the latest model X. We have 5 in stock. Would you like to purchase it?"

[0186] Step 4:

[0187] When the user responds with "Yes, I will purchase" via their smart device, the server initiates the payment process. The input is the user's indication of their purchase intent, and the output is the payment screen displayed on the payment terminal. The server generates the payment screen using a QR code or electronic payment service and displays it to the user.

[0188] Step 5:

[0189] When a user completes the payment process using a QR code or electronic payment service, a payment completion notification is sent to the server. The input is the payment completion notification, and the output is an instruction to pick up the goods. The server then instructs the delivery robot to pick up the goods.

[0190] Step 6:

[0191] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The input is a pick-up instruction from the server, and the output is the delivery of the items to the user or handover station.

[0192] Step 7:

[0193] The server updates the store's inventory information once it confirms that the items have been picked up. The input is a notification that the items have been picked up, and the output is an update to the inventory database. The server updates the inventory information in real time and automatically generates replenishment orders as needed.

[0194] Through these processing steps, the operation of unmanned stores becomes more efficient, the user shopping experience is improved, and costs are reduced.

[0195] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0196] This invention relates to a system for operating an unmanned shop that recognizes user emotions and optimizes customer service, inventory management, and product handover based on that information. The system detects user arrivals, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, handover, and inventory management. The specific operating procedures and program processing of this system are described below.

[0197] System Overview

[0198] This system includes the following elements:

[0199] 1. Sensors that detect user visits

[0200] 2. Generative artificial intelligence programs that interact with users

[0201] 3. Database for managing product inventory information

[0202] 4. Terminal device for processing payments

[0203] 5. Robots that handle product delivery.

[0204] 6. An artificial intelligence program for managing store inventory.

[0205] 7. Emotion engine that recognizes user emotions

[0206] User visit detection

[0207] When a user arrives at the unmanned shop, a sensor installed at the shop entrance detects the user's movement and notifies the server. The sensor's detection activates a generative artificial intelligence, initiating an initial interaction with the user.

[0208] AI-powered customer service and emotion engines

[0209] The generative artificial intelligence interacts with the user. Simultaneously, the emotion engine analyzes the user's facial expressions and voice, and feeds that data back to the generative AI. For example, if the user appears tired, the generative AI will respond by using a gentler tone.

[0210] Product search and selection

[0211] Generative artificial intelligence analyzes user statements and suggests the most suitable products. For example, if a user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?" Furthermore, an emotion engine recognizes the user's emotions and adjusts the suggestions accordingly. For instance, if the user appears confused, the generative AI will add additional explanations.

[0212] For the product selected by the user, the server accesses the inventory database to check the stock status. The server then sends this inventory information to a generative artificial intelligence (AI), which informs the user.

[0213] Payment processing

[0214] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code or credit card on the terminal. After the server confirms the payment is complete, the item handover process begins.

[0215] Product handover

[0216] Once payment is complete, the server instructs the delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user receives the items from the delivery robot.

[0217] Store inventory management

[0218] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[0219] Specific example

[0220] For example, if a user visits the unmanned shop "TECH STORE" and wants to purchase a new smartphone, the process would be as follows:

[0221] 1. When a user arrives at the store, a terminal (sensor) detects the user and notifies the server.

[0222] 2. The server activates the generative artificial intelligence, which then speaks to the user, saying, "Welcome, what kind of product are you looking for?"

[0223] 3. The user responds, "I want a new smartphone."

[0224] 4. The generative AI checks inventory via the server, and if the emotion engine detects positive emotions from the user's facial expressions, it suggests, "We recommend the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0225] 5. When the user responds with "Yes, I will purchase," the server displays the payment screen on the payment terminal, and the user makes the payment using a QR code.

[0226] 6. The server confirms that payment is complete and sends instructions to the robot to pick up the items.

[0227] 7. The robot picks up the products and prepares them for handing over to the user.

[0228] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[0229] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[0230] The following describes the processing flow.

[0231] Step 1:

[0232] The user reaches the entrance of the unmanned shop. A terminal (sensor) detects the user's movement and sends that information to the server.

[0233] Step 2:

[0234] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI displays a greeting message to the user saying, "Welcome, what product are you looking for?"

[0235] Step 3:

[0236] A generative AI analyzes the user's statements and inputs information about the products the user desires. Simultaneously, an emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This emotion data is then fed back to the generative AI.

[0237] Step 4:

[0238] The user says, "I want a new smartphone." The generative artificial intelligence analyzes this information, and if the emotion engine determines that the user's emotion is positive, the generative AI suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0239] Step 5:

[0240] The user selects a product based on the suggestion. For example, they might respond, "Yes, I'll buy it." The server receives this information and checks the inventory database to confirm the product's availability.

[0241] Step 6:

[0242] The server sends inventory information to the generative artificial intelligence. The generative AI notifies the user that the item is in stock. It displays the message, "Inventory confirmed. Payment will now be processed."

[0243] Step 7:

[0244] The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the payment screen and prompts the user to make a payment.

[0245] Step 8:

[0246] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[0247] Step 9:

[0248] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the specified goods from the warehouse and either hands them directly to the user or places them at a designated delivery station.

[0249] Step 10:

[0250] A robot receives the goods, places them at a delivery station, and notifies the user that the goods are ready. The user then picks up the goods from the station.

[0251] Step 11:

[0252] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[0253] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, the use of an emotion engine enables flexible and appropriate customer service that responds to the user's emotions.

[0254] (Example 2)

[0255] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0256] Current unmanned shop systems lack effective interaction with users and emotional recognition, making it difficult to provide optimal customer service tailored to individual needs. Furthermore, the accuracy of automation in inventory management and product handover is low, hindering improvements in operational efficiency.

[0257] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0258] In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for analyzing the user's emotions, means for adjusting the content of the dialogue with the user according to the analyzed emotions, means for checking product inventory according to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for managing the store's inventory. This enables flexible customer service based on the user's emotions, as well as highly accurate inventory management and product delivery.

[0259] "Means of detecting user arrival" refers to sensors or devices used to determine when a user has reached a store.

[0260] "Generative artificial intelligence" refers to AI systems that perform natural language processing and dialogue generation.

[0261] "Means of interacting with the user" refers to the function by which generative artificial intelligence communicates with the user through voice or text.

[0262] "Means of analyzing user emotions" refers to emotion engines and algorithms that analyze a user's facial expressions and voice data to determine their emotional state.

[0263] "Means of adjusting the content of conversations with the user according to analyzed emotions" refers to a function in which generative artificial intelligence changes the tone and content of conversations based on the user's emotional data obtained from the emotion engine.

[0264] "Methods for checking product inventory" refers to the function of accessing a database system that manages product inventory information to check how much of a particular product is in stock.

[0265] "Means of processing payment" refers to terminal devices used by users to pay for selected products using credit cards, QR codes, or other methods.

[0266] "Methods of delivering goods by robot" refers to sales robots that pick up goods from a warehouse after payment is completed and hand them over to the user, or place them at a delivery station.

[0267] "Methods for managing store inventory" refers to an AI system that monitors inventory status in real time and automatically issues inventory replenishment orders as needed.

[0268] This invention is a system that streamlines the operation of unmanned shops and provides flexible customer service based on user emotions. The system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, delivery, and inventory management. Specific embodiments are described below.

[0269] System Configuration

[0270] The main hardware and software components of this system are as follows:

[0271] 1. Sensors to detect user arrival: When a user arrives at the unmanned shop, a motion sensor installed at the shop entrance detects the user's movement and notifies the server.

[0272] 2. Generative AI Program: The server launches a generative AI such as OpenAI GPT-3 (registered trademark) and engages in an initial interaction with the user.

[0273] 3. Emotion Analysis Engine: Using an emotion engine such as Affectiva, the system analyzes the user's emotions from their facial expressions and voice, and feeds that data back into the generative artificial intelligence.

[0274] 4. Database for managing product inventory information: Product inventory information is managed using a database system such as MySQL (registered trademark).

[0275] 5. Payment terminals: Users make payments using QR codes or credit cards at payment terminals such as Clover Station.

[0276] 6. Robots for product delivery: Robots such as Fetch Robotics pick up products and hand them directly to the user.

[0277] 7. Artificial intelligence program for store inventory management: AI using TENSORFLOW® and other technologies monitors inventory levels and automatically issues replenishment instructions.

[0278] Processing explanation in natural language

[0279] When the sensor detects a user, a notification is sent to the server, and the generative artificial intelligence is activated. The generative AI asks the user, "Welcome, what product are you looking for?" Through this initial dialogue, the user's requests are collected.

[0280] When a user says, "I want a new smartphone," their statement is collected via the microphone and converted to text by Google® Cloud Speech-to-Text. Generative artificial intelligence uses this transcribed information to make appropriate product suggestions. For example, it might suggest, "How about the latest smartphone model X?" Simultaneously, an emotion engine analyzes the user's facial expressions and voice and provides feedback to the generative AI. If the user appears confused, the generative AI provides additional explanations.

[0281] When a user selects a product, the server accesses the inventory database to check its availability. For example, an SQL query such as "SELECT stock FROM products WHERE item='Smartphone Model X'" is used. If the item is in stock, the generative artificial intelligence informs the user, "There are 5 in stock. Would you like to purchase it?"

[0282] When the user decides to purchase, the server displays a payment screen on the payment terminal. The user makes the payment using a QR code or a credit card. When a payment completion notification is sent from the payment terminal to the server, the server sends a product pickup instruction to the robot. The robot picks up the product from the warehouse and either directly delivers it to the user or places it at a designated delivery station. When the user receives the product, the inventory database is updated, and the store management AI monitors the inventory status and automatically issues a replenishment instruction.

[0283] Specific Example

[0284] For example, when a user visits the unmanned store "TECH STORE" and wants to purchase a smartphone, the following process occurs.

[0285] 1. When the user arrives at the store, the sensor detects the user and notifies the server.

[0286] 2. The server activates the generative AI, and the generative AI greets the user with "Welcome. What kind of product are you looking for?" ​​​​​​​​​​​​​​​​​​7. The robot picks up the products and prepares them for handing over to the user.

[0292] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[0293] Example of a prompt

[0294] A user has arrived at the TECH STORE. A sensor detects the user, and customer service using generative artificial intelligence begins.

[0295] 1. "Welcome, what kind of product are you looking for?"

[0296] 2. The process for optimal product recommendations and inventory management when a user answers "new smartphone".

[0297] 3. Instructions for payment processing and product delivery.

[0298] 4. Instructions for automatic replenishment when store inventory levels drop.

[0299] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[0300] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0301] Step 1:

[0302] A terminal (sensor) detects a user's arrival. When a user passes through the store entrance, a motion sensor detects their movement and sends a signal to the server. At this point, the input is the sensor signal, and the output is a notification to the server.

[0303] Step 2:

[0304] The server activates a generative artificial intelligence (e.g., OpenAI GPT-3). Upon receiving a notification from the sensor, the server activates the generative artificial intelligence and prepares for the initial conversation. The input is the notification from the sensor, and the output is the activation of the generative artificial intelligence.

[0305] Step 3:

[0306] The generative artificial intelligence starts a conversation with the user. The generative artificial intelligence asks the user, either verbally or in text, "Welcome. What kind of product are you looking for?" This conversation is output through the terminal's audio device or display. The input is the generative artificial intelligence's conversation prompt, and the output is the question to the user.

[0307] Step 4:

[0308] The user states a product request to the generative artificial intelligence. For example, the user answers, "A new smartphone." This statement is collected through the terminal's microphone and sent to the server as voice data. The input is the user's voice data, and the output is the transmission of the voice data to the server.

[0309] Step 5:

[0310] The server converts the voice data into text and passes it to the generative artificial intelligence. Google Cloud Speech-to-Text is used to convert the voice into text. The input is the user's voice data, and the output is the transmission of the text data to the generative artificial intelligence.

[0311] Step 6:

[0312] The generative artificial intelligence analyzes the user's request and proposes an appropriate product. For example, the generative artificial intelligence makes a proposal such as, "How about the latest smartphone model X?" The generative artificial intelligence generates this text and displays it to the user through the terminal's display or audio device. The input is the user's text data, and the output is the product proposal.

[0313] Step 7:

[0314] The emotion engine analyzes the user's facial expressions and provides feedback to the generative artificial intelligence. The user's facial expressions are captured by a camera and analyzed in real time. Emotional data is transmitted to the generative artificial intelligence via a server. The input is the user's facial expression data, and the output is emotional feedback to the generative artificial intelligence.

[0315] Step 8:

[0316] The server checks the product's inventory status. After the generative artificial intelligence understands the user's request, the server accesses the database to check the product's inventory. For example, it executes a query such as "SELECT stock FROM products WHERE item='Smartphone Model X'". The input is the request data for the generative artificial intelligence, and the output is the retrieved inventory data.

[0317] Step 9:

[0318] The server sends inventory information to a generative artificial intelligence (AI) system, which then notifies the user. Based on the inventory information, the AI ​​informs the user that "There are 5 items in stock." The input is inventory data, and the output is a notification to the user.

[0319] Step 10:

[0320] The user indicates their intention to purchase. For example, they might say, "Yes, I will purchase it." This statement is collected through the device's microphone and sent to the server. The input is the user's intention to purchase, and the output is the transmission of audio data to the server.

[0321] Step 11:

[0322] The server displays the payment screen on the payment terminal. The terminal displays a screen for the user to make a payment. Input is the user's purchase intention data, and output is instructions for the payment terminal.

[0323] Step 12:

[0324] The user makes a payment. Payment is made using a QR code or credit card, and a payment completion notification is sent to the server. The input is the user's payment information, and the output is the payment completion notification.

[0325] Step 13:

[0326] The server sends a pickup instruction to the delivery robot. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at the delivery station. The input is a payment completion notification, and the output is a pickup instruction to the robot.

[0327] Step 14:

[0328] A robot picks up the product and hands it to the user. When the user receives the product, the robot waits at a designated location and hands it directly to the user, or places the product at a station. The input is a pickup instruction from the server, and the output is the handover of the product to the user.

[0329] Step 15:

[0330] The server updates the inventory database, and the store management AI monitors the inventory status. After the product is handed over to the user, the inventory database is updated, and the AI ​​monitoring the inventory status automatically issues replenishment instructions as needed. Input is product handover information, and output is inventory data updates and replenishment instructions.

[0331] (Application Example 2)

[0332] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0333] Conventional unmanned store systems were unable to recognize user emotions and optimize customer service and product recommendations accordingly. This led to problems such as decreased customer satisfaction. Furthermore, they did not achieve sufficient efficiency in inventory management and product handover.

[0334] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for recognizing the user's emotions, means for suggesting appropriate products according to the user's requests, means for checking product inventory, means for processing payment for the products selected by the user, means for having a robot deliver the products after payment is completed, and means for managing the store's inventory. This enables flexible customer service that responds to the user's emotions, efficient inventory management, and product delivery.

[0335] "Means for detecting user visits" refer to devices such as sensors and cameras that detect when a user approaches or reaches the system.

[0336] "Means for engaging in dialogue with a user using generative artificial intelligence" refers to a device that uses an artificial intelligence program employing natural language processing technology to engage in voice or text-based dialogue with a user.

[0337] "Means of recognizing user emotions" refer to artificial intelligence programs or sensors that analyze a user's facial expressions and voice to identify their emotions.

[0338] "A means of suggesting appropriate products in response to user requests" refers to a generative artificial intelligence program that presents the most suitable products based on the user's statements and actions.

[0339] "Methods for checking product inventory" refer to database systems that record and track the number and condition of products in stores and warehouses.

[0340] "Means of processing payment for products selected by the user" refers to terminal devices or payment systems that process payment for products selected by the user.

[0341] "A means of delivering goods by robot after payment is completed" refers to a robot that picks up the goods after payment is completed and hands them over to the user.

[0342] "A means of managing store inventory" refers to an artificial intelligence program that monitors the inventory status of products in a store in real time and issues replenishment orders as needed.

[0343] This invention is a system that recognizes the emotions of users in unmanned stores and optimizes customer service, inventory management, and product handover based on that information. Detailed embodiments for carrying out this invention are described below.

[0344] System Overview

[0345] This system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search, selection, payment, delivery, and inventory management. The system includes the following elements:

[0346] 1. Means of detecting user visits: Sensors and cameras are used to detect user movements and notify the server.

[0347] 2. Means of interacting with users using generative artificial intelligence: Interaction with users is conducted using an artificial intelligence program that employs natural language processing.

[0348] 3. Means of recognizing user emotions: The system analyzes facial expressions and voice using cameras and microphones, and recognizes user emotions using an emotion engine.

[0349] 4. Means for suggesting appropriate products in response to user requests: A generative artificial intelligence program that suggests the optimal product based on the user's statements and emotions.

[0350] 5. Method for checking product inventory: Access the inventory database in the store or warehouse to check the stock status.

[0351] 6. Means for processing payment for the items selected by the user: Payment terminals (such as QR code readers or credit card readers) for the user to pay for the items selected by the user.

[0352] 7. A means of handing over goods by robot after payment is completed: A robot to pick up the goods after payment is completed and hand them over to the user.

[0353] 8. Means of managing store inventory: An artificial intelligence program that monitors the inventory status of products in the store in real time and issues replenishment instructions as needed.

[0354] Operation Flow Description

[0355] 1. The server detects a user's arrival based on data from sensors and activates a generative artificial intelligence program.

[0356] 2. The generative artificial intelligence program initiates an initial conversation with the user through natural language processing to confirm the product the user is looking for.

[0357] 3. The camera and microphone acquire the user's facial expression and voice data, and the emotion engine analyzes this data to recognize the user's emotions.

[0358] 4. The generative artificial intelligence program suggests appropriate products in a tone that matches the user's emotions and checks inventory status from the database.

[0359] 5. Once the user selects an item, the server displays the payment screen on the payment terminal, and the user makes the payment.

[0360] 6. Once payment is complete, the server sends instructions to the product delivery robot, which then delivers the picked-up items to the user.

[0361] 7. Finally, the server updates the inventory database and manages the store's inventory.

[0362] Examples of specific cases and prompt statements

[0363] Specific example

[0364] A user enters an unmanned convenience store wearing smart glasses. The smart glasses use facial recognition and emotion analysis to analyze the user's facial expressions, and generative artificial intelligence asks, "Welcome, what are you looking for today?" When the user replies, "I'm looking for a new smartphone," the system checks the inventory and suggests, "We have the latest smartphone in stock. Would you like to purchase it?" If the user approves, the total price is displayed, and after the user pays, a robot hands over the product.

[0365] Example of a prompt

[0366] Input: A user, looking tired, enters an unmanned convenience store and is looking for a new smartphone.

[0367] Output: We will suggest the latest smartphones, and after payment, a robot will hand over the product.

[0368] The above describes specific embodiments for implementing the present invention. This system enables flexible customer service tailored to the user's emotions, efficient inventory management, and product delivery.

[0369] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0370] Step 1:

[0371] When a user enters the store, it is detected by a sensor.

[0372] Input: User reaches store

[0373] Output: The sensor detects a user's arrival and notifies the server.

[0374] Data processing / calculation: Sensors detect user movements and send that information to the server.

[0375] Operation: A sensor installed on the device detects movement and immediately sends that information to the server.

[0376] Step 2:

[0377] Launching a Generative Artificial Intelligence Program

[0378] Input: Visitor information from sensors

[0379] Output: Launch of a generative artificial intelligence program

[0380] Data processing / calculation: The server receives signals from sensors and launches a generative artificial intelligence program.

[0381] Operation: The server detects the user's arrival and launches a generative artificial intelligence program.

[0382] Step 3:

[0383] Starting the initial interaction with the user

[0384] Input: Activation signal for a generative artificial intelligence program

[0385] Output: Voice message "Welcome, what product are you looking for?"

[0386] Data Processing / Computation: Generative artificial intelligence performs natural language generation and creates initial dialogue messages.

[0387] Operation: The server plays a voice message to the user using a generative artificial intelligence program.

[0388] Step 4:

[0389] The system captures the user's facial expressions and voice, and analyzes them using an emotion engine.

[0390] Input: Camera video, microphone audio

[0391] Output: User sentiment data

[0392] Data processing / calculation: The emotion engine analyzes the user's emotions using camera video and audio data.

[0393] How it works: The device's built-in camera and microphone capture the user's face and voice, and an emotion engine analyzes the data.

[0394] Step 5:

[0395] Product proposals tailored to user requirements

[0396] Input: User statements and sentiment data

[0397] Output: Suggestion messages such as "How about the latest smartphone model X?"

[0398] Data Processing / Calculation: Generative artificial intelligence programs suggest optimal products based on user statements and emotions.

[0399] Operation: The server generates suggestions using a generative artificial intelligence program based on the user's requests and emotional data, and plays them back as voice messages.

[0400] Step 6:

[0401] Checking the inventory database

[0402] Input: Proposed product information

[0403] Output: Inventory status

[0404] Data processing / calculation: The server accesses the inventory database and checks the inventory status of the suggested products.

[0405] Operation: The server queries the database to retrieve current inventory information.

[0406] Step 7:

[0407] Display of payment screen

[0408] Input: User's expression of intent to purchase the product.

[0409] Output: Display of payment screen

[0410] Data processing / calculation: The server sends payment information to the payment terminal.

[0411] Operation: When the user indicates their intention to purchase, the payment screen will appear on the device.

[0412] Step 8:

[0413] Payment processing

[0414] Input: User's payment information (credit card information, QR code, etc.)

[0415] Output: Payment completion confirmation message

[0416] Data processing / calculation: The payment system verifies the user's payment information and completes the payment.

[0417] Operation: The user makes a payment at the payment terminal, and the server confirms that the payment processing is complete.

[0418] Step 9:

[0419] Product pickup and handover

[0420] Input: Payment completion information

[0421] Output: Product handover complete

[0422] Data processing / calculation: The server instructs the robot to pick up the products and hand them over to the user.

[0423] Operation: Once payment is complete, the robot picks up the goods from the warehouse and hands them over to the user.

[0424] Step 10:

[0425] Updating and managing inventory data

[0426] Input: Information about the product provided to the user.

[0427] Output: Updated inventory data

[0428] Data processing / calculation: The server updates the inventory database, and the inventory management AI monitors the inventory status in real time.

[0429] Operation: After the goods are delivered, the inventory database is updated, and replenishment orders are automatically issued as needed.

[0430] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0431] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0432] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0433] [Second Embodiment]

[0434] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0435] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0436] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0437] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0438] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0440] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0441] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0442] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0443] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0444] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0445] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0446] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The following describes the specific operating procedures and program processing of this system in natural language.

[0447] System Overview

[0448] This system detects user visits, uses generative artificial intelligence to engage in dialogue, and automates product search and selection, payment, delivery, and inventory management. The detailed operation of each element is described below.

[0449] User visit detection

[0450] When a user arrives at the unmanned shop, a sensor installed at the entrance detects the user. The information from this sensor acts as a trigger to activate the artificial intelligence customer service system, which will be described next.

[0451] AI-powered customer service

[0452] When the sensor detects a user, the server activates generative artificial intelligence and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, if the user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?"

[0453] Product search and selection

[0454] When a user selects a smartphone based on the suggestions of the generative artificial intelligence, the server accesses the inventory database to check the stock status of that product. If the product is in stock, the server notifies the generative artificial intelligence, which then informs the user.

[0455] Payment processing

[0456] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code or credit card on the terminal. After payment is complete, the server receives the information and begins the next step in the product delivery process.

[0457] Product handover

[0458] Once payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[0459] Store inventory management

[0460] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[0461] Specific example

[0462] For example, suppose a user visits an unmanned shop called "TECH STORE" and wants to buy a new smartphone. In this case, the process would be as follows:

[0463] 1. When a user reaches the entrance, the terminal (sensor) detects the user.

[0464] 2. The server activates the generative artificial intelligence and speaks to the user.

[0465] 3. The generative artificial intelligence asks, "Welcome. What kind of product are you looking for?"

[0466] 4. The user responds, "I want a new smartphone."

[0467] 5. The generative AI checks the inventory via the server and notifies the user, "We have the latest Model X. There are 5 in stock. Would you like to purchase it?"

[0468] 6. The user responds, "Yes, I will purchase it."

[0469] 7. The server displays the payment screen on the payment terminal, and the user completes the payment using the QR code.

[0470] 8. The server confirms that the payment is complete and sends instructions to the robot to pick up the goods.

[0471] 9. The robot hands the product to the user, and the user receives the product.

[0472] 10. The terminal (store management AI) recognizes the decrease in inventory and sends a replenishment instruction to the server.

[0473] In this way, this system enables the operation of automated, unmanned shops, providing reduced labor costs and improved operational efficiency.

[0474] The following describes the processing flow.

[0475] Step 1:

[0476] The user reaches the entrance of the unmanned shop. A terminal (a sensor installed at the entrance) detects the user's movement and sends that information to the server.

[0477] Step 2:

[0478] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI automatically displays a greeting message to the user.

[0479] Step 3:

[0480] A user who sees a message displayed by a generative artificial intelligence asks questions about the product they want.

[0481] Generative artificial intelligence analyzes user statements and suggests the most suitable products. If the user says, "I want a new smartphone," it will suggest, "How about the latest smartphone model X?"

[0482] Step 4:

[0483] Upon receiving a user's request, the server accesses the inventory database to check the stock status of the requested product. The server then sends the inventory information to a generative artificial intelligence (AI), which notifies the user. For example, it might display a message such as, "There are 5 items left in stock."

[0484] Step 5:

[0485] The user selects an item and indicates their intention to purchase it. The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the user's payment information.

[0486] Step 6:

[0487] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[0488] Step 7:

[0489] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the goods from the warehouse and prepares them for delivery to the user.

[0490] Step 8:

[0491] A robot delivers the product to a designated pick-up station and notifies the user that the product is ready. The user then picks up the product from the station.

[0492] Step 9:

[0493] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[0494] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, it significantly reduces labor costs and improves operational efficiency.

[0495] (Example 1)

[0496] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0497] Conventional unmanned stores often rely on manual or semi-automatic processes for tasks such as detecting user arrival, selecting products, processing payments, handing over products, and managing inventory, resulting in reduced overall operational efficiency. High operating costs and frequent stockouts were also problematic. Furthermore, the mechanical nature of user interaction made it difficult to provide a smooth purchasing experience. This invention aims to solve these problems, streamline the operation of unmanned stores, and reduce costs.

[0498] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0499] In this invention, the server includes means for detecting a user's arrival, means for transmitting the detected information to the server, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for monitoring inventory status for store operations and automatically generating replenishment instructions. This automates the entire process from the user's arrival to the completion of purchase, significantly improving operational efficiency and enabling cost reduction.

[0500] "Means of detecting user visits" refers to a system that uses devices such as sensors and cameras to detect when a user visits a store and collects that information.

[0501] "Means for transmitting detected information to a server" refers to a system that includes communication devices and protocols for transmitting user visitor information collected by sensors, cameras, etc., to a server.

[0502] "Means of engaging in dialogue with users using generative artificial intelligence" refers to a system that utilizes the latest generative artificial intelligence technology to enable natural dialogue with users through voice or text.

[0503] "A means of checking product inventory in response to user requests" refers to a system that queries a database for inventory information of products selected or requested by the user and retrieves the results.

[0504] "A means of processing payment for products selected by the user" refers to a system that processes the payment for products that the user has decided to purchase using payment methods such as QR codes or credit cards.

[0505] "A method of handing over goods by robot after payment is completed" refers to a system in which, after payment is completed, a robot picks up the designated goods and either hands them directly to the user or places them at a handover station.

[0506] "A means of monitoring inventory status and automatically generating replenishment instructions for store operations" refers to a system that constantly monitors inventory within a store, and when inventory levels become low, the server automatically generates replenishment instructions, thereby streamlining store operations.

[0507] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The system detects user arrivals, engages in dialogue using generative artificial intelligence, and automates product search and selection, payment, delivery, and inventory management. The following describes the specific operating procedures of this system.

[0508] The system includes the following main hardware and software components:

[0509] 1. Sensors: Installed at the entrance to detect user arrivals. For example, infrared sensors or cameras are used.

[0510] 2. Server: Receives information from sensors and activates generative artificial intelligence. It also accesses the inventory database, processes payments, and sends instructions to robots.

[0511] 3. Generative Artificial Intelligence: Enables voice or text-based interaction with the user. This artificial intelligence performs tasks such as answering questions and suggesting products.

[0512] 4. Inventory Database: A database that manages the inventory status of products in real time.

[0513] 5. Payment terminal: A terminal that includes a QR code reader and credit card reader to process payments for the items selected by the user.

[0514] 6. Robot: After payment is completed, the robot picks up the goods from the warehouse and either hands them to the user or places them at a designated handover station.

[0515] 7. Store Management AI: An artificial intelligence program that constantly monitors inventory levels and automatically generates replenishment instructions.

[0516] As a concrete example, the process for a user visiting an unmanned shop and wanting to purchase a new smartphone would be as follows:

[0517] 1. When a user reaches the store entrance, a sensor installed on the terminal detects the user and sends that information to the server.

[0518] 2. The server receives the information, activates the generative artificial intelligence, and speaks to the user, saying, "Welcome. What kind of product are you looking for?"

[0519] 3. When a user says, "I want a new smartphone," the generative artificial intelligence suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0520] 4. Once the user decides to make a purchase, the server displays the payment screen on the payment terminal.

[0521] 5. Once the user scans the QR code and completes the payment, the server sends instructions to the robot to pick up the items.

[0522] 6. The robot retrieves a smartphone from the warehouse and hands it to the user.

[0523] 7. The store management AI checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment order.

[0524] This system automates the entire process, significantly improving operational efficiency and reducing costs. Furthermore, users can enjoy a smooth and intuitive purchasing experience.

[0525] The above describes embodiments for carrying out the present invention.

[0526] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0527] Step 1:

[0528] When a user reaches the store entrance, a sensor installed on the terminal detects the user. The sensor captures the user's movements and transmits the data to a server. The sensor's input is the detection of the user's movement or presence, and its output is a signal containing the detection information. This signal notifies the server that the user has entered the store. Infrared sensors and cameras are used specifically for this purpose.

[0529] Step 2:

[0530] When the server receives information from the sensor that a user has arrived, it activates the generative artificial intelligence (AI). The server's input is the detection information from the sensor, and its output is the command to activate the generative AI. The generative AI then begins a voice or text-based conversation with the user. For example, the server might send the command "A user has arrived. Please begin the conversation" to the generative AI.

[0531] Step 3:

[0532] The generative artificial intelligence (AI) initiates a conversation with the user and listens to their requests. The input to the AI ​​is the startup command from the server and the user's statements, and the output is the response. If the AI ​​greets the user with "Welcome. What product are you looking for?" and the user replies "I want a new smartphone," the AI ​​proceeds to the next step based on that request.

[0533] Step 4:

[0534] The server receives a user request, accesses the inventory database, and checks the product's stock status. The server's input is the user's request information from the generative artificial intelligence, and its output is the inventory check result. Specifically, the server sends a query to the database saying, "Please check the inventory of Model X smartphones," and the database responds, "There are 5 in stock."

[0535] Step 5:

[0536] The generative artificial intelligence provides product information to the user based on inventory check results from the server. The input to the generative AI is the inventory check results, and the output is a notification to the user. Specifically, it notifies the user, "We have the latest smartphone model X. There are 5 in stock. Would you like to purchase it?"

[0537] Step 6:

[0538] Once the user selects an item and decides to purchase it, the server displays a payment screen on the payment terminal. The server's input is the user's purchase decision and selected item information, and its output is instructions for the payment terminal. Specifically, the server sends the message "Process payment for Model X smartphone" to the payment terminal.

[0539] Step 7:

[0540] Users pay using a QR code or credit card at a payment terminal. The terminal receives payment instructions from the server and the user's payment information, and the output is a confirmation that the payment is complete. Specifically, the user scans a QR code with their smartphone, and the payment amount is displayed on the terminal.

[0541] Step 8:

[0542] Once the server confirms payment is complete, it sends an instruction to the robot to pick up the item. The server's input is payment completion information, and its output is an instruction to the robot. Specifically, it commands the robot to "Pick up the Model X smartphone and deliver it to the user."

[0543] Step 9:

[0544] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The robot receives instructions from a server as input, and its output is the completion of product delivery. For example, the robot might take a smartphone from a shelf and hand it to the user.

[0545] Step 10:

[0546] The store management AI on the terminal checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment instruction. The input to the store management AI is inventory data, and the output is a replenishment instruction to the server. For example, if the inventory falls to two or fewer units, it automatically notifies the server with "Inventory replenishment is needed."

[0547] (Application Example 1)

[0548] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0549] In operating unmanned stores, there is a need for specific methods and systems to achieve efficiency and cost reduction by fully automating everything from user arrival to product purchase, handover, and inventory management. In this process, rapid and accurate product recommendations and payment processing are required to enhance the user experience.

[0550] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0551] In this invention, the server includes means for detecting a user's arrival, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, means for managing the store's inventory, means for detecting the user using the user's smart device and beacon device, means for generative artificial intelligence to suggest products based on the user's statements, and means for processing payment via QR code or electronic payment service. This enables improved user experience, increased operational efficiency, and cost reduction in unmanned stores.

[0552] A "user" is a customer who visits an unmanned store and intends to purchase goods.

[0553] "Means of detecting visits" refer to devices such as sensors and beacon devices that detect a user's presence when they access a store.

[0554] "Generative artificial intelligence" refers to artificial intelligence programs that generate appropriate responses and suggestions based on user statements and requests.

[0555] "Means for checking product inventory" refers to a database system that manages the inventory status of products and provides that information to users upon request.

[0556] "Means of processing payment" refers to terminal devices and software used to process payments for products selected by the user through QR codes or electronic payment services.

[0557] "A means of delivering goods by robot" refers to a robotic system that picks up goods from a warehouse after payment is completed and delivers them to the user.

[0558] "A means of managing store inventory" refers to an artificial intelligence program that constantly monitors the inventory of products in a store and automatically instructs replenishment as needed.

[0559] A "smart device" is a portable electronic device that can connect to the internet, such as a smartphone or tablet.

[0560] A "beacon device" is a small device that provides location information via wireless communication and interacts with nearby smart devices.

[0561] A "QR code" is a type of two-dimensional barcode, a pattern used to efficiently transmit information.

[0562] "Electronic payment services" are services that allow payments to be completed electronically in an online or mobile environment (e.g., credit cards, various payment apps).

[0563] A "prompt" is an initial word or phrase that a generative artificial intelligence uses to initiate a conversation with a user.

[0564] System Overview

[0565] The unmanned store operation system accurately detects user arrivals, interacts with users using generative artificial intelligence, and automates product search and selection, payment, handover, and inventory management. The system of the present invention is composed of the following hardware and software.

[0566] User visit detection

[0567] First, when a user arrives at the unmanned store, their smart device is detected by a beacon device installed at the entrance. This allows the system to recognize the user's arrival as a trigger.

[0568] AI-powered customer service

[0569] When a user's arrival is detected by a beacon, the server activates generative artificial intelligence (for example, OpenAI's GPT-4 model) and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, the following prompts are used.

[0570] Prompt message: "Welcome. What kind of product are you looking for?"

[0571] Product search and selection

[0572] When a user selects a product based on the suggestions of a generative artificial intelligence, the server accesses the store's inventory database to check the product's availability. If the product is in stock, the generative AI notifies the user and confirms their intention to purchase.

[0573] Payment processing

[0574] Once a user selects an item and indicates their intention to purchase, the server displays a payment screen on the payment terminal. The user pays using a QR code or an electronic payment service (e.g., Stripe API or Square API) on the terminal. After the payment is complete, the information is sent to the server.

[0575] Product handover

[0576] After payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[0577] Store inventory management

[0578] The server constantly monitors the store's inventory levels and automatically detects when replenishment is needed if inventory levels drop. This prevents stockouts and improves the efficiency of store operations.

[0579] Specific example

[0580] Suppose a user visits an unmanned store and wants to purchase the latest smartphone. In this case, the process would proceed as follows:

[0581] 1. When a user arrives at the store entrance, a beacon device detects the user's smart device, and the arrival is recognized as a trigger.

[0582] 2. The server starts the generative artificial intelligence and sends the following prompt message to the user.

[0583] "Welcome. What kind of product are you looking for?"

[0584] 3. When a user responds with "I want a new smartphone," the generative artificial intelligence retrieves appropriate product information from the store's inventory database and suggests it to the user.

[0585] "We have the latest Model X in stock. We have 5 units available. Would you like to purchase one?"

[0586] 4. When the user responds with "Yes, I will purchase," the payment screen is displayed.

[0587] 5. The user completes the payment using a QR code or electronic payment service, and that information is sent to the server.

[0588] 6. The server instructs the delivery robot to pick up the product, and the robot delivers the product to the user.

[0589] 7. The server updates the store's inventory and automatically issues instructions if replenishment is needed.

[0590] The system described above streamlines the operation of unmanned stores, improves the user shopping experience, and reduces costs.

[0591] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0592] Step 1:

[0593] When a user arrives at an unmanned store, a beacon device installed at the entrance detects the user's smart device. The input is the ID signal of the smart device connected to the beacon device, and the output is a visitor notification sent to the server. Upon receiving this notification, the server recognizes the user's arrival and initiates the next processing step.

[0594] Step 2:

[0595] The server activates the generative artificial intelligence and begins interacting with the user. The input is a visitor notification, and the output is the initial prompt message sent to the user. Specifically, the generative AI generates the prompt message "Welcome. What products are you looking for?" and displays it on the smart device.

[0596] Step 3:

[0597] When a user responds via a smart device with "I want a new smartphone," the server sends that input to a generative artificial intelligence (AI). The input is the user's statement, and the output is the AI's response to it. The AI ​​accesses an inventory database, checks the stock status of the latest smartphone model, and then generates a response such as, "We have the latest model X. We have 5 in stock. Would you like to purchase it?"

[0598] Step 4:

[0599] When the user responds with "Yes, I will purchase" via their smart device, the server initiates the payment process. The input is the user's indication of their purchase intent, and the output is the payment screen displayed on the payment terminal. The server generates the payment screen using a QR code or electronic payment service and displays it to the user.

[0600] Step 5:

[0601] When a user completes the payment process using a QR code or electronic payment service, a payment completion notification is sent to the server. The input is the payment completion notification, and the output is an instruction to pick up the goods. The server then instructs the delivery robot to pick up the goods.

[0602] Step 6:

[0603] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The input is a pick-up instruction from the server, and the output is the delivery of the items to the user or handover station.

[0604] Step 7:

[0605] The server updates the store's inventory information once it confirms that the items have been picked up. The input is a notification that the items have been picked up, and the output is an update to the inventory database. The server updates the inventory information in real time and automatically generates replenishment orders as needed.

[0606] Through these processing steps, the operation of unmanned stores becomes more efficient, the user shopping experience is improved, and costs are reduced.

[0607] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0608] This invention relates to a system for operating an unmanned shop that recognizes user emotions and optimizes customer service, inventory management, and product handover based on that information. The system detects user arrivals, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, handover, and inventory management. The specific operating procedures and program processing of this system are described below.

[0609] System Overview

[0610] This system includes the following elements:

[0611] 1. Sensors that detect user visits

[0612] 2. Generative artificial intelligence programs that interact with users

[0613] 3. Database for managing product inventory information

[0614] 4. Terminal device for processing payments

[0615] 5. Robots that handle product delivery.

[0616] 6. An artificial intelligence program for managing store inventory.

[0617] 7. Emotion engine that recognizes user emotions

[0618] User visit detection

[0619] When a user arrives at the unmanned shop, a sensor installed at the shop entrance detects the user's movement and notifies the server. The sensor's detection activates a generative artificial intelligence, initiating an initial interaction with the user.

[0620] AI-powered customer service and emotion engines

[0621] The generative artificial intelligence interacts with the user. Simultaneously, the emotion engine analyzes the user's facial expressions and voice, and feeds that data back to the generative AI. For example, if the user appears tired, the generative AI will respond by using a gentler tone.

[0622] Product search and selection

[0623] Generative artificial intelligence analyzes user statements and suggests the most suitable products. For example, if a user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?" Furthermore, an emotion engine recognizes the user's emotions and adjusts the suggestions accordingly. For instance, if the user appears confused, the generative AI will add additional explanations.

[0624] For the product selected by the user, the server accesses the inventory database to check the stock status. The server then sends this inventory information to a generative artificial intelligence (AI), which informs the user.

[0625] Payment processing

[0626] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code or credit card on the terminal. After the server confirms the payment is complete, the item handover process begins.

[0627] Product handover

[0628] Once payment is complete, the server instructs the delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user receives the items from the delivery robot.

[0629] Store inventory management

[0630] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[0631] Specific example

[0632] For example, if a user visits the unmanned shop "TECH STORE" and wants to purchase a new smartphone, the process would be as follows:

[0633] 1. When a user arrives at the store, a terminal (sensor) detects the user and notifies the server.

[0634] 2. The server activates the generative artificial intelligence, which then speaks to the user, saying, "Welcome, what kind of product are you looking for?"

[0635] 3. The user responds, "I want a new smartphone."

[0636] 4. The generative AI checks inventory via the server, and if the emotion engine detects positive emotions from the user's facial expressions, it suggests, "We recommend the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0637] 5. When the user responds with "Yes, I will purchase," the server displays the payment screen on the payment terminal, and the user makes the payment using a QR code.

[0638] 6. The server confirms that payment is complete and sends instructions to the robot to pick up the items.

[0639] 7. The robot picks up the products and prepares them for handing over to the user.

[0640] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[0641] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[0642] The following describes the processing flow.

[0643] Step 1:

[0644] The user reaches the entrance of the unmanned shop. A terminal (sensor) detects the user's movement and sends that information to the server.

[0645] Step 2:

[0646] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI displays a greeting message to the user saying, "Welcome, what product are you looking for?"

[0647] Step 3:

[0648] A generative AI analyzes the user's statements and inputs information about the products the user desires. Simultaneously, an emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This emotion data is then fed back to the generative AI.

[0649] Step 4:

[0650] The user says, "I want a new smartphone." The generative artificial intelligence analyzes this information, and if the emotion engine determines that the user's emotion is positive, the generative AI suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0651] Step 5:

[0652] The user selects a product based on the suggestion. For example, they might respond, "Yes, I'll buy it." The server receives this information and checks the inventory database to confirm the product's availability.

[0653] Step 6:

[0654] The server sends inventory information to the generative artificial intelligence. The generative AI notifies the user that the item is in stock. It displays the message, "Inventory confirmed. Payment will now be processed."

[0655] Step 7:

[0656] The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the payment screen and prompts the user to make a payment.

[0657] Step 8:

[0658] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[0659] Step 9:

[0660] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the specified goods from the warehouse and either hands them directly to the user or places them at a designated delivery station.

[0661] Step 10:

[0662] A robot receives the goods, places them at a delivery station, and notifies the user that the goods are ready. The user then picks up the goods from the station.

[0663] Step 11:

[0664] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[0665] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, the use of an emotion engine enables flexible and appropriate customer service that responds to the user's emotions.

[0666] (Example 2)

[0667] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0668] Current unmanned shop systems lack effective interaction with users and emotional recognition, making it difficult to provide optimal customer service tailored to individual needs. Furthermore, the accuracy of automation in inventory management and product handover is low, hindering improvements in operational efficiency.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0670] In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for analyzing the user's emotions, means for adjusting the content of the dialogue with the user according to the analyzed emotions, means for checking product inventory according to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for managing the store's inventory. This enables flexible customer service based on the user's emotions, as well as highly accurate inventory management and product delivery.

[0671] "Means of detecting user arrival" refers to sensors or devices used to determine when a user has reached a store.

[0672] "Generative artificial intelligence" refers to AI systems that perform natural language processing and dialogue generation.

[0673] "Means of interacting with the user" refers to the function by which generative artificial intelligence communicates with the user through voice or text.

[0674] "Means of analyzing user emotions" refers to emotion engines and algorithms that analyze a user's facial expressions and voice data to determine their emotional state.

[0675] "Means of adjusting the content of conversations with the user according to analyzed emotions" refers to a function in which generative artificial intelligence changes the tone and content of conversations based on the user's emotional data obtained from the emotion engine.

[0676] "Methods for checking product inventory" refers to the function of accessing a database system that manages product inventory information to check how much of a particular product is in stock.

[0677] "Means of processing payment" refers to terminal devices used by users to pay for selected products using credit cards, QR codes, or other methods.

[0678] "Methods of delivering goods by robot" refers to sales robots that pick up goods from a warehouse after payment is completed and hand them over to the user, or place them at a delivery station.

[0679] "Methods for managing store inventory" refers to an AI system that monitors inventory status in real time and automatically issues inventory replenishment orders as needed.

[0680] This invention is a system that streamlines the operation of unmanned shops and provides flexible customer service based on user emotions. The system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, delivery, and inventory management. Specific embodiments are described below.

[0681] System Configuration

[0682] The main hardware and software components of this system are as follows:

[0683] 1. Sensors to detect user arrival: When a user arrives at the unmanned shop, a motion sensor installed at the shop entrance detects the user's movement and notifies the server.

[0684] 2. Generative AI Program: The server launches a generative AI such as OpenAI GPT-3 and engages in an initial interaction with the user.

[0685] 3. Emotion Analysis Engine: Using an emotion engine such as Affectiva, the system analyzes the user's emotions from their facial expressions and voice, and feeds that data back into the generative artificial intelligence.

[0686] 4. Database for managing product inventory information: Product inventory information is managed using a database system such as MySQL.

[0687] 5. Payment terminals: Users make payments using QR codes or credit cards at payment terminals such as Clover Station.

[0688] 6. Robots for product delivery: Robots such as Fetch Robotics pick up products and hand them directly to the user.

[0689] 7. Artificial intelligence program for store inventory management: An AI using TensorFlow, etc., monitors inventory status and automatically issues replenishment orders.

[0690] Processing explanation in natural language

[0691] When the sensor detects a user, a notification is sent to the server, and the generative artificial intelligence is activated. The generative AI asks the user, "Welcome, what product are you looking for?" Through this initial dialogue, the user's requests are collected.

[0692] When a user says, "I want a new smartphone," their statement is collected via the microphone and converted to text by Google Cloud Speech-to-Text. Generative artificial intelligence uses this transcribed information to make appropriate product suggestions. For example, it might suggest, "How about the latest smartphone model X?" Simultaneously, an emotion engine analyzes the user's facial expressions and voice and provides feedback to the generative AI. If the user appears confused, the generative AI provides additional explanations.

[0693] When a user selects a product, the server accesses the inventory database to check its availability. For example, an SQL query such as "SELECT stock FROM products WHERE item='Smartphone Model X'" is used. If the item is in stock, the generative artificial intelligence informs the user, "There are 5 in stock. Would you like to purchase it?"

[0694] Once a user decides to purchase, the server displays a payment screen on the payment terminal. The user makes the payment using a QR code or credit card. When a payment completion notification is sent from the payment terminal to the server, the server sends an instruction to the robot to pick up the product. The robot picks up the product from the warehouse and either hands it directly to the user or places it at a designated handover station. Once the user receives the product, the inventory database is updated, and the store management AI monitors the inventory status and automatically issues replenishment instructions.

[0695] Specific example

[0696] For example, if a user visits the unmanned shop "TECH STORE" and wants to purchase a smartphone, the process would be as follows:

[0697] 1. When a user arrives at the store, a sensor detects the user and notifies the server.

[0698] 2. The server activates the generative artificial intelligence, which then says, "Welcome, what product are you looking for?"

[0699] 3. The user responds, "I want a new smartphone."

[0700] 4. The generative AI checks inventory via the server, and if the emotion engine detects positive emotions from the user's facial expressions, it suggests, "We recommend the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0701] 5. When the user responds with "Yes, I will purchase," the server displays the payment screen on the payment terminal, and the user makes the payment using a QR code.

[0702] 6. The server confirms that payment is complete and sends instructions to the robot to pick up the items.

[0703] 7. The robot picks up the products and prepares them for handing over to the user.

[0704] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[0705] Example of a prompt

[0706] A user has arrived at the TECH STORE. A sensor detects the user, and customer service using generative artificial intelligence begins.

[0707] 1. "Welcome, what kind of product are you looking for?"

[0708] 2. The process for optimal product recommendations and inventory management when a user answers "new smartphone".

[0709] 3. Instructions for payment processing and product delivery.

[0710] 4. Instructions for automatic replenishment when store inventory levels drop.

[0711] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[0712] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0713] Step 1:

[0714] A terminal (sensor) detects a user's arrival. When a user passes through the store entrance, a motion sensor detects their movement and sends a signal to the server. At this point, the input is the sensor signal, and the output is a notification to the server.

[0715] Step 2:

[0716] The server starts a generative artificial intelligence (e.g., OpenAI GPT-3). Upon receiving a notification from a sensor, the server starts the generative AI and prepares for initial interaction. The input is the notification from the sensor, and the output is the startup of the generative AI.

[0717] Step 3:

[0718] The generative artificial intelligence (AI) initiates a dialogue with the user. The AI ​​asks the user, either verbally or in text, "Welcome, what product are you looking for?" This dialogue is output through the terminal's audio device or display. The input is the AI's dialogue prompt, and the output is the question to the user.

[0719] Step 4:

[0720] The user makes a product request to the generative artificial intelligence. For example, the AI ​​might respond, "A new smartphone." This statement is collected through the device's microphone and sent to the server as audio data. The input is the user's audio data, and the output is the transmission of audio data to the server.

[0721] Step 5:

[0722] The server converts the audio data into text and passes it to the generative artificial intelligence. Google Cloud Speech-to-Text is used to convert the audio to text. The input is the user's audio data, and the output is the transmission of text data to the generative artificial intelligence.

[0723] Step 6:

[0724] Generative artificial intelligence analyzes user requests and suggests appropriate products. For example, it might suggest, "How about the latest smartphone model X?" The generative AI generates this text, which is then displayed to the user via the device's display or audio device. The input is the user's text data, and the output is a product suggestion.

[0725] Step 7:

[0726] The emotion engine analyzes the user's facial expressions and provides feedback to the generative artificial intelligence. The user's facial expressions are captured by a camera and analyzed in real time. Emotional data is transmitted to the generative artificial intelligence via a server. The input is the user's facial expression data, and the output is emotional feedback to the generative artificial intelligence.

[0727] Step 8:

[0728] The server checks the product's inventory status. After the generative artificial intelligence understands the user's request, the server accesses the database to check the product's inventory. For example, it executes a query such as "SELECT stock FROM products WHERE item='Smartphone Model X'". The input is the request data for the generative artificial intelligence, and the output is the retrieved inventory data.

[0729] Step 9:

[0730] The server sends inventory information to a generative artificial intelligence (AI) system, which then notifies the user. Based on the inventory information, the AI ​​informs the user that "There are 5 items in stock." The input is inventory data, and the output is a notification to the user.

[0731] Step 10:

[0732] The user indicates their intention to purchase. For example, they might say, "Yes, I will purchase it." This statement is collected through the device's microphone and sent to the server. The input is the user's intention to purchase, and the output is the transmission of audio data to the server.

[0733] Step 11:

[0734] The server displays the payment screen on the payment terminal. The terminal displays a screen for the user to make a payment. Input is the user's purchase intention data, and output is instructions for the payment terminal.

[0735] Step 12:

[0736] The user makes a payment. Payment is made using a QR code or credit card, and a payment completion notification is sent to the server. The input is the user's payment information, and the output is the payment completion notification.

[0737] Step 13:

[0738] The server sends a pickup instruction to the delivery robot. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at the delivery station. The input is a payment completion notification, and the output is a pickup instruction to the robot.

[0739] Step 14:

[0740] A robot picks up the product and hands it to the user. When the user receives the product, the robot waits at a designated location and hands it directly to the user, or places the product at a station. The input is a pickup instruction from the server, and the output is the handover of the product to the user.

[0741] Step 15:

[0742] The server updates the inventory database, and the store management AI monitors the inventory status. After the product is handed over to the user, the inventory database is updated, and the AI ​​monitoring the inventory status automatically issues replenishment instructions as needed. Input is product handover information, and output is inventory data updates and replenishment instructions.

[0743] (Application Example 2)

[0744] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0745] Conventional unmanned store systems were unable to recognize user emotions and optimize customer service and product recommendations accordingly. This led to problems such as decreased customer satisfaction. Furthermore, they did not achieve sufficient efficiency in inventory management and product handover.

[0746] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for recognizing the user's emotions, means for suggesting appropriate products according to the user's requests, means for checking product inventory, means for processing payment for the products selected by the user, means for having a robot deliver the products after payment is completed, and means for managing the store's inventory. This enables flexible customer service that responds to the user's emotions, efficient inventory management, and product delivery.

[0747] "Means for detecting user visits" refer to devices such as sensors and cameras that detect when a user approaches or reaches the system.

[0748] "Means for engaging in dialogue with a user using generative artificial intelligence" refers to a device that uses an artificial intelligence program employing natural language processing technology to engage in voice or text-based dialogue with a user.

[0749] "Means of recognizing user emotions" refer to artificial intelligence programs or sensors that analyze a user's facial expressions and voice to identify their emotions.

[0750] "A means of suggesting appropriate products in response to user requests" refers to a generative artificial intelligence program that presents the most suitable products based on the user's statements and actions.

[0751] "Methods for checking product inventory" refer to database systems that record and track the number and condition of products in stores and warehouses.

[0752] "Means of processing payment for products selected by the user" refers to terminal devices or payment systems that process payment for products selected by the user.

[0753] "A means of delivering goods by robot after payment is completed" refers to a robot that picks up the goods after payment is completed and hands them over to the user.

[0754] "A means of managing store inventory" refers to an artificial intelligence program that monitors the inventory status of products in a store in real time and issues replenishment orders as needed.

[0755] This invention is a system that recognizes the emotions of users in unmanned stores and optimizes customer service, inventory management, and product handover based on that information. Detailed embodiments for carrying out this invention are described below.

[0756] System Overview

[0757] This system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search, selection, payment, delivery, and inventory management. The system includes the following elements:

[0758] 1. Means of detecting user visits: Sensors and cameras are used to detect user movements and notify the server.

[0759] 2. Means of interacting with users using generative artificial intelligence: Interaction with users is conducted using an artificial intelligence program that employs natural language processing.

[0760] 3. Means of recognizing user emotions: The system analyzes facial expressions and voice using cameras and microphones, and recognizes user emotions using an emotion engine.

[0761] 4. Means for suggesting appropriate products in response to user requests: A generative artificial intelligence program that suggests the optimal product based on the user's statements and emotions.

[0762] 5. Method for checking product inventory: Access the inventory database in the store or warehouse to check the stock status.

[0763] 6. Means for processing payment for the items selected by the user: Payment terminals (such as QR code readers or credit card readers) for the user to pay for the items selected by the user.

[0764] 7. A means of handing over goods by robot after payment is completed: A robot to pick up the goods after payment is completed and hand them over to the user.

[0765] 8. Means of managing store inventory: An artificial intelligence program that monitors the inventory status of products in the store in real time and issues replenishment instructions as needed.

[0766] Operation Flow Description

[0767] 1. The server detects a user's arrival based on data from sensors and activates a generative artificial intelligence program.

[0768] 2. The generative artificial intelligence program initiates an initial conversation with the user through natural language processing to confirm the product the user is looking for.

[0769] 3. The camera and microphone acquire the user's facial expression and voice data, and the emotion engine analyzes this data to recognize the user's emotions.

[0770] 4. The generative artificial intelligence program suggests appropriate products in a tone that matches the user's emotions and checks inventory status from the database.

[0771] 5. Once the user selects an item, the server displays the payment screen on the payment terminal, and the user makes the payment.

[0772] 6. Once payment is complete, the server sends instructions to the product delivery robot, which then delivers the picked-up items to the user.

[0773] 7. Finally, the server updates the inventory database and manages the store's inventory.

[0774] Examples of specific cases and prompt statements

[0775] Specific example

[0776] A user enters an unmanned convenience store wearing smart glasses. The smart glasses use facial recognition and emotion analysis to analyze the user's facial expressions, and generative artificial intelligence asks, "Welcome, what are you looking for today?" When the user replies, "I'm looking for a new smartphone," the system checks the inventory and suggests, "We have the latest smartphone in stock. Would you like to purchase it?" If the user approves, the total price is displayed, and after the user pays, a robot hands over the product.

[0777] Example of a prompt

[0778] Input: A user, looking tired, enters an unmanned convenience store and is looking for a new smartphone.

[0779] Output: We will suggest the latest smartphones, and after payment, a robot will hand over the product.

[0780] The above describes specific embodiments for implementing the present invention. This system enables flexible customer service tailored to the user's emotions, efficient inventory management, and product delivery.

[0781] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0782] Step 1:

[0783] When a user enters the store, it is detected by a sensor.

[0784] Input: User reaches store

[0785] Output: The sensor detects a user's arrival and notifies the server.

[0786] Data processing / calculation: Sensors detect user movements and send that information to the server.

[0787] Operation: A sensor installed on the device detects movement and immediately sends that information to the server.

[0788] Step 2:

[0789] Launching a Generative Artificial Intelligence Program

[0790] Input: Visitor information from sensors

[0791] Output: Launch of a generative artificial intelligence program

[0792] Data processing / calculation: The server receives signals from sensors and launches a generative artificial intelligence program.

[0793] Operation: The server detects the user's arrival and launches a generative artificial intelligence program.

[0794] Step 3:

[0795] Starting the initial interaction with the user

[0796] Input: Activation signal for a generative artificial intelligence program

[0797] Output: Voice message "Welcome, what product are you looking for?"

[0798] Data Processing / Computation: Generative artificial intelligence performs natural language generation and creates initial dialogue messages.

[0799] Operation: The server plays a voice message to the user using a generative artificial intelligence program.

[0800] Step 4:

[0801] The system captures the user's facial expressions and voice, and analyzes them using an emotion engine.

[0802] Input: Camera video, microphone audio

[0803] Output: User sentiment data

[0804] Data processing / calculation: The emotion engine analyzes the user's emotions using camera video and audio data.

[0805] How it works: The device's built-in camera and microphone capture the user's face and voice, and an emotion engine analyzes the data.

[0806] Step 5:

[0807] Product proposals tailored to user requirements

[0808] Input: User statements and sentiment data

[0809] Output: Suggestion messages such as "How about the latest smartphone model X?"

[0810] Data Processing / Calculation: Generative artificial intelligence programs suggest optimal products based on user statements and emotions.

[0811] Operation: The server generates suggestions using a generative artificial intelligence program based on the user's requests and emotional data, and plays them back as voice messages.

[0812] Step 6:

[0813] Checking the inventory database

[0814] Input: Proposed product information

[0815] Output: Inventory status

[0816] Data processing / calculation: The server accesses the inventory database and checks the inventory status of the suggested products.

[0817] Operation: The server queries the database to retrieve current inventory information.

[0818] Step 7:

[0819] Display of payment screen

[0820] Input: User's expression of intent to purchase the product.

[0821] Output: Display of payment screen

[0822] Data processing / calculation: The server sends payment information to the payment terminal.

[0823] Operation: When the user indicates their intention to purchase, the payment screen will appear on the device.

[0824] Step 8:

[0825] Payment processing

[0826] Input: User's payment information (credit card information, QR code, etc.)

[0827] Output: Payment completion confirmation message

[0828] Data processing / calculation: The payment system verifies the user's payment information and completes the payment.

[0829] Operation: The user makes a payment at the payment terminal, and the server confirms that the payment processing is complete.

[0830] Step 9:

[0831] Product pickup and handover

[0832] Input: Payment completion information

[0833] Output: Product handover complete

[0834] Data processing / calculation: The server instructs the robot to pick up the products and hand them over to the user.

[0835] Operation: Once payment is complete, the robot picks up the goods from the warehouse and hands them over to the user.

[0836] Step 10:

[0837] Updating and managing inventory data

[0838] Input: Information about the product provided to the user.

[0839] Output: Updated inventory data

[0840] Data processing / calculation: The server updates the inventory database, and the inventory management AI monitors the inventory status in real time.

[0841] Operation: After the goods are delivered, the inventory database is updated, and replenishment orders are automatically issued as needed.

[0842] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0843] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0844] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0845] [Third Embodiment]

[0846] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0847] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0848] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0849] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0850] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0851] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0852] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0853] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0854] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0855] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0856] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0857] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0858] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The following describes the specific operating procedures and program processing of this system in natural language.

[0859] System Overview

[0860] This system detects user visits, uses generative artificial intelligence to engage in dialogue, and automates product search and selection, payment, delivery, and inventory management. The detailed operation of each element is described below.

[0861] User visit detection

[0862] When a user arrives at the unmanned shop, a sensor installed at the entrance detects the user. The information from this sensor acts as a trigger to activate the artificial intelligence customer service system, which will be described next.

[0863] AI-powered customer service

[0864] When the sensor detects a user, the server activates generative artificial intelligence and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, if the user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?"

[0865] Product search and selection

[0866] When a user selects a smartphone based on the suggestions of the generative artificial intelligence, the server accesses the inventory database to check the stock status of that product. If the product is in stock, the server notifies the generative artificial intelligence, which then informs the user.

[0867] Payment processing

[0868] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code or credit card on the terminal. After payment is complete, the server receives the information and begins the next step in the product delivery process.

[0869] Product handover

[0870] Once payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[0871] Store inventory management

[0872] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[0873] Specific example

[0874] For example, suppose a user visits an unmanned shop called "TECH STORE" and wants to buy a new smartphone. In this case, the process would be as follows:

[0875] 1. When a user reaches the entrance, the terminal (sensor) detects the user.

[0876] 2. The server activates the generative artificial intelligence and speaks to the user.

[0877] 3. The generative artificial intelligence asks, "Welcome. What kind of product are you looking for?"

[0878] 4. The user responds, "I want a new smartphone."

[0879] 5. The generative AI checks the inventory via the server and notifies the user, "We have the latest Model X. There are 5 in stock. Would you like to purchase it?"

[0880] 6. The user responds, "Yes, I will purchase it."

[0881] 7. The server displays the payment screen on the payment terminal, and the user completes the payment using the QR code.

[0882] 8. The server confirms that the payment is complete and sends instructions to the robot to pick up the goods.

[0883] 9. The robot hands the product to the user, and the user receives the product.

[0884] 10. The terminal (store management AI) recognizes the decrease in inventory and sends a replenishment instruction to the server.

[0885] In this way, this system enables the operation of automated, unmanned shops, providing reduced labor costs and improved operational efficiency.

[0886] The following describes the processing flow.

[0887] Step 1:

[0888] The user reaches the entrance of the unmanned shop. A terminal (a sensor installed at the entrance) detects the user's movement and sends that information to the server.

[0889] Step 2:

[0890] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI automatically displays a greeting message to the user.

[0891] Step 3:

[0892] A user who sees a message displayed by a generative artificial intelligence asks questions about the product they want.

[0893] Generative artificial intelligence analyzes user statements and suggests the most suitable products. If the user says, "I want a new smartphone," it will suggest, "How about the latest smartphone model X?"

[0894] Step 4:

[0895] Upon receiving a user's request, the server accesses the inventory database to check the stock status of the requested product. The server then sends the inventory information to a generative artificial intelligence (AI), which notifies the user. For example, it might display a message such as, "There are 5 items left in stock."

[0896] Step 5:

[0897] The user selects an item and indicates their intention to purchase it. The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the user's payment information.

[0898] Step 6:

[0899] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[0900] Step 7:

[0901] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the goods from the warehouse and prepares them for delivery to the user.

[0902] Step 8:

[0903] A robot delivers the product to a designated pick-up station and notifies the user that the product is ready. The user then picks up the product from the station.

[0904] Step 9:

[0905] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[0906] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, it significantly reduces labor costs and improves operational efficiency.

[0907] (Example 1)

[0908] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0909] Conventional unmanned stores often rely on manual or semi-automatic processes for tasks such as detecting user arrival, selecting products, processing payments, handing over products, and managing inventory, resulting in reduced overall operational efficiency. High operating costs and frequent stockouts were also problematic. Furthermore, the mechanical nature of user interaction made it difficult to provide a smooth purchasing experience. This invention aims to solve these problems, streamline the operation of unmanned stores, and reduce costs.

[0910] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0911] In this invention, the server includes means for detecting a user's arrival, means for transmitting the detected information to the server, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for monitoring inventory status for store operations and automatically generating replenishment instructions. This automates the entire process from the user's arrival to the completion of purchase, significantly improving operational efficiency and enabling cost reduction.

[0912] "Means of detecting user visits" refers to a system that uses devices such as sensors and cameras to detect when a user visits a store and collects that information.

[0913] "Means for transmitting detected information to a server" refers to a system that includes communication devices and protocols for transmitting user visitor information collected by sensors, cameras, etc., to a server.

[0914] "Means of engaging in dialogue with users using generative artificial intelligence" refers to a system that utilizes the latest generative artificial intelligence technology to enable natural dialogue with users through voice or text.

[0915] "A means of checking product inventory in response to user requests" refers to a system that queries a database for inventory information of products selected or requested by the user and retrieves the results.

[0916] "A means of processing payment for products selected by the user" refers to a system that processes the payment for products that the user has decided to purchase using payment methods such as QR codes or credit cards.

[0917] "A method of handing over goods by robot after payment is completed" refers to a system in which, after payment is completed, a robot picks up the designated goods and either hands them directly to the user or places them at a handover station.

[0918] "A means of monitoring inventory status and automatically generating replenishment instructions for store operations" refers to a system that constantly monitors inventory within a store, and when inventory levels become low, the server automatically generates replenishment instructions, thereby streamlining store operations.

[0919] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The system detects user arrivals, engages in dialogue using generative artificial intelligence, and automates product search and selection, payment, delivery, and inventory management. The following describes the specific operating procedures of this system.

[0920] The system includes the following main hardware and software components:

[0921] 1. Sensors: Installed at the entrance to detect user arrivals. For example, infrared sensors or cameras are used.

[0922] 2. Server: Receives information from sensors and activates generative artificial intelligence. It also accesses the inventory database, processes payments, and sends instructions to robots.

[0923] 3. Generative Artificial Intelligence: Enables voice or text-based interaction with the user. This artificial intelligence performs tasks such as answering questions and suggesting products.

[0924] 4. Inventory Database: A database that manages the inventory status of products in real time.

[0925] 5. Payment terminal: A terminal that includes a QR code reader and credit card reader to process payments for the items selected by the user.

[0926] 6. Robot: After payment is completed, the robot picks up the goods from the warehouse and either hands them to the user or places them at a designated handover station.

[0927] 7. Store Management AI: An artificial intelligence program that constantly monitors inventory levels and automatically generates replenishment instructions.

[0928] As a concrete example, the process for a user visiting an unmanned shop and wanting to purchase a new smartphone would be as follows:

[0929] 1. When a user reaches the store entrance, a sensor installed on the terminal detects the user and sends that information to the server.

[0930] 2. The server receives the information, activates the generative artificial intelligence, and speaks to the user, saying, "Welcome. What kind of product are you looking for?"

[0931] 3. When a user says, "I want a new smartphone," the generative artificial intelligence suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[0932] 4. Once the user decides to make a purchase, the server displays the payment screen on the payment terminal.

[0933] 5. Once the user scans the QR code and completes the payment, the server sends instructions to the robot to pick up the items.

[0934] 6. The robot retrieves a smartphone from the warehouse and hands it to the user.

[0935] 7. The store management AI checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment order.

[0936] This system automates the entire process, significantly improving operational efficiency and reducing costs. Furthermore, users can enjoy a smooth and intuitive purchasing experience.

[0937] The above describes embodiments for carrying out the present invention.

[0938] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0939] Step 1:

[0940] When a user reaches the store entrance, a sensor installed on the terminal detects the user. The sensor captures the user's movements and transmits the data to a server. The sensor's input is the detection of the user's movement or presence, and its output is a signal containing the detection information. This signal notifies the server that the user has entered the store. Infrared sensors and cameras are used specifically for this purpose.

[0941] Step 2:

[0942] When the server receives information from the sensor that a user has arrived, it activates the generative artificial intelligence (AI). The server's input is the detection information from the sensor, and its output is the command to activate the generative AI. The generative AI then begins a voice or text-based conversation with the user. For example, the server might send the command "A user has arrived. Please begin the conversation" to the generative AI.

[0943] Step 3:

[0944] The generative artificial intelligence (AI) initiates a conversation with the user and listens to their requests. The input to the AI ​​is the startup command from the server and the user's statements, and the output is the response. If the AI ​​greets the user with "Welcome. What product are you looking for?" and the user replies "I want a new smartphone," the AI ​​proceeds to the next step based on that request.

[0945] Step 4:

[0946] The server receives a user request, accesses the inventory database, and checks the product's stock status. The server's input is the user's request information from the generative artificial intelligence, and its output is the inventory check result. Specifically, the server sends a query to the database saying, "Please check the inventory of Model X smartphones," and the database responds, "There are 5 in stock."

[0947] Step 5:

[0948] The generative artificial intelligence provides product information to the user based on inventory check results from the server. The input to the generative AI is the inventory check results, and the output is a notification to the user. Specifically, it notifies the user, "We have the latest smartphone model X. There are 5 in stock. Would you like to purchase it?"

[0949] Step 6:

[0950] Once the user selects an item and decides to purchase it, the server displays a payment screen on the payment terminal. The server's input is the user's purchase decision and selected item information, and its output is instructions for the payment terminal. Specifically, the server sends the message "Process payment for Model X smartphone" to the payment terminal.

[0951] Step 7:

[0952] Users pay using a QR code or credit card at a payment terminal. The terminal receives payment instructions from the server and the user's payment information, and the output is a confirmation that the payment is complete. Specifically, the user scans a QR code with their smartphone, and the payment amount is displayed on the terminal.

[0953] Step 8:

[0954] Once the server confirms payment is complete, it sends an instruction to the robot to pick up the item. The server's input is payment completion information, and its output is an instruction to the robot. Specifically, it commands the robot to "Pick up the Model X smartphone and deliver it to the user."

[0955] Step 9:

[0956] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The robot receives instructions from a server as input, and its output is the completion of product delivery. For example, the robot might take a smartphone from a shelf and hand it to the user.

[0957] Step 10:

[0958] The store management AI on the terminal checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment instruction. The input to the store management AI is inventory data, and the output is a replenishment instruction to the server. For example, if the inventory falls to two or fewer units, it automatically notifies the server with "Inventory replenishment is needed."

[0959] (Application Example 1)

[0960] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0961] In operating unmanned stores, there is a need for specific methods and systems to achieve efficiency and cost reduction by fully automating everything from user arrival to product purchase, handover, and inventory management. In this process, rapid and accurate product recommendations and payment processing are required to enhance the user experience.

[0962] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0963] In this invention, the server includes means for detecting a user's arrival, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, means for managing the store's inventory, means for detecting the user using the user's smart device and beacon device, means for generative artificial intelligence to suggest products based on the user's statements, and means for processing payment via QR code or electronic payment service. This enables improved user experience, increased operational efficiency, and cost reduction in unmanned stores.

[0964] A "user" is a customer who visits an unmanned store and intends to purchase goods.

[0965] "Means of detecting visits" refer to devices such as sensors and beacon devices that detect a user's presence when they access a store.

[0966] "Generative artificial intelligence" refers to artificial intelligence programs that generate appropriate responses and suggestions based on user statements and requests.

[0967] "Means for checking product inventory" refers to a database system that manages the inventory status of products and provides that information to users upon request.

[0968] "Means of processing payment" refers to terminal devices and software used to process payments for products selected by the user through QR codes or electronic payment services.

[0969] "A means of delivering goods by robot" refers to a robotic system that picks up goods from a warehouse after payment is completed and delivers them to the user.

[0970] "A means of managing store inventory" refers to an artificial intelligence program that constantly monitors the inventory of products in a store and automatically instructs replenishment as needed.

[0971] A "smart device" is a portable electronic device that can connect to the internet, such as a smartphone or tablet.

[0972] A "beacon device" is a small device that provides location information via wireless communication and interacts with nearby smart devices.

[0973] A "QR code" is a type of two-dimensional barcode, a pattern used to efficiently transmit information.

[0974] "Electronic payment services" are services that allow payments to be completed electronically in an online or mobile environment (e.g., credit cards, various payment apps).

[0975] A "prompt" is an initial word or phrase that a generative artificial intelligence uses to initiate a conversation with a user.

[0976] System Overview

[0977] The unmanned store operation system accurately detects user arrivals, interacts with users using generative artificial intelligence, and automates product search and selection, payment, handover, and inventory management. The system of the present invention is composed of the following hardware and software.

[0978] User visit detection

[0979] First, when a user arrives at the unmanned store, their smart device is detected by a beacon device installed at the entrance. This allows the system to recognize the user's arrival as a trigger.

[0980] AI-powered customer service

[0981] When a user's arrival is detected by a beacon, the server activates generative artificial intelligence (for example, OpenAI's GPT-4 model) and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, the following prompts are used.

[0982] Prompt message: "Welcome. What kind of product are you looking for?"

[0983] Product search and selection

[0984] When a user selects a product based on the suggestions of a generative artificial intelligence, the server accesses the store's inventory database to check the product's availability. If the product is in stock, the generative AI notifies the user and confirms their intention to purchase.

[0985] Payment processing

[0986] Once a user selects an item and indicates their intention to purchase, the server displays a payment screen on the payment terminal. The user pays using a QR code or an electronic payment service (e.g., Stripe API or Square API) on the terminal. After the payment is complete, the information is sent to the server.

[0987] Product handover

[0988] After payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[0989] Store inventory management

[0990] The server constantly monitors the store's inventory levels and automatically detects when replenishment is needed if inventory levels drop. This prevents stockouts and improves the efficiency of store operations.

[0991] Specific example

[0992] Suppose a user visits an unmanned store and wants to purchase the latest smartphone. In this case, the process would proceed as follows:

[0993] 1. When a user arrives at the store entrance, a beacon device detects the user's smart device, and the arrival is recognized as a trigger.

[0994] 2. The server starts the generative artificial intelligence and sends the following prompt message to the user.

[0995] "Welcome. What kind of product are you looking for?"

[0996] 3. When a user responds with "I want a new smartphone," the generative artificial intelligence retrieves appropriate product information from the store's inventory database and suggests it to the user.

[0997] "We have the latest Model X in stock. We have 5 units available. Would you like to purchase one?"

[0998] 4. When the user responds with "Yes, I will purchase," the payment screen is displayed.

[0999] 5. The user completes the payment using a QR code or electronic payment service, and that information is sent to the server.

[1000] 6. The server instructs the delivery robot to pick up the product, and the robot delivers the product to the user.

[1001] 7. The server updates the store's inventory and automatically issues instructions if replenishment is needed.

[1002] The system described above streamlines the operation of unmanned stores, improves the user shopping experience, and reduces costs.

[1003] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1004] Step 1:

[1005] When a user arrives at an unmanned store, a beacon device installed at the entrance detects the user's smart device. The input is the ID signal of the smart device connected to the beacon device, and the output is a visitor notification sent to the server. Upon receiving this notification, the server recognizes the user's arrival and initiates the next processing step.

[1006] Step 2:

[1007] The server activates the generative artificial intelligence and begins interacting with the user. The input is a visitor notification, and the output is the initial prompt message sent to the user. Specifically, the generative AI generates the prompt message "Welcome. What products are you looking for?" and displays it on the smart device.

[1008] Step 3:

[1009] When a user responds via a smart device with "I want a new smartphone," the server sends that input to a generative artificial intelligence (AI). The input is the user's statement, and the output is the AI's response to it. The AI ​​accesses an inventory database, checks the stock status of the latest smartphone model, and then generates a response such as, "We have the latest model X. We have 5 in stock. Would you like to purchase it?"

[1010] Step 4:

[1011] When the user responds with "Yes, I will purchase" via their smart device, the server initiates the payment process. The input is the user's indication of their purchase intent, and the output is the payment screen displayed on the payment terminal. The server generates the payment screen using a QR code or electronic payment service and displays it to the user.

[1012] Step 5:

[1013] When a user completes the payment process using a QR code or electronic payment service, a payment completion notification is sent to the server. The input is the payment completion notification, and the output is an instruction to pick up the goods. The server then instructs the delivery robot to pick up the goods.

[1014] Step 6:

[1015] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The input is a pick-up instruction from the server, and the output is the delivery of the items to the user or handover station.

[1016] Step 7:

[1017] The server updates the store's inventory information once it confirms that the items have been picked up. The input is a notification that the items have been picked up, and the output is an update to the inventory database. The server updates the inventory information in real time and automatically generates replenishment orders as needed.

[1018] Through these processing steps, the operation of unmanned stores becomes more efficient, the user shopping experience is improved, and costs are reduced.

[1019] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1020] This invention relates to a system for operating an unmanned shop that recognizes user emotions and optimizes customer service, inventory management, and product handover based on that information. The system detects user arrivals, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, handover, and inventory management. The specific operating procedures and program processing of this system are described below.

[1021] System Overview

[1022] This system includes the following elements:

[1023] 1. Sensors that detect user visits

[1024] 2. Generative artificial intelligence programs that interact with users

[1025] 3. Database for managing product inventory information

[1026] 4. Terminal device for processing payments

[1027] 5. Robots that handle product delivery.

[1028] 6. An artificial intelligence program for managing store inventory.

[1029] 7. Emotion engine that recognizes user emotions

[1030] User visit detection

[1031] When a user arrives at the unmanned shop, a sensor installed at the shop entrance detects the user's movement and notifies the server. The sensor's detection activates a generative artificial intelligence, initiating an initial interaction with the user.

[1032] AI-powered customer service and emotion engines

[1033] The generative artificial intelligence interacts with the user. Simultaneously, the emotion engine analyzes the user's facial expressions and voice, and feeds that data back to the generative AI. For example, if the user appears tired, the generative AI will respond by using a gentler tone.

[1034] Product search and selection

[1035] Generative artificial intelligence analyzes user statements and suggests the most suitable products. For example, if a user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?" Furthermore, an emotion engine recognizes the user's emotions and adjusts the suggestions accordingly. For instance, if the user appears confused, the generative AI will add additional explanations.

[1036] For the product selected by the user, the server accesses the inventory database to check the stock status. The server then sends this inventory information to a generative artificial intelligence (AI), which informs the user.

[1037] Payment processing

[1038] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code or credit card on the terminal. After the server confirms the payment is complete, the item handover process begins.

[1039] Product handover

[1040] Once payment is complete, the server instructs the delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user receives the items from the delivery robot.

[1041] Store inventory management

[1042] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[1043] Specific example

[1044] For example, if a user visits the unmanned shop "TECH STORE" and wants to purchase a new smartphone, the process would be as follows:

[1045] 1. When a user arrives at the store, a terminal (sensor) detects the user and notifies the server.

[1046] 2. The server activates the generative artificial intelligence, which then speaks to the user, saying, "Welcome, what kind of product are you looking for?"

[1047] 3. The user responds, "I want a new smartphone."

[1048] 4. The generative AI checks inventory via the server, and if the emotion engine detects positive emotions from the user's facial expressions, it suggests, "We recommend the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[1049] 5. When the user responds with "Yes, I will purchase," the server displays the payment screen on the payment terminal, and the user makes the payment using a QR code.

[1050] 6. The server confirms that payment is complete and sends instructions to the robot to pick up the items.

[1051] 7. The robot picks up the products and prepares them for handing over to the user.

[1052] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[1053] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[1054] The following describes the processing flow.

[1055] Step 1:

[1056] The user reaches the entrance of the unmanned shop. A terminal (sensor) detects the user's movement and sends that information to the server.

[1057] Step 2:

[1058] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI displays a greeting message to the user saying, "Welcome, what product are you looking for?"

[1059] Step 3:

[1060] A generative AI analyzes the user's statements and inputs information about the products the user desires. Simultaneously, an emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This emotion data is then fed back to the generative AI.

[1061] Step 4:

[1062] The user says, "I want a new smartphone." The generative artificial intelligence analyzes this information, and if the emotion engine determines that the user's emotion is positive, the generative AI suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[1063] Step 5:

[1064] The user selects a product based on the suggestion. For example, they might respond, "Yes, I'll buy it." The server receives this information and checks the inventory database to confirm the product's availability.

[1065] Step 6:

[1066] The server sends inventory information to the generative artificial intelligence. The generative AI notifies the user that the item is in stock. It displays the message, "Inventory confirmed. Payment will now be processed."

[1067] Step 7:

[1068] The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the payment screen and prompts the user to make a payment.

[1069] Step 8:

[1070] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[1071] Step 9:

[1072] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the specified goods from the warehouse and either hands them directly to the user or places them at a designated delivery station.

[1073] Step 10:

[1074] A robot receives the goods, places them at a delivery station, and notifies the user that the goods are ready. The user then picks up the goods from the station.

[1075] Step 11:

[1076] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[1077] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, the use of an emotion engine enables flexible and appropriate customer service that responds to the user's emotions.

[1078] (Example 2)

[1079] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1080] Current unmanned shop systems lack effective interaction with users and emotional recognition, making it difficult to provide optimal customer service tailored to individual needs. Furthermore, the accuracy of automation in inventory management and product handover is low, hindering improvements in operational efficiency.

[1081] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1082] In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for analyzing the user's emotions, means for adjusting the content of the dialogue with the user according to the analyzed emotions, means for checking product inventory according to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for managing the store's inventory. This enables flexible customer service based on the user's emotions, as well as highly accurate inventory management and product delivery.

[1083] "Means of detecting user arrival" refers to sensors or devices used to determine when a user has reached a store.

[1084] "Generative artificial intelligence" refers to AI systems that perform natural language processing and dialogue generation.

[1085] "Means of interacting with the user" refers to the function by which generative artificial intelligence communicates with the user through voice or text.

[1086] "Means of analyzing user emotions" refers to emotion engines and algorithms that analyze a user's facial expressions and voice data to determine their emotional state.

[1087] "Means of adjusting the content of conversations with the user according to analyzed emotions" refers to a function in which generative artificial intelligence changes the tone and content of conversations based on the user's emotional data obtained from the emotion engine.

[1088] "Methods for checking product inventory" refers to the function of accessing a database system that manages product inventory information to check how much of a particular product is in stock.

[1089] "Means of processing payment" refers to terminal devices used by users to pay for selected products using credit cards, QR codes, or other methods.

[1090] "Methods of delivering goods by robot" refers to sales robots that pick up goods from a warehouse after payment is completed and hand them over to the user, or place them at a delivery station.

[1091] "Methods for managing store inventory" refers to an AI system that monitors inventory status in real time and automatically issues inventory replenishment orders as needed.

[1092] This invention is a system that streamlines the operation of unmanned shops and provides flexible customer service based on user emotions. The system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, delivery, and inventory management. Specific embodiments are described below.

[1093] System Configuration

[1094] The main hardware and software components of this system are as follows:

[1095] 1. Sensors to detect user arrival: When a user arrives at the unmanned shop, a motion sensor installed at the shop entrance detects the user's movement and notifies the server.

[1096] 2. Generative AI Program: The server launches a generative AI such as OpenAI GPT-3 and engages in an initial interaction with the user.

[1097] 3. Emotion Analysis Engine: Using an emotion engine such as Affectiva, the system analyzes the user's emotions from their facial expressions and voice, and feeds that data back into the generative artificial intelligence.

[1098] 4. Database for managing product inventory information: Product inventory information is managed using a database system such as MySQL.

[1099] 5. Payment terminals: Users make payments using QR codes or credit cards at payment terminals such as Clover Station.

[1100] 6. Robots for product delivery: Robots such as Fetch Robotics pick up products and hand them directly to the user.

[1101] 7. Artificial intelligence program for store inventory management: An AI using TensorFlow, etc., monitors inventory status and automatically issues replenishment orders.

[1102] Processing explanation in natural language

[1103] When the sensor detects a user, a notification is sent to the server, and the generative artificial intelligence is activated. The generative AI asks the user, "Welcome, what product are you looking for?" Through this initial dialogue, the user's requests are collected.

[1104] When a user says, "I want a new smartphone," their statement is collected via the microphone and converted to text by Google Cloud Speech-to-Text. Generative artificial intelligence uses this transcribed information to make appropriate product suggestions. For example, it might suggest, "How about the latest smartphone model X?" Simultaneously, an emotion engine analyzes the user's facial expressions and voice and provides feedback to the generative AI. If the user appears confused, the generative AI provides additional explanations.

[1105] When a user selects a product, the server accesses the inventory database to check its availability. For example, an SQL query such as "SELECT stock FROM products WHERE item='Smartphone Model X'" is used. If the item is in stock, the generative artificial intelligence informs the user, "There are 5 in stock. Would you like to purchase it?"

[1106] Once a user decides to purchase, the server displays a payment screen on the payment terminal. The user makes the payment using a QR code or credit card. When a payment completion notification is sent from the payment terminal to the server, the server sends an instruction to the robot to pick up the product. The robot picks up the product from the warehouse and either hands it directly to the user or places it at a designated handover station. Once the user receives the product, the inventory database is updated, and the store management AI monitors the inventory status and automatically issues replenishment instructions.

[1107] Specific example

[1108] For example, if a user visits the unmanned shop "TECH STORE" and wants to purchase a smartphone, the process would be as follows:

[1109] 1. When a user arrives at the store, a sensor detects the user and notifies the server.

[1110] 2. The server activates the generative artificial intelligence, which then says, "Welcome, what product are you looking for?"

[1111] 3. The user responds, "I want a new smartphone."

[1112] 4. The generative AI checks inventory via the server, and if the emotion engine detects positive emotions from the user's facial expressions, it suggests, "We recommend the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[1113] 5. When the user responds with "Yes, I will purchase," the server displays the payment screen on the payment terminal, and the user makes the payment using a QR code.

[1114] 6. The server confirms that payment is complete and sends instructions to the robot to pick up the items.

[1115] 7. The robot picks up the products and prepares them for handing over to the user.

[1116] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[1117] Example of a prompt

[1118] A user has arrived at the TECH STORE. A sensor detects the user, and customer service using generative artificial intelligence begins.

[1119] 1. "Welcome, what kind of product are you looking for?"

[1120] 2. The process for optimal product recommendations and inventory management when a user answers "new smartphone".

[1121] 3. Instructions for payment processing and product delivery.

[1122] 4. Instructions for automatic replenishment when store inventory levels drop.

[1123] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[1124] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1125] Step 1:

[1126] A terminal (sensor) detects a user's arrival. When a user passes through the store entrance, a motion sensor detects their movement and sends a signal to the server. At this point, the input is the sensor signal, and the output is a notification to the server.

[1127] Step 2:

[1128] The server starts a generative artificial intelligence (e.g., OpenAI GPT-3). Upon receiving a notification from a sensor, the server starts the generative AI and prepares for initial interaction. The input is the notification from the sensor, and the output is the startup of the generative AI.

[1129] Step 3:

[1130] The generative artificial intelligence (AI) initiates a dialogue with the user. The AI ​​asks the user, either verbally or in text, "Welcome, what product are you looking for?" This dialogue is output through the terminal's audio device or display. The input is the AI's dialogue prompt, and the output is the question to the user.

[1131] Step 4:

[1132] The user makes a product request to the generative artificial intelligence. For example, the AI ​​might respond, "A new smartphone." This statement is collected through the device's microphone and sent to the server as audio data. The input is the user's audio data, and the output is the transmission of audio data to the server.

[1133] Step 5:

[1134] The server converts the audio data into text and passes it to the generative artificial intelligence. Google Cloud Speech-to-Text is used to convert the audio to text. The input is the user's audio data, and the output is the transmission of text data to the generative artificial intelligence.

[1135] Step 6:

[1136] Generative artificial intelligence analyzes user requests and suggests appropriate products. For example, it might suggest, "How about the latest smartphone model X?" The generative AI generates this text, which is then displayed to the user via the device's display or audio device. The input is the user's text data, and the output is a product suggestion.

[1137] Step 7:

[1138] The emotion engine analyzes the user's facial expressions and provides feedback to the generative artificial intelligence. The user's facial expressions are captured by a camera and analyzed in real time. Emotional data is transmitted to the generative artificial intelligence via a server. The input is the user's facial expression data, and the output is emotional feedback to the generative artificial intelligence.

[1139] Step 8:

[1140] The server checks the product's inventory status. After the generative artificial intelligence understands the user's request, the server accesses the database to check the product's inventory. For example, it executes a query such as "SELECT stock FROM products WHERE item='Smartphone Model X'". The input is the request data for the generative artificial intelligence, and the output is the retrieved inventory data.

[1141] Step 9:

[1142] The server sends inventory information to a generative artificial intelligence (AI) system, which then notifies the user. Based on the inventory information, the AI ​​informs the user that "There are 5 items in stock." The input is inventory data, and the output is a notification to the user.

[1143] Step 10:

[1144] The user indicates their intention to purchase. For example, they might say, "Yes, I will purchase it." This statement is collected through the device's microphone and sent to the server. The input is the user's intention to purchase, and the output is the transmission of audio data to the server.

[1145] Step 11:

[1146] The server displays the payment screen on the payment terminal. The terminal displays a screen for the user to make a payment. Input is the user's purchase intention data, and output is instructions for the payment terminal.

[1147] Step 12:

[1148] The user makes a payment. Payment is made using a QR code or credit card, and a payment completion notification is sent to the server. The input is the user's payment information, and the output is the payment completion notification.

[1149] Step 13:

[1150] The server sends a pickup instruction to the delivery robot. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at the delivery station. The input is a payment completion notification, and the output is a pickup instruction to the robot.

[1151] Step 14:

[1152] A robot picks up the product and hands it to the user. When the user receives the product, the robot waits at a designated location and hands it directly to the user, or places the product at a station. The input is a pickup instruction from the server, and the output is the handover of the product to the user.

[1153] Step 15:

[1154] The server updates the inventory database, and the store management AI monitors the inventory status. After the product is handed over to the user, the inventory database is updated, and the AI ​​monitoring the inventory status automatically issues replenishment instructions as needed. Input is product handover information, and output is inventory data updates and replenishment instructions.

[1155] (Application Example 2)

[1156] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1157] Conventional unmanned store systems were unable to recognize user emotions and optimize customer service and product recommendations accordingly. This led to problems such as decreased customer satisfaction. Furthermore, they did not achieve sufficient efficiency in inventory management and product handover.

[1158] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for recognizing the user's emotions, means for suggesting appropriate products according to the user's requests, means for checking product inventory, means for processing payment for the products selected by the user, means for having a robot deliver the products after payment is completed, and means for managing the store's inventory. This enables flexible customer service that responds to the user's emotions, efficient inventory management, and product delivery.

[1159] "Means for detecting user visits" refer to devices such as sensors and cameras that detect when a user approaches or reaches the system.

[1160] "Means for engaging in dialogue with a user using generative artificial intelligence" refers to a device that uses an artificial intelligence program employing natural language processing technology to engage in voice or text-based dialogue with a user.

[1161] "Means of recognizing user emotions" refer to artificial intelligence programs or sensors that analyze a user's facial expressions and voice to identify their emotions.

[1162] "A means of suggesting appropriate products in response to user requests" refers to a generative artificial intelligence program that presents the most suitable products based on the user's statements and actions.

[1163] "Methods for checking product inventory" refer to database systems that record and track the number and condition of products in stores and warehouses.

[1164] "Means of processing payment for products selected by the user" refers to terminal devices or payment systems that process payment for products selected by the user.

[1165] "A means of delivering goods by robot after payment is completed" refers to a robot that picks up the goods after payment is completed and hands them over to the user.

[1166] "A means of managing store inventory" refers to an artificial intelligence program that monitors the inventory status of products in a store in real time and issues replenishment orders as needed.

[1167] This invention is a system that recognizes the emotions of users in unmanned stores and optimizes customer service, inventory management, and product handover based on that information. Detailed embodiments for carrying out this invention are described below.

[1168] System Overview

[1169] This system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search, selection, payment, delivery, and inventory management. The system includes the following elements:

[1170] 1. Means of detecting user visits: Sensors and cameras are used to detect user movements and notify the server.

[1171] 2. Means of interacting with users using generative artificial intelligence: Interaction with users is conducted using an artificial intelligence program that employs natural language processing.

[1172] 3. Means of recognizing user emotions: The system analyzes facial expressions and voice using cameras and microphones, and recognizes user emotions using an emotion engine.

[1173] 4. Means for suggesting appropriate products in response to user requests: A generative artificial intelligence program that suggests the optimal product based on the user's statements and emotions.

[1174] 5. Method for checking product inventory: Access the inventory database in the store or warehouse to check the stock status.

[1175] 6. Means for processing payment for the items selected by the user: Payment terminals (such as QR code readers or credit card readers) for the user to pay for the items selected by the user.

[1176] 7. A means of handing over goods by robot after payment is completed: A robot to pick up the goods after payment is completed and hand them over to the user.

[1177] 8. Means of managing store inventory: An artificial intelligence program that monitors the inventory status of products in the store in real time and issues replenishment instructions as needed.

[1178] Operation Flow Description

[1179] 1. The server detects a user's arrival based on data from sensors and activates a generative artificial intelligence program.

[1180] 2. The generative artificial intelligence program initiates an initial conversation with the user through natural language processing to confirm the product the user is looking for.

[1181] 3. The camera and microphone acquire the user's facial expression and voice data, and the emotion engine analyzes this data to recognize the user's emotions.

[1182] 4. The generative artificial intelligence program suggests appropriate products in a tone that matches the user's emotions and checks inventory status from the database.

[1183] 5. Once the user selects an item, the server displays the payment screen on the payment terminal, and the user makes the payment.

[1184] 6. Once payment is complete, the server sends instructions to the product delivery robot, which then delivers the picked-up items to the user.

[1185] 7. Finally, the server updates the inventory database and manages the store's inventory.

[1186] Examples of specific cases and prompt statements

[1187] Specific example

[1188] A user enters an unmanned convenience store wearing smart glasses. The smart glasses use facial recognition and emotion analysis to analyze the user's facial expressions, and generative artificial intelligence asks, "Welcome, what are you looking for today?" When the user replies, "I'm looking for a new smartphone," the system checks the inventory and suggests, "We have the latest smartphone in stock. Would you like to purchase it?" If the user approves, the total price is displayed, and after the user pays, a robot hands over the product.

[1189] Example of a prompt

[1190] Input: A user, looking tired, enters an unmanned convenience store and is looking for a new smartphone.

[1191] Output: We will suggest the latest smartphones, and after payment, a robot will hand over the product.

[1192] The above describes specific embodiments for implementing the present invention. This system enables flexible customer service tailored to the user's emotions, efficient inventory management, and product delivery.

[1193] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1194] Step 1:

[1195] When a user enters the store, it is detected by a sensor.

[1196] Input: User reaches store

[1197] Output: The sensor detects a user's arrival and notifies the server.

[1198] Data processing / calculation: Sensors detect user movements and send that information to the server.

[1199] Operation: A sensor installed on the device detects movement and immediately sends that information to the server.

[1200] Step 2:

[1201] Launching a Generative Artificial Intelligence Program

[1202] Input: Visitor information from sensors

[1203] Output: Launch of a generative artificial intelligence program

[1204] Data processing / calculation: The server receives signals from sensors and launches a generative artificial intelligence program.

[1205] Operation: The server detects the user's arrival and launches a generative artificial intelligence program.

[1206] Step 3:

[1207] Starting the initial interaction with the user

[1208] Input: Activation signal for a generative artificial intelligence program

[1209] Output: Voice message "Welcome, what product are you looking for?"

[1210] Data Processing / Computation: Generative artificial intelligence performs natural language generation and creates initial dialogue messages.

[1211] Operation: The server plays a voice message to the user using a generative artificial intelligence program.

[1212] Step 4:

[1213] The system captures the user's facial expressions and voice, and analyzes them using an emotion engine.

[1214] Input: Camera video, microphone audio

[1215] Output: User sentiment data

[1216] Data processing / calculation: The emotion engine analyzes the user's emotions using camera video and audio data.

[1217] How it works: The device's built-in camera and microphone capture the user's face and voice, and an emotion engine analyzes the data.

[1218] Step 5:

[1219] Product proposals tailored to user requirements

[1220] Input: User statements and sentiment data

[1221] Output: Suggestion messages such as "How about the latest smartphone model X?"

[1222] Data Processing / Calculation: Generative artificial intelligence programs suggest optimal products based on user statements and emotions.

[1223] Operation: The server generates suggestions using a generative artificial intelligence program based on the user's requests and emotional data, and plays them back as voice messages.

[1224] Step 6:

[1225] Checking the inventory database

[1226] Input: Proposed product information

[1227] Output: Inventory status

[1228] Data processing / calculation: The server accesses the inventory database and checks the inventory status of the suggested products.

[1229] Operation: The server queries the database to retrieve current inventory information.

[1230] Step 7:

[1231] Display of payment screen

[1232] Input: User's expression of intent to purchase the product.

[1233] Output: Display of payment screen

[1234] Data processing / calculation: The server sends payment information to the payment terminal.

[1235] Operation: When the user indicates their intention to purchase, the payment screen will appear on the device.

[1236] Step 8:

[1237] Payment processing

[1238] Input: User's payment information (credit card information, QR code, etc.)

[1239] Output: Payment completion confirmation message

[1240] Data processing / calculation: The payment system verifies the user's payment information and completes the payment.

[1241] Operation: The user makes a payment at the payment terminal, and the server confirms that the payment processing is complete.

[1242] Step 9:

[1243] Product pickup and handover

[1244] Input: Payment completion information

[1245] Output: Product handover complete

[1246] Data processing / calculation: The server instructs the robot to pick up the products and hand them over to the user.

[1247] Operation: Once payment is complete, the robot picks up the goods from the warehouse and hands them over to the user.

[1248] Step 10:

[1249] Updating and managing inventory data

[1250] Input: Information about the product provided to the user.

[1251] Output: Updated inventory data

[1252] Data processing / calculation: The server updates the inventory database, and the inventory management AI monitors the inventory status in real time.

[1253] Operation: After the goods are delivered, the inventory database is updated, and replenishment orders are automatically issued as needed.

[1254] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1255] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1256] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1257] [Fourth Embodiment]

[1258] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1259] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1260] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1261] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1262] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1263] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1264] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1265] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1266] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1267] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1268] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1269] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1270] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1271] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The following describes the specific operating procedures and program processing of this system in natural language.

[1272] System Overview

[1273] This system detects user visits, uses generative artificial intelligence to engage in dialogue, and automates product search and selection, payment, delivery, and inventory management. The detailed operation of each element is described below.

[1274] User visit detection

[1275] When a user arrives at the unmanned shop, a sensor installed at the entrance detects the user. The information from this sensor acts as a trigger to activate the artificial intelligence customer service system, which will be described next.

[1276] AI-powered customer service

[1277] When the sensor detects a user, the server activates generative artificial intelligence and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, if the user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?"

[1278] Product search and selection

[1279] When a user selects a smartphone based on the suggestions of the generative artificial intelligence, the server accesses the inventory database to check the stock status of that product. If the product is in stock, the server notifies the generative artificial intelligence, which then informs the user.

[1280] Payment processing

[1281] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code or credit card on the terminal. After payment is complete, the server receives the information and begins the next step in the product delivery process.

[1282] Product handover

[1283] Once payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[1284] Store inventory management

[1285] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[1286] Specific example

[1287] For example, suppose a user visits an unmanned shop called "TECH STORE" and wants to buy a new smartphone. In this case, the process would be as follows:

[1288] 1. When a user reaches the entrance, the terminal (sensor) detects the user.

[1289] 2. The server activates the generative artificial intelligence and speaks to the user.

[1290] 3. The generative artificial intelligence asks, "Welcome. What kind of product are you looking for?"

[1291] 4. The user responds, "I want a new smartphone."

[1292] 5. The generative AI checks the inventory via the server and notifies the user, "We have the latest Model X. There are 5 in stock. Would you like to purchase it?"

[1293] 6. The user responds, "Yes, I will purchase it."

[1294] 7. The server displays the payment screen on the payment terminal, and the user completes the payment using the QR code.

[1295] 8. The server confirms that the payment is complete and sends instructions to the robot to pick up the goods.

[1296] 9. The robot hands the product to the user, and the user receives the product.

[1297] 10. The terminal (store management AI) recognizes the decrease in inventory and sends a replenishment instruction to the server.

[1298] In this way, this system enables the operation of automated, unmanned shops, providing reduced labor costs and improved operational efficiency.

[1299] The following describes the processing flow.

[1300] Step 1:

[1301] The user reaches the entrance of the unmanned shop. A terminal (a sensor installed at the entrance) detects the user's movement and sends that information to the server.

[1302] Step 2:

[1303] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI automatically displays a greeting message to the user.

[1304] Step 3:

[1305] A user who sees a message displayed by a generative artificial intelligence asks questions about the product they want.

[1306] Generative artificial intelligence analyzes user statements and suggests the most suitable products. If the user says, "I want a new smartphone," it will suggest, "How about the latest smartphone model X?"

[1307] Step 4:

[1308] Upon receiving a user's request, the server accesses the inventory database to check the stock status of the requested product. The server then sends the inventory information to a generative artificial intelligence (AI), which notifies the user. For example, it might display a message such as, "There are 5 items left in stock."

[1309] Step 5:

[1310] The user selects an item and indicates their intention to purchase it. The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the user's payment information.

[1311] Step 6:

[1312] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[1313] Step 7:

[1314] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the goods from the warehouse and prepares them for delivery to the user.

[1315] Step 8:

[1316] A robot delivers the product to a designated pick-up station and notifies the user that the product is ready. The user then picks up the product from the station.

[1317] Step 9:

[1318] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[1319] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, it significantly reduces labor costs and improves operational efficiency.

[1320] (Example 1)

[1321] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1322] Conventional unmanned stores often rely on manual or semi-automatic processes for tasks such as detecting user arrival, selecting products, processing payments, handing over products, and managing inventory, resulting in reduced overall operational efficiency. High operating costs and frequent stockouts were also problematic. Furthermore, the mechanical nature of user interaction made it difficult to provide a smooth purchasing experience. This invention aims to solve these problems, streamline the operation of unmanned stores, and reduce costs.

[1323] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1324] In this invention, the server includes means for detecting a user's arrival, means for transmitting the detected information to the server, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for monitoring inventory status for store operations and automatically generating replenishment instructions. This automates the entire process from the user's arrival to the completion of purchase, significantly improving operational efficiency and enabling cost reduction.

[1325] "Means of detecting user visits" refers to a system that uses devices such as sensors and cameras to detect when a user visits a store and collects that information.

[1326] "Means for transmitting detected information to a server" refers to a system that includes communication devices and protocols for transmitting user visitor information collected by sensors, cameras, etc., to a server.

[1327] "Means of engaging in dialogue with users using generative artificial intelligence" refers to a system that utilizes the latest generative artificial intelligence technology to enable natural dialogue with users through voice or text.

[1328] "A means of checking product inventory in response to user requests" refers to a system that queries a database for inventory information of products selected or requested by the user and retrieves the results.

[1329] "A means of processing payment for products selected by the user" refers to a system that processes the payment for products that the user has decided to purchase using payment methods such as QR codes or credit cards.

[1330] "A method of handing over goods by robot after payment is completed" refers to a system in which, after payment is completed, a robot picks up the designated goods and either hands them directly to the user or places them at a handover station.

[1331] "A means of monitoring inventory status and automatically generating replenishment instructions for store operations" refers to a system that constantly monitors inventory within a store, and when inventory levels become low, the server automatically generates replenishment instructions, thereby streamlining store operations.

[1332] This invention is a system designed to streamline the operation of unmanned shops and reduce costs. The system detects user arrivals, engages in dialogue using generative artificial intelligence, and automates product search and selection, payment, delivery, and inventory management. The following describes the specific operating procedures of this system.

[1333] The system includes the following main hardware and software components:

[1334] 1. Sensors: Installed at the entrance to detect user arrivals. For example, infrared sensors or cameras are used.

[1335] 2. Server: Receives information from sensors and activates generative artificial intelligence. It also accesses the inventory database, processes payments, and sends instructions to robots.

[1336] 3. Generative Artificial Intelligence: Enables voice or text-based interaction with the user. This artificial intelligence performs tasks such as answering questions and suggesting products.

[1337] 4. Inventory Database: A database that manages the inventory status of products in real time.

[1338] 5. Payment terminal: A terminal that includes a QR code reader and credit card reader to process payments for the items selected by the user.

[1339] 6. Robot: After payment is completed, the robot picks up the goods from the warehouse and either hands them to the user or places them at a designated handover station.

[1340] 7. Store Management AI: An artificial intelligence program that constantly monitors inventory levels and automatically generates replenishment instructions.

[1341] As a concrete example, the process for a user visiting an unmanned shop and wanting to purchase a new smartphone would be as follows:

[1342] 1. When a user reaches the store entrance, a sensor installed on the terminal detects the user and sends that information to the server.

[1343] 2. The server receives the information, activates the generative artificial intelligence, and speaks to the user, saying, "Welcome. What kind of product are you looking for?"

[1344] 3. When a user says, "I want a new smartphone," the generative artificial intelligence suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[1345] 4. Once the user decides to make a purchase, the server displays the payment screen on the payment terminal.

[1346] 5. Once the user scans the QR code and completes the payment, the server sends instructions to the robot to pick up the items.

[1347] 6. The robot retrieves a smartphone from the warehouse and hands it to the user.

[1348] 7. The store management AI checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment order.

[1349] This system automates the entire process, significantly improving operational efficiency and reducing costs. Furthermore, users can enjoy a smooth and intuitive purchasing experience.

[1350] The above describes embodiments for carrying out the present invention.

[1351] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1352] Step 1:

[1353] When a user reaches the store entrance, a sensor installed on the terminal detects the user. The sensor captures the user's movements and transmits the data to a server. The sensor's input is the detection of the user's movement or presence, and its output is a signal containing the detection information. This signal notifies the server that the user has entered the store. Infrared sensors and cameras are used specifically for this purpose.

[1354] Step 2:

[1355] When the server receives information from the sensor that a user has arrived, it activates the generative artificial intelligence (AI). The server's input is the detection information from the sensor, and its output is the command to activate the generative AI. The generative AI then begins a voice or text-based conversation with the user. For example, the server might send the command "A user has arrived. Please begin the conversation" to the generative AI.

[1356] Step 3:

[1357] The generative artificial intelligence (AI) initiates a conversation with the user and listens to their requests. The input to the AI ​​is the startup command from the server and the user's statements, and the output is the response. If the AI ​​greets the user with "Welcome. What product are you looking for?" and the user replies "I want a new smartphone," the AI ​​proceeds to the next step based on that request.

[1358] Step 4:

[1359] The server receives a user request, accesses the inventory database, and checks the product's stock status. The server's input is the user's request information from the generative artificial intelligence, and its output is the inventory check result. Specifically, the server sends a query to the database saying, "Please check the inventory of Model X smartphones," and the database responds, "There are 5 in stock."

[1360] Step 5:

[1361] The generative artificial intelligence provides product information to the user based on inventory check results from the server. The input to the generative AI is the inventory check results, and the output is a notification to the user. Specifically, it notifies the user, "We have the latest smartphone model X. There are 5 in stock. Would you like to purchase it?"

[1362] Step 6:

[1363] Once the user selects an item and decides to purchase it, the server displays a payment screen on the payment terminal. The server's input is the user's purchase decision and selected item information, and its output is instructions for the payment terminal. Specifically, the server sends the message "Process payment for Model X smartphone" to the payment terminal.

[1364] Step 7:

[1365] Users pay using a QR code or credit card at a payment terminal. The terminal receives payment instructions from the server and the user's payment information, and the output is a confirmation that the payment is complete. Specifically, the user scans a QR code with their smartphone, and the payment amount is displayed on the terminal.

[1366] Step 8:

[1367] Once the server confirms payment is complete, it sends an instruction to the robot to pick up the item. The server's input is payment completion information, and its output is an instruction to the robot. Specifically, it commands the robot to "Pick up the Model X smartphone and deliver it to the user."

[1368] Step 9:

[1369] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The robot receives instructions from a server as input, and its output is the completion of product delivery. For example, the robot might take a smartphone from a shelf and hand it to the user.

[1370] Step 10:

[1371] The store management AI on the terminal checks the inventory status, recognizes when inventory is low, notifies the server, and generates a replenishment instruction. The input to the store management AI is inventory data, and the output is a replenishment instruction to the server. For example, if the inventory falls to two or fewer units, it automatically notifies the server with "Inventory replenishment is needed."

[1372] (Application Example 1)

[1373] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1374] In operating unmanned stores, there is a need for specific methods and systems to achieve efficiency and cost reduction by fully automating everything from user arrival to product purchase, handover, and inventory management. In this process, rapid and accurate product recommendations and payment processing are required to enhance the user experience.

[1375] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1376] In this invention, the server includes means for detecting a user's arrival, means for interacting with the user using generative artificial intelligence, means for checking product inventory in response to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, means for managing the store's inventory, means for detecting the user using the user's smart device and beacon device, means for generative artificial intelligence to suggest products based on the user's statements, and means for processing payment via QR code or electronic payment service. This enables improved user experience, increased operational efficiency, and cost reduction in unmanned stores.

[1377] A "user" is a customer who visits an unmanned store and intends to purchase goods.

[1378] "Means of detecting visits" refer to devices such as sensors and beacon devices that detect a user's presence when they access a store.

[1379] "Generative artificial intelligence" refers to artificial intelligence programs that generate appropriate responses and suggestions based on user statements and requests.

[1380] "Means for checking product inventory" refers to a database system that manages the inventory status of products and provides that information to users upon request.

[1381] "Means of processing payment" refers to terminal devices and software used to process payments for products selected by the user through QR codes or electronic payment services.

[1382] "A means of delivering goods by robot" refers to a robotic system that picks up goods from a warehouse after payment is completed and delivers them to the user.

[1383] "A means of managing store inventory" refers to an artificial intelligence program that constantly monitors the inventory of products in a store and automatically instructs replenishment as needed.

[1384] A "smart device" is a portable electronic device that can connect to the internet, such as a smartphone or tablet.

[1385] A "beacon device" is a small device that provides location information via wireless communication and interacts with nearby smart devices.

[1386] A "QR code" is a type of two-dimensional barcode, a pattern used to efficiently transmit information.

[1387] "Electronic payment services" are services that allow payments to be completed electronically in an online or mobile environment (e.g., credit cards, various payment apps).

[1388] A "prompt" is an initial word or phrase that a generative artificial intelligence uses to initiate a conversation with a user.

[1389] System Overview

[1390] The unmanned store operation system accurately detects user arrivals, interacts with users using generative artificial intelligence, and automates product search and selection, payment, handover, and inventory management. The system of the present invention is composed of the following hardware and software.

[1391] User visit detection

[1392] First, when a user arrives at the unmanned store, their smart device is detected by a beacon device installed at the entrance. This allows the system to recognize the user's arrival as a trigger.

[1393] AI-powered customer service

[1394] When a user's arrival is detected by a beacon, the server activates generative artificial intelligence (for example, OpenAI's GPT-4 model) and begins interacting with the user. The generative AI provides appropriate information in response to the user's questions and requests, assisting in product selection. For example, the following prompts are used.

[1395] Prompt message: "Welcome. What kind of product are you looking for?"

[1396] Product search and selection

[1397] When a user selects a product based on the suggestions of a generative artificial intelligence, the server accesses the store's inventory database to check the product's availability. If the product is in stock, the generative AI notifies the user and confirms their intention to purchase.

[1398] Payment processing

[1399] Once a user selects an item and indicates their intention to purchase, the server displays a payment screen on the payment terminal. The user pays using a QR code or an electronic payment service (e.g., Stripe API or Square API) on the terminal. After the payment is complete, the information is sent to the server.

[1400] Product handover

[1401] After payment is complete, the server instructs a delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user completes the purchase process by receiving the items from the delivery robot.

[1402] Store inventory management

[1403] The server constantly monitors the store's inventory levels and automatically detects when replenishment is needed if inventory levels drop. This prevents stockouts and improves the efficiency of store operations.

[1404] Specific example

[1405] Suppose a user visits an unmanned store and wants to purchase the latest smartphone. In this case, the process would proceed as follows:

[1406] 1. When a user arrives at the store entrance, a beacon device detects the user's smart device, and the arrival is recognized as a trigger.

[1407] 2. The server starts the generative artificial intelligence and sends the following prompt message to the user.

[1408] "Welcome. What kind of product are you looking for?"

[1409] 3. When a user responds with "I want a new smartphone," the generative artificial intelligence retrieves appropriate product information from the store's inventory database and suggests it to the user.

[1410] "We have the latest Model X in stock. We have 5 units available. Would you like to purchase one?"

[1411] 4. When the user responds with "Yes, I will purchase," the payment screen is displayed.

[1412] 5. The user completes the payment using a QR code or electronic payment service, and that information is sent to the server.

[1413] 6. The server instructs the delivery robot to pick up the product, and the robot delivers the product to the user.

[1414] 7. The server updates the store's inventory and automatically issues instructions if replenishment is needed.

[1415] The system described above streamlines the operation of unmanned stores, improves the user shopping experience, and reduces costs.

[1416] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1417] Step 1:

[1418] When a user arrives at an unmanned store, a beacon device installed at the entrance detects the user's smart device. The input is the ID signal of the smart device connected to the beacon device, and the output is a visitor notification sent to the server. Upon receiving this notification, the server recognizes the user's arrival and initiates the next processing step.

[1419] Step 2:

[1420] The server activates the generative artificial intelligence and begins interacting with the user. The input is a visitor notification, and the output is the initial prompt message sent to the user. Specifically, the generative AI generates the prompt message "Welcome. What products are you looking for?" and displays it on the smart device.

[1421] Step 3:

[1422] When a user responds via a smart device with "I want a new smartphone," the server sends that input to a generative artificial intelligence (AI). The input is the user's statement, and the output is the AI's response to it. The AI ​​accesses an inventory database, checks the stock status of the latest smartphone model, and then generates a response such as, "We have the latest model X. We have 5 in stock. Would you like to purchase it?"

[1423] Step 4:

[1424] When the user responds with "Yes, I will purchase" via their smart device, the server initiates the payment process. The input is the user's indication of their purchase intent, and the output is the payment screen displayed on the payment terminal. The server generates the payment screen using a QR code or electronic payment service and displays it to the user.

[1425] Step 5:

[1426] When a user completes the payment process using a QR code or electronic payment service, a payment completion notification is sent to the server. The input is the payment completion notification, and the output is an instruction to pick up the goods. The server then instructs the delivery robot to pick up the goods.

[1427] Step 6:

[1428] The robot picks up designated items from the warehouse and either hands them directly to the user or places them at a designated handover station. The input is a pick-up instruction from the server, and the output is the delivery of the items to the user or handover station.

[1429] Step 7:

[1430] The server updates the store's inventory information once it confirms that the items have been picked up. The input is a notification that the items have been picked up, and the output is an update to the inventory database. The server updates the inventory information in real time and automatically generates replenishment orders as needed.

[1431] Through these processing steps, the operation of unmanned stores becomes more efficient, the user shopping experience is improved, and costs are reduced.

[1432] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1433] This invention relates to a system for operating an unmanned shop that recognizes user emotions and optimizes customer service, inventory management, and product handover based on that information. The system detects user arrivals, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, handover, and inventory management. The specific operating procedures and program processing of this system are described below.

[1434] System Overview

[1435] This system includes the following elements:

[1436] 1. Sensors that detect user visits

[1437] 2. Generative artificial intelligence programs that interact with users

[1438] 3. Database for managing product inventory information

[1439] 4. Terminal device for processing payments

[1440] 5. Robots that handle product delivery.

[1441] 6. An artificial intelligence program for managing store inventory.

[1442] 7. Emotion engine that recognizes user emotions

[1443] User visit detection

[1444] When a user arrives at the unmanned shop, a sensor installed at the shop entrance detects the user's movement and notifies the server. The sensor's detection activates a generative artificial intelligence, initiating an initial interaction with the user.

[1445] AI-powered customer service and emotion engines

[1446] The generative artificial intelligence interacts with the user. Simultaneously, the emotion engine analyzes the user's facial expressions and voice, and feeds that data back to the generative AI. For example, if the user appears tired, the generative AI will respond by using a gentler tone.

[1447] Product search and selection

[1448] Generative artificial intelligence analyzes user statements and suggests the most suitable products. For example, if a user says, "I want a new smartphone," the generative AI might suggest, "How about the latest smartphone model X?" Furthermore, an emotion engine recognizes the user's emotions and adjusts the suggestions accordingly. For instance, if the user appears confused, the generative AI will add additional explanations.

[1449] For the product selected by the user, the server accesses the inventory database to check the stock status. The server then sends this inventory information to a generative artificial intelligence (AI), which informs the user.

[1450] Payment processing

[1451] Once the user selects an item, the server displays a payment screen on the payment terminal. The user pays using a QR code or credit card on the terminal. After the server confirms the payment is complete, the item handover process begins.

[1452] Product handover

[1453] Once payment is complete, the server instructs the delivery robot to pick up the items. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at a designated delivery station. The user receives the items from the delivery robot.

[1454] Store inventory management

[1455] The store management AI constantly monitors inventory levels. If inventory levels drop, the server detects that replenishment is needed and automatically generates the necessary instructions. This prevents stockouts and improves the efficiency of store operations.

[1456] Specific example

[1457] For example, if a user visits the unmanned shop "TECH STORE" and wants to purchase a new smartphone, the process would be as follows:

[1458] 1. When a user arrives at the store, a terminal (sensor) detects the user and notifies the server.

[1459] 2. The server activates the generative artificial intelligence, which then speaks to the user, saying, "Welcome, what kind of product are you looking for?"

[1460] 3. The user responds, "I want a new smartphone."

[1461] 4. The generative AI checks inventory via the server, and if the emotion engine detects positive emotions from the user's facial expressions, it suggests, "We recommend the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[1462] 5. When the user responds with "Yes, I will purchase," the server displays the payment screen on the payment terminal, and the user makes the payment using a QR code.

[1463] 6. The server confirms that payment is complete and sends instructions to the robot to pick up the items.

[1464] 7. The robot picks up the products and prepares them for handing over to the user.

[1465] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[1466] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[1467] The following describes the processing flow.

[1468] Step 1:

[1469] The user reaches the entrance of the unmanned shop. A terminal (sensor) detects the user's movement and sends that information to the server.

[1470] Step 2:

[1471] The server receives data from the sensors and activates the generative artificial intelligence. The generative AI displays a greeting message to the user saying, "Welcome, what product are you looking for?"

[1472] Step 3:

[1473] A generative AI analyzes the user's statements and inputs information about the products the user desires. Simultaneously, an emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This emotion data is then fed back to the generative AI.

[1474] Step 4:

[1475] The user says, "I want a new smartphone." The generative artificial intelligence analyzes this information, and if the emotion engine determines that the user's emotion is positive, the generative AI suggests, "We have the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[1476] Step 5:

[1477] The user selects a product based on the suggestion. For example, they might respond, "Yes, I'll buy it." The server receives this information and checks the inventory database to confirm the product's availability.

[1478] Step 6:

[1479] The server sends inventory information to the generative artificial intelligence. The generative AI notifies the user that the item is in stock. It displays the message, "Inventory confirmed. Payment will now be processed."

[1480] Step 7:

[1481] The server instructs the payment terminal to display the payment screen. The terminal (payment device) displays the payment screen and prompts the user to make a payment.

[1482] Step 8:

[1483] The user pays for the goods using a QR code or credit card at the payment device. The server confirms that the payment has been completed.

[1484] Step 9:

[1485] After payment is complete, the server sends instructions to the delivery robot to pick up and deliver the goods. The robot picks up the specified goods from the warehouse and either hands them directly to the user or places them at a designated delivery station.

[1486] Step 10:

[1487] A robot receives the goods, places them at a delivery station, and notifies the user that the goods are ready. The user then picks up the goods from the station.

[1488] Step 11:

[1489] Once the product handover is complete, the terminal (store management AI) updates the inventory data. If the inventory is low, the server automatically issues a command to replenish the stock and prepare for the next order.

[1490] This series of processes allows unmanned shops to operate efficiently, enabling users to purchase and receive goods without any problems. Furthermore, the use of an emotion engine enables flexible and appropriate customer service that responds to the user's emotions.

[1491] (Example 2)

[1492] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1493] Current unmanned shop systems lack effective interaction with users and emotional recognition, making it difficult to provide optimal customer service tailored to individual needs. Furthermore, the accuracy of automation in inventory management and product handover is low, hindering improvements in operational efficiency.

[1494] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1495] In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for analyzing the user's emotions, means for adjusting the content of the dialogue with the user according to the analyzed emotions, means for checking product inventory according to the user's request, means for processing payment for the product selected by the user, means for having a robot deliver the product after payment is completed, and means for managing the store's inventory. This enables flexible customer service based on the user's emotions, as well as highly accurate inventory management and product delivery.

[1496] "Means of detecting user arrival" refers to sensors or devices used to determine when a user has reached a store.

[1497] "Generative artificial intelligence" refers to AI systems that perform natural language processing and dialogue generation.

[1498] "Means of interacting with the user" refers to the function by which generative artificial intelligence communicates with the user through voice or text.

[1499] "Means of analyzing user emotions" refers to emotion engines and algorithms that analyze a user's facial expressions and voice data to determine their emotional state.

[1500] "Means of adjusting the content of conversations with the user according to analyzed emotions" refers to a function in which generative artificial intelligence changes the tone and content of conversations based on the user's emotional data obtained from the emotion engine.

[1501] "Methods for checking product inventory" refers to the function of accessing a database system that manages product inventory information to check how much of a particular product is in stock.

[1502] "Means of processing payment" refers to terminal devices used by users to pay for selected products using credit cards, QR codes, or other methods.

[1503] "Methods of delivering goods by robot" refers to sales robots that pick up goods from a warehouse after payment is completed and hand them over to the user, or place them at a delivery station.

[1504] "Methods for managing store inventory" refers to an AI system that monitors inventory status in real time and automatically issues inventory replenishment orders as needed.

[1505] This invention is a system that streamlines the operation of unmanned shops and provides flexible customer service based on user emotions. The system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search and selection, payment, delivery, and inventory management. Specific embodiments are described below.

[1506] System Configuration

[1507] The main hardware and software components of this system are as follows:

[1508] 1. Sensors to detect user arrival: When a user arrives at the unmanned shop, a motion sensor installed at the shop entrance detects the user's movement and notifies the server.

[1509] 2. Generative AI Program: The server launches a generative AI such as OpenAI GPT-3 and engages in an initial interaction with the user.

[1510] 3. Emotion Analysis Engine: Using an emotion engine such as Affectiva, the system analyzes the user's emotions from their facial expressions and voice, and feeds that data back into the generative artificial intelligence.

[1511] 4. Database for managing product inventory information: Product inventory information is managed using a database system such as MySQL.

[1512] 5. Payment terminals: Users make payments using QR codes or credit cards at payment terminals such as Clover Station.

[1513] 6. Robots for product delivery: Robots such as Fetch Robotics pick up products and hand them directly to the user.

[1514] 7. Artificial intelligence program for store inventory management: An AI using TensorFlow, etc., monitors inventory status and automatically issues replenishment orders.

[1515] Processing explanation in natural language

[1516] When the sensor detects a user, a notification is sent to the server, and the generative artificial intelligence is activated. The generative AI asks the user, "Welcome, what product are you looking for?" Through this initial dialogue, the user's requests are collected.

[1517] When a user says, "I want a new smartphone," their statement is collected via the microphone and converted to text by Google Cloud Speech-to-Text. Generative artificial intelligence uses this transcribed information to make appropriate product suggestions. For example, it might suggest, "How about the latest smartphone model X?" Simultaneously, an emotion engine analyzes the user's facial expressions and voice and provides feedback to the generative AI. If the user appears confused, the generative AI provides additional explanations.

[1518] When a user selects a product, the server accesses the inventory database to check its availability. For example, an SQL query such as "SELECT stock FROM products WHERE item='Smartphone Model X'" is used. If the item is in stock, the generative artificial intelligence informs the user, "There are 5 in stock. Would you like to purchase it?"

[1519] Once a user decides to purchase, the server displays a payment screen on the payment terminal. The user makes the payment using a QR code or credit card. When a payment completion notification is sent from the payment terminal to the server, the server sends an instruction to the robot to pick up the product. The robot picks up the product from the warehouse and either hands it directly to the user or places it at a designated handover station. Once the user receives the product, the inventory database is updated, and the store management AI monitors the inventory status and automatically issues replenishment instructions.

[1520] Specific example

[1521] For example, if a user visits the unmanned shop "TECH STORE" and wants to purchase a smartphone, the process would be as follows:

[1522] 1. When a user arrives at the store, a sensor detects the user and notifies the server.

[1523] 2. The server activates the generative artificial intelligence, which then says, "Welcome, what product are you looking for?"

[1524] 3. The user responds, "I want a new smartphone."

[1525] 4. The generative AI checks inventory via the server, and if the emotion engine detects positive emotions from the user's facial expressions, it suggests, "We recommend the latest smartphone model X. We have 5 in stock. Would you like to purchase it?"

[1526] 5. When the user responds with "Yes, I will purchase," the server displays the payment screen on the payment terminal, and the user makes the payment using a QR code.

[1527] 6. The server confirms that payment is complete and sends instructions to the robot to pick up the items.

[1528] 7. The robot picks up the products and prepares them for handing over to the user.

[1529] 8. When the user receives the product, the terminal (store management AI) updates the inventory data and issues replenishment instructions as needed.

[1530] Example of a prompt

[1531] A user has arrived at the TECH STORE. A sensor detects the user, and customer service using generative artificial intelligence begins.

[1532] 1. "Welcome, what kind of product are you looking for?"

[1533] 2. The process for optimal product recommendations and inventory management when a user answers "new smartphone".

[1534] 3. Instructions for payment processing and product delivery.

[1535] 4. Instructions for automatic replenishment when store inventory levels drop.

[1536] In this way, the system enables the efficient operation of automated, unmanned shops and provides flexible customer service based on user emotions.

[1537] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1538] Step 1:

[1539] A terminal (sensor) detects a user's arrival. When a user passes through the store entrance, a motion sensor detects their movement and sends a signal to the server. At this point, the input is the sensor signal, and the output is a notification to the server.

[1540] Step 2:

[1541] The server starts a generative artificial intelligence (e.g., OpenAI GPT-3). Upon receiving a notification from a sensor, the server starts the generative AI and prepares for initial interaction. The input is the notification from the sensor, and the output is the startup of the generative AI.

[1542] Step 3:

[1543] The generative artificial intelligence (AI) initiates a dialogue with the user. The AI ​​asks the user, either verbally or in text, "Welcome, what product are you looking for?" This dialogue is output through the terminal's audio device or display. The input is the AI's dialogue prompt, and the output is the question to the user.

[1544] Step 4:

[1545] The user makes a product request to the generative artificial intelligence. For example, the AI ​​might respond, "A new smartphone." This statement is collected through the device's microphone and sent to the server as audio data. The input is the user's audio data, and the output is the transmission of audio data to the server.

[1546] Step 5:

[1547] The server converts the audio data into text and passes it to the generative artificial intelligence. Google Cloud Speech-to-Text is used to convert the audio to text. The input is the user's audio data, and the output is the transmission of text data to the generative artificial intelligence.

[1548] Step 6:

[1549] Generative artificial intelligence analyzes user requests and suggests appropriate products. For example, it might suggest, "How about the latest smartphone model X?" The generative AI generates this text, which is then displayed to the user via the device's display or audio device. The input is the user's text data, and the output is a product suggestion.

[1550] Step 7:

[1551] The emotion engine analyzes the user's facial expressions and provides feedback to the generative artificial intelligence. The user's facial expressions are captured by a camera and analyzed in real time. Emotional data is transmitted to the generative artificial intelligence via a server. The input is the user's facial expression data, and the output is emotional feedback to the generative artificial intelligence.

[1552] Step 8:

[1553] The server checks the product's inventory status. After the generative artificial intelligence understands the user's request, the server accesses the database to check the product's inventory. For example, it executes a query such as "SELECT stock FROM products WHERE item='Smartphone Model X'". The input is the request data for the generative artificial intelligence, and the output is the retrieved inventory data.

[1554] Step 9:

[1555] The server sends inventory information to a generative artificial intelligence (AI) system, which then notifies the user. Based on the inventory information, the AI ​​informs the user that "There are 5 items in stock." The input is inventory data, and the output is a notification to the user.

[1556] Step 10:

[1557] The user indicates their intention to purchase. For example, they might say, "Yes, I will purchase it." This statement is collected through the device's microphone and sent to the server. The input is the user's intention to purchase, and the output is the transmission of audio data to the server.

[1558] Step 11:

[1559] The server displays the payment screen on the payment terminal. The terminal displays a screen for the user to make a payment. Input is the user's purchase intention data, and output is instructions for the payment terminal.

[1560] Step 12:

[1561] The user makes a payment. Payment is made using a QR code or credit card, and a payment completion notification is sent to the server. The input is the user's payment information, and the output is the payment completion notification.

[1562] Step 13:

[1563] The server sends a pickup instruction to the delivery robot. The robot picks up the specified items from the warehouse and either hands them directly to the user or places them at the delivery station. The input is a payment completion notification, and the output is a pickup instruction to the robot.

[1564] Step 14:

[1565] A robot picks up the product and hands it to the user. When the user receives the product, the robot waits at a designated location and hands it directly to the user, or places the product at a station. The input is a pickup instruction from the server, and the output is the handover of the product to the user.

[1566] Step 15:

[1567] The server updates the inventory database, and the store management AI monitors the inventory status. After the product is handed over to the user, the inventory database is updated, and the AI ​​monitoring the inventory status automatically issues replenishment instructions as needed. Input is product handover information, and output is inventory data updates and replenishment instructions.

[1568] (Application Example 2)

[1569] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1570] Conventional unmanned store systems were unable to recognize user emotions and optimize customer service and product recommendations accordingly. This led to problems such as decreased customer satisfaction. Furthermore, they did not achieve sufficient efficiency in inventory management and product handover.

[1571] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for detecting a user's arrival, means for engaging in dialogue with the user using generative artificial intelligence, means for recognizing the user's emotions, means for suggesting appropriate products according to the user's requests, means for checking product inventory, means for processing payment for the products selected by the user, means for having a robot deliver the products after payment is completed, and means for managing the store's inventory. This enables flexible customer service that responds to the user's emotions, efficient inventory management, and product delivery.

[1572] "Means for detecting user visits" refer to devices such as sensors and cameras that detect when a user approaches or reaches the system.

[1573] "Means for engaging in dialogue with a user using generative artificial intelligence" refers to a device that uses an artificial intelligence program employing natural language processing technology to engage in voice or text-based dialogue with a user.

[1574] "Means of recognizing user emotions" refer to artificial intelligence programs or sensors that analyze a user's facial expressions and voice to identify their emotions.

[1575] "A means of suggesting appropriate products in response to user requests" refers to a generative artificial intelligence program that presents the most suitable products based on the user's statements and actions.

[1576] "Methods for checking product inventory" refer to database systems that record and track the number and condition of products in stores and warehouses.

[1577] "Means of processing payment for products selected by the user" refers to terminal devices or payment systems that process payment for products selected by the user.

[1578] "A means of delivering goods by robot after payment is completed" refers to a robot that picks up the goods after payment is completed and hands them over to the user.

[1579] "A means of managing store inventory" refers to an artificial intelligence program that monitors the inventory status of products in a store in real time and issues replenishment orders as needed.

[1580] This invention is a system that recognizes the emotions of users in unmanned stores and optimizes customer service, inventory management, and product handover based on that information. Detailed embodiments for carrying out this invention are described below.

[1581] System Overview

[1582] This system detects user visits, engages in dialogue using generative artificial intelligence and an emotion engine, and automates product search, selection, payment, delivery, and inventory management. The system includes the following elements:

[1583] 1. Means of detecting user visits: Sensors and cameras are used to detect user movements and notify the server.

[1584] 2. Means of interacting with users using generative artificial intelligence: Interaction with users is conducted using an artificial intelligence program that employs natural language processing.

[1585] 3. Means of recognizing user emotions: The system analyzes facial expressions and voice using cameras and microphones, and recognizes user emotions using an emotion engine.

[1586] 4. Means for suggesting appropriate products in response to user requests: A generative artificial intelligence program that suggests the optimal product based on the user's statements and emotions.

[1587] 5. Method for checking product inventory: Access the inventory database in the store or warehouse to check the stock status.

[1588] 6. Means for processing payment for the items selected by the user: Payment terminals (such as QR code readers or credit card readers) for the user to pay for the items selected by the user.

[1589] 7. A means of handing over goods by robot after payment is completed: A robot to pick up the goods after payment is completed and hand them over to the user.

[1590] 8. Means of managing store inventory: An artificial intelligence program that monitors the inventory status of products in the store in real time and issues replenishment instructions as needed.

[1591] Operation Flow Description

[1592] 1. The server detects a user's arrival based on data from sensors and activates a generative artificial intelligence program.

[1593] 2. The generative artificial intelligence program initiates an initial conversation with the user through natural language processing to confirm the product the user is looking for.

[1594] 3. The camera and microphone acquire the user's facial expression and voice data, and the emotion engine analyzes this data to recognize the user's emotions.

[1595] 4. The generative artificial intelligence program suggests appropriate products in a tone that matches the user's emotions and checks inventory status from the database.

[1596] 5. Once the user selects an item, the server displays the payment screen on the payment terminal, and the user makes the payment.

[1597] 6. Once payment is complete, the server sends instructions to the product delivery robot, which then delivers the picked-up items to the user.

[1598] 7. Finally, the server updates the inventory database and manages the store's inventory.

[1599] Examples of specific cases and prompt statements

[1600] Specific example

[1601] A user enters an unmanned convenience store wearing smart glasses. The smart glasses use facial recognition and emotion analysis to analyze the user's facial expressions, and generative artificial intelligence asks, "Welcome, what are you looking for today?" When the user replies, "I'm looking for a new smartphone," the system checks the inventory and suggests, "We have the latest smartphone in stock. Would you like to purchase it?" If the user approves, the total price is displayed, and after the user pays, a robot hands over the product.

[1602] Example of a prompt

[1603] Input: A user, looking tired, enters an unmanned convenience store and is looking for a new smartphone.

[1604] Output: We will suggest the latest smartphones, and after payment, a robot will hand over the product.

[1605] The above describes specific embodiments for implementing the present invention. This system enables flexible customer service tailored to the user's emotions, efficient inventory management, and product delivery.

[1606] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1607] Step 1:

[1608] When a user enters the store, it is detected by a sensor.

[1609] Input: User reaches store

[1610] Output: The sensor detects a user's arrival and notifies the server.

[1611] Data processing / calculation: Sensors detect user movements and send that information to the server.

[1612] Operation: A sensor installed on the device detects movement and immediately sends that information to the server.

[1613] Step 2:

[1614] Launching a Generative Artificial Intelligence Program

[1615] Input: Visitor information from sensors

[1616] Output: Launch of a generative artificial intelligence program

[1617] Data processing / calculation: The server receives signals from sensors and launches a generative artificial intelligence program.

[1618] Operation: The server detects the user's arrival and launches a generative artificial intelligence program.

[1619] Step 3:

[1620] Starting the initial interaction with the user

[1621] Input: Activation signal for a generative artificial intelligence program

[1622] Output: Voice message "Welcome, what product are you looking for?"

[1623] Data Processing / Computation: Generative artificial intelligence performs natural language generation and creates initial dialogue messages.

[1624] Operation: The server plays a voice message to the user using a generative artificial intelligence program.

[1625] Step 4:

[1626] The system captures the user's facial expressions and voice, and analyzes them using an emotion engine.

[1627] Input: Camera video, microphone audio

[1628] Output: User sentiment data

[1629] Data processing / calculation: The emotion engine analyzes the user's emotions using camera video and audio data.

[1630] How it works: The device's built-in camera and microphone capture the user's face and voice, and an emotion engine analyzes the data.

[1631] Step 5:

[1632] Product proposals tailored to user requirements

[1633] Input: User statements and sentiment data

[1634] Output: Suggestion messages such as "How about the latest smartphone model X?"

[1635] Data Processing / Calculation: Generative artificial intelligence programs suggest optimal products based on user statements and emotions.

[1636] Operation: The server generates suggestions using a generative artificial intelligence program based on the user's requests and emotional data, and plays them back as voice messages.

[1637] Step 6:

[1638] Checking the inventory database

[1639] Input: Proposed product information

[1640] Output: Inventory status

[1641] Data processing / calculation: The server accesses the inventory database and checks the inventory status of the suggested products.

[1642] Operation: The server queries the database to retrieve current inventory information.

[1643] Step 7:

[1644] Display of payment screen

[1645] Input: User's expression of intent to purchase the product.

[1646] Output: Display of payment screen

[1647] Data processing / calculation: The server sends payment information to the payment terminal.

[1648] Operation: When the user indicates their intention to purchase, the payment screen will appear on the device.

[1649] Step 8:

[1650] Payment processing

[1651] Input: User's payment information (credit card information, QR code, etc.)

[1652] Output: Payment completion confirmation message

[1653] Data processing / calculation: The payment system verifies the user's payment information and completes the payment.

[1654] Operation: The user makes a payment at the payment terminal, and the server confirms that the payment processing is complete.

[1655] Step 9:

[1656] Product pickup and handover

[1657] Input: Payment completion information

[1658] Output: Product handover complete

[1659] Data processing / calculation: The server instructs the robot to pick up the products and hand them over to the user.

[1660] Operation: Once payment is complete, the robot picks up the goods from the warehouse and hands them over to the user.

[1661] Step 10:

[1662] Updating and managing inventory data

[1663] Input: Information about the product provided to the user.

[1664] Output: Updated inventory data

[1665] Data processing / calculation: The server updates the inventory database, and the inventory management AI monitors the inventory status in real time.

[1666] Operation: After the goods are delivered, the inventory database is updated, and replenishment orders are automatically issued as needed.

[1667] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1668] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1669] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1670] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1671] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1672] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1673] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1674] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1675] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1676] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1677] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1678] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1679] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1680] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1681] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1682] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1683] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1684] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1685] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1686] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1687] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1688] The following is further disclosed regarding the embodiments described above.

[1689] (Claim 1)

[1690] A means of detecting user visits,

[1691] A means of interacting with users using generative artificial intelligence,

[1692] A means of checking product inventory in response to user requests,

[1693] A means of processing payment for the product selected by the user,

[1694] A method of having a robot deliver the goods after payment is completed,

[1695] Methods for managing store inventory,

[1696] A system that includes this.

[1697] (Claim 2)

[1698] A sensor that detects the arrival of a user,

[1699] A generative artificial intelligence program that interacts with users,

[1700] A database for managing product inventory information,

[1701] A terminal device for processing payments,

[1702] A robot that handles the handover of goods,

[1703] An artificial intelligence program for managing store inventory,

[1704] The system according to claim 1, including the following:

[1705] (Claim 3)

[1706] The system according to claim 1, further comprising a means for a generative artificial intelligence program that assists in interaction with a user to suggest appropriate products based on the user's statements.

[1707] "Example 1"

[1708] (Claim 1)

[1709] A means of detecting user visits,

[1710] A means of sending the detected information to the server,

[1711] A means of interacting with users using generative artificial intelligence,

[1712] A means of checking product inventory in response to user requests,

[1713] A means of processing payment for the product selected by the user,

[1714] A method of having a robot deliver the goods after payment is completed,

[1715] A means for monitoring inventory status and automatically generating replenishment instructions for store operations,

[1716] A system that includes this.

[1717] (Claim 2)

[1718] A sensor that detects the arrival of a user,

[1719] A generative artificial intelligence program that interacts with users,

[1720] A database for managing product inventory information,

[1721] A terminal device for processing payments,

[1722] A robot that handles the handover of goods,

[1723] A store management artificial intelligence program that monitors inventory status and generates replenishment orders,

[1724] The system according to claim 1, including the following:

[1725] (Claim 3)

[1726] A generative artificial intelligence program that supports interaction with users provides a means to suggest appropriate products based on the user's statements,

[1727] The system according to claim 1, further comprising means for notifying the inventory status by referring to an inventory database.

[1728] "Application Example 1"

[1729] (Claim 1)

[1730] A means of detecting user visits,

[1731] A means of interacting with users using generative artificial intelligence,

[1732] A means of checking product inventory in response to user requests,

[1733] A means of processing payment for the product selected by the user,

[1734] A method of having a robot deliver the goods after payment is completed,

[1735] Methods for managing store inventory,

[1736] A means of detecting a user using the user's smart device and beacon device,

[1737] A method by which generative artificial intelligence suggests products based on user statements,

[1738] A means of processing payments via QR codes or electronic payment services,

[1739] A system that includes this.

[1740] (Claim 2)

[1741] A sensor that detects the arrival of a user,

[1742] A generative artificial intelligence program that interacts with users,

[1743] A database for managing product inventory information,

[1744] A terminal device for processing payments,

[1745] A robot that handles the handover of goods,

[1746] An artificial intelligence program for managing store inventory,

[1747] The user's smart device and beacon device,

[1748] A system that processes payments via QR codes or electronic payment services,

[1749] The system according to claim 1, including the following:

[1750] (Claim 3)

[1751] A generative artificial intelligence program that supports interaction with users provides a means to suggest appropriate products based on the user's statements,

[1752] The system according to claim 1, further comprising means for detecting a user's arrival using a beacon device and for a generative artificial intelligence to initiate a dialogue using a prompt sentence.

[1753] "Example 2 of combining an emotion engine"

[1754] (Claim 1)

[1755] A means of detecting user visits,

[1756] A means of interacting with users using generative artificial intelligence,

[1757] A means of analyzing user emotions,

[1758] A means of adjusting the content of conversations with the user according to the analyzed emotions,

[1759] A means of checking product inventory in response to user requests,

[1760] A means of processing payment for the product selected by the user,

[1761] A method of having a robot deliver the goods after payment is completed,

[1762] Methods for managing store inventory,

[1763] A system that includes this.

[1764] (Claim 2)

[1765] A sensor that detects the arrival of a user,

[1766] A generative artificial intelligence program that interacts with users,

[1767] An emotion engine that analyzes user emotions,

[1768] A database for managing product inventory information,

[1769] A terminal device for processing payments,

[1770] A robot that handles the handover of goods,

[1771] An artificial intelligence program for managing store inventory,

[1772] The system according to claim 1, including the following:

[1773] (Claim 3)

[1774] The system according to claim 1, further comprising a means for a generative artificial intelligence program that assists in interaction with a user to suggest appropriate products based on the user's statements.

[1775] "Application example 2 when combining with an emotional engine"

[1776] (Claim 1)

[1777] A means of detecting user visits,

[1778] A means of interacting with users using generative artificial intelligence,

[1779] Means of recognizing user emotions,

[1780] A means of proposing appropriate products in response to user requests,

[1781] Methods for checking product inventory,

[1782] A means of processing payment for the product selected by the user,

[1783] A method of having a robot deliver the goods after payment is completed,

[1784] Methods for managing store inventory,

[1785] A system that includes this.

[1786] (Claim 2)

[1787] A sensor that detects the arrival of a user,

[1788] A generative artificial intelligence program that interacts with users,

[1789] An emotion engine that recognizes the user's emotions,

[1790] A database for managing product inventory information,

[1791] A terminal device for processing payments,

[1792] A ...

Claims

1. A means of detecting user visits, A means of interacting with users using generative artificial intelligence, A means of checking product inventory in response to user requests, A means of processing payment for the product selected by the user, A method of having a robot deliver the goods after payment is completed, Methods for managing store inventory, A system that includes this.

2. A sensor that detects the arrival of a user, A generative artificial intelligence program that interacts with users, A database for managing product inventory information, A terminal device for processing payments, A robot that handles the handover of goods, An artificial intelligence program for managing store inventory, The system according to claim 1, including the following:

3. The system according to claim 1, further comprising a means for a generative artificial intelligence program that assists in interaction with a user to suggest appropriate products based on the user's statements.

Images