System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to launch a supermarket application using a mobile device; means for automatically scanning information of a product by a sensor and transmitting corresponding product information to a server; means for performing automatic payment based on the product information received from the server; means for receiving a user's shopping request using a voice recognition function and displaying a placement location of the corresponding product on a map; and means for adding a discounted product to a shopping list in conjunction with leaflet information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional supermarkets have had problems with long checkout queues and the resulting congestion. Customers also spend a lot of time searching for products, which can lead to customers forgetting to buy what they need. This results in lost sales opportunities for the supermarket. Furthermore, users spend a lot of time planning their meals, making efficient shopping difficult. There is a need to solve these problems. [Means for solving the problem]

[0005] The present invention solves the above problems by providing the user with a shopping list and recommended product information by having them launch a supermarket application on their mobile device. Automatically scanning product information using sensors and sending the relevant product information to a server eliminates waiting in line at the cash register. Automatic payment based on product information received from the server increases shopping efficiency and reduces congestion. The system receives a user's shopping request using a voice recognition function and displays the location of the relevant product on a map, reducing the time it takes to find a product. Furthermore, the system provides a means for adding discounted products to a shopping list in conjunction with flyer information, enabling users to shop efficiently and economically.

[0006] "User" refers to a general consumer who uses the supermarket application to shop.

[0007] "Mobile device" refers to a portable information terminal such as a smartphone or tablet.

[0008] "Supermarket application" refers to software that allows users to search for products, create shopping lists, make payments, etc.

[0009] "Launch" refers to the act of launching an application on a mobile device and making it available for use.

[0010] A "sensor" refers to a device attached to a basket that automatically reads product information.

[0011] "Product information" refers to detailed data such as product name, price, and stock status.

[0012] "Server" refers to a computer system that processes product information and user requests and transmits the results to a mobile device.

[0013] "Automatic payment" refers to the process by which a user completes payment using a pre-set payment method when purchasing a product without manual input.

[0014] "Voice recognition function" refers to technology that converts a user's voice input into text data.

[0015] A "shopping request" refers to a user's instructions to an application via voice or text regarding a specific product or meal plan.

[0016] "View on map" refers to visualizing product locations and optimal shopping routes on the screen of a mobile device.

[0017] "Flyer information" refers to digital data of promotional materials including information on discounted products and special sales offered by supermarkets.

[0018] "Discounted Items" refers to items offered at a discount below the regular price.

[0019] A "shopping list" refers to a list of items that a user plans to purchase. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention relates to a smart system for improving shopping experience in supermarkets, which has multiple functions to enable users to efficiently shop using mobile devices.

[0042] An embodiment of the system is described as follows.

[0043] 1. Starting the application and loading initial data

[0044] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[0045] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[0046] 2. Product scanning and automatic payment

[0047] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[0048] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[0049] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[0050] 3. Voice recognition for product search and navigation

[0051] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[0052] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[0053] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[0054] 4. Linking with flyer information

[0055] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[0056] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[0057] Examples:

[0058] When a user launches the app, a list of past purchases and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[0059] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[0060] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[0061] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[0062] In this way, the system can greatly enhance the user's shopping experience.

[0063] The processing flow will be explained below.

[0064] (Program processing steps)

[0065] Starting the application and loading initial data

[0066] Step 1:

[0067] A user taps to launch an application on their mobile device.

[0068] Step 2:

[0069] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[0070] Step 3:

[0071] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[0072] Step 4:

[0073] The device displays the received data, and the user can check their past purchase history and recommended products.

[0074] Product scanning and automatic payment

[0075] Step 1:

[0076] The user places the items they want to purchase in the cart.

[0077] Step 2:

[0078] An application on the terminal reads product information from sensors attached to the basket.

[0079] Step 3:

[0080] The terminal transmits the product information to the server.

[0081] Step 4:

[0082] The server analyzes the product information and checks whether the corresponding product exists in the database.

[0083] Step 5:

[0084] The server sends product price and stock information to the terminal.

[0085] Step 6:

[0086] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[0087] Step 7:

[0088] When the user completes their purchase, they press the "Confirm payment" button within the application.

[0089] Step 8:

[0090] The terminal transmits the information necessary to finalize the payment to the server.

[0091] Step 9:

[0092] The server automatically settles the payment using the user's registered payment method.

[0093] Step 10:

[0094] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[0095] Voice recognition for product search and navigation

[0096] Step 1:

[0097] The user makes a voice search request for a specific dish or product.

[0098] Step 2:

[0099] The device converts the voice input into text data and sends the information to the server.

[0100] Step 3:

[0101] The server creates a list of required items based on the request and identifies the location of each item.

[0102] Step 4:

[0103] The server sends the product list and placement information to the terminal.

[0104] Step 5:

[0105] The device displays the received product list and location information as a map, and guides the user to the optimal shopping route.

[0106] Linking with flyer information

[0107] Step 1:

[0108] The terminal obtains the latest flyer information from the server.

[0109] Step 2:

[0110] The flyer information acquired by the device is displayed within the application.

[0111] Step 3:

[0112] A user sees a flyer and adds discounted items to a list.

[0113] Step 4:

[0114] The terminal transmits the added discount product information to the server.

[0115] Step 5:

[0116] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[0117] Step 6:

[0118] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] Conventional supermarket shopping systems have problems with the complicated management of purchased items, payment, and related information acquisition. In particular, the user shopping experience is degraded by manual product scanning, long waiting times at the cash register, the inconvenience of voice-activated product searches, and the difficulty of checking sale information and recommended products. Furthermore, the decentralized nature of these functions and the lack of an integrated system hinders user satisfaction.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes means for automatically reading product information and transmitting the relevant product information, means for providing recommended product information based on past purchase history, means for receiving a user's shopping request using voice recognition and displaying product location guidance, means for adding sale items to a shopping list in conjunction with public relations information, and means for displaying the total amount in real time. This enables users to shop efficiently, significantly improving the shopping experience, and is expected to increase overall convenience through the integrated system.

[0124] A "mobile device" is an electronic device that a user can carry and that can run applications.

[0125] "Information processing device software" refers to an application that runs on a mobile device and assists users in shopping.

[0126] A "detection device" is a hardware device that automatically reads product information, such as an RFID reader.

[0127] "Data processing device" refers to a server or cloud service that receives product information, analyzes it, and executes various processes.

[0128] "Automatic payment" is a feature that allows users to complete payment without manual intervention when purchasing products.

[0129] The "voice recognition function" is a technology that converts a user's voice input into text data and processes the content.

[0130] A "shopping request" is a request made by a user to an application to request a specific product or dish.

[0131] "Guide display" is a function that visually shows the location of products and shopping routes to the user.

[0132] "Publicity information" refers to information that includes information about special sales and discounted products offered by supermarkets.

[0133] "Special sale items" are items offered at a lower price than usual.

[0134] "Past purchase history" is a record of data on products previously purchased by a user.

[0135] "Recommended product information" is information that suggests products suitable for a user based on the user's past purchase history, etc.

[0136] "Real-time display" is a function that instantly updates and displays information in response to operations or changes made by the user.

[0137] The "total amount" is the total amount paid by the user for all products purchased.

[0138] This invention relates to an integrated system for significantly improving the shopping experience in supermarkets, which provides a wide range of functions for users to efficiently shop using mobile devices.

[0139] In one embodiment of this system, a user first launches an application using a mobile device. The mobile device used here is a portable information processing device such as a smartphone or tablet. When the user launches the app, the device sends a request to a server to obtain the user's past purchase history and recommended product information. This server is a high-performance data processing device that contains a user database.

[0140] Next, when a user places items in a basket in the supermarket, a detection device such as an RFID reader attached to the basket automatically reads the product information. This product information is sent to a server via the terminal. The server references a product database to check the product price and stock information. The price information is returned to the terminal in real time, and the total price of the items the user has placed in the basket is displayed on the terminal screen.

[0141] Furthermore, if a user wishes to search for a specific dish or product, they can make a request using the voice recognition function. This voice recognition is performed using Google Voice Assistant. The device converts the user's voice input into text data and sends that information to the server. The server then identifies the required product list and product location, and sends it back to the device. Based on this, the device displays a map and navigates the optimal shopping route.

[0142] The application also incorporates the latest supermarket promotional information. Users can check information on special sales and discounted products through the app and add these products to their shopping list. The added product information is synchronized with the server database, and the optimal shopping route is recalculated.

[0143] As a concrete example, when a user launches an app, a list of past purchase history and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, with discounted butter recommended next to them. When a user places an apple in their basket at the supermarket, a sensor detects the apple and sends that information to the device. The device updates the total price in real time based on the information received from the server. When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and where to place them on a map. If the user looks at a flyer and adds discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[0144] Examples of prompts for the generative AI model include, "What are the features of an app that can improve the efficiency of shopping at the supermarket?", "Please explain how a smart system that uses voice recognition to search for specific products works", and "Please explain how a shopping app with an automatic payment system works."

[0145] In this way, the system provides a multifunctional platform for comprehensively improving the user's shopping experience.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] Starting the application and getting initial data

[0149] Executor: User and device

[0150] Input: Tap the application icon on your mobile device

[0151] Action: User taps the icon on their mobile device to launch the application.

[0152] Data processing: The device sends an HTTP request to the server, sending the user ID and authentication information.

[0153] Output: Get the user's past purchase history and recommended product information from the server

[0154] Specific operation: The server searches the user database for purchase history, generates product information using a recommendation algorithm, and sends it back to the device. The device analyzes this information and displays it on the screen.

[0155] Step 2:

[0156] Product scanning and information transmission

[0157] Actors: Users, devices, and sensors

[0158] Input: User adds product to cart

[0159] How it works: When an item is placed in the cart, the RFID reader captures the item information.

[0160] Data processing: The sensor reads the data from the RFID tag and sends it to the device. The device then sends this data to the server.

[0161] Output: Get product price and availability information from the server

[0162] Specific operation: The server queries the database based on the product ID, checks the price and stock of the corresponding product, and sends it back to the terminal, which updates the total price on the screen in real time.

[0163] Step 3:

[0164] Product search using voice recognition

[0165] Executor: User and device

[0166] Input: User's voice input (e.g. "I want to make curry")

[0167] Action: User makes a voice request

[0168] Data processing: The device's voice recognition function (Google Voice Assistant) converts the voice into text data and sends it to the server.

[0169] Output: Get the list of required products and their locations from the server

[0170] Specific operation: The server queries the database to obtain a list of the specified products and their locations, and sends this to the device, which then displays it using a map display engine.

[0171] Step 4:

[0172] Check flyer information and add it to your product list

[0173] Executor: User and device

[0174] Input: Publicity information (flyer) within the application

[0175] What happens: A user browses promotions within the app and sees discounted products.

[0176] Data processing: When a user adds an item to their shopping list, the device sends this information to the server.

[0177] Output: Get the updated optimal shopping route from the server

[0178] Specific operation: The server recalculates the optimal route based on the new product list and sends it to the device. The device uses its map display engine to display the updated route to the user.

[0179] Step 5:

[0180] Automatic payment

[0181] Executor: User and device

[0182] Input: User taps the "Pay" button

[0183] Action: User performs payment operation

[0184] Data processing: The device sends payment information (e.g., credit card information) to the server.

[0185] Output: Receives a confirmation message from the server and displays the payment result to the user.

[0186] What happens: The server processes the payment using a payment processor (e.g., a money processing system) and, if successful, sends a confirmation message back to the terminal, which displays this information on its screen.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] Today's consumers expect an efficient and comfortable shopping experience at supermarkets and other stores. They also have growing demands for reduced checkout wait times, improved in-store navigation, and product search capabilities using voice recognition. While meeting these demands, stores must also operate efficiently. However, the systems needed to achieve this are not yet fully in place, and the development of technology to reduce shopper stress is a challenge.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes means for a user to launch a store application using a mobile terminal, means for automatically scanning product information with a sensor and transmitting the relevant product information to the server, means for automatically making a payment based on the product information received from the server, means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map, means for adding discounted products to a shopping list in conjunction with the latest information, means for acquiring recommended product information and past purchase history and displaying them to the user, and means for optimally navigating the store based on the map display, thereby enabling the user to shop efficiently and comfortably.

[0192] A "mobile terminal" is an information terminal that can be carried by a user, and mainly refers to a smartphone or tablet terminal.

[0193] A "store application" is software that users use to assist them in shopping at physical stores, and is an application that runs on a mobile device.

[0194] A "sensor" is a device that detects physical phenomena and converts them into electronic signals, and is used to scan product information.

[0195] "Product information" refers to data relating to a specific product, such as product identification information, price information, and inventory information.

[0196] A "server" is a computer system that functions as a central processing unit, receives requests from clients, and processes data.

[0197] "Automatic payment" is a function that automatically calculates the price of the product purchased by the user and completes the payment using a payment method registered in advance.

[0198] "Voice recognition function" is a technology that inputs human speech as voice data and converts it into text data.

[0199] A "shopping request" is a request made by a user to retrieve information about a particular product.

[0200] "Displaying on a map" refers to visually displaying the layout of the store and showing the location of specific products.

[0201] "Latest news" refers to information about the latest discounts and special offers offered by supermarkets and stores.

[0202] "Discounted items" refer to items that are sold at a lower price than the regular price.

[0203] A "shopping list" is a list of items that a user plans to purchase.

[0204] "Recommended product information" refers to information for recommending specific products based on the user's past purchase history and preferences.

[0205] "Past purchase history" refers to a record of information about products previously purchased by a user.

[0206] "Navigation" is a function that guides the user to their destination, showing the location of the desired product within the store and providing the optimal route.

[0207] This invention relates to an advanced support system for users to efficiently shop using mobile devices. The system combines automatic product information scanning, voice recognition search, navigation, and payment functions to enhance the in-store shopping experience.

[0208] Hardware and software used

[0209] Hardware:

[0210] Mobile device (iOS or Android)

[0211] Sensors that scan product information (e.g., barcode scanners)

[0212] software:

[0213] Mobile applications (using Flutter or React Native as development frameworks)

[0214] Server side (using Node.js and Firebase)

[0215] Speech Recognition API (Google Cloud Speech-to-Text)

[0216] Payment system (Stripe, PayPal)

[0217] Database (Firestore)

[0218] Program overview and processing content

[0219] Start the application and load the initial data:

[0220] When a user launches an application on their mobile device, a request is sent to the server to retrieve the user's past purchase history and recommended product information. This information is displayed on the mobile device screen, allowing the user to check a list of past purchases and recommended products.

[0221] Product scanning and automatic payment:

[0222] When a user places an item in their cart, they scan the product barcode with the camera on their mobile device. The scanned product information is sent to a server via the mobile device, where the product price and stock information are confirmed. Once shopping is complete, payment is automatically made within the app, and payment is completed using the payment method registered in advance.

[0223] Voice-activated product search and navigation:

[0224] The user searches for a specific product using voice recognition. The mobile device converts the voice input into text data and sends it to the server. Based on the user's request, the server retrieves the list of required products and their location data and sends it to the mobile device. Based on this, a map of the store is displayed and the user is guided to the optimal shopping route.

[0225] Stay up-to-date with discounted items and add them to your shopping list:

[0226] The application incorporates the latest store information, allowing users to check discounted and special offers. When a discounted item is added to the shopping list, the system recalculates the optimal shopping route on the server and displays it on the mobile device.

[0227] Specific examples

[0228] For example, when a user launches the app on their mobile device, they can voice-input a request such as "I want to get the ingredients for pasta for dinner." Through voice recognition, the app displays ingredients such as "tomato sauce, pasta, olive oil, and cheese" and their locations, and suggests the shortest route within the store.

[0229] When a user places an item in their cart, the product information is instantly sent to the server, and the total price is updated in real time. Once shopping is complete, payment can be completed with just a few taps on a mobile device, eliminating the need to wait in long lines.

[0230] Example prompt sentence:

[0231] "I want to make pasta."

[0232] "Tell me what's for dinner."

[0233] This allows the user to shop efficiently and comfortably.

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1:

[0236] Launching the application

[0237] When a user launches an application on a mobile device, the device sends the user's authentication information to the server. Input includes a user ID and authentication token. The server receives this, retrieves the user's past purchase history and recommended product information, and returns it to the device. As output, the user's past purchase history and recommended product information are displayed on the device. Specific operations include retrieving and displaying data after sending the authentication information.

[0238] Step 2:

[0239] Product Scan

[0240] The user picks up an item and scans the barcode with the camera on their mobile device. The input contains the item's barcode information. The device reads the barcode and sends the item information to the server. The server retrieves the item's price and stock information from the product database and returns it to the device. The output is the item's price and stock information displayed on the screen. During this process, the device detects the item's barcode and sends the data to the server.

[0241] Step 3:

[0242] Automatic payment

[0243] The user completes their purchase and pays within the app. Input includes the total price of the items purchased by the user and the registered payment method. The terminal sends the payment information to the server, which then calls a payment system (e.g., Stripe or PayPal) to complete the payment. Output is a confirmation message on the terminal that the payment was successful. Specific actions include starting and completing the payment process.

[0244] Step 4:

[0245] Product search using voice recognition

[0246] The user inputs a request by voice, such as "I want to make pasta." The input includes voice data. The device converts the voice data into text using the Google Cloud Speech-to-Text API and sends this text data to the server. Based on the request, the server identifies the list of required products and their locations and returns this to the device. As output, the product list and their locations are displayed on a map. Specific operations include voice recognition and data transmission.

[0247] Step 5:

[0248] Proposal of the best shopping route

[0249] Based on the location data received from the server, the terminal displays the optimal shopping route on a map within the store. The input includes multiple product location data. Based on this data, the terminal calculates the optimal route using a shortest path algorithm and presents it to the user. As an output, the optimal route is displayed on a map. Specific operations include analyzing map data and displaying the route.

[0250] Step 6:

[0251] Linking with the latest information

[0252] Within the application, the user checks the latest discounted items and special offers and adds them to a shopping list. The input includes the latest information data. The server sends the latest information to the user's device, which displays it. When the user adds discounted items to the list, the server again calculates the optimal shopping route and returns it to the device. The output is the updated optimal route displayed on the map. Specific operations include retrieving the latest information and updating the list.

[0253] Step 7:

[0254] Displaying recommended product information

[0255] Based on the user's past purchase history and the database, the server calculates recommended products and sends them to the terminal. The input includes past purchase history data. The server analyzes this and recommends products suitable for the user. The output is the recommended products displayed on the screen. Specific operations include data analysis and information display.

[0256] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0257] This invention relates to a smart system that improves the shopping experience in supermarkets, and in particular to a system that recognizes the user's emotions by combining emotion engines and makes appropriate shopping suggestions based on those emotions. The system has the following functions:

[0258] 1. Starting the application and loading initial data

[0259] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[0260] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[0261] 2. Product scanning and automatic payment

[0262] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[0263] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[0264] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[0265] 3. Voice recognition for product search and navigation

[0266] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[0267] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[0268] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[0269] 4. Linking with flyer information

[0270] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[0271] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[0272] 5. User Emotion Recognition by Emotion Engine

[0273] When a user uses an application, the device uses a camera and microphone to recognize emotions from the user's facial expressions and voice.

[0274] The device sends the collected emotional data to a server, which then uses an emotion engine to analyze the user's emotional state.

[0275] Based on the analysis results, the server provides the user with recommended product information according to their emotional state.

[0276] Examples:

[0277] When a user launches the app, their past purchase history and recommended products are displayed. For example, the milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[0278] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[0279] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[0280] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[0281] If the user looks tired, the emotion engine will recognize this and the server will recommend products that have a relaxing effect (such as aromatic candles or herbal tea).

[0282] In this way, the system can provide various shopping suggestions depending on the user's emotional state, significantly improving the user's shopping experience.

[0283] The processing flow will be explained below.

[0284] Starting the application and loading initial data

[0285] Step 1:

[0286] A user taps to launch an application on their mobile device.

[0287] Step 2:

[0288] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[0289] Step 3:

[0290] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[0291] Step 4:

[0292] The device displays the received data, and the user can check their past purchase history and recommended products.

[0293] Product scanning and automatic payment

[0294] Step 1:

[0295] The user places the items they want to purchase in the cart.

[0296] Step 2:

[0297] The terminal reads product information from sensors attached to the basket.

[0298] Step 3:

[0299] The terminal transmits the product information to the server.

[0300] Step 4:

[0301] The server analyzes the product information and checks whether the corresponding product exists in the database.

[0302] Step 5:

[0303] The server sends product price and stock information to the terminal.

[0304] Step 6:

[0305] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[0306] Step 7:

[0307] When the user completes their purchase, they press the "Confirm payment" button within the application.

[0308] Step 8:

[0309] The terminal transmits the information necessary to finalize the payment to the server.

[0310] Step 9:

[0311] The server automatically settles the payment using the user's registered payment method.

[0312] Step 10:

[0313] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[0314] Voice recognition for product search and navigation

[0315] Step 1:

[0316] The user makes a voice search request for a specific dish or product.

[0317] Step 2:

[0318] The device converts the voice input into text data and sends the information to the server.

[0319] Step 3:

[0320] The server creates a list of required items based on the request and identifies the location of each item.

[0321] Step 4:

[0322] The server sends the product list and location data to the terminal.

[0323] Step 5:

[0324] The device receives a list of products and displays their location information as a map, guiding users to the optimal shopping route.

[0325] Linking with flyer information

[0326] Step 1:

[0327] The terminal obtains the latest flyer information from the server.

[0328] Step 2:

[0329] The flyer information acquired by the device is displayed within the application.

[0330] Step 3:

[0331] A user sees a flyer and adds discounted items to a list.

[0332] Step 4:

[0333] The terminal transmits the added discount product information to the server.

[0334] Step 5:

[0335] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[0336] Step 6:

[0337] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[0338] Recognizing user emotions with an emotion engine

[0339] Step 1:

[0340] When a user is using an application, the device uses a camera and microphone to collect the user's facial expressions and voice.

[0341] Step 2:

[0342] The terminal transmits the collected facial expression and voice data to the server.

[0343] Step 3:

[0344] The server uses an emotion engine to analyze the user's emotional state.

[0345] Step 4:

[0346] Based on the analysis results, the server creates recommended product information according to the user's emotional state and sends it to the terminal.

[0347] Step 5:

[0348] The device will display recommended products that reflect your emotional state.

[0349] Examples:

[0350] If a user has a tired expression while using the app, the device will recognize that expression with its camera and send it to the server.

[0351] The server uses an emotion engine to detect "fatigue," and then lists recommended products that have a relaxing effect (e.g., aroma candles and herbal tea) and sends them to the device.

[0352] The device will suggest products that will help users feel refreshed by displaying "Recommended products: Aroma candles, herbal tea" on the screen.

[0353] In this way, the system can make individualized suggestions based on the user's emotional state, making the shopping experience more personal and convenient.

[0354] Example 2

[0355] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] Modern supermarkets require systems that enable users to enjoy an efficient and stress-free shopping experience. However, existing systems struggle to promote user shopping behavior, and the product search and payment processes can be time-consuming. Furthermore, they do not consider user emotions when suggesting products, and thus fail to provide a personalized shopping experience.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0358] In this invention, the server includes means for a user to launch a store management system application using a mobile communication device, means for automatically reading product information using a sensor and transmitting the product information to a data processing device, means for automatically making a payment based on the product information received from the data processing device, means for receiving a user's shopping request using a voice recognition function and displaying the location of the corresponding product on a guidance display device, means for adding discounted products to a shopping list in conjunction with advertising information, and means for analyzing emotions from the user's facial expressions and voice using an emotion recognition engine and providing recommended product information based on the analysis results. This enables users to not only efficiently search for and purchase products, but also receive personalized shopping suggestions.

[0359] A "mobile communication device" is a communication device that a user can carry around, and primarily refers to devices such as smartphones and tablets.

[0360] A "store management system" refers to an application used to manage products and support users' shopping experiences in retail stores such as supermarkets.

[0361] "Sensor" means a device for detecting and reading product information, including RFID readers and barcode scanners.

[0362] "Data processing device" refers to a server or database that analyzes product information sent from the sensor and provides the necessary information to the user.

[0363] "Automatic payment" refers to the process of automatically paying the total amount of the products purchased by the user using a payment method registered in advance.

[0364] "Voice recognition function" refers to technology that converts a user's voice input into text data and then provides the system with appropriate information based on that data.

[0365] A "guidance display device" is a device used to display product locations and optimal shopping routes within a store, and primarily refers to the screen of a user's mobile communication device.

[0366] "Advertising information" refers to supermarket flyers and discounted product information, and is used to provide users with special sale information.

[0367] An "emotion recognition engine" refers to an algorithm that analyzes a user's emotional state from their facial expressions and voice, and recommends appropriate products based on the results.

[0368] "Personalized shopping suggestions" refers to a feature that suggests products and services that are suitable for each individual user based on their past purchase history and emotional state.

[0369] This invention relates to a store management system for improving the shopping experience in supermarkets, particularly a system that uses mobile communication devices, data processing devices, sensors, and an emotion recognition engine to analyze a user's emotions and make appropriate suggestions based on the analysis.

[0370] When a user launches an application, the mobile communication device (e.g., a smartphone or tablet) sends a request to a data processing device (server) to obtain the user's past purchase history and recommended product information. This allows the device to display a list of previously purchased products and the store's recommended products. As a specific example, previously purchased milk or bread may be displayed, with discounted butter recommended next to them.

[0371] When a user selects an item in a supermarket, a sensor (e.g., an RFID reader or barcode scanner) automatically reads the item's information and transmits it to a data processing device via a mobile communication device. The data processing device analyzes the item's information, checks the price and stock information of the item, and displays the total price on the terminal in real time. For example, when an apple is placed in a basket, the sensor detects the apple and transmits the information to the terminal, instantly updating the total price on the screen.

[0372] Furthermore, if a user wishes to search for a specific dish or product, the request is made using a voice recognition function. The mobile communication device converts the user's voice input into text data and sends it to the data processing device. The data processing device creates a list of necessary products based on the user's request and identifies the location of each product. Based on the received information, the mobile communication device displays the product locations on a map and navigates the optimal shopping route. For example, in response to a request to "make curry," a list of necessary ingredients and their locations are displayed on a map.

[0373] The application incorporates the latest store advertising information, allowing users to check information on discounted and special sale items. Users can add discounted items to their shopping list, and this information is synchronized with the data processing device's database. This allows the optimal shopping route to be recalculated and displayed on the mobile communication device. For example, if a user sees a flyer and adds a discounted curry roux to their shopping list, the optimal route recalculated by the data processing device will be displayed on the device.

[0374] Furthermore, to understand the user's emotional state, facial expressions and voice are collected using the camera and microphone of the mobile communication device and analyzed by an emotion recognition engine. The emotion recognition engine analyzes the user's emotional state and provides appropriate product information based on that analysis. For example, if the user looks tired, it may recommend relaxing aroma candles or herbal tea.

[0375] This allows users to not only efficiently search and purchase products, but also receive personalized shopping suggestions.

[0376] Prompt statement example:

[0377] 1. "I want to develop a system to simplify supermarket shopping. It includes functionality to recognize user emotions and recommend products based on those emotions. Can you please explain the detailed process of this system?"

[0378] 2. "Please explain in detail how smart systems can be used to improve the shopping experience at supermarkets. In particular, I would like to know about the use of voice recognition and emotion engines."

[0379] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0380] Step 1:

[0381] A user launches a store management system application using a mobile communication device. When the user launches the app, the device sends a request for initial data to the server. The server obtains the user's past purchase history and recommended product information and sends this to the device. The device displays the received data on the screen, allowing the user to view a list of products they have purchased in the past and recommended products from the supermarket. For example, a list of milk, bread, and butter that the user has purchased in the past, as well as discounted products, is displayed.

[0382] input:

[0383] User launches app

[0384] output:

[0385] Past purchase history

[0386] Recommended product information

[0387] Step 2:

[0388] When a user places an item in a basket at a supermarket, a sensor automatically reads the item's information. This information is sent to a server via the terminal. The server analyzes the received product information and checks the price and stock information of the item. The analyzed information is sent to the terminal in real time, and the terminal updates and displays the total price. For example, when a user puts an apple in the basket, the sensor reads the apple's information, and the terminal updates the total price in real time.

[0389] input:

[0390] Product information added to the user's cart

[0391] output:

[0392] Real-time total amount

[0393] Product price and availability information

[0394] Step 3:

[0395] When a user wants to search for a specific dish or product, they make a request using the voice recognition function. The user's voice input is converted into text data by the device and sent to the server. The server creates a product list based on the user's voice request and identifies the location of each product. The device uses the information from the server to display the product locations on a map and navigates the user to the optimal shopping route. For example, if a user makes a voice request such as "I want to make curry," a list of the necessary ingredients and their locations will be displayed on a map.

[0396] input:

[0397] User voice request

[0398] output:

[0399] Location of food and products

[0400] The best shopping route

[0401] Step 4:

[0402] The user can check the latest store advertising information embedded in the device. The user can then check the displayed discounted and special sale items and add the discounted items to their shopping list. The added product information is synchronized with the server's database, and the server sends a recalculated optimal shopping route to the device. The device displays the new route information, allowing the user to shop efficiently. For example, if the user sees a flyer and adds a discounted curry roux to their shopping list, a new optimal route will be displayed on the device.

[0403] input:

[0404] Adding discounted products to users

[0405] output:

[0406] Updated shopping route

[0407] Discounted product information

[0408] Step 5:

[0409] While the user is using the application, the device uses a camera and microphone to collect the user's facial expressions and voice. The device sends the collected emotional data to a server, which then uses an emotion recognition engine to analyze the user's emotional state. Based on the analysis results, the server generates recommended product information according to the user's emotional state and sends it to the device. The device then displays this information to the user. For example, if the user looks tired, the emotion recognition engine will recommend products with a relaxing effect (aroma candles or herbal tea).

[0410] input:

[0411] User emotion data

[0412] output:

[0413] Emotion-based product recommendations

[0414] (Application example 2)

[0415] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] Traditional shopping experiences in brick-and-mortar stores often require users to search around the store or wait in line at the cash register for long periods of time, resulting in inefficiency. Furthermore, product recommendations are often made without taking the user's emotional state into consideration, which can lead to lower user satisfaction. There is a need to improve the user's shopping experience by solving these problems.

[0417] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for the user to launch an application for the facility using a mobile information terminal; means for automatically reading product information using a detector and transmitting the relevant product information to the information processing device; means for making an automatic payment based on the product information received from the information processing device; means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map; means for adding discounted products to a shopping list in conjunction with advertising information; and means for recognizing the user's emotional state using facial expression analysis and voice analysis and providing recommended product information based on the emotional state. This makes it possible to make individual shopping suggestions according to the user's emotional state, thereby providing an efficient and satisfying shopping experience.

[0418] A "personal digital assistant" is an electronic device that can be carried by a user and is a device for executing applications and performing data communication.

[0419] "Store facilities" refer to physical stores such as supermarkets and other retail outlets where customers visit in person to shop.

[0420] An "application" is a program that runs on a mobile information terminal and provides support functions to help users shop efficiently.

[0421] A "detector" is a device that automatically reads product information and is a sensor that obtains information using barcodes, RFID, etc.

[0422] The "information processing device" is a computer system such as a server, which receives, analyzes, and stores the product information transmitted from the detector.

[0423] "Automatic payment" is a system in which a user automatically pays for products purchased via an information processing device without having to line up at the cash register.

[0424] The "voice recognition function" is a technology that converts the user's voice into text data, and is a function that receives and analyzes shopping requests.

[0425] "Advertising information" is information about discounted products and special sales offered by the facility, and is displayed within the application to support the user's shopping experience.

[0426] "Discounted products" are products offered at special prices based on advertising information, and are intended to encourage users to consume.

[0427] "Facial expression analysis" is a technology that uses a camera to analyze a user's facial expressions and identify their emotional state.

[0428] "Voice analysis" is a technology that uses a microphone to analyze the nuances of a user's voice and words and identify their emotional state.

[0429] "Emotional state" refers to the user's current psychological state, and expresses emotions such as happiness, sadness, fatigue, etc.

[0430] "Recommended product information" is information that suggests products that the user may be interested in based on the user's past purchase history and current emotional state.

[0431] The present invention relates to a smart system that enables a user to launch an application at a shopping facility using a mobile information terminal to improve the shopping experience. Specific embodiments of the present invention will be described in detail below.

[0432] System Configuration

[0433] This system is composed of several elements, including a mobile information terminal, a detector, an information processing device, a voice recognition function, an advertising information system, a facial expression analysis and voice analysis system, and an automatic payment system.

[0434] Hardware and software used

[0435] Mobile information terminal: An electronic device, such as a smartphone or tablet, that runs applications and can be easily operated by the user.

[0436] Detector: A sensor that reads product information, such as a barcode scanner or RFID reader.

[0437] Information processing device: A system that runs on a server or in a cloud computing environment and performs data analysis and information storage.

[0438] Speech recognition function: Technology that converts speech to text, such as Google Speech Recognition API.

[0439] Advertising information system: A platform that manages special sales information and discounted products at visiting facilities.

[0440] Facial and voice analysis systems: Technologies for identifying a user's emotional state. For example, camera devices and Adaptive Recognition APIs, microphone devices and voice analysis software.

[0441] Automated payment systems: payment gateways and mobile payment solutions.

[0442] Processing flow

[0443] 1. Launching the application

[0444] When a user starts an application for a facility using a mobile information terminal, the terminal transmits a request to an information processing device to obtain past purchase history and recommended product information.

[0445] 2. Product Scan

[0446] When a user places an item in the cart, the detector automatically reads the item information and sends it to an information processing device, which analyzes the information and updates the total price in real time.

[0447] 3. Automatic payments

[0448] Once shopping is complete, the terminal will process the payment through an automatic payment system based on the product information received from the information processing device, thereby reducing waiting times at the cash register.

[0449] 4. Product search using voice recognition

[0450] When a user makes a shopping request using the voice recognition function, the terminal converts the voice request into text data and sends it to the information processing device. The information processing device identifies the location of the product based on the user's request and displays it as a map on the terminal.

[0451] 5. Linking with advertising information

[0452] The application displays the latest advertising information and allows users to add discounted items to their shopping list, which causes the information processing device to recalculate the optimal shopping route and display it to the user.

[0453] 6. Product Recommendations Based on Emotion Recognition

[0454] The device's camera and microphone collect the user's facial expressions and voice, which are then sent to an information processing device. The device uses facial expression and voice analysis systems to recognize the user's emotional state and provide recommended product information based on that emotion.

[0455] Specific examples

[0456] Example of displaying past purchase history: When a user launches the app, the app displays the milk and bread they previously purchased and suggests related products.

[0457] Product scanning and real-time price updates: When a user places an apple in the basket, the detector reads the product information and the total price is updated in real time on the terminal screen.

[0458] Example of a voice recognition request: When a user requests by voice, "I want to make curry," the necessary ingredients and their locations are displayed on a map.

[0459] A concrete example of emotion recognition: If a user has a tired expression, the facial expression analysis system will recognize this and suggest relaxing aroma candles or herbal tea.

[0460] Prompt Sentence Examples

[0461] User: Please use the camera to analyze your face.

[0462] ---

[0463] Facial image:

[0464] [User's face image]

[0465] ---

[0466] System: The user appears tired. Try a relaxing scented candle or herbal tea.

[0467] In this way, the present invention allows for personalized shopping suggestions based on the user's emotional state, providing an efficient and satisfying shopping experience.

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] The user launches an application for the facility they are visiting using their mobile information terminal. They enter their user ID, and the terminal sends a request to the server. This request includes the user ID, and the server uses this information to obtain the user's past purchase history and recommended product information, which it then sends to the terminal. The terminal then displays the received information on its screen.

[0471] Step 2:

[0472] When a user places an item in the cart, a detector (such as a barcode scanner or RFID reader) reads the item's identification information. This identification information is sent to the server via the terminal. The server analyzes the item's price and inventory information and calculates the total price in real time. This information is then sent back to the terminal and displayed on the terminal screen.

[0473] Step 3:

[0474] Once the user has completed their purchase, they press the "Pay" button within the application. The device sends all of the purchase information to the server, which then uses an automated payment system to process the payment. Payment completion information is then sent from the server to the device, and the user is notified.

[0475] Step 4:

[0476] When a user wants to search for a specific product, they input a voice command using the voice recognition function. The device converts the user's voice into text using the Google Speech Recognition API and sends it to the server. The server uses this text data to search for product location information and sends it to the device as a map. The device then displays the map on the screen and navigates the user to the optimal shopping route.

[0477] Step 5:

[0478] The latest advertising information is displayed within the application. The user checks discounted items and selects the ID of the discounted item to add it to their shopping list. The device sends this information to the server, which then recalculates the optimal shopping route and sends it to the device. The device then displays the new route.

[0479] Step 6:

[0480] The device's camera and microphone are used to collect the user's facial expressions and voice, which are then sent to the server. The server then uses facial and voice analysis systems to recognize the user's emotional state. It then calculates recommended product information based on the user's emotional state and sends this to the device. The device then displays recommended product information based on the user's emotion on the screen.

[0481] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0482] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0483] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0484] [Second embodiment]

[0485] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0486] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0487] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0488] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0489] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0490] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0491] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0492] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0493] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0494] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0495] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0496] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0497] The present invention relates to a smart system for improving shopping experience in supermarkets, which has multiple functions to enable users to efficiently shop using mobile devices.

[0498] An embodiment of the system is described as follows.

[0499] 1. Starting the application and loading initial data

[0500] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[0501] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[0502] 2. Product scanning and automatic payment

[0503] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[0504] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[0505] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[0506] 3. Voice recognition for product search and navigation

[0507] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[0508] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[0509] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[0510] 4. Linking with flyer information

[0511] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[0512] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[0513] Examples:

[0514] When a user launches the app, a list of past purchases and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[0515] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[0516] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[0517] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[0518] In this way, the system can greatly enhance the user's shopping experience.

[0519] The processing flow will be explained below.

[0520] (Program processing steps)

[0521] Starting the application and loading initial data

[0522] Step 1:

[0523] A user taps to launch an application on their mobile device.

[0524] Step 2:

[0525] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[0526] Step 3:

[0527] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[0528] Step 4:

[0529] The device displays the received data, and the user can check their past purchase history and recommended products.

[0530] Product scanning and automatic payment

[0531] Step 1:

[0532] The user places the items they want to purchase in the cart.

[0533] Step 2:

[0534] An application on the terminal reads product information from sensors attached to the basket.

[0535] Step 3:

[0536] The terminal transmits the product information to the server.

[0537] Step 4:

[0538] The server analyzes the product information and checks whether the corresponding product exists in the database.

[0539] Step 5:

[0540] The server sends product price and stock information to the terminal.

[0541] Step 6:

[0542] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[0543] Step 7:

[0544] When the user completes their purchase, they press the "Confirm payment" button within the application.

[0545] Step 8:

[0546] The terminal transmits the information necessary to finalize the payment to the server.

[0547] Step 9:

[0548] The server automatically settles the payment using the user's registered payment method.

[0549] Step 10:

[0550] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[0551] Voice recognition for product search and navigation

[0552] Step 1:

[0553] The user makes a voice search request for a specific dish or product.

[0554] Step 2:

[0555] The device converts the voice input into text data and sends the information to the server.

[0556] Step 3:

[0557] The server creates a list of required items based on the request and identifies the location of each item.

[0558] Step 4:

[0559] The server sends the product list and placement information to the terminal.

[0560] Step 5:

[0561] The device displays the received product list and location information as a map, and guides the user to the optimal shopping route.

[0562] Linking with flyer information

[0563] Step 1:

[0564] The terminal obtains the latest flyer information from the server.

[0565] Step 2:

[0566] The flyer information acquired by the device is displayed within the application.

[0567] Step 3:

[0568] A user sees a flyer and adds discounted items to a list.

[0569] Step 4:

[0570] The terminal transmits the added discount product information to the server.

[0571] Step 5:

[0572] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[0573] Step 6:

[0574] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[0575] Example 1

[0576] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0577] Conventional supermarket shopping systems have problems with the complicated management of purchased items, payment, and related information acquisition. In particular, the user shopping experience is degraded by manual product scanning, long waiting times at the cash register, the inconvenience of voice-activated product searches, and the difficulty of checking sale information and recommended products. Furthermore, the decentralized nature of these functions and the lack of an integrated system hinders user satisfaction.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0579] In this invention, the server includes means for automatically reading product information and transmitting the relevant product information, means for providing recommended product information based on past purchase history, means for receiving a user's shopping request using voice recognition and displaying product location guidance, means for adding sale items to a shopping list in conjunction with public relations information, and means for displaying the total amount in real time. This enables users to shop efficiently, significantly improving the shopping experience, and is expected to increase overall convenience through the integrated system.

[0580] A "mobile device" is an electronic device that a user can carry and that can run applications.

[0581] "Information processing device software" refers to an application that runs on a mobile device and assists users in shopping.

[0582] A "detection device" is a hardware device that automatically reads product information, such as an RFID reader.

[0583] "Data processing device" refers to a server or cloud service that receives product information, analyzes it, and executes various processes.

[0584] "Automatic payment" is a feature that allows users to complete payment without manual intervention when purchasing products.

[0585] The "voice recognition function" is a technology that converts a user's voice input into text data and processes the content.

[0586] A "shopping request" is a request made by a user to an application to request a specific product or dish.

[0587] "Guide display" is a function that visually shows the location of products and shopping routes to the user.

[0588] "Publicity information" refers to information that includes information about special sales and discounted products offered by supermarkets.

[0589] "Special sale items" are items offered at a lower price than usual.

[0590] "Past purchase history" is a record of data on products previously purchased by a user.

[0591] "Recommended product information" is information that suggests products suitable for a user based on the user's past purchase history, etc.

[0592] "Real-time display" is a function that instantly updates and displays information in response to operations or changes made by the user.

[0593] The "total amount" is the total amount paid by the user for all products purchased.

[0594] This invention relates to an integrated system for significantly improving the shopping experience in supermarkets, which provides a wide range of functions for users to efficiently shop using mobile devices.

[0595] In one embodiment of this system, a user first launches an application using a mobile device. The mobile device used here is a portable information processing device such as a smartphone or tablet. When the user launches the app, the device sends a request to a server to obtain the user's past purchase history and recommended product information. This server is a high-performance data processing device that contains a user database.

[0596] Next, when a user places items in a basket in the supermarket, a detection device such as an RFID reader attached to the basket automatically reads the product information. This product information is sent to a server via the terminal. The server references a product database to check the product price and stock information. The price information is returned to the terminal in real time, and the total price of the items the user has placed in the basket is displayed on the terminal screen.

[0597] Furthermore, if a user wishes to search for a specific dish or product, they can make a request using the voice recognition function. This voice recognition is performed using Google Voice Assistant. The device converts the user's voice input into text data and sends that information to the server. The server then identifies the required product list and product location, and sends it back to the device. Based on this, the device displays a map and navigates the optimal shopping route.

[0598] The application also incorporates the latest supermarket promotional information. Users can check information on special sales and discounted products through the app and add these products to their shopping list. The added product information is synchronized with the server database, and the optimal shopping route is recalculated.

[0599] As a concrete example, when a user launches an app, a list of past purchase history and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, with discounted butter recommended next to them. When a user places an apple in their basket at the supermarket, a sensor detects the apple and sends that information to the device. The device updates the total price in real time based on the information received from the server. When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and where to place them on a map. If the user looks at a flyer and adds discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[0600] Examples of prompts for the generative AI model include, "What are the features of an app that can improve the efficiency of shopping at the supermarket?", "Please explain how a smart system that uses voice recognition to search for specific products works", and "Please explain how a shopping app with an automatic payment system works."

[0601] In this way, the system provides a multifunctional platform for comprehensively improving the user's shopping experience.

[0602] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0603] Step 1:

[0604] Starting the application and getting initial data

[0605] Executor: User and device

[0606] Input: Tap the application icon on your mobile device

[0607] Action: User taps the icon on their mobile device to launch the application.

[0608] Data processing: The device sends an HTTP request to the server, sending the user ID and authentication information.

[0609] Output: Get the user's past purchase history and recommended product information from the server

[0610] Specific operation: The server searches the user database for purchase history, generates product information using a recommendation algorithm, and sends it back to the device. The device analyzes this information and displays it on the screen.

[0611] Step 2:

[0612] Product scanning and information transmission

[0613] Actors: Users, devices, and sensors

[0614] Input: User adds product to cart

[0615] How it works: When an item is placed in the cart, the RFID reader captures the item information.

[0616] Data processing: The sensor reads the data from the RFID tag and sends it to the device. The device then sends this data to the server.

[0617] Output: Get product price and availability information from the server

[0618] Specific operation: The server queries the database based on the product ID, checks the price and stock of the corresponding product, and sends it back to the terminal, which updates the total price on the screen in real time.

[0619] Step 3:

[0620] Product search using voice recognition

[0621] Executor: User and device

[0622] Input: User's voice input (e.g. "I want to make curry")

[0623] Action: User makes a voice request

[0624] Data processing: The device's voice recognition function (Google Voice Assistant) converts the voice into text data and sends it to the server.

[0625] Output: Get the list of required products and their locations from the server

[0626] Specific operation: The server queries the database to obtain a list of the specified products and their locations, and sends this to the device, which then displays it using a map display engine.

[0627] Step 4:

[0628] Check flyer information and add it to your product list

[0629] Executor: User and device

[0630] Input: Publicity information (flyer) within the application

[0631] What happens: A user browses promotions within the app and sees discounted products.

[0632] Data processing: When a user adds an item to their shopping list, the device sends this information to the server.

[0633] Output: Get the updated optimal shopping route from the server

[0634] Specific operation: The server recalculates the optimal route based on the new product list and sends it to the device. The device uses its map display engine to display the updated route to the user.

[0635] Step 5:

[0636] Automatic payment

[0637] Executor: User and device

[0638] Input: User taps the "Pay" button

[0639] Action: User performs payment operation

[0640] Data processing: The device sends payment information (e.g., credit card information) to the server.

[0641] Output: Receives a confirmation message from the server and displays the payment result to the user.

[0642] What happens: The server processes the payment using a payment processor (e.g., a money processing system) and, if successful, sends a confirmation message back to the terminal, which displays this information on its screen.

[0643] (Application example 1)

[0644] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0645] Today's consumers expect an efficient and comfortable shopping experience at supermarkets and other stores. They also have growing demands for reduced checkout wait times, improved in-store navigation, and product search capabilities using voice recognition. While meeting these demands, stores must also operate efficiently. However, the systems needed to achieve this are not yet fully in place, and the development of technology to reduce shopper stress is a challenge.

[0646] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0647] In this invention, the server includes means for a user to launch a store application using a mobile terminal, means for automatically scanning product information with a sensor and transmitting the relevant product information to the server, means for automatically making a payment based on the product information received from the server, means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map, means for adding discounted products to a shopping list in conjunction with the latest information, means for acquiring recommended product information and past purchase history and displaying them to the user, and means for optimally navigating the store based on the map display, thereby enabling the user to shop efficiently and comfortably.

[0648] A "mobile terminal" is an information terminal that can be carried by a user, and mainly refers to a smartphone or tablet terminal.

[0649] A "store application" is software that users use to assist them in shopping at physical stores, and is an application that runs on a mobile device.

[0650] A "sensor" is a device that detects physical phenomena and converts them into electronic signals, and is used to scan product information.

[0651] "Product information" refers to data relating to a specific product, such as product identification information, price information, and inventory information.

[0652] A "server" is a computer system that functions as a central processing unit, receives requests from clients, and processes data.

[0653] "Automatic payment" is a function that automatically calculates the price of the product purchased by the user and completes the payment using a payment method registered in advance.

[0654] "Voice recognition function" is a technology that inputs human speech as voice data and converts it into text data.

[0655] A "shopping request" is a request made by a user to retrieve information about a particular product.

[0656] "Displaying on a map" refers to visually displaying the layout of the store and showing the location of specific products.

[0657] "Latest news" refers to information about the latest discounts and special offers offered by supermarkets and stores.

[0658] "Discounted items" refer to items that are sold at a lower price than the regular price.

[0659] A "shopping list" is a list of items that a user plans to purchase.

[0660] "Recommended product information" refers to information for recommending specific products based on the user's past purchase history and preferences.

[0661] "Past purchase history" refers to a record of information about products previously purchased by a user.

[0662] "Navigation" is a function that guides the user to their destination, showing the location of the desired product within the store and providing the optimal route.

[0663] This invention relates to an advanced support system for users to efficiently shop using mobile devices. The system combines automatic product information scanning, voice recognition search, navigation, and payment functions to enhance the in-store shopping experience.

[0664] Hardware and software used

[0665] Hardware:

[0666] Mobile device (iOS or Android)

[0667] Sensors that scan product information (e.g., barcode scanners)

[0668] software:

[0669] Mobile applications (using Flutter or React Native as development frameworks)

[0670] Server side (using Node.js and Firebase)

[0671] Speech Recognition API (Google Cloud Speech-to-Text)

[0672] Payment system (Stripe, PayPal)

[0673] Database (Firestore)

[0674] Program overview and processing content

[0675] Start the application and load the initial data:

[0676] When a user launches an application on their mobile device, a request is sent to the server to retrieve the user's past purchase history and recommended product information. This information is displayed on the mobile device screen, allowing the user to check a list of past purchases and recommended products.

[0677] Product scanning and automatic payment:

[0678] When a user places an item in their cart, they scan the product barcode with the camera on their mobile device. The scanned product information is sent to a server via the mobile device, where the product price and stock information are confirmed. Once shopping is complete, payment is automatically made within the app, and payment is completed using the payment method registered in advance.

[0679] Voice-activated product search and navigation:

[0680] The user searches for a specific product using voice recognition. The mobile device converts the voice input into text data and sends it to the server. Based on the user's request, the server retrieves the list of required products and their location data and sends it to the mobile device. Based on this, a map of the store is displayed and the user is guided to the optimal shopping route.

[0681] Stay up-to-date with discounted items and add them to your shopping list:

[0682] The application incorporates the latest store information, allowing users to check discounted and special offers. When a discounted item is added to the shopping list, the system recalculates the optimal shopping route on the server and displays it on the mobile device.

[0683] Specific examples

[0684] For example, when a user launches the app on their mobile device, they can voice-input a request such as "I want to get the ingredients for pasta for dinner." Through voice recognition, the app displays ingredients such as "tomato sauce, pasta, olive oil, and cheese" and their locations, and suggests the shortest route within the store.

[0685] When a user places an item in their cart, the product information is instantly sent to the server, and the total price is updated in real time. Once shopping is complete, payment can be completed with just a few taps on a mobile device, eliminating the need to wait in long lines.

[0686] Example prompt sentence:

[0687] "I want to make pasta."

[0688] "Tell me what's for dinner."

[0689] This allows the user to shop efficiently and comfortably.

[0690] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0691] Step 1:

[0692] Launching the application

[0693] When a user launches an application on a mobile device, the device sends the user's authentication information to the server. Input includes a user ID and authentication token. The server receives this, retrieves the user's past purchase history and recommended product information, and returns it to the device. As output, the user's past purchase history and recommended product information are displayed on the device. Specific operations include retrieving and displaying data after sending the authentication information.

[0694] Step 2:

[0695] Product Scan

[0696] The user picks up an item and scans the barcode with the camera on their mobile device. The input contains the item's barcode information. The device reads the barcode and sends the item information to the server. The server retrieves the item's price and stock information from the product database and returns it to the device. The output is the item's price and stock information displayed on the screen. During this process, the device detects the item's barcode and sends the data to the server.

[0697] Step 3:

[0698] Automatic payment

[0699] The user completes their purchase and pays within the app. Input includes the total price of the items purchased by the user and the registered payment method. The terminal sends the payment information to the server, which then calls a payment system (e.g., Stripe or PayPal) to complete the payment. Output is a confirmation message on the terminal that the payment was successful. Specific actions include starting and completing the payment process.

[0700] Step 4:

[0701] Product search using voice recognition

[0702] The user inputs a request by voice, such as "I want to make pasta." The input includes voice data. The device converts the voice data into text using the Google Cloud Speech-to-Text API and sends this text data to the server. Based on the request, the server identifies the list of required products and their locations and returns this to the device. As output, the product list and their locations are displayed on a map. Specific operations include voice recognition and data transmission.

[0703] Step 5:

[0704] Proposal of the best shopping route

[0705] Based on the location data received from the server, the terminal displays the optimal shopping route on a map within the store. The input includes multiple product location data. Based on this data, the terminal calculates the optimal route using a shortest path algorithm and presents it to the user. As an output, the optimal route is displayed on a map. Specific operations include analyzing map data and displaying the route.

[0706] Step 6:

[0707] Linking with the latest information

[0708] Within the application, the user checks the latest discounted items and special offers and adds them to a shopping list. The input includes the latest information data. The server sends the latest information to the user's device, which displays it. When the user adds discounted items to the list, the server again calculates the optimal shopping route and returns it to the device. The output is the updated optimal route displayed on the map. Specific operations include retrieving the latest information and updating the list.

[0709] Step 7:

[0710] Displaying recommended product information

[0711] Based on the user's past purchase history and the database, the server calculates recommended products and sends them to the terminal. The input includes past purchase history data. The server analyzes this and recommends products suitable for the user. The output is the recommended products displayed on the screen. Specific operations include data analysis and information display.

[0712] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0713] This invention relates to a smart system that improves the shopping experience in supermarkets, and in particular to a system that recognizes the user's emotions by combining emotion engines and makes appropriate shopping suggestions based on those emotions. The system has the following functions:

[0714] 1. Starting the application and loading initial data

[0715] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[0716] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[0717] 2. Product scanning and automatic payment

[0718] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[0719] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[0720] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[0721] 3. Voice recognition for product search and navigation

[0722] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[0723] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[0724] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[0725] 4. Linking with flyer information

[0726] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[0727] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[0728] 5. User Emotion Recognition by Emotion Engine

[0729] When a user uses an application, the device uses a camera and microphone to recognize emotions from the user's facial expressions and voice.

[0730] The device sends the collected emotional data to a server, which then uses an emotion engine to analyze the user's emotional state.

[0731] Based on the analysis results, the server provides the user with recommended product information according to their emotional state.

[0732] Examples:

[0733] When a user launches the app, their past purchase history and recommended products are displayed. For example, the milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[0734] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[0735] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[0736] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[0737] If the user looks tired, the emotion engine will recognize this and the server will recommend products that have a relaxing effect (such as aromatic candles or herbal tea).

[0738] In this way, the system can provide various shopping suggestions depending on the user's emotional state, significantly improving the user's shopping experience.

[0739] The processing flow will be explained below.

[0740] Starting the application and loading initial data

[0741] Step 1:

[0742] A user taps to launch an application on their mobile device.

[0743] Step 2:

[0744] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[0745] Step 3:

[0746] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[0747] Step 4:

[0748] The device displays the received data, and the user can check their past purchase history and recommended products.

[0749] Product scanning and automatic payment

[0750] Step 1:

[0751] The user places the items they want to purchase in the cart.

[0752] Step 2:

[0753] The terminal reads product information from sensors attached to the basket.

[0754] Step 3:

[0755] The terminal transmits the product information to the server.

[0756] Step 4:

[0757] The server analyzes the product information and checks whether the corresponding product exists in the database.

[0758] Step 5:

[0759] The server sends product price and stock information to the terminal.

[0760] Step 6:

[0761] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[0762] Step 7:

[0763] When the user completes their purchase, they press the "Confirm payment" button within the application.

[0764] Step 8:

[0765] The terminal transmits the information necessary to finalize the payment to the server.

[0766] Step 9:

[0767] The server automatically settles the payment using the user's registered payment method.

[0768] Step 10:

[0769] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[0770] Voice recognition for product search and navigation

[0771] Step 1:

[0772] The user makes a voice search request for a specific dish or product.

[0773] Step 2:

[0774] The device converts the voice input into text data and sends the information to the server.

[0775] Step 3:

[0776] The server creates a list of required items based on the request and identifies the location of each item.

[0777] Step 4:

[0778] The server sends the product list and location data to the terminal.

[0779] Step 5:

[0780] The device receives a list of products and displays their location information as a map, guiding users to the optimal shopping route.

[0781] Linking with flyer information

[0782] Step 1:

[0783] The terminal obtains the latest flyer information from the server.

[0784] Step 2:

[0785] The flyer information acquired by the device is displayed within the application.

[0786] Step 3:

[0787] A user sees a flyer and adds discounted items to a list.

[0788] Step 4:

[0789] The terminal transmits the added discount product information to the server.

[0790] Step 5:

[0791] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[0792] Step 6:

[0793] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[0794] Recognizing user emotions with an emotion engine

[0795] Step 1:

[0796] When a user is using an application, the device uses a camera and microphone to collect the user's facial expressions and voice.

[0797] Step 2:

[0798] The terminal transmits the collected facial expression and voice data to the server.

[0799] Step 3:

[0800] The server uses an emotion engine to analyze the user's emotional state.

[0801] Step 4:

[0802] Based on the analysis results, the server creates recommended product information according to the user's emotional state and sends it to the terminal.

[0803] Step 5:

[0804] The device will display recommended products that reflect your emotional state.

[0805] Examples:

[0806] If a user has a tired expression while using the app, the device will recognize that expression with its camera and send it to the server.

[0807] The server uses an emotion engine to detect "fatigue," and then lists recommended products that have a relaxing effect (e.g., aroma candles and herbal tea) and sends them to the device.

[0808] The device will suggest products that will help users feel refreshed by displaying "Recommended products: Aroma candles, herbal tea" on the screen.

[0809] In this way, the system can make individualized suggestions based on the user's emotional state, making the shopping experience more personal and convenient.

[0810] Example 2

[0811] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0812] Modern supermarkets require systems that enable users to enjoy an efficient and stress-free shopping experience. However, existing systems struggle to promote user shopping behavior, and the product search and payment processes can be time-consuming. Furthermore, they do not consider user emotions when suggesting products, and thus fail to provide a personalized shopping experience.

[0813] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0814] In this invention, the server includes means for a user to launch a store management system application using a mobile communication device, means for automatically reading product information using a sensor and transmitting the product information to a data processing device, means for automatically making a payment based on the product information received from the data processing device, means for receiving a user's shopping request using a voice recognition function and displaying the location of the corresponding product on a guidance display device, means for adding discounted products to a shopping list in conjunction with advertising information, and means for analyzing emotions from the user's facial expressions and voice using an emotion recognition engine and providing recommended product information based on the analysis results. This enables users to not only efficiently search for and purchase products, but also receive personalized shopping suggestions.

[0815] A "mobile communication device" is a communication device that a user can carry around, and primarily refers to devices such as smartphones and tablets.

[0816] A "store management system" refers to an application used to manage products and support users' shopping experiences in retail stores such as supermarkets.

[0817] "Sensor" means a device for detecting and reading product information, including RFID readers and barcode scanners.

[0818] "Data processing device" refers to a server or database that analyzes product information sent from the sensor and provides the necessary information to the user.

[0819] "Automatic payment" refers to the process of automatically paying the total amount of the products purchased by the user using a payment method registered in advance.

[0820] "Voice recognition function" refers to technology that converts a user's voice input into text data and then provides the system with appropriate information based on that data.

[0821] A "guidance display device" is a device used to display product locations and optimal shopping routes within a store, and primarily refers to the screen of a user's mobile communication device.

[0822] "Advertising information" refers to supermarket flyers and discounted product information, and is used to provide users with special sale information.

[0823] An "emotion recognition engine" refers to an algorithm that analyzes a user's emotional state from their facial expressions and voice, and recommends appropriate products based on the results.

[0824] "Personalized shopping suggestions" refers to a feature that suggests products and services that are suitable for each individual user based on their past purchase history and emotional state.

[0825] This invention relates to a store management system for improving the shopping experience in supermarkets, particularly a system that uses mobile communication devices, data processing devices, sensors, and an emotion recognition engine to analyze a user's emotions and make appropriate suggestions based on the analysis.

[0826] When a user launches an application, the mobile communication device (e.g., a smartphone or tablet) sends a request to a data processing device (server) to obtain the user's past purchase history and recommended product information. This allows the device to display a list of previously purchased products and the store's recommended products. As a specific example, previously purchased milk or bread may be displayed, with discounted butter recommended next to them.

[0827] When a user selects an item in a supermarket, a sensor (e.g., an RFID reader or barcode scanner) automatically reads the item's information and transmits it to a data processing device via a mobile communication device. The data processing device analyzes the item's information, checks the price and stock information of the item, and displays the total price on the terminal in real time. For example, when an apple is placed in a basket, the sensor detects the apple and transmits the information to the terminal, instantly updating the total price on the screen.

[0828] Furthermore, if a user wishes to search for a specific dish or product, the request is made using a voice recognition function. The mobile communication device converts the user's voice input into text data and sends it to the data processing device. The data processing device creates a list of necessary products based on the user's request and identifies the location of each product. Based on the received information, the mobile communication device displays the product locations on a map and navigates the optimal shopping route. For example, in response to a request to "make curry," a list of necessary ingredients and their locations are displayed on a map.

[0829] The application incorporates the latest store advertising information, allowing users to check information on discounted and special sale items. Users can add discounted items to their shopping list, and this information is synchronized with the data processing device's database. This allows the optimal shopping route to be recalculated and displayed on the mobile communication device. For example, if a user sees a flyer and adds a discounted curry roux to their shopping list, the optimal route recalculated by the data processing device will be displayed on the device.

[0830] Furthermore, to understand the user's emotional state, facial expressions and voice are collected using the camera and microphone of the mobile communication device and analyzed by an emotion recognition engine. The emotion recognition engine analyzes the user's emotional state and provides appropriate product information based on that analysis. For example, if the user looks tired, it may recommend relaxing aroma candles or herbal tea.

[0831] This allows users to not only efficiently search and purchase products, but also receive personalized shopping suggestions.

[0832] Prompt statement example:

[0833] 1. "I want to develop a system to simplify supermarket shopping. It includes functionality to recognize user emotions and recommend products based on those emotions. Can you please explain the detailed process of this system?"

[0834] 2. "Please explain in detail how smart systems can be used to improve the shopping experience at supermarkets. In particular, I would like to know about the use of voice recognition and emotion engines."

[0835] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0836] Step 1:

[0837] A user launches a store management system application using a mobile communication device. When the user launches the app, the device sends a request for initial data to the server. The server obtains the user's past purchase history and recommended product information and sends this to the device. The device displays the received data on the screen, allowing the user to view a list of products they have purchased in the past and recommended products from the supermarket. For example, a list of milk, bread, and butter that the user has purchased in the past, as well as discounted products, is displayed.

[0838] input:

[0839] User launches app

[0840] output:

[0841] Past purchase history

[0842] Recommended product information

[0843] Step 2:

[0844] When a user places an item in a basket at a supermarket, a sensor automatically reads the item's information. This information is sent to a server via the terminal. The server analyzes the received product information and checks the price and stock information of the item. The analyzed information is sent to the terminal in real time, and the terminal updates and displays the total price. For example, when a user puts an apple in the basket, the sensor reads the apple's information, and the terminal updates the total price in real time.

[0845] input:

[0846] Product information added to the user's cart

[0847] output:

[0848] Real-time total amount

[0849] Product price and availability information

[0850] Step 3:

[0851] When a user wants to search for a specific dish or product, they make a request using the voice recognition function. The user's voice input is converted into text data by the device and sent to the server. The server creates a product list based on the user's voice request and identifies the location of each product. The device uses the information from the server to display the product locations on a map and navigates the user to the optimal shopping route. For example, if a user makes a voice request such as "I want to make curry," a list of the necessary ingredients and their locations will be displayed on a map.

[0852] input:

[0853] User voice request

[0854] output:

[0855] Location of food and products

[0856] The best shopping route

[0857] Step 4:

[0858] The user can check the latest store advertising information embedded in the device. The user can then check the displayed discounted and special sale items and add the discounted items to their shopping list. The added product information is synchronized with the server's database, and the server sends a recalculated optimal shopping route to the device. The device displays the new route information, allowing the user to shop efficiently. For example, if the user sees a flyer and adds a discounted curry roux to their shopping list, a new optimal route will be displayed on the device.

[0859] input:

[0860] Adding discounted products to users

[0861] output:

[0862] Updated shopping route

[0863] Discounted product information

[0864] Step 5:

[0865] While the user is using the application, the device uses a camera and microphone to collect the user's facial expressions and voice. The device sends the collected emotional data to a server, which then uses an emotion recognition engine to analyze the user's emotional state. Based on the analysis results, the server generates recommended product information according to the user's emotional state and sends it to the device. The device then displays this information to the user. For example, if the user looks tired, the emotion recognition engine will recommend products with a relaxing effect (aroma candles or herbal tea).

[0866] input:

[0867] User emotion data

[0868] output:

[0869] Emotion-based product recommendations

[0870] (Application example 2)

[0871] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0872] Traditional shopping experiences in brick-and-mortar stores often require users to search around the store or wait in line at the cash register for long periods of time, resulting in inefficiency. Furthermore, product recommendations are often made without taking the user's emotional state into consideration, which can lead to lower user satisfaction. There is a need to improve the user's shopping experience by solving these problems.

[0873] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for the user to launch an application for the facility using a mobile information terminal; means for automatically reading product information using a detector and transmitting the relevant product information to the information processing device; means for making an automatic payment based on the product information received from the information processing device; means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map; means for adding discounted products to a shopping list in conjunction with advertising information; and means for recognizing the user's emotional state using facial expression analysis and voice analysis and providing recommended product information based on the emotional state. This makes it possible to make individual shopping suggestions according to the user's emotional state, thereby providing an efficient and satisfying shopping experience.

[0874] A "personal digital assistant" is an electronic device that can be carried by a user and is a device for executing applications and performing data communication.

[0875] "Store facilities" refer to physical stores such as supermarkets and other retail outlets where customers visit in person to shop.

[0876] An "application" is a program that runs on a mobile information terminal and provides support functions to help users shop efficiently.

[0877] A "detector" is a device that automatically reads product information and is a sensor that obtains information using barcodes, RFID, etc.

[0878] The "information processing device" is a computer system such as a server, which receives, analyzes, and stores the product information transmitted from the detector.

[0879] "Automatic payment" is a system in which a user automatically pays for products purchased via an information processing device without having to line up at the cash register.

[0880] The "voice recognition function" is a technology that converts the user's voice into text data, and is a function that receives and analyzes shopping requests.

[0881] "Advertising information" is information about discounted products and special sales offered by the facility, and is displayed within the application to support the user's shopping experience.

[0882] "Discounted products" are products offered at special prices based on advertising information, and are intended to encourage users to consume.

[0883] "Facial expression analysis" is a technology that uses a camera to analyze a user's facial expressions and identify their emotional state.

[0884] "Voice analysis" is a technology that uses a microphone to analyze the nuances of a user's voice and words and identify their emotional state.

[0885] "Emotional state" refers to the user's current psychological state, and expresses emotions such as happiness, sadness, fatigue, etc.

[0886] "Recommended product information" is information that suggests products that the user may be interested in based on the user's past purchase history and current emotional state.

[0887] The present invention relates to a smart system that enables a user to launch an application at a shopping facility using a mobile information terminal to improve the shopping experience. Specific embodiments of the present invention will be described in detail below.

[0888] System Configuration

[0889] This system is composed of several elements, including a mobile information terminal, a detector, an information processing device, a voice recognition function, an advertising information system, a facial expression analysis and voice analysis system, and an automatic payment system.

[0890] Hardware and software used

[0891] Mobile information terminal: An electronic device, such as a smartphone or tablet, that runs applications and can be easily operated by the user.

[0892] Detector: A sensor that reads product information, such as a barcode scanner or RFID reader.

[0893] Information processing device: A system that runs on a server or in a cloud computing environment and performs data analysis and information storage.

[0894] Speech recognition function: Technology that converts speech to text, such as Google Speech Recognition API.

[0895] Advertising information system: A platform that manages special sales information and discounted products at visiting facilities.

[0896] Facial and voice analysis systems: Technologies for identifying a user's emotional state. For example, camera devices and Adaptive Recognition APIs, microphone devices and voice analysis software.

[0897] Automated payment systems: payment gateways and mobile payment solutions.

[0898] Processing flow

[0899] 1. Launching the application

[0900] When a user starts an application for a facility using a mobile information terminal, the terminal transmits a request to an information processing device to obtain past purchase history and recommended product information.

[0901] 2. Product Scan

[0902] When a user places an item in the cart, the detector automatically reads the item information and sends it to an information processing device, which analyzes the information and updates the total price in real time.

[0903] 3. Automatic payments

[0904] Once shopping is complete, the terminal will process the payment through an automatic payment system based on the product information received from the information processing device, thereby reducing waiting times at the cash register.

[0905] 4. Product search using voice recognition

[0906] When a user makes a shopping request using the voice recognition function, the terminal converts the voice request into text data and sends it to the information processing device. The information processing device identifies the location of the product based on the user's request and displays it as a map on the terminal.

[0907] 5. Linking with advertising information

[0908] The application displays the latest advertising information and allows users to add discounted items to their shopping list, which causes the information processing device to recalculate the optimal shopping route and display it to the user.

[0909] 6. Product Recommendations Based on Emotion Recognition

[0910] The device's camera and microphone collect the user's facial expressions and voice, which are then sent to an information processing device. The device uses facial expression and voice analysis systems to recognize the user's emotional state and provide recommended product information based on that emotion.

[0911] Specific examples

[0912] Example of displaying past purchase history: When a user launches the app, the app displays the milk and bread they previously purchased and suggests related products.

[0913] Product scanning and real-time price updates: When a user places an apple in the basket, the detector reads the product information and the total price is updated in real time on the terminal screen.

[0914] Example of a voice recognition request: When a user requests by voice, "I want to make curry," the necessary ingredients and their locations are displayed on a map.

[0915] A concrete example of emotion recognition: If a user has a tired expression, the facial expression analysis system will recognize this and suggest relaxing aroma candles or herbal tea.

[0916] Prompt Sentence Examples

[0917] User: Please use the camera to analyze your face.

[0918] ---

[0919] Facial image:

[0920] [User's face image]

[0921] ---

[0922] System: The user appears tired. Try a relaxing scented candle or herbal tea.

[0923] In this way, the present invention allows for personalized shopping suggestions based on the user's emotional state, providing an efficient and satisfying shopping experience.

[0924] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0925] Step 1:

[0926] The user launches an application for the facility they are visiting using their mobile information terminal. They enter their user ID, and the terminal sends a request to the server. This request includes the user ID, and the server uses this information to obtain the user's past purchase history and recommended product information, which it then sends to the terminal. The terminal then displays the received information on its screen.

[0927] Step 2:

[0928] When a user places an item in the cart, a detector (such as a barcode scanner or RFID reader) reads the item's identification information. This identification information is sent to the server via the terminal. The server analyzes the item's price and inventory information and calculates the total price in real time. This information is then sent back to the terminal and displayed on the terminal screen.

[0929] Step 3:

[0930] Once the user has completed their purchase, they press the "Pay" button within the application. The device sends all of the purchase information to the server, which then uses an automated payment system to process the payment. Payment completion information is then sent from the server to the device, and the user is notified.

[0931] Step 4:

[0932] When a user wants to search for a specific product, they input a voice command using the voice recognition function. The device converts the user's voice into text using the Google Speech Recognition API and sends it to the server. The server uses this text data to search for product location information and sends it to the device as a map. The device then displays the map on the screen and navigates the user to the optimal shopping route.

[0933] Step 5:

[0934] The latest advertising information is displayed within the application. The user checks discounted items and selects the ID of the discounted item to add it to their shopping list. The device sends this information to the server, which then recalculates the optimal shopping route and sends it to the device. The device then displays the new route.

[0935] Step 6:

[0936] The device's camera and microphone are used to collect the user's facial expressions and voice, which are then sent to the server. The server then uses facial and voice analysis systems to recognize the user's emotional state. It then calculates recommended product information based on the user's emotional state and sends this to the device. The device then displays recommended product information based on the user's emotion on the screen.

[0937] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0938] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0939] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0940] [Third embodiment]

[0941] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0942] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0943] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0944] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0945] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0946] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0947] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0948] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0949] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0950] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0951] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0952] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0953] The present invention relates to a smart system for improving shopping experience in supermarkets, which has multiple functions to enable users to efficiently shop using mobile devices.

[0954] An embodiment of the system is described as follows.

[0955] 1. Starting the application and loading initial data

[0956] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[0957] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[0958] 2. Product scanning and automatic payment

[0959] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[0960] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[0961] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[0962] 3. Voice recognition for product search and navigation

[0963] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[0964] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[0965] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[0966] 4. Linking with flyer information

[0967] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[0968] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[0969] Examples:

[0970] When a user launches the app, a list of past purchases and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[0971] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[0972] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[0973] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[0974] In this way, the system can greatly enhance the user's shopping experience.

[0975] The processing flow will be explained below.

[0976] (Program processing steps)

[0977] Starting the application and loading initial data

[0978] Step 1:

[0979] A user taps to launch an application on their mobile device.

[0980] Step 2:

[0981] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[0982] Step 3:

[0983] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[0984] Step 4:

[0985] The device displays the received data, and the user can check their past purchase history and recommended products.

[0986] Product scanning and automatic payment

[0987] Step 1:

[0988] The user places the items they want to purchase in the cart.

[0989] Step 2:

[0990] An application on the terminal reads product information from sensors attached to the basket.

[0991] Step 3:

[0992] The terminal transmits the product information to the server.

[0993] Step 4:

[0994] The server analyzes the product information and checks whether the corresponding product exists in the database.

[0995] Step 5:

[0996] The server sends product price and stock information to the terminal.

[0997] Step 6:

[0998] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[0999] Step 7:

[1000] When the user completes their purchase, they press the "Confirm payment" button within the application.

[1001] Step 8:

[1002] The terminal transmits the information necessary to finalize the payment to the server.

[1003] Step 9:

[1004] The server automatically settles the payment using the user's registered payment method.

[1005] Step 10:

[1006] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[1007] Voice recognition for product search and navigation

[1008] Step 1:

[1009] The user makes a voice search request for a specific dish or product.

[1010] Step 2:

[1011] The device converts the voice input into text data and sends the information to the server.

[1012] Step 3:

[1013] The server creates a list of required items based on the request and identifies the location of each item.

[1014] Step 4:

[1015] The server sends the product list and placement information to the terminal.

[1016] Step 5:

[1017] The device displays the received product list and location information as a map, and guides the user to the optimal shopping route.

[1018] Linking with flyer information

[1019] Step 1:

[1020] The terminal obtains the latest flyer information from the server.

[1021] Step 2:

[1022] The flyer information acquired by the device is displayed within the application.

[1023] Step 3:

[1024] A user sees a flyer and adds discounted items to a list.

[1025] Step 4:

[1026] The terminal transmits the added discount product information to the server.

[1027] Step 5:

[1028] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[1029] Step 6:

[1030] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[1031] Example 1

[1032] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] Conventional supermarket shopping systems have problems with the complicated management of purchased items, payment, and related information acquisition. In particular, the user shopping experience is degraded by manual product scanning, long waiting times at the cash register, the inconvenience of voice-activated product searches, and the difficulty of checking sale information and recommended products. Furthermore, the decentralized nature of these functions and the lack of an integrated system hinders user satisfaction.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1035] In this invention, the server includes means for automatically reading product information and transmitting the relevant product information, means for providing recommended product information based on past purchase history, means for receiving a user's shopping request using voice recognition and displaying product location guidance, means for adding sale items to a shopping list in conjunction with public relations information, and means for displaying the total amount in real time. This enables users to shop efficiently, significantly improving the shopping experience, and is expected to increase overall convenience through the integrated system.

[1036] A "mobile device" is an electronic device that a user can carry and that can run applications.

[1037] "Information processing device software" refers to an application that runs on a mobile device and assists users in shopping.

[1038] A "detection device" is a hardware device that automatically reads product information, such as an RFID reader.

[1039] "Data processing device" refers to a server or cloud service that receives product information, analyzes it, and executes various processes.

[1040] "Automatic payment" is a feature that allows users to complete payment without manual intervention when purchasing products.

[1041] The "voice recognition function" is a technology that converts a user's voice input into text data and processes the content.

[1042] A "shopping request" is a request made by a user to an application to request a specific product or dish.

[1043] "Guide display" is a function that visually shows the location of products and shopping routes to the user.

[1044] "Publicity information" refers to information that includes information about special sales and discounted products offered by supermarkets.

[1045] "Special sale items" are items offered at a lower price than usual.

[1046] "Past purchase history" is a record of data on products previously purchased by a user.

[1047] "Recommended product information" is information that suggests products suitable for a user based on the user's past purchase history, etc.

[1048] "Real-time display" is a function that instantly updates and displays information in response to operations or changes made by the user.

[1049] The "total amount" is the total amount paid by the user for all products purchased.

[1050] This invention relates to an integrated system for significantly improving the shopping experience in supermarkets, which provides a wide range of functions for users to efficiently shop using mobile devices.

[1051] In one embodiment of this system, a user first launches an application using a mobile device. The mobile device used here is a portable information processing device such as a smartphone or tablet. When the user launches the app, the device sends a request to a server to obtain the user's past purchase history and recommended product information. This server is a high-performance data processing device that contains a user database.

[1052] Next, when a user places items in a basket in the supermarket, a detection device such as an RFID reader attached to the basket automatically reads the product information. This product information is sent to a server via the terminal. The server references a product database to check the product price and stock information. The price information is returned to the terminal in real time, and the total price of the items the user has placed in the basket is displayed on the terminal screen.

[1053] Furthermore, if a user wishes to search for a specific dish or product, they can make a request using the voice recognition function. This voice recognition is performed using Google Voice Assistant. The device converts the user's voice input into text data and sends that information to the server. The server then identifies the required product list and product location, and sends it back to the device. Based on this, the device displays a map and navigates the optimal shopping route.

[1054] The application also incorporates the latest supermarket promotional information. Users can check information on special sales and discounted products through the app and add these products to their shopping list. The added product information is synchronized with the server database, and the optimal shopping route is recalculated.

[1055] As a concrete example, when a user launches an app, a list of past purchase history and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, with discounted butter recommended next to them. When a user places an apple in their basket at the supermarket, a sensor detects the apple and sends that information to the device. The device updates the total price in real time based on the information received from the server. When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and where to place them on a map. If the user looks at a flyer and adds discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[1056] Examples of prompts for the generative AI model include, "What are the features of an app that can improve the efficiency of shopping at the supermarket?", "Please explain how a smart system that uses voice recognition to search for specific products works", and "Please explain how a shopping app with an automatic payment system works."

[1057] In this way, the system provides a multifunctional platform for comprehensively improving the user's shopping experience.

[1058] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1059] Step 1:

[1060] Starting the application and getting initial data

[1061] Executor: User and device

[1062] Input: Tap the application icon on your mobile device

[1063] Action: User taps the icon on their mobile device to launch the application.

[1064] Data processing: The device sends an HTTP request to the server, sending the user ID and authentication information.

[1065] Output: Get the user's past purchase history and recommended product information from the server

[1066] Specific operation: The server searches the user database for purchase history, generates product information using a recommendation algorithm, and sends it back to the device. The device analyzes this information and displays it on the screen.

[1067] Step 2:

[1068] Product scanning and information transmission

[1069] Actors: Users, devices, and sensors

[1070] Input: User adds product to cart

[1071] How it works: When an item is placed in the cart, the RFID reader captures the item information.

[1072] Data processing: The sensor reads the data from the RFID tag and sends it to the device. The device then sends this data to the server.

[1073] Output: Get product price and availability information from the server

[1074] Specific operation: The server queries the database based on the product ID, checks the price and stock of the corresponding product, and sends it back to the terminal, which updates the total price on the screen in real time.

[1075] Step 3:

[1076] Product search using voice recognition

[1077] Executor: User and device

[1078] Input: User's voice input (e.g. "I want to make curry")

[1079] Action: User makes a voice request

[1080] Data processing: The device's voice recognition function (Google Voice Assistant) converts the voice into text data and sends it to the server.

[1081] Output: Get the list of required products and their locations from the server

[1082] Specific operation: The server queries the database to obtain a list of the specified products and their locations, and sends this to the device, which then displays it using a map display engine.

[1083] Step 4:

[1084] Check flyer information and add it to your product list

[1085] Executor: User and device

[1086] Input: Publicity information (flyer) within the application

[1087] What happens: A user browses promotions within the app and sees discounted products.

[1088] Data processing: When a user adds an item to their shopping list, the device sends this information to the server.

[1089] Output: Get the updated optimal shopping route from the server

[1090] Specific operation: The server recalculates the optimal route based on the new product list and sends it to the device. The device uses its map display engine to display the updated route to the user.

[1091] Step 5:

[1092] Automatic payment

[1093] Executor: User and device

[1094] Input: User taps the "Pay" button

[1095] Action: User performs payment operation

[1096] Data processing: The device sends payment information (e.g., credit card information) to the server.

[1097] Output: Receives a confirmation message from the server and displays the payment result to the user.

[1098] What happens: The server processes the payment using a payment processor (e.g., a money processing system) and, if successful, sends a confirmation message back to the terminal, which displays this information on its screen.

[1099] (Application example 1)

[1100] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1101] Today's consumers expect an efficient and comfortable shopping experience at supermarkets and other stores. They also have growing demands for reduced checkout wait times, improved in-store navigation, and product search capabilities using voice recognition. While meeting these demands, stores must also operate efficiently. However, the systems needed to achieve this are not yet fully in place, and the development of technology to reduce shopper stress is a challenge.

[1102] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1103] In this invention, the server includes means for a user to launch a store application using a mobile terminal, means for automatically scanning product information with a sensor and transmitting the relevant product information to the server, means for automatically making a payment based on the product information received from the server, means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map, means for adding discounted products to a shopping list in conjunction with the latest information, means for acquiring recommended product information and past purchase history and displaying them to the user, and means for optimally navigating the store based on the map display, thereby enabling the user to shop efficiently and comfortably.

[1104] A "mobile terminal" is an information terminal that can be carried by a user, and mainly refers to a smartphone or tablet terminal.

[1105] A "store application" is software that users use to assist them in shopping at physical stores, and is an application that runs on a mobile device.

[1106] A "sensor" is a device that detects physical phenomena and converts them into electronic signals, and is used to scan product information.

[1107] "Product information" refers to data relating to a specific product, such as product identification information, price information, and inventory information.

[1108] A "server" is a computer system that functions as a central processing unit, receives requests from clients, and processes data.

[1109] "Automatic payment" is a function that automatically calculates the price of the product purchased by the user and completes the payment using a payment method registered in advance.

[1110] "Voice recognition function" is a technology that inputs human speech as voice data and converts it into text data.

[1111] A "shopping request" is a request made by a user to retrieve information about a particular product.

[1112] "Displaying on a map" refers to visually displaying the layout of the store and showing the location of specific products.

[1113] "Latest news" refers to information about the latest discounts and special offers offered by supermarkets and stores.

[1114] "Discounted items" refer to items that are sold at a lower price than the regular price.

[1115] A "shopping list" is a list of items that a user plans to purchase.

[1116] "Recommended product information" refers to information for recommending specific products based on the user's past purchase history and preferences.

[1117] "Past purchase history" refers to a record of information about products previously purchased by a user.

[1118] "Navigation" is a function that guides the user to their destination, showing the location of the desired product within the store and providing the optimal route.

[1119] This invention relates to an advanced support system for users to efficiently shop using mobile devices. The system combines automatic product information scanning, voice recognition search, navigation, and payment functions to enhance the in-store shopping experience.

[1120] Hardware and software used

[1121] Hardware:

[1122] Mobile device (iOS or Android)

[1123] Sensors that scan product information (e.g., barcode scanners)

[1124] software:

[1125] Mobile applications (using Flutter or React Native as development frameworks)

[1126] Server side (using Node.js and Firebase)

[1127] Speech Recognition API (Google Cloud Speech-to-Text)

[1128] Payment system (Stripe, PayPal)

[1129] Database (Firestore)

[1130] Program overview and processing content

[1131] Start the application and load the initial data:

[1132] When a user launches an application on their mobile device, a request is sent to the server to retrieve the user's past purchase history and recommended product information. This information is displayed on the mobile device screen, allowing the user to check a list of past purchases and recommended products.

[1133] Product scanning and automatic payment:

[1134] When a user places an item in their cart, they scan the product barcode with the camera on their mobile device. The scanned product information is sent to a server via the mobile device, where the product price and stock information are confirmed. Once shopping is complete, payment is automatically made within the app, and payment is completed using the payment method registered in advance.

[1135] Voice-activated product search and navigation:

[1136] The user searches for a specific product using voice recognition. The mobile device converts the voice input into text data and sends it to the server. Based on the user's request, the server retrieves the list of required products and their location data and sends it to the mobile device. Based on this, a map of the store is displayed and the user is guided to the optimal shopping route.

[1137] Stay up-to-date with discounted items and add them to your shopping list:

[1138] The application incorporates the latest store information, allowing users to check discounted and special offers. When a discounted item is added to the shopping list, the system recalculates the optimal shopping route on the server and displays it on the mobile device.

[1139] Specific examples

[1140] For example, when a user launches the app on their mobile device, they can voice-input a request such as "I want to get the ingredients for pasta for dinner." Through voice recognition, the app displays ingredients such as "tomato sauce, pasta, olive oil, and cheese" and their locations, and suggests the shortest route within the store.

[1141] When a user places an item in their cart, the product information is instantly sent to the server, and the total price is updated in real time. Once shopping is complete, payment can be completed with just a few taps on a mobile device, eliminating the need to wait in long lines.

[1142] Example prompt sentence:

[1143] "I want to make pasta."

[1144] "Tell me what's for dinner."

[1145] This allows the user to shop efficiently and comfortably.

[1146] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1147] Step 1:

[1148] Launching the application

[1149] When a user launches an application on a mobile device, the device sends the user's authentication information to the server. Input includes a user ID and authentication token. The server receives this, retrieves the user's past purchase history and recommended product information, and returns it to the device. As output, the user's past purchase history and recommended product information are displayed on the device. Specific operations include retrieving and displaying data after sending the authentication information.

[1150] Step 2:

[1151] Product Scan

[1152] The user picks up an item and scans the barcode with the camera on their mobile device. The input contains the item's barcode information. The device reads the barcode and sends the item information to the server. The server retrieves the item's price and stock information from the product database and returns it to the device. The output is the item's price and stock information displayed on the screen. During this process, the device detects the item's barcode and sends the data to the server.

[1153] Step 3:

[1154] Automatic payment

[1155] The user completes their purchase and pays within the app. Input includes the total price of the items purchased by the user and the registered payment method. The terminal sends the payment information to the server, which then calls a payment system (e.g., Stripe or PayPal) to complete the payment. Output is a confirmation message on the terminal that the payment was successful. Specific actions include starting and completing the payment process.

[1156] Step 4:

[1157] Product search using voice recognition

[1158] The user inputs a request by voice, such as "I want to make pasta." The input includes voice data. The device converts the voice data into text using the Google Cloud Speech-to-Text API and sends this text data to the server. Based on the request, the server identifies the list of required products and their locations and returns this to the device. As output, the product list and their locations are displayed on a map. Specific operations include voice recognition and data transmission.

[1159] Step 5:

[1160] Proposal of the best shopping route

[1161] Based on the location data received from the server, the terminal displays the optimal shopping route on a map within the store. The input includes multiple product location data. Based on this data, the terminal calculates the optimal route using a shortest path algorithm and presents it to the user. As an output, the optimal route is displayed on a map. Specific operations include analyzing map data and displaying the route.

[1162] Step 6:

[1163] Linking with the latest information

[1164] Within the application, the user checks the latest discounted items and special offers and adds them to a shopping list. The input includes the latest information data. The server sends the latest information to the user's device, which displays it. When the user adds discounted items to the list, the server again calculates the optimal shopping route and returns it to the device. The output is the updated optimal route displayed on the map. Specific operations include retrieving the latest information and updating the list.

[1165] Step 7:

[1166] Displaying recommended product information

[1167] Based on the user's past purchase history and the database, the server calculates recommended products and sends them to the terminal. The input includes past purchase history data. The server analyzes this and recommends products suitable for the user. The output is the recommended products displayed on the screen. Specific operations include data analysis and information display.

[1168] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1169] This invention relates to a smart system that improves the shopping experience in supermarkets, and in particular to a system that recognizes the user's emotions by combining emotion engines and makes appropriate shopping suggestions based on those emotions. The system has the following functions:

[1170] 1. Starting the application and loading initial data

[1171] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[1172] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[1173] 2. Product scanning and automatic payment

[1174] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[1175] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[1176] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[1177] 3. Voice recognition for product search and navigation

[1178] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[1179] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[1180] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[1181] 4. Linking with flyer information

[1182] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[1183] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[1184] 5. User Emotion Recognition by Emotion Engine

[1185] When a user uses an application, the device uses a camera and microphone to recognize emotions from the user's facial expressions and voice.

[1186] The device sends the collected emotional data to a server, which then uses an emotion engine to analyze the user's emotional state.

[1187] Based on the analysis results, the server provides the user with recommended product information according to their emotional state.

[1188] Examples:

[1189] When a user launches the app, their past purchase history and recommended products are displayed. For example, the milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[1190] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[1191] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[1192] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[1193] If the user looks tired, the emotion engine will recognize this and the server will recommend products that have a relaxing effect (such as aromatic candles or herbal tea).

[1194] In this way, the system can provide various shopping suggestions depending on the user's emotional state, significantly improving the user's shopping experience.

[1195] The processing flow will be explained below.

[1196] Starting the application and loading initial data

[1197] Step 1:

[1198] A user taps to launch an application on their mobile device.

[1199] Step 2:

[1200] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[1201] Step 3:

[1202] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[1203] Step 4:

[1204] The device displays the received data, and the user can check their past purchase history and recommended products.

[1205] Product scanning and automatic payment

[1206] Step 1:

[1207] The user places the items they want to purchase in the cart.

[1208] Step 2:

[1209] The terminal reads product information from sensors attached to the basket.

[1210] Step 3:

[1211] The terminal transmits the product information to the server.

[1212] Step 4:

[1213] The server analyzes the product information and checks whether the corresponding product exists in the database.

[1214] Step 5:

[1215] The server sends product price and stock information to the terminal.

[1216] Step 6:

[1217] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[1218] Step 7:

[1219] When the user completes their purchase, they press the "Confirm payment" button within the application.

[1220] Step 8:

[1221] The terminal transmits the information necessary to finalize the payment to the server.

[1222] Step 9:

[1223] The server automatically settles the payment using the user's registered payment method.

[1224] Step 10:

[1225] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[1226] Voice recognition for product search and navigation

[1227] Step 1:

[1228] The user makes a voice search request for a specific dish or product.

[1229] Step 2:

[1230] The device converts the voice input into text data and sends the information to the server.

[1231] Step 3:

[1232] The server creates a list of required items based on the request and identifies the location of each item.

[1233] Step 4:

[1234] The server sends the product list and location data to the terminal.

[1235] Step 5:

[1236] The device receives a list of products and displays their location information as a map, guiding users to the optimal shopping route.

[1237] Linking with flyer information

[1238] Step 1:

[1239] The terminal obtains the latest flyer information from the server.

[1240] Step 2:

[1241] The flyer information acquired by the device is displayed within the application.

[1242] Step 3:

[1243] A user sees a flyer and adds discounted items to a list.

[1244] Step 4:

[1245] The terminal transmits the added discount product information to the server.

[1246] Step 5:

[1247] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[1248] Step 6:

[1249] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[1250] Recognizing user emotions with an emotion engine

[1251] Step 1:

[1252] When a user is using an application, the device uses a camera and microphone to collect the user's facial expressions and voice.

[1253] Step 2:

[1254] The terminal transmits the collected facial expression and voice data to the server.

[1255] Step 3:

[1256] The server uses an emotion engine to analyze the user's emotional state.

[1257] Step 4:

[1258] Based on the analysis results, the server creates recommended product information according to the user's emotional state and sends it to the terminal.

[1259] Step 5:

[1260] The device will display recommended products that reflect your emotional state.

[1261] Examples:

[1262] If a user has a tired expression while using the app, the device will recognize that expression with its camera and send it to the server.

[1263] The server uses an emotion engine to detect "fatigue," and then lists recommended products that have a relaxing effect (e.g., aroma candles and herbal tea) and sends them to the device.

[1264] The device will suggest products that will help users feel refreshed by displaying "Recommended products: Aroma candles, herbal tea" on the screen.

[1265] In this way, the system can make individualized suggestions based on the user's emotional state, making the shopping experience more personal and convenient.

[1266] Example 2

[1267] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1268] Modern supermarkets require systems that enable users to enjoy an efficient and stress-free shopping experience. However, existing systems struggle to promote user shopping behavior, and the product search and payment processes can be time-consuming. Furthermore, they do not consider user emotions when suggesting products, and thus fail to provide a personalized shopping experience.

[1269] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1270] In this invention, the server includes means for a user to launch a store management system application using a mobile communication device, means for automatically reading product information using a sensor and transmitting the product information to a data processing device, means for automatically making a payment based on the product information received from the data processing device, means for receiving a user's shopping request using a voice recognition function and displaying the location of the corresponding product on a guidance display device, means for adding discounted products to a shopping list in conjunction with advertising information, and means for analyzing emotions from the user's facial expressions and voice using an emotion recognition engine and providing recommended product information based on the analysis results. This enables users to not only efficiently search for and purchase products, but also receive personalized shopping suggestions.

[1271] A "mobile communication device" is a communication device that a user can carry around, and primarily refers to devices such as smartphones and tablets.

[1272] A "store management system" refers to an application used to manage products and support users' shopping experiences in retail stores such as supermarkets.

[1273] "Sensor" means a device for detecting and reading product information, including RFID readers and barcode scanners.

[1274] "Data processing device" refers to a server or database that analyzes product information sent from the sensor and provides the necessary information to the user.

[1275] "Automatic payment" refers to the process of automatically paying the total amount of the products purchased by the user using a payment method registered in advance.

[1276] "Voice recognition function" refers to technology that converts a user's voice input into text data and then provides the system with appropriate information based on that data.

[1277] A "guidance display device" is a device used to display product locations and optimal shopping routes within a store, and primarily refers to the screen of a user's mobile communication device.

[1278] "Advertising information" refers to supermarket flyers and discounted product information, and is used to provide users with special sale information.

[1279] An "emotion recognition engine" refers to an algorithm that analyzes a user's emotional state from their facial expressions and voice, and recommends appropriate products based on the results.

[1280] "Personalized shopping suggestions" refers to a feature that suggests products and services that are suitable for each individual user based on their past purchase history and emotional state.

[1281] This invention relates to a store management system for improving the shopping experience in supermarkets, particularly a system that uses mobile communication devices, data processing devices, sensors, and an emotion recognition engine to analyze a user's emotions and make appropriate suggestions based on the analysis.

[1282] When a user launches an application, the mobile communication device (e.g., a smartphone or tablet) sends a request to a data processing device (server) to obtain the user's past purchase history and recommended product information. This allows the device to display a list of previously purchased products and the store's recommended products. As a specific example, previously purchased milk or bread may be displayed, with discounted butter recommended next to them.

[1283] When a user selects an item in a supermarket, a sensor (e.g., an RFID reader or barcode scanner) automatically reads the item's information and transmits it to a data processing device via a mobile communication device. The data processing device analyzes the item's information, checks the price and stock information of the item, and displays the total price on the terminal in real time. For example, when an apple is placed in a basket, the sensor detects the apple and transmits the information to the terminal, instantly updating the total price on the screen.

[1284] Furthermore, if a user wishes to search for a specific dish or product, the request is made using a voice recognition function. The mobile communication device converts the user's voice input into text data and sends it to the data processing device. The data processing device creates a list of necessary products based on the user's request and identifies the location of each product. Based on the received information, the mobile communication device displays the product locations on a map and navigates the optimal shopping route. For example, in response to a request to "make curry," a list of necessary ingredients and their locations are displayed on a map.

[1285] The application incorporates the latest store advertising information, allowing users to check information on discounted and special sale items. Users can add discounted items to their shopping list, and this information is synchronized with the data processing device's database. This allows the optimal shopping route to be recalculated and displayed on the mobile communication device. For example, if a user sees a flyer and adds a discounted curry roux to their shopping list, the optimal route recalculated by the data processing device will be displayed on the device.

[1286] Furthermore, to understand the user's emotional state, facial expressions and voice are collected using the camera and microphone of the mobile communication device and analyzed by an emotion recognition engine. The emotion recognition engine analyzes the user's emotional state and provides appropriate product information based on that analysis. For example, if the user looks tired, it may recommend relaxing aroma candles or herbal tea.

[1287] This allows users to not only efficiently search and purchase products, but also receive personalized shopping suggestions.

[1288] Prompt statement example:

[1289] 1. "I want to develop a system to simplify supermarket shopping. It includes functionality to recognize user emotions and recommend products based on those emotions. Can you please explain the detailed process of this system?"

[1290] 2. "Please explain in detail how smart systems can be used to improve the shopping experience at supermarkets. In particular, I would like to know about the use of voice recognition and emotion engines."

[1291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] A user launches a store management system application using a mobile communication device. When the user launches the app, the device sends a request for initial data to the server. The server obtains the user's past purchase history and recommended product information and sends this to the device. The device displays the received data on the screen, allowing the user to view a list of products they have purchased in the past and recommended products from the supermarket. For example, a list of milk, bread, and butter that the user has purchased in the past, as well as discounted products, is displayed.

[1294] input:

[1295] User launches app

[1296] output:

[1297] Past purchase history

[1298] Recommended product information

[1299] Step 2:

[1300] When a user places an item in a basket at a supermarket, a sensor automatically reads the item's information. This information is sent to a server via the terminal. The server analyzes the received product information and checks the price and stock information of the item. The analyzed information is sent to the terminal in real time, and the terminal updates and displays the total price. For example, when a user puts an apple in the basket, the sensor reads the apple's information, and the terminal updates the total price in real time.

[1301] input:

[1302] Product information added to the user's cart

[1303] output:

[1304] Real-time total amount

[1305] Product price and availability information

[1306] Step 3:

[1307] When a user wants to search for a specific dish or product, they make a request using the voice recognition function. The user's voice input is converted into text data by the device and sent to the server. The server creates a product list based on the user's voice request and identifies the location of each product. The device uses the information from the server to display the product locations on a map and navigates the user to the optimal shopping route. For example, if a user makes a voice request such as "I want to make curry," a list of the necessary ingredients and their locations will be displayed on a map.

[1308] input:

[1309] User voice request

[1310] output:

[1311] Location of food and products

[1312] The best shopping route

[1313] Step 4:

[1314] The user can check the latest store advertising information embedded in the device. The user can then check the displayed discounted and special sale items and add the discounted items to their shopping list. The added product information is synchronized with the server's database, and the server sends a recalculated optimal shopping route to the device. The device displays the new route information, allowing the user to shop efficiently. For example, if the user sees a flyer and adds a discounted curry roux to their shopping list, a new optimal route will be displayed on the device.

[1315] input:

[1316] Adding discounted products to users

[1317] output:

[1318] Updated shopping route

[1319] Discounted product information

[1320] Step 5:

[1321] While the user is using the application, the device uses a camera and microphone to collect the user's facial expressions and voice. The device sends the collected emotional data to a server, which then uses an emotion recognition engine to analyze the user's emotional state. Based on the analysis results, the server generates recommended product information according to the user's emotional state and sends it to the device. The device then displays this information to the user. For example, if the user looks tired, the emotion recognition engine will recommend products with a relaxing effect (aroma candles or herbal tea).

[1322] input:

[1323] User emotion data

[1324] output:

[1325] Emotion-based product recommendations

[1326] (Application example 2)

[1327] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1328] Traditional shopping experiences in brick-and-mortar stores often require users to search around the store or wait in line at the cash register for long periods of time, resulting in inefficiency. Furthermore, product recommendations are often made without taking the user's emotional state into consideration, which can lead to lower user satisfaction. There is a need to improve the user's shopping experience by solving these problems.

[1329] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for the user to launch an application for the facility using a mobile information terminal; means for automatically reading product information using a detector and transmitting the relevant product information to the information processing device; means for making an automatic payment based on the product information received from the information processing device; means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map; means for adding discounted products to a shopping list in conjunction with advertising information; and means for recognizing the user's emotional state using facial expression analysis and voice analysis and providing recommended product information based on the emotional state. This makes it possible to make individual shopping suggestions according to the user's emotional state, thereby providing an efficient and satisfying shopping experience.

[1330] A "personal digital assistant" is an electronic device that can be carried by a user and is a device for executing applications and performing data communication.

[1331] "Store facilities" refer to physical stores such as supermarkets and other retail outlets where customers visit in person to shop.

[1332] An "application" is a program that runs on a mobile information terminal and provides support functions to help users shop efficiently.

[1333] A "detector" is a device that automatically reads product information and is a sensor that obtains information using barcodes, RFID, etc.

[1334] The "information processing device" is a computer system such as a server, which receives, analyzes, and stores the product information transmitted from the detector.

[1335] "Automatic payment" is a system in which a user automatically pays for products purchased via an information processing device without having to line up at the cash register.

[1336] The "voice recognition function" is a technology that converts the user's voice into text data, and is a function that receives and analyzes shopping requests.

[1337] "Advertising information" is information about discounted products and special sales offered by the facility, and is displayed within the application to support the user's shopping experience.

[1338] "Discounted products" are products offered at special prices based on advertising information, and are intended to encourage users to consume.

[1339] "Facial expression analysis" is a technology that uses a camera to analyze a user's facial expressions and identify their emotional state.

[1340] "Voice analysis" is a technology that uses a microphone to analyze the nuances of a user's voice and words and identify their emotional state.

[1341] "Emotional state" refers to the user's current psychological state, and expresses emotions such as happiness, sadness, fatigue, etc.

[1342] "Recommended product information" is information that suggests products that the user may be interested in based on the user's past purchase history and current emotional state.

[1343] The present invention relates to a smart system that enables a user to launch an application at a shopping facility using a mobile information terminal to improve the shopping experience. Specific embodiments of the present invention will be described in detail below.

[1344] System Configuration

[1345] This system is composed of several elements, including a mobile information terminal, a detector, an information processing device, a voice recognition function, an advertising information system, a facial expression analysis and voice analysis system, and an automatic payment system.

[1346] Hardware and software used

[1347] Mobile information terminal: An electronic device, such as a smartphone or tablet, that runs applications and can be easily operated by the user.

[1348] Detector: A sensor that reads product information, such as a barcode scanner or RFID reader.

[1349] Information processing device: A system that runs on a server or in a cloud computing environment and performs data analysis and information storage.

[1350] Speech recognition function: Technology that converts speech to text, such as Google Speech Recognition API.

[1351] Advertising information system: A platform that manages special sales information and discounted products at visiting facilities.

[1352] Facial and voice analysis systems: Technologies for identifying a user's emotional state. For example, camera devices and Adaptive Recognition APIs, microphone devices and voice analysis software.

[1353] Automated payment systems: payment gateways and mobile payment solutions.

[1354] Processing flow

[1355] 1. Launching the application

[1356] When a user starts an application for a facility using a mobile information terminal, the terminal transmits a request to an information processing device to obtain past purchase history and recommended product information.

[1357] 2. Product Scan

[1358] When a user places an item in the cart, the detector automatically reads the item information and sends it to an information processing device, which analyzes the information and updates the total price in real time.

[1359] 3. Automatic payments

[1360] Once shopping is complete, the terminal will process the payment through an automatic payment system based on the product information received from the information processing device, thereby reducing waiting times at the cash register.

[1361] 4. Product search using voice recognition

[1362] When a user makes a shopping request using the voice recognition function, the terminal converts the voice request into text data and sends it to the information processing device. The information processing device identifies the location of the product based on the user's request and displays it as a map on the terminal.

[1363] 5. Linking with advertising information

[1364] The application displays the latest advertising information and allows users to add discounted items to their shopping list, which causes the information processing device to recalculate the optimal shopping route and display it to the user.

[1365] 6. Product Recommendations Based on Emotion Recognition

[1366] The device's camera and microphone collect the user's facial expressions and voice, which are then sent to an information processing device. The device uses facial expression and voice analysis systems to recognize the user's emotional state and provide recommended product information based on that emotion.

[1367] Specific examples

[1368] Example of displaying past purchase history: When a user launches the app, the app displays the milk and bread they previously purchased and suggests related products.

[1369] Product scanning and real-time price updates: When a user places an apple in the basket, the detector reads the product information and the total price is updated in real time on the terminal screen.

[1370] Example of a voice recognition request: When a user requests by voice, "I want to make curry," the necessary ingredients and their locations are displayed on a map.

[1371] A concrete example of emotion recognition: If a user has a tired expression, the facial expression analysis system will recognize this and suggest relaxing aroma candles or herbal tea.

[1372] Prompt Sentence Examples

[1373] User: Please use the camera to analyze your face.

[1374] ---

[1375] Facial image:

[1376] [User's face image]

[1377] ---

[1378] System: The user appears tired. Try a relaxing scented candle or herbal tea.

[1379] In this way, the present invention allows for personalized shopping suggestions based on the user's emotional state, providing an efficient and satisfying shopping experience.

[1380] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1381] Step 1:

[1382] The user launches an application for the facility they are visiting using their mobile information terminal. They enter their user ID, and the terminal sends a request to the server. This request includes the user ID, and the server uses this information to obtain the user's past purchase history and recommended product information, which it then sends to the terminal. The terminal then displays the received information on its screen.

[1383] Step 2:

[1384] When a user places an item in the cart, a detector (such as a barcode scanner or RFID reader) reads the item's identification information. This identification information is sent to the server via the terminal. The server analyzes the item's price and inventory information and calculates the total price in real time. This information is then sent back to the terminal and displayed on the terminal screen.

[1385] Step 3:

[1386] Once the user has completed their purchase, they press the "Pay" button within the application. The device sends all of the purchase information to the server, which then uses an automated payment system to process the payment. Payment completion information is then sent from the server to the device, and the user is notified.

[1387] Step 4:

[1388] When a user wants to search for a specific product, they input a voice command using the voice recognition function. The device converts the user's voice into text using the Google Speech Recognition API and sends it to the server. The server uses this text data to search for product location information and sends it to the device as a map. The device then displays the map on the screen and navigates the user to the optimal shopping route.

[1389] Step 5:

[1390] The latest advertising information is displayed within the application. The user checks discounted items and selects the ID of the discounted item to add it to their shopping list. The device sends this information to the server, which then recalculates the optimal shopping route and sends it to the device. The device then displays the new route.

[1391] Step 6:

[1392] The device's camera and microphone are used to collect the user's facial expressions and voice, which are then sent to the server. The server then uses facial and voice analysis systems to recognize the user's emotional state. It then calculates recommended product information based on the user's emotional state and sends this to the device. The device then displays recommended product information based on the user's emotion on the screen.

[1393] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1394] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1395] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1396] [Fourth embodiment]

[1397] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1398] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1399] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1400] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1401] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1403] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1404] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1405] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1406] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1407] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1408] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1409] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1410] The present invention relates to a smart system for improving shopping experience in supermarkets, which has multiple functions to enable users to efficiently shop using mobile devices.

[1411] An embodiment of the system is described as follows.

[1412] 1. Starting the application and loading initial data

[1413] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[1414] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[1415] 2. Product scanning and automatic payment

[1416] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[1417] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[1418] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[1419] 3. Voice recognition for product search and navigation

[1420] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[1421] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[1422] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[1423] 4. Linking with flyer information

[1424] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[1425] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[1426] Examples:

[1427] When a user launches the app, a list of past purchases and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[1428] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[1429] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[1430] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[1431] In this way, the system can greatly enhance the user's shopping experience.

[1432] The processing flow will be explained below.

[1433] (Program processing steps)

[1434] Starting the application and loading initial data

[1435] Step 1:

[1436] A user taps to launch an application on their mobile device.

[1437] Step 2:

[1438] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[1439] Step 3:

[1440] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[1441] Step 4:

[1442] The device displays the received data, and the user can check their past purchase history and recommended products.

[1443] Product scanning and automatic payment

[1444] Step 1:

[1445] The user places the items they want to purchase in the cart.

[1446] Step 2:

[1447] An application on the terminal reads product information from sensors attached to the basket.

[1448] Step 3:

[1449] The terminal transmits the product information to the server.

[1450] Step 4:

[1451] The server analyzes the product information and checks whether the corresponding product exists in the database.

[1452] Step 5:

[1453] The server sends product price and stock information to the terminal.

[1454] Step 6:

[1455] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[1456] Step 7:

[1457] When the user completes their purchase, they press the "Confirm payment" button within the application.

[1458] Step 8:

[1459] The terminal transmits the information necessary to finalize the payment to the server.

[1460] Step 9:

[1461] The server automatically settles the payment using the user's registered payment method.

[1462] Step 10:

[1463] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[1464] Voice recognition for product search and navigation

[1465] Step 1:

[1466] The user makes a voice search request for a specific dish or product.

[1467] Step 2:

[1468] The device converts the voice input into text data and sends the information to the server.

[1469] Step 3:

[1470] The server creates a list of required items based on the request and identifies the location of each item.

[1471] Step 4:

[1472] The server sends the product list and placement information to the terminal.

[1473] Step 5:

[1474] The device displays the received product list and location information as a map, and guides the user to the optimal shopping route.

[1475] Linking with flyer information

[1476] Step 1:

[1477] The terminal obtains the latest flyer information from the server.

[1478] Step 2:

[1479] The flyer information acquired by the device is displayed within the application.

[1480] Step 3:

[1481] A user sees a flyer and adds discounted items to a list.

[1482] Step 4:

[1483] The terminal transmits the added discount product information to the server.

[1484] Step 5:

[1485] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[1486] Step 6:

[1487] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[1488] Example 1

[1489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1490] Conventional supermarket shopping systems have problems with the complicated management of purchased items, payment, and related information acquisition. In particular, the user shopping experience is degraded by manual product scanning, long waiting times at the cash register, the inconvenience of voice-activated product searches, and the difficulty of checking sale information and recommended products. Furthermore, the decentralized nature of these functions and the lack of an integrated system hinders user satisfaction.

[1491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1492] In this invention, the server includes means for automatically reading product information and transmitting the relevant product information, means for providing recommended product information based on past purchase history, means for receiving a user's shopping request using voice recognition and displaying product location guidance, means for adding sale items to a shopping list in conjunction with public relations information, and means for displaying the total amount in real time. This enables users to shop efficiently, significantly improving the shopping experience, and is expected to increase overall convenience through the integrated system.

[1493] A "mobile device" is an electronic device that a user can carry and that can run applications.

[1494] "Information processing device software" refers to an application that runs on a mobile device and assists users in shopping.

[1495] A "detection device" is a hardware device that automatically reads product information, such as an RFID reader.

[1496] "Data processing device" refers to a server or cloud service that receives product information, analyzes it, and executes various processes.

[1497] "Automatic payment" is a feature that allows users to complete payment without manual intervention when purchasing products.

[1498] The "voice recognition function" is a technology that converts a user's voice input into text data and processes the content.

[1499] A "shopping request" is a request made by a user to an application to request a specific product or dish.

[1500] "Guide display" is a function that visually shows the location of products and shopping routes to the user.

[1501] "Publicity information" refers to information that includes information about special sales and discounted products offered by supermarkets.

[1502] "Special sale items" are items offered at a lower price than usual.

[1503] "Past purchase history" is a record of data on products previously purchased by a user.

[1504] "Recommended product information" is information that suggests products suitable for a user based on the user's past purchase history, etc.

[1505] "Real-time display" is a function that instantly updates and displays information in response to operations or changes made by the user.

[1506] The "total amount" is the total amount paid by the user for all products purchased.

[1507] This invention relates to an integrated system for significantly improving the shopping experience in supermarkets, which provides a wide range of functions for users to efficiently shop using mobile devices.

[1508] In one embodiment of this system, a user first launches an application using a mobile device. The mobile device used here is a portable information processing device such as a smartphone or tablet. When the user launches the app, the device sends a request to a server to obtain the user's past purchase history and recommended product information. This server is a high-performance data processing device that contains a user database.

[1509] Next, when a user places items in a basket in the supermarket, a detection device such as an RFID reader attached to the basket automatically reads the product information. This product information is sent to a server via the terminal. The server references a product database to check the product price and stock information. The price information is returned to the terminal in real time, and the total price of the items the user has placed in the basket is displayed on the terminal screen.

[1510] Furthermore, if a user wishes to search for a specific dish or product, they can make a request using the voice recognition function. This voice recognition is performed using Google Voice Assistant. The device converts the user's voice input into text data and sends that information to the server. The server then identifies the required product list and product location, and sends it back to the device. Based on this, the device displays a map and navigates the optimal shopping route.

[1511] The application also incorporates the latest supermarket promotional information. Users can check information on special sales and discounted products through the app and add these products to their shopping list. The added product information is synchronized with the server database, and the optimal shopping route is recalculated.

[1512] As a concrete example, when a user launches an app, a list of past purchase history and recommended products is displayed. For example, milk and bread that the user previously purchased are displayed, with discounted butter recommended next to them. When a user places an apple in their basket at the supermarket, a sensor detects the apple and sends that information to the device. The device updates the total price in real time based on the information received from the server. When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and where to place them on a map. If the user looks at a flyer and adds discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[1513] Examples of prompts for the generative AI model include, "What are the features of an app that can improve the efficiency of shopping at the supermarket?", "Please explain how a smart system that uses voice recognition to search for specific products works", and "Please explain how a shopping app with an automatic payment system works."

[1514] In this way, the system provides a multifunctional platform for comprehensively improving the user's shopping experience.

[1515] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1516] Step 1:

[1517] Starting the application and getting initial data

[1518] Executor: User and device

[1519] Input: Tap the application icon on your mobile device

[1520] Action: User taps the icon on their mobile device to launch the application.

[1521] Data processing: The device sends an HTTP request to the server, sending the user ID and authentication information.

[1522] Output: Get the user's past purchase history and recommended product information from the server

[1523] Specific operation: The server searches the user database for purchase history, generates product information using a recommendation algorithm, and sends it back to the device. The device analyzes this information and displays it on the screen.

[1524] Step 2:

[1525] Product scanning and information transmission

[1526] Actors: Users, devices, and sensors

[1527] Input: User adds product to cart

[1528] How it works: When an item is placed in the cart, the RFID reader captures the item information.

[1529] Data processing: The sensor reads the data from the RFID tag and sends it to the device. The device then sends this data to the server.

[1530] Output: Get product price and availability information from the server

[1531] Specific operation: The server queries the database based on the product ID, checks the price and stock of the corresponding product, and sends it back to the terminal, which updates the total price on the screen in real time.

[1532] Step 3:

[1533] Product search using voice recognition

[1534] Executor: User and device

[1535] Input: User's voice input (e.g. "I want to make curry")

[1536] Action: User makes a voice request

[1537] Data processing: The device's voice recognition function (Google Voice Assistant) converts the voice into text data and sends it to the server.

[1538] Output: Get the list of required products and their locations from the server

[1539] Specific operation: The server queries the database to obtain a list of the specified products and their locations, and sends this to the device, which then displays it using a map display engine.

[1540] Step 4:

[1541] Check flyer information and add it to your product list

[1542] Executor: User and device

[1543] Input: Publicity information (flyer) within the application

[1544] What happens: A user browses promotions within the app and sees discounted products.

[1545] Data processing: When a user adds an item to their shopping list, the device sends this information to the server.

[1546] Output: Get the updated optimal shopping route from the server

[1547] Specific operation: The server recalculates the optimal route based on the new product list and sends it to the device. The device uses its map display engine to display the updated route to the user.

[1548] Step 5:

[1549] Automatic payment

[1550] Executor: User and device

[1551] Input: User taps the "Pay" button

[1552] Action: User performs payment operation

[1553] Data processing: The device sends payment information (e.g., credit card information) to the server.

[1554] Output: Receives a confirmation message from the server and displays the payment result to the user.

[1555] What happens: The server processes the payment using a payment processor (e.g., a money processing system) and, if successful, sends a confirmation message back to the terminal, which displays this information on its screen.

[1556] (Application example 1)

[1557] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1558] Today's consumers expect an efficient and comfortable shopping experience at supermarkets and other stores. They also have growing demands for reduced checkout wait times, improved in-store navigation, and product search capabilities using voice recognition. While meeting these demands, stores must also operate efficiently. However, the systems needed to achieve this are not yet fully in place, and the development of technology to reduce shopper stress is a challenge.

[1559] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1560] In this invention, the server includes means for a user to launch a store application using a mobile terminal, means for automatically scanning product information with a sensor and transmitting the relevant product information to the server, means for automatically making a payment based on the product information received from the server, means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map, means for adding discounted products to a shopping list in conjunction with the latest information, means for acquiring recommended product information and past purchase history and displaying them to the user, and means for optimally navigating the store based on the map display, thereby enabling the user to shop efficiently and comfortably.

[1561] A "mobile terminal" is an information terminal that can be carried by a user, and mainly refers to a smartphone or tablet terminal.

[1562] A "store application" is software that users use to assist them in shopping at physical stores, and is an application that runs on a mobile device.

[1563] A "sensor" is a device that detects physical phenomena and converts them into electronic signals, and is used to scan product information.

[1564] "Product information" refers to data relating to a specific product, such as product identification information, price information, and inventory information.

[1565] A "server" is a computer system that functions as a central processing unit, receives requests from clients, and processes data.

[1566] "Automatic payment" is a function that automatically calculates the price of the product purchased by the user and completes the payment using a payment method registered in advance.

[1567] "Voice recognition function" is a technology that inputs human speech as voice data and converts it into text data.

[1568] A "shopping request" is a request made by a user to retrieve information about a particular product.

[1569] "Displaying on a map" refers to visually displaying the layout of the store and showing the location of specific products.

[1570] "Latest news" refers to information about the latest discounts and special offers offered by supermarkets and stores.

[1571] "Discounted items" refer to items that are sold at a lower price than the regular price.

[1572] A "shopping list" is a list of items that a user plans to purchase.

[1573] "Recommended product information" refers to information for recommending specific products based on the user's past purchase history and preferences.

[1574] "Past purchase history" refers to a record of information about products previously purchased by a user.

[1575] "Navigation" is a function that guides the user to their destination, showing the location of the desired product within the store and providing the optimal route.

[1576] This invention relates to an advanced support system for users to efficiently shop using mobile devices. The system combines automatic product information scanning, voice recognition search, navigation, and payment functions to enhance the in-store shopping experience.

[1577] Hardware and software used

[1578] Hardware:

[1579] Mobile device (iOS or Android)

[1580] Sensors that scan product information (e.g., barcode scanners)

[1581] software:

[1582] Mobile applications (using Flutter or React Native as development frameworks)

[1583] Server side (using Node.js and Firebase)

[1584] Speech Recognition API (Google Cloud Speech-to-Text)

[1585] Payment system (Stripe, PayPal)

[1586] Database (Firestore)

[1587] Program overview and processing content

[1588] Start the application and load the initial data:

[1589] When a user launches an application on their mobile device, a request is sent to the server to retrieve the user's past purchase history and recommended product information. This information is displayed on the mobile device screen, allowing the user to check a list of past purchases and recommended products.

[1590] Product scanning and automatic payment:

[1591] When a user places an item in their cart, they scan the product barcode with the camera on their mobile device. The scanned product information is sent to a server via the mobile device, where the product price and stock information are confirmed. Once shopping is complete, payment is automatically made within the app, and payment is completed using the payment method registered in advance.

[1592] Voice-activated product search and navigation:

[1593] The user searches for a specific product using voice recognition. The mobile device converts the voice input into text data and sends it to the server. Based on the user's request, the server retrieves the list of required products and their location data and sends it to the mobile device. Based on this, a map of the store is displayed and the user is guided to the optimal shopping route.

[1594] Stay up-to-date with discounted items and add them to your shopping list:

[1595] The application incorporates the latest store information, allowing users to check discounted and special offers. When a discounted item is added to the shopping list, the system recalculates the optimal shopping route on the server and displays it on the mobile device.

[1596] Specific examples

[1597] For example, when a user launches the app on their mobile device, they can voice-input a request such as "I want to get the ingredients for pasta for dinner." Through voice recognition, the app displays ingredients such as "tomato sauce, pasta, olive oil, and cheese" and their locations, and suggests the shortest route within the store.

[1598] When a user places an item in their cart, the product information is instantly sent to the server, and the total price is updated in real time. Once shopping is complete, payment can be completed with just a few taps on a mobile device, eliminating the need to wait in long lines.

[1599] Example prompt sentence:

[1600] "I want to make pasta."

[1601] "Tell me what's for dinner."

[1602] This allows the user to shop efficiently and comfortably.

[1603] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1604] Step 1:

[1605] Launching the application

[1606] When a user launches an application on a mobile device, the device sends the user's authentication information to the server. Input includes a user ID and authentication token. The server receives this, retrieves the user's past purchase history and recommended product information, and returns it to the device. As output, the user's past purchase history and recommended product information are displayed on the device. Specific operations include retrieving and displaying data after sending the authentication information.

[1607] Step 2:

[1608] Product Scan

[1609] The user picks up an item and scans the barcode with the camera on their mobile device. The input contains the item's barcode information. The device reads the barcode and sends the item information to the server. The server retrieves the item's price and stock information from the product database and returns it to the device. The output is the item's price and stock information displayed on the screen. During this process, the device detects the item's barcode and sends the data to the server.

[1610] Step 3:

[1611] Automatic payment

[1612] The user completes their purchase and pays within the app. Input includes the total price of the items purchased by the user and the registered payment method. The terminal sends the payment information to the server, which then calls a payment system (e.g., Stripe or PayPal) to complete the payment. Output is a confirmation message on the terminal that the payment was successful. Specific actions include starting and completing the payment process.

[1613] Step 4:

[1614] Product search using voice recognition

[1615] The user inputs a request by voice, such as "I want to make pasta." The input includes voice data. The device converts the voice data into text using the Google Cloud Speech-to-Text API and sends this text data to the server. Based on the request, the server identifies the list of required products and their locations and returns this to the device. As output, the product list and their locations are displayed on a map. Specific operations include voice recognition and data transmission.

[1616] Step 5:

[1617] Proposal of the best shopping route

[1618] Based on the location data received from the server, the terminal displays the optimal shopping route on a map within the store. The input includes multiple product location data. Based on this data, the terminal calculates the optimal route using a shortest path algorithm and presents it to the user. As an output, the optimal route is displayed on a map. Specific operations include analyzing map data and displaying the route.

[1619] Step 6:

[1620] Linking with the latest information

[1621] Within the application, the user checks the latest discounted items and special offers and adds them to a shopping list. The input includes the latest information data. The server sends the latest information to the user's device, which displays it. When the user adds discounted items to the list, the server again calculates the optimal shopping route and returns it to the device. The output is the updated optimal route displayed on the map. Specific operations include retrieving the latest information and updating the list.

[1622] Step 7:

[1623] Displaying recommended product information

[1624] Based on the user's past purchase history and the database, the server calculates recommended products and sends them to the terminal. The input includes past purchase history data. The server analyzes this and recommends products suitable for the user. The output is the recommended products displayed on the screen. Specific operations include data analysis and information display.

[1625] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1626] This invention relates to a smart system that improves the shopping experience in supermarkets, and in particular to a system that recognizes the user's emotions by combining emotion engines and makes appropriate shopping suggestions based on those emotions. The system has the following functions:

[1627] 1. Starting the application and loading initial data

[1628] When a user launches an application on a mobile device, the device sends a request to the server to obtain the user's past purchase history and recommended product information.

[1629] The acquired information is displayed on the device screen, allowing the user to check a list of products they have purchased in the past and products recommended by the supermarket.

[1630] 2. Product scanning and automatic payment

[1631] When a user places an item in a basket in a supermarket, a sensor attached to the basket reads the product information, which is then sent to the server via the terminal.

[1632] The server analyzes the product information and checks the price and stock information of the relevant product, which then updates the total price of the products added to the user's cart in real time.

[1633] Once shopping is complete, users can make an automatic payment within the app, which uses a pre-registered payment method to complete the payment, reducing waiting time at the register.

[1634] 3. Voice recognition for product search and navigation

[1635] If a user wishes to search for a specific dish or product, they can make a request using the device's voice recognition function.

[1636] The device converts the user's voice input into text data and sends that information to the server, which then creates a list of required products based on the user's request and identifies the location of each product.

[1637] Based on the information received from the server, the device displays the product locations on a map and navigates the optimal shopping route.

[1638] 4. Linking with flyer information

[1639] The application incorporates the latest supermarket flyer information, allowing users to check information on discounted products and special offers through the app.

[1640] Based on the flyer information, users can add discounted products to their shopping list, which is then synchronized with the server's database and the optimal shopping route is recalculated.

[1641] 5. User Emotion Recognition by Emotion Engine

[1642] When a user uses an application, the device uses a camera and microphone to recognize emotions from the user's facial expressions and voice.

[1643] The device sends the collected emotional data to a server, which then uses an emotion engine to analyze the user's emotional state.

[1644] Based on the analysis results, the server provides the user with recommended product information according to their emotional state.

[1645] Examples:

[1646] When a user launches the app, their past purchase history and recommended products are displayed. For example, the milk and bread that the user previously purchased are displayed, and discounted butter is recommended next to them.

[1647] When a user puts an apple in their basket at the supermarket, a sensor detects the apple and sends the information to the terminal, which then updates the total amount in real time based on the information received from the server.

[1648] When a user makes a voice request such as "I want to make curry," the device sends the request to the server and displays a list of the necessary ingredients and their locations on a map.

[1649] When a user looks at a flyer and adds a discounted curry roux to their shopping list, the server recalculates the optimal route and displays it on the device.

[1650] If the user looks tired, the emotion engine will recognize this and the server will recommend products that have a relaxing effect (such as aromatic candles or herbal tea).

[1651] In this way, the system can provide various shopping suggestions depending on the user's emotional state, significantly improving the user's shopping experience.

[1652] The processing flow will be explained below.

[1653] Starting the application and loading initial data

[1654] Step 1:

[1655] A user taps to launch an application on their mobile device.

[1656] Step 2:

[1657] The device sends the user ID to the server and requests information about past purchase history and recommended products.

[1658] Step 3:

[1659] The server retrieves data on past purchase history and recommended products based on the user ID and sends it to the device.

[1660] Step 4:

[1661] The device displays the received data, and the user can check their past purchase history and recommended products.

[1662] Product scanning and automatic payment

[1663] Step 1:

[1664] The user places the items they want to purchase in the cart.

[1665] Step 2:

[1666] The terminal reads product information from sensors attached to the basket.

[1667] Step 3:

[1668] The terminal transmits the product information to the server.

[1669] Step 4:

[1670] The server analyzes the product information and checks whether the corresponding product exists in the database.

[1671] Step 5:

[1672] The server sends product price and stock information to the terminal.

[1673] Step 6:

[1674] The product prices received by the terminal are reflected in the total amount in real time and displayed on the terminal screen.

[1675] Step 7:

[1676] When the user completes their purchase, they press the "Confirm payment" button within the application.

[1677] Step 8:

[1678] The terminal transmits the information necessary to finalize the payment to the server.

[1679] Step 9:

[1680] The server automatically settles the payment using the user's registered payment method.

[1681] Step 10:

[1682] The server sends payment completion information to the terminal, and the terminal displays a notification that "payment has been completed."

[1683] Voice recognition for product search and navigation

[1684] Step 1:

[1685] The user makes a voice search request for a specific dish or product.

[1686] Step 2:

[1687] The device converts the voice input into text data and sends the information to the server.

[1688] Step 3:

[1689] The server creates a list of required items based on the request and identifies the location of each item.

[1690] Step 4:

[1691] The server sends the product list and location data to the terminal.

[1692] Step 5:

[1693] The device receives a list of products and displays their location information as a map, guiding users to the optimal shopping route.

[1694] Linking with flyer information

[1695] Step 1:

[1696] The terminal obtains the latest flyer information from the server.

[1697] Step 2:

[1698] The flyer information acquired by the device is displayed within the application.

[1699] Step 3:

[1700] A user sees a flyer and adds discounted items to a list.

[1701] Step 4:

[1702] The terminal transmits the added discount product information to the server.

[1703] Step 5:

[1704] The server calculates the optimal shopping route, including discounted items, and sends it to the terminal.

[1705] Step 6:

[1706] Based on the information received by the device, the optimal route is redisplayed and the user is guided thereto.

[1707] Recognizing user emotions with an emotion engine

[1708] Step 1:

[1709] When a user is using an application, the device uses a camera and microphone to collect the user's facial expressions and voice.

[1710] Step 2:

[1711] The terminal transmits the collected facial expression and voice data to the server.

[1712] Step 3:

[1713] The server uses an emotion engine to analyze the user's emotional state.

[1714] Step 4:

[1715] Based on the analysis results, the server creates recommended product information according to the user's emotional state and sends it to the terminal.

[1716] Step 5:

[1717] The device will display recommended products that reflect your emotional state.

[1718] Examples:

[1719] If a user has a tired expression while using the app, the device will recognize that expression with its camera and send it to the server.

[1720] The server uses an emotion engine to detect "fatigue," and then lists recommended products that have a relaxing effect (e.g., aroma candles and herbal tea) and sends them to the device.

[1721] The device will suggest products that will help users feel refreshed by displaying "Recommended products: Aroma candles, herbal tea" on the screen.

[1722] In this way, the system can make individualized suggestions based on the user's emotional state, making the shopping experience more personal and convenient.

[1723] Example 2

[1724] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1725] Modern supermarkets require systems that enable users to enjoy an efficient and stress-free shopping experience. However, existing systems struggle to promote user shopping behavior, and the product search and payment processes can be time-consuming. Furthermore, they do not consider user emotions when suggesting products, and thus fail to provide a personalized shopping experience.

[1726] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1727] In this invention, the server includes means for a user to launch a store management system application using a mobile communication device, means for automatically reading product information using a sensor and transmitting the product information to a data processing device, means for automatically making a payment based on the product information received from the data processing device, means for receiving a user's shopping request using a voice recognition function and displaying the location of the corresponding product on a guidance display device, means for adding discounted products to a shopping list in conjunction with advertising information, and means for analyzing emotions from the user's facial expressions and voice using an emotion recognition engine and providing recommended product information based on the analysis results. This enables users to not only efficiently search for and purchase products, but also receive personalized shopping suggestions.

[1728] A "mobile communication device" is a communication device that a user can carry around, and primarily refers to devices such as smartphones and tablets.

[1729] A "store management system" refers to an application used to manage products and support users' shopping experiences in retail stores such as supermarkets.

[1730] "Sensor" means a device for detecting and reading product information, including RFID readers and barcode scanners.

[1731] "Data processing device" refers to a server or database that analyzes product information sent from the sensor and provides the necessary information to the user.

[1732] "Automatic payment" refers to the process of automatically paying the total amount of the products purchased by the user using a payment method registered in advance.

[1733] "Voice recognition function" refers to technology that converts a user's voice input into text data and then provides the system with appropriate information based on that data.

[1734] A "guidance display device" is a device used to display product locations and optimal shopping routes within a store, and primarily refers to the screen of a user's mobile communication device.

[1735] "Advertising information" refers to supermarket flyers and discounted product information, and is used to provide users with special sale information.

[1736] An "emotion recognition engine" refers to an algorithm that analyzes a user's emotional state from their facial expressions and voice, and recommends appropriate products based on the results.

[1737] "Personalized shopping suggestions" refers to a feature that suggests products and services that are suitable for each individual user based on their past purchase history and emotional state.

[1738] This invention relates to a store management system for improving the shopping experience in supermarkets, particularly a system that uses mobile communication devices, data processing devices, sensors, and an emotion recognition engine to analyze a user's emotions and make appropriate suggestions based on the analysis.

[1739] When a user launches an application, the mobile communication device (e.g., a smartphone or tablet) sends a request to a data processing device (server) to obtain the user's past purchase history and recommended product information. This allows the device to display a list of previously purchased products and the store's recommended products. As a specific example, previously purchased milk or bread may be displayed, with discounted butter recommended next to them.

[1740] When a user selects an item in a supermarket, a sensor (e.g., an RFID reader or barcode scanner) automatically reads the item's information and transmits it to a data processing device via a mobile communication device. The data processing device analyzes the item's information, checks the price and stock information of the item, and displays the total price on the terminal in real time. For example, when an apple is placed in a basket, the sensor detects the apple and transmits the information to the terminal, instantly updating the total price on the screen.

[1741] Furthermore, if a user wishes to search for a specific dish or product, the request is made using a voice recognition function. The mobile communication device converts the user's voice input into text data and sends it to the data processing device. The data processing device creates a list of necessary products based on the user's request and identifies the location of each product. Based on the received information, the mobile communication device displays the product locations on a map and navigates the optimal shopping route. For example, in response to a request to "make curry," a list of necessary ingredients and their locations are displayed on a map.

[1742] The application incorporates the latest store advertising information, allowing users to check information on discounted and special sale items. Users can add discounted items to their shopping list, and this information is synchronized with the data processing device's database. This allows the optimal shopping route to be recalculated and displayed on the mobile communication device. For example, if a user sees a flyer and adds a discounted curry roux to their shopping list, the optimal route recalculated by the data processing device will be displayed on the device.

[1743] Furthermore, to understand the user's emotional state, facial expressions and voice are collected using the camera and microphone of the mobile communication device and analyzed by an emotion recognition engine. The emotion recognition engine analyzes the user's emotional state and provides appropriate product information based on that analysis. For example, if the user looks tired, it may recommend relaxing aroma candles or herbal tea.

[1744] This allows users to not only efficiently search and purchase products, but also receive personalized shopping suggestions.

[1745] Prompt statement example:

[1746] 1. "I want to develop a system to simplify supermarket shopping. It includes functionality to recognize user emotions and recommend products based on those emotions. Can you please explain the detailed process of this system?"

[1747] 2. "Please explain in detail how smart systems can be used to improve the shopping experience at supermarkets. In particular, I would like to know about the use of voice recognition and emotion engines."

[1748] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1749] Step 1:

[1750] A user launches a store management system application using a mobile communication device. When the user launches the app, the device sends a request for initial data to the server. The server obtains the user's past purchase history and recommended product information and sends this to the device. The device displays the received data on the screen, allowing the user to view a list of products they have purchased in the past and recommended products from the supermarket. For example, a list of milk, bread, and butter that the user has purchased in the past, as well as discounted products, is displayed.

[1751] input:

[1752] User launches app

[1753] output:

[1754] Past purchase history

[1755] Recommended product information

[1756] Step 2:

[1757] When a user places an item in a basket at a supermarket, a sensor automatically reads the item's information. This information is sent to a server via the terminal. The server analyzes the received product information and checks the price and stock information of the item. The analyzed information is sent to the terminal in real time, and the terminal updates and displays the total price. For example, when a user puts an apple in the basket, the sensor reads the apple's information, and the terminal updates the total price in real time.

[1758] input:

[1759] Product information added to the user's cart

[1760] output:

[1761] Real-time total amount

[1762] Product price and availability information

[1763] Step 3:

[1764] When a user wants to search for a specific dish or product, they make a request using the voice recognition function. The user's voice input is converted into text data by the device and sent to the server. The server creates a product list based on the user's voice request and identifies the location of each product. The device uses the information from the server to display the product locations on a map and navigates the user to the optimal shopping route. For example, if a user makes a voice request such as "I want to make curry," a list of the necessary ingredients and their locations will be displayed on a map.

[1765] input:

[1766] User voice request

[1767] output:

[1768] Location of food and products

[1769] The best shopping route

[1770] Step 4:

[1771] The user can check the latest store advertising information embedded in the device. The user can then check the displayed discounted and special sale items and add the discounted items to their shopping list. The added product information is synchronized with the server's database, and the server sends a recalculated optimal shopping route to the device. The device displays the new route information, allowing the user to shop efficiently. For example, if the user sees a flyer and adds a discounted curry roux to their shopping list, a new optimal route will be displayed on the device.

[1772] input:

[1773] Adding discounted products to users

[1774] output:

[1775] Updated shopping route

[1776] Discounted product information

[1777] Step 5:

[1778] While the user is using the application, the device uses a camera and microphone to collect the user's facial expressions and voice. The device sends the collected emotional data to a server, which then uses an emotion recognition engine to analyze the user's emotional state. Based on the analysis results, the server generates recommended product information according to the user's emotional state and sends it to the device. The device then displays this information to the user. For example, if the user looks tired, the emotion recognition engine will recommend products with a relaxing effect (aroma candles or herbal tea).

[1779] input:

[1780] User emotion data

[1781] output:

[1782] Emotion-based product recommendations

[1783] (Application example 2)

[1784] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1785] Traditional shopping experiences in brick-and-mortar stores often require users to search around the store or wait in line at the cash register for long periods of time, resulting in inefficiency. Furthermore, product recommendations are often made without taking the user's emotional state into consideration, which can lead to lower user satisfaction. There is a need to improve the user's shopping experience by solving these problems.

[1786] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for the user to launch an application for the facility using a mobile information terminal; means for automatically reading product information using a detector and transmitting the relevant product information to the information processing device; means for making an automatic payment based on the product information received from the information processing device; means for receiving a shopping request from the user using a voice recognition function and displaying the location of the relevant product on a map; means for adding discounted products to a shopping list in conjunction with advertising information; and means for recognizing the user's emotional state using facial expression analysis and voice analysis and providing recommended product information based on the emotional state. This makes it possible to make individual shopping suggestions according to the user's emotional state, thereby providing an efficient and satisfying shopping experience.

[1787] A "personal digital assistant" is an electronic device that can be carried by a user and is a device for executing applications and performing data communication.

[1788] "Store facilities" refer to physical stores such as supermarkets and other retail outlets where customers visit in person to shop.

[1789] An "application" is a program that runs on a mobile information terminal and provides support functions to help users shop efficiently.

[1790] A "detector" is a device that automatically reads product information and is a sensor that obtains information using barcodes, RFID, etc.

[1791] The "information processing device" is a computer system such as a server, which receives, analyzes, and stores the product information transmitted from the detector.

[1792] "Automatic payment" is a system in which a user automatically pays for products purchased via an information processing device without having to line up at the cash register.

[1793] The "voice recognition function" is a technology that converts the user's voice into text data, and is a function that receives and analyzes shopping requests.

[1794] "Advertising information" is information about discounted products and special sales offered by the facility, and is displayed within the application to support the user's shopping experience.

[1795] "Discounted products" are products offered at special prices based on advertising information, and are intended to encourage users to consume.

[1796] "Facial expression analysis" is a technology that uses a camera to analyze a user's facial expressions and identify their emotional state.

[1797] "Voice analysis" is a technology that uses a microphone to analyze the nuances of a user's voice and words and identify their emotional state.

[1798] "Emotional state" refers to the user's current psychological state, and expresses emotions such as happiness, sadness, fatigue, etc.

[1799] "Recommended product information" is information that suggests products that the user may be interested in based on the user's past purchase history and current emotional state.

[1800] The present invention relates to a smart system that enables a user to launch an application at a shopping facility using a mobile information terminal to improve the shopping experience. Specific embodiments of the present invention will be described in detail below.

[1801] System Configuration

[1802] This system is composed of several elements, including a mobile information terminal, a detector, an information processing device, a voice recognition function, an advertising information system, a facial expression analysis and voice analysis system, and an automatic payment system.

[1803] Hardware and software used

[1804] Mobile information terminal: An electronic device, such as a smartphone or tablet, that runs applications and can be easily operated by the user.

[1805] Detector: A sensor that reads product information, such as a barcode scanner or RFID reader.

[1806] Information processing device: A system that runs on a server or in a cloud computing environment and performs data analysis and information storage.

[1807] Speech recognition function: Technology that converts speech to text, such as Google Speech Recognition API.

[1808] Advertising information system: A platform that manages special sales information and discounted products at visiting facilities.

[1809] Facial and voice analysis systems: Technologies for identifying a user's emotional state. For example, camera devices and Adaptive Recognition APIs, microphone devices and voice analysis software.

[1810] Automated payment systems: payment gateways and mobile payment solutions.

[1811] Processing flow

[1812] 1. Launching the application

[1813] When a user starts an application for a facility using a mobile information terminal, the terminal transmits a request to an information processing device to obtain past purchase history and recommended product information.

[1814] 2. Product Scan

[1815] When a user places an item in the cart, the detector automatically reads the item information and sends it to an information processing device, which analyzes the information and updates the total price in real time.

[1816] 3. Automatic payments

[1817] Once shopping is complete, the terminal will process the payment through an automatic payment system based on the product information received from the information processing device, thereby reducing waiting times at the cash register.

[1818] 4. Product search using voice recognition

[1819] When a user makes a shopping request using the voice recognition function, the terminal converts the voice request into text data and sends it to the information processing device. The information processing device identifies the location of the product based on the user's request and displays it as a map on the terminal.

[1820] 5. Linking with advertising information

[1821] The application displays the latest advertising information and allows users to add discounted items to their shopping list, which causes the information processing device to recalculate the optimal shopping route and display it to the user.

[1822] 6. Product Recommendations Based on Emotion Recognition

[1823] The device's camera and microphone collect the user's facial expressions and voice, which are then sent to an information processing device. The device uses facial expression and voice analysis systems to recognize the user's emotional state and provide recommended product information based on that emotion.

[1824] Specific examples

[1825] Example of displaying past purchase history: When a user launches the app, the app displays the milk and bread they previously purchased and suggests related products.

[1826] Product scanning and real-time price updates: When a user places an apple in the basket, the detector reads the product information and the total price is updated in real time on the terminal screen.

[1827] Example of a voice recognition request: When a user requests by voice, "I want to make curry," the necessary ingredients and their locations are displayed on a map.

[1828] A concrete example of emotion recognition: If a user has a tired expression, the facial expression analysis system will recognize this and suggest relaxing aroma candles or herbal tea.

[1829] Prompt Sentence Examples

[1830] User: Please use the camera to analyze your face.

[1831] ---

[1832] Facial image:

[1833] [User's face image]

[1834] ---

[1835] System: The user appears tired. Try a relaxing scented candle or herbal tea.

[1836] In this way, the present invention allows for personalized shopping suggestions based on the user's emotional state, providing an efficient and satisfying shopping experience.

[1837] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1838] Step 1:

[1839] The user launches an application for the facility they are visiting using their mobile information terminal. They enter their user ID, and the terminal sends a request to the server. This request includes the user ID, and the server uses this information to obtain the user's past purchase history and recommended product information, which it then sends to the terminal. The terminal then displays the received information on its screen.

[1840] Step 2:

[1841] When a user places an item in the cart, a detector (such as a barcode scanner or RFID reader) reads the item's identification information. This identification information is sent to the server via the terminal. The server analyzes the item's price and inventory information and calculates the total price in real time. This information is then sent back to the terminal and displayed on the terminal screen.

[1842] Step 3:

[1843] Once the user has completed their purchase, they press the "Pay" button within the application. The device sends all of the purchase information to the server, which then uses an automated payment system to process the payment. Payment completion information is then sent from the server to the device, and the user is notified.

[1844] Step 4:

[1845] When a user wants to search for a specific product, they input a voice command using the voice recognition function. The device converts the user's voice into text using the Google Speech Recognition API and sends it to the server. The server uses this text data to search for product location information and sends it to the device as a map. The device then displays the map on the screen and navigates the user to the optimal shopping route.

[1846] Step 5:

[1847] The latest advertising information is displayed within the application. The user checks discounted items and selects the ID of the discounted item to add it to their shopping list. The device sends this information to the server, which then recalculates the optimal shopping route and sends it to the device. The device then displays the new route.

[1848] Step 6:

[1849] The device's camera and microphone are used to collect the user's facial expressions and voice, which are then sent to the server. The server then uses facial and voice analysis systems to recognize the user's emotional state. It then calculates recommended product information based on the user's emotional state and sends this to the device. The device then displays recommended product information based on the user's emotion on the screen.

[1850] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1851] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1852] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1853] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1854] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1855] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1856] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1857] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1858] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1859] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1860] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1861] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1862] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1863] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1864] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1865] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1866] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1867] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1868] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1869] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1870] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1871] The following is further disclosed regarding the above embodiment.

[1872] (Claim 1)

[1873] means for a user to launch a supermarket application using a mobile device;

[1874] a means for automatically scanning product information by a sensor and transmitting the corresponding product information to a server;

[1875] means for automatically making payments based on the product information received from the server;

[1876] a means for receiving a shopping request from a user using a voice recognition function and displaying the location of the corresponding product on a map;

[1877] The system is linked to flyer information and includes a means for adding discounted items to a shopping list.

[1878] (Claim 2)

[1879] The system according to claim 1, which proposes an optimal shopping route based on voice recognition and displays the route on a map.

[1880] (Claim 3)

[1881] 2. The system according to claim 1, wherein recommended product information is provided based on the user's past purchase history.

[1882] "Example 1"

[1883] (Claim 1)

[1884] A means for a user to activate software of an information processing device using a mobile device;

[1885] means for automatically reading product information by a detection device and transmitting the corresponding product information to a data processing device;

[1886] means for making an automatic payment based on the product information received from the data processing device;

[1887] a means for receiving a shopping request from a user using a voice recognition function and displaying a guide showing the location of the corresponding product;

[1888] Linking with public relations information, there is a way to add special sale items to your shopping list,

[1889] A means for providing recommended product information based on the user's past purchase history;

[1890] A way to display the total amount in real time

[1891] ...

[1892] A system including:

[1893] (Claim 2)

[1894] The system according to claim 1, wherein the system proposes an optimal shopping route based on voice recognition and displays guidance.

[1895] (Claim 3)

[1896] 2. The system according to claim 1, wherein recommended product information is provided based on the user's past purchase history.

[1897] "Application Example 1"

[1898] (Claim 1)

[1899] A means for a user to launch a store application using a mobile terminal;

[1900] a means for automatically scanning product information by a sensor and transmitting the corresponding product information to a server;

[1901] means for automatically making payments based on the product information received from the server;

[1902] a means for receiving a shopping request from a user using a voice recognition function and displaying the location of the corresponding product on a map;

[1903] Link with the latest information and add discounted items to your shopping list.

[1904] A means to obtain recommended product information and past purchase history and display them to users,

[1905] The system includes a means for optimal navigation within the store based on a map display.

[1906] (Claim 2)

[1907] The system according to claim 1, which proposes an optimal shopping route based on voice recognition and displays the route on a map.

[1908] (Claim 3)

[1909] 2. The system according to claim 1, wherein recommended product information is provided based on the user's past purchase history.

[1910] "Example 2: Combining Emotion Engines"

[1911] (Claim 1)

[1912] A means for a user to launch a store management system application using a mobile communication device;

[1913] means for automatically reading product information by a sensor and transmitting the product information to a data processing device;

[1914] means for automatically making payments based on the product information received from the data processing device;

[1915] a means for receiving a shopping request from a user using a voice recognition function and displaying the location of the corresponding product on a guidance display device;

[1916] A way to add discounted products to your shopping list in conjunction with advertising information,

[1917] A system that includes a means for analyzing emotions from a user's facial expressions and voice using an emotion recognition engine and providing recommended product information based on the analysis results.

[1918] (Claim 2)

[1919] The system according to claim 1, wherein the system proposes an optimal shopping route based on voice recognition and displays the route on a guidance display device.

[1920] (Claim 3)

[1921] 2. The system according to claim 1, wherein recommended product information is provided based on the user's past purchase history.

[1922] "Application example 2 when combining emotion engines"

[1923] (Claim 1)

[1924] A means for a user to launch an application for a facility using a mobile information terminal;

[1925] means for automatically reading product information by a detector and transmitting the corresponding product information to an information processing device;

[1926] means for making an automatic payment based on the product information received from the information processing device;

[1927] a means for receiving a shopping request from a user using a voice recognition function and displaying the location of the corresponding product on a map;

[1928] A way to add discounted products to your shopping list in conjunction with advertising information,

[1929] A means for recognizing the emotional state of a user using facial expression analysis and voice analysis, and providing recommended product information based on the emotional state;

[1930] A system including:

[1931] (Claim 2)

[1932] The system according to claim 1, which proposes an optimal shopping route based on voice recognition and displays the route on a map.

[1933] (Claim 3)

[1934] 10. The system of claim 1, wherein the system provides recommended product information based on the user's past purchase history and emotional state. [Explanation of symbols]

[1935] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for a user to launch a supermarket application using a mobile device; a means for automatically scanning product information by a sensor and transmitting the corresponding product information to a server; means for automatically making payments based on the product information received from the server; a means for receiving a shopping request from a user using a voice recognition function and displaying the location of the corresponding product on a map; The system is linked to flyer information and includes a means for adding discounted items to a shopping list.

2. 2. The system according to claim 1, wherein the system proposes an optimal shopping route based on speech recognition and displays the route on a map.

3. The system according to claim 1, wherein recommended product information is provided based on the user's past purchase history.

Images