Information processing method, information processing apparatus, and storage medium

Abstract

To enable a visually handicapped person to smoothly act at ease in an actual facility such as a store.SOLUTION: An information processing method executed in an information processing system in which a user terminal and a management server are connected via a predetermined network includes outputting information based on an action of a user in a real facility from the user terminal to the user by voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] Generally, when people with visual impairments such as blindness or low vision shop at physical stores such as convenience stores or supermarkets, they need to find and speak to store staff as soon as they enter the store to receive assistance with guiding them around the store, selecting products, and making payments.

[0003] On the other hand, in recent years, an increasing number of stores have been introducing self-checkouts and going unmanned in order to alleviate labor shortages and reduce labor costs.

[0004] For example, Patent Document 1 below discloses a technology in which a two-dimensional code is displayed on an open / close product shelf in a vending machine, an unmanned store, etc., a user reads the two-dimensional code on the product shelf with a purchaser terminal such as a smartphone, and when the user's removal of a product from the product shelf is detected from an image of a camera installed in the store, the removed product is added to the user's virtual cart. In Patent Document 1, payment for the products in the virtual cart is made using an online payment service. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2023-126727 Summary of the Invention [Problem to be solved by the invention]

[0006] However, as stores move towards self-checkouts and unmanned stores, the environment supporting the shopping of visually impaired people deteriorates, with fewer or no staff available to assist them, making it difficult for them to shop safely and smoothly.

[0007] For example, in the above-mentioned Patent Document 1, if there is no staff member in the store to assist with shopping, it is difficult for visually impaired people to reach the shelf where the desired product is displayed. Even if they reach the desired shelf, they are unable to check whether the product they pick up from the shelf is of satisfactory quality, and they may end up purchasing a product that is close to its expiration date or a product different from what they wanted, which presents a problem that makes it difficult for visually impaired people to shop with peace of mind.

[0008] Furthermore, not only in stores, but also in various other facilities (including restaurants, hotels, sports gyms, and private homes), such as public or community facilities such as city halls, libraries, and train stations (or station buildings), the current situation is such that it is difficult for visually impaired people to move around smoothly and safely. This makes it difficult for visually impaired people to move around safely and smoothly.

[0009] Therefore, an object of the present disclosure is to provide an information processing method, an information processing device, and a program that enable visually impaired people to move around safely and smoothly in actual facilities such as stores. [Means for solving the problem]

[0010] An information processing method according to one embodiment of the present disclosure is an information processing method executed in an information processing system in which a user terminal and a management server are connected via a predetermined network, and includes outputting information based on the user's behavior within a physical facility to the user via audio from the user terminal.

[0011] An information processing device according to one embodiment of the present disclosure is an information processing device connected to a management server via a predetermined network, and includes an information acquisition means for acquiring information based on a user's behavior within a physical facility, and an audio output means for audio outputting information based on the user's behavior within the physical facility to the user.

[0012] A program according to one embodiment of the present disclosure is a program for causing a processor provided in an information processing device connected to a management server via a predetermined network to function, and causes the processor to execute a process of acquiring information based on a user's behavior within a physical facility and a process of outputting information based on the user's behavior within the physical facility to the user by voice. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic diagram illustrating an example of a schematic configuration of a product sales system as an information processing system according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating a more detailed example of the functional configuration of a management server according to the first embodiment of the present disclosure. [Figure 3] 1 is a diagram showing an example of store information managed in a store information database (DB) according to a first embodiment of the present disclosure. [Figure 4] FIG. 2 is a diagram showing an example of product shelf information managed in a product shelf information DB according to the first embodiment of the present disclosure. [Figure 5] 2 is a diagram showing an example of basic product information managed in a basic product information DB according to the first embodiment of the present disclosure. FIG. [Figure 6] FIG. 2 is a diagram showing an example of product supplemental information managed in a product supplemental information DB according to the first embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram showing an example of review-related information managed in a review-related information DB according to the first embodiment of the present disclosure. [Figure 8] 2 is a diagram showing an example of usage-related information managed in a usage-related information DB according to the first embodiment of the present disclosure; FIG. [Figure 9] FIG. 2 is a block diagram illustrating a schematic configuration example of a user terminal according to the first embodiment of the present disclosure. [Figure 10] 4 is a flowchart showing an operation example of the product sales system according to the first embodiment of the present disclosure. [Figure 11] FIG. 3 is a diagram showing an example of a standby screen according to the first embodiment of the present disclosure. [Figure 12] FIG. 2 is a diagram showing an example of a product shelf guide screen according to the first embodiment of the present disclosure. [Figure 13] FIG. 2 is a diagram showing an example of a basic product information providing screen according to the first embodiment of the present disclosure. [Figure 14] FIG. 10 is a diagram showing an example of another standby screen according to the first embodiment of the present disclosure. [Figure 15] FIG. 10 is a diagram showing an example of a product additional information providing screen according to the first embodiment of the present disclosure. [Figure 16] FIG. 2 is a diagram showing an example of a product confirmation screen in a virtual cart according to the first embodiment of the present disclosure. [Figure 17] 10 is a flowchart showing a specific example of check-in processing according to the first embodiment of the present disclosure. [Figure 18] FIG. 3 is a diagram showing an example of a store information providing screen according to the first embodiment of the present disclosure. [Figure 19] 10 is a flowchart showing a specific example of a payment process according to the first embodiment of the present disclosure. [Figure 20] FIG. 10 is a diagram showing an example of a payment error notification screen according to the first embodiment of the present disclosure. [Figure 21] FIG. 10 is a diagram showing an example of a virtual cart product cancellation operation request screen according to the first embodiment of the present disclosure. [Figure 22] FIG. 10 is a diagram showing an example of an alternative payment selection screen according to the first embodiment of the present disclosure (part 1). [Figure 23] FIG. 10 is a diagram showing an example of an alternative payment selection screen according to the first embodiment of the present disclosure (part 2). [Figure 24] 10 is a flowchart illustrating an example of a payment approval process according to the first embodiment of the present disclosure. [Figure 25]FIG. 10 is a schematic diagram illustrating an example of a schematic configuration of a product sales system as an information processing system according to a second embodiment of the present disclosure. [Figure 26] FIG. 10 is a block diagram illustrating a more detailed configuration example of a management server according to a second embodiment of the present disclosure. [Figure 27] 10 is a flowchart showing an operation example of the product sales system according to the second embodiment of the present disclosure. [Figure 28] FIG. 10 is a schematic diagram illustrating an example of a schematic configuration of a product sales system as an information processing system according to a third embodiment of the present disclosure. [Figure 29] FIG. 11 is a block diagram illustrating a more detailed configuration example of a management server according to a third embodiment of the present disclosure. [Figure 30] 10 is a flowchart illustrating an example of the operation of the product sales system according to the third embodiment of the present disclosure. [Figure 31] FIG. 1 is a hardware configuration diagram illustrating an example of an information processing device that realizes various functions according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Several embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that in the following embodiments, the same components are assigned the same reference numerals, and redundant description will be omitted. Furthermore, the following description will exemplify a case in which a visually impaired user U purchases a product P at a physical store T, but is not limited to this. For example, the user U may not be visually impaired. Furthermore, instead of the physical store T, various actual facilities (including ordinary homes as facilities; hereinafter, also referred to as physical facilities) such as public or public facilities such as city halls, libraries, and train stations (or station buildings) as well as other facilities such as restaurants, hotels, and sports gyms, or ordinary homes, may also be used.

[0015] First Embodiment First, a first embodiment of the present disclosure will be described in detail with reference to the drawings.

[0016] (Product Sales System 1) 1 is a schematic diagram showing an example of the overall configuration of a product sales system as an information processing system according to a first embodiment of the present disclosure. As shown in FIG. 1, the product sales system 1 according to this embodiment includes a management server 10, a payment server 20, a store system 40, and a user terminal 50, all of which are connected via a network 90.

[0017] The network 90 may be a predetermined network such as the Internet or a wide area network (WAN). A virtual private network such as an Internet VPN (Virtual Private Network) or an IP-VPN (Internet Protocol Virtual Private Network) may be established between the management server 10 and the payment server 20 or between the management server 10 and the store system 40. Furthermore, a review site may be provided on the network 90 where purchasers and users can post reviews of products and the like and view posted reviews.

[0018] Management Server 10 The management server 10 is a server for comprehensively managing the services (hereinafter also referred to as purchase support services) provided to the user U through the user terminal 50 according to this embodiment. This management server 10 may be configured with a single or multiple linked physical servers, or may be configured with a virtual server such as a cloud server. The configuration of the management server 10 will be described in more detail later with reference to FIG. 2 etc.

[0019] Payment Server 20 The payment server 20 is a server for executing payments (payments) for products purchased by the user U at the store 40. This payment server 20 may be a server operated by a payment agency that handles cashless payments online, such as credit card payments, ID payments, carrier payments, and electronic money payments.

[0020] In this description, credit card payment may refer to online payment using a credit card. ID payment may refer to online payment in which payment is made by logging in to an external service using an account ID and password, such as PayPay (registered trademark), Amazon Pay (registered trademark), or Rakuten Pay (registered trademark). Carrier payment may refer to online payment in which payment is made through authentication using the ID and password of each carrier company. Electronic money payment may refer to online payment in which payment is made from a pre-charged balance, such as Rakuten Edy (registered trademark) or Mobile Suica (registered trademark).

[0021] Physical store T The physical store T may be a store such as a supermarket, convenience store, or shopping mall where users U can actually pick up, select, and purchase products that are on display.

[0022] A store-side system 40 may be installed in the physical store T to manage product ordering, arrival, inventory, product prices, manufacturing dates and times, best-before (use-by) dates, display positions, etc. This store-side system 40 may include, for example, an information processing terminal such as a personal computer or a tablet terminal. The store-side system 40 may also include a POS terminal in a POS (Point Of Sale) system. The store-side system 40 may be a system that includes a POS system, or may be a system separate from the POS system.

[0023] A local area network (LAN) 49 connected to the network 90 via a gateway may be established in the physical store T. The LAN 49 may include a wireless LAN (including Wi-Fi). In this case, the user terminal 50 of the user U present in the physical store T may be able to connect to the LAN 49 via the wireless LAN. However, this is not limited to this, and the user terminal 50 may also be able to connect to the network 90 via a wireless LAN established in the physical store T separately from the LAN 49.

[0024] In a physical store T, product shelves R are installed on which various products P are displayed. An IC tag (hereinafter also referred to as a shelf tag) 41 that can read information by contact or contactless may be provided in a predetermined position on each product shelf R (for example, on the right side of each shelf). This shelf tag 41 may store information (hereinafter also referred to as product shelf information) about the category of products (for example, bread, milk, etc.) displayed on this product shelf R.

[0025] Each product P displayed on the product shelf R may be provided with an individual IC tag (hereinafter also referred to as product tag) 42. This product tag 42 may store basic information (hereinafter also referred to as basic product information) about this product P. The basic product information may include a product ID for uniquely identifying the product P to which this product tag 42 is attached (or the production lot of this product P).

[0026] Here, for example, RFID (Radio Frequency Identification) tags can be used as the IC tags 41 and 42. However, there is no limitation to this, and various modifications can be made to the IC tags as long as they can be read by a simple operation such as bringing the user terminal 50 close to or in contact with the IC tags.

[0027] Furthermore, if the user terminal 50 has a camera function, a two-dimensional code such as a QR code (registered trademark) or a one-dimensional code such as a barcode may be used instead of the product tag 42. In this case, the user U can obtain information about the product P (at least the product ID) by positioning the product P within the angle of view of the camera of the user terminal 50. At this time, to prevent the user terminal 50 from reading codes of products P other than the product P that the user U is holding, the three-dimensional range for reading the code may be limited using a distance measuring sensor such as a ToF (Time of Flight) sensor provided in the user terminal 50. Furthermore, the user U may be guided by voice or the like to indicate the position of the code on the product P, thereby guiding the user so that the code is within the angle of view of the camera.

[0028] For example, one or more cameras 43 may be installed near the entrance or exit of the physical store T to manage users entering (checking in) and leaving (checking out) the physical store T. In addition, one or more cameras 43 may be installed in the sales floor of the physical store T, positioned to minimize blind spots in order to detect users' movements and behavior.

[0029] The camera 43 may be connected to the store system 40 via a dedicated line such as a LAN 49 or a USB (Universal Serial Bus) and may transmit to the store system 40 video images obtained by continuous shooting or still images or video images obtained by periodic or intermittent shooting. In response to this, the store system 40 may perform predetermined processing on the input video images or still images (hereinafter, these will be collectively referred to as image data). Alternatively, when the management server 10 processes the image data, the image data acquired by the camera 43 may be sent directly to the management server 10 via the LAN 49 and the network 90, or may be transferred from the store system 40 to the management server 10.

[0030] User terminal 50 As mentioned above, the purchase support service provided by this embodiment is provided to user U via a user terminal 50 carried by user U when entering physical store T. The user terminal 50 may be a mobile terminal capable of information processing and network communication, such as a smartphone, mobile phone, or tablet terminal. Furthermore, the user terminal 50 may be owned by user U, or may be a mobile terminal loaned to user U by the physical store T or a service provider of the purchase support service according to this embodiment.

[0031] Here, since the purpose of the purchase support service according to this embodiment is to support purchases for visually impaired people, various contents related to this service are provided to the user U mainly as audio output from earphones 51 (which may be headphones, a headset, or the like) connected to the user terminal 50 (and / or audio output from the user terminal 50). Note that the connection between the user terminal 50 and the earphones 51 may be wired or wireless.

[0032] 30 review sites The review site 30 may be a site that collects and publishes user evaluations, such as ratings and comments posted by general consumers, regarding various products P. This review site 30 may be a site operated by a company, such as Tabelog (registered trademark) or Gurunavi (registered trademark), that is different from the service vendor that provides the purchase support service according to this embodiment, or may be a site operated by the service vendor according to this embodiment.

[0033] (Management Server 10) Next, the management server 10 according to this embodiment will be described in more detail. Note that, although this embodiment illustrates a case in which the management server 10 provides all of the functions of the purchase support service, this is not limiting, and it is also possible to configure the store-side system 40 to provide some or all of the functions exemplified below.

[0034] Fig. 2 is a block diagram showing a more detailed example of the functional configuration of the management server according to this embodiment. As shown in Fig. 2, the management server 10 includes a biometric information processing unit 101, an authentication processing unit 103, a check-in management unit 104, a virtual cart product management unit 105, a payment request unit 106, a communication unit 108, a store information management unit 111, a product shelf information management unit 112, a product basic information management unit 113, a product supplemental information management unit 114, a review-related information management unit 115, and a usage-related information management unit 116.

[0035] Furthermore, an authentication information database (DB) 102, a payment information DB 107, a store information DB 121, a product shelf information DB 122, a product basic information DB 123, a product supplemental information DB 124, a review-related information DB 125, and a usage-related information DB 126 are connected to the management server 10. These DBs may be configured as cloud storage or the like located on the network 90, or may be configured as a recording device or the like connected to or built into the management server 10.

[0036] Communications Department 108 The communication unit 108 is configured by, for example, a network interface card (NIC) and transmits and receives various data via the network 90.

[0037] Biometric information processing unit 101, authentication information DB 102, and authentication processing unit 103 The biometric information processing unit 101 extracts biometric information of the user U from image data acquired by the camera 43 of the physical store T. The extracted biometric information may be used to authenticate whether the user U has entered / exited the physical store T, approve payment for the user U, etc.

[0038] The authentication information DB102 stores authentication information for authenticating the user U, such as a user ID and password for uniquely identifying the user U, user information such as the name and address of the user U, and biometric information extracted from image data captured in the past of the user U.

[0039] The authentication processing unit 103 compares the biometric information extracted from the image data by the biometric information processing unit 101 with the authentication information stored in the authentication information DB 102 to identify which user the person detected in the physical store T is, and approve payment for user U.

[0040] The technology for extracting biometric information of user U from image data and the technology for enabling authentication using the extracted biometric information and secure communication during authentication (such as synchronization control and network separation) may utilize, for example, the technology disclosed in Japanese Patent No. 6551765. That is, the technology disclosed in Japanese Patent No. 6551765 may be applied to the embodiments of the present disclosure as appropriate. However, in the present embodiment, various processes, such as extracting biometric information from image data and approving payment for user U using the biometric information, may be executed by the store system 40, the management server 10, or the payment server 20. For simplicity of explanation, the following describes an example in which various processes are executed by the management system 10.

[0041] Check-in Management Department 104 The check-in management unit 104 manages the entry / exit of the user U into the physical store T based on the authentication result by the authentication processing unit 103. Then, when the check-in management unit 104 determines that the user U has entered the physical store T, it activates the virtual cart for this user U. Also, when the check-in management unit 104 determines that the user U has left the physical store T, it resets the virtual cart set for this user U.

[0042] In this embodiment, a camera is installed near the entrance / exit of the physical store T, and the user U is identified based on biometric information extracted from the camera image. The user U's entry / exit to the physical store T is determined by determining from the camera image which direction the user U passed through the entrance / exit. However, the method for determining the user U's entry / exit to the physical store T is not limited to this method. For example, an IC tag may be installed at the entrance / exit, and the user U may read this IC tag with the user terminal 50 when entering and / or leaving the store, thereby determining the user U's entry / exit to the physical store T. Alternatively, location information may be acquired from a GNSS (Global Navigation Satellite System) function or the like provided in the user terminal 50, and the user U's entry / exit to the physical store T may be determined based on this location information and the location of the physical store T.

[0043] · Virtual cart product management section 105 The virtual cart product management unit 105 manages a list of products P that the user U has instructed to purchase (i.e., products P stocked in the virtual cart). For example, when the user U instructs the purchase of a new product P, the virtual cart product management unit 105 newly registers the instructed product P in the virtual cart. When the user U instructs the deletion of a registered product P, the virtual cart product management unit 105 either deletes the specified product P from the virtual cart or reduces the number of the specified product P from the virtual cart.

[0044] Furthermore, the virtual cart product management unit 105 calculates the total price for the product P in the virtual cart. The calculation of the total price may be performed, for example, when the user U checks the products in the virtual cart or when the user U makes a payment, or when the product P is added to or deleted from the virtual cart.

[0045] Payment request unit 106 and payment information DB 107 The payment information DB 107 holds information for making a payment (also referred to as payment information), such as credit card information of the user U. When making a payment for products in the virtual cart, the payment request unit 106 requests the payment server 20 to make a payment for the total amount via the network 90, based on the payment information held in the payment information DB 107. Then, when the payment server 20 notifies the user U that the payment has been completed, the payment request unit 106 notifies the user U that the payment has been completed.

[0046] The number of pieces of payment information for each user U registered in the payment information DB 107 is not limited to one. For example, multiple pieces of payment information may be registered in the payment information DB 107, and one of these pieces of payment information may be managed as primary payment information.

[0047] Furthermore, biometric information extracted from a camera image may be used when the payment request unit 106 requests payment from the payment server 20. For example, when it is determined from a camera image that the user U has checked out (left the store) from the physical store T, the payment request unit 106 may be configured to automatically request the payment server 20 to make payment for the product P that the user U has taken out of the physical store T, based on the authentication result using the biometric information by the authentication processing unit 103. This allows the user U to complete payment for the purchased product P without using a payment medium such as the user terminal 50 or a credit card. For example, the technology disclosed in Japanese Patent No. 6551765 may be used for such payment request processing.

[0048] Store information management unit 111 and store information DB 121 The store information management unit 111 uses a store information DB 121 to manage information (hereinafter also referred to as store information) about the real store T that provides the purchase support service according to this embodiment.

[0049] An example of store information managed in the store information DB according to this embodiment is shown in Fig. 3. As illustrated in Fig. 3, the store information may include a store account, a password, a store ID, a store name, access information, and advertising information. In addition to these, the store information may also include information about the facilities of the physical store T, such as the location of restrooms and elevators.

[0050] The store account may be an account name that the physical store T set with the service provider (or was assigned by the service provider) when the physical store T signed a contract as the store side (service provider) for the purchase support service. The password may be authentication information required when the physical store T logs in to the store account managed by the management server 10.

[0051] The store ID may be identification information for uniquely identifying the physical store T. This store ID may be a value assigned to the physical store T by the service provider at the time of the contract.

[0052] The store name may be the name of the physical store T. The access information may be information for accessing the physical store T, such as the address, telephone number, URL (Uniform Resource Locator) of the store's homepage, business days, and business hours of the physical store T. The advertising information may be information about daily advertisements, such as the prices of products sold at the physical store T and special offers.

[0053] Of the store information managed in the store information DB121, at least the store name, access information, and advertising information may be registered or updated as needed by staff at the physical store T by logging in to the management server 10 from the store system 40 using a store account and password.

[0054] Product shelf information management unit 112 and product shelf information DB 122 The product shelf information management unit 112 manages information (hereinafter also referred to as product shelf information) regarding which product P is displayed in which position (such as on which row or on the left or right side) on each product shelf R installed in the physical store T using the product shelf information DB 122.

[0055] An example of product shelf information managed in the product shelf information DB 122 according to this embodiment is shown in Fig. 4. As exemplified in Fig. 4, the product shelf information can include a store ID, a product shelf ID, product shelf guide information, and a product ID.

[0056] The product shelf ID may be identification information for uniquely identifying each of multiple product shelves R installed in the physical store T. The product shelf guidance information may be information (e.g., text information) for providing voice guidance to the user U about the categories of products P displayed on the product shelves R.

[0057] The product ID may be identification information for uniquely identifying the product P. This product ID may be information that can uniquely identify each product ID, or may be information that can uniquely identify the lot in which the product P was manufactured.

[0058] Of the product shelf information managed in the product shelf information DB122, at least the product shelf ID, product shelf guide information, and product ID may be added or updated as needed by staff of the physical store T by logging in to the management server 10 from the store system 40 using a store account and password.

[0059] Furthermore, the product shelf information is information stored in shelf tags 41 attached to product shelves R in the physical store T. Storing the product shelf information in the shelf tags 41 may be performed appropriately in the store-side system 50, for example.

[0060] Product basic information management unit 113 and product basic information DB 123 The basic product information management unit 113 uses the basic product information DB 123 to manage basic information about the product P, such as the product name, price, and expiration date (hereinafter also referred to as basic product information), for each store and product.

[0061] An example of the basic product information managed in the basic product information DB 123 according to this embodiment is shown in Fig. 5. As illustrated in Fig. 5, if the product P is a food product, the basic product information may include, in addition to the product name, price, and best-before (use-by) date described above, items that comply with the food labeling standards under the Food Labeling Act and that the general consumer is likely to pay particular attention to or notice when purchasing, such as a store ID, product ID, manufacturer, content volume, storage method, and labeling of functional health foods. The basic product information may also include information generated depending on the time of purchase, such as point redemption information for a product P whose best-before (use-by) date is approaching.

[0062] The product name may be the name of the product P. The manufacturer may be the name of the manufacturer of the product P. The price may be the selling price of the product P at the physical store T. The best-before (use-by) date may be the best-before (use-by) date of the product P set by the manufacturer or the physical store P. The storage method may be information regarding the storage method of the product P. The health functional food indication may be information indicating if the product P is a nutrient functional food, a food for specified health uses (FOSHU), or a food with functional claims.

[0063] Of the basic product information managed in the basic product information DB 123, at least the price may be entered by a staff member of the physical store T logging in to the management server 10 using a store account and password from the store-side system 40. Other information may be entered directly into the management server 10 by a service provider of the purchase support service, or by a staff member of the physical store T logging in to the management server 10 using a store account and password from the store-side system 40, or by a staff member of the manufacturer accessing the management server 10 from a terminal installed on the manufacturer's side and entering the information.

[0064] The basic product information is information stored in a product tag 42 attached to the product P. The basic product information may be stored in the product tag 42, for example, in the store system 50 as appropriate.

[0065] Product supplemental information management unit 114 and product supplemental information DB 124 The product supplemental information management unit 114 uses the product supplemental information DB 124 to manage information about the product P that is not included in the product basic information (hereinafter also referred to as product supplemental information) for each store and product.

[0066] An example of product supplemental information managed in the product supplemental information DB 124 according to this embodiment is shown in FIG. 6. As illustrated in FIG. 6, if product P is a food product, the product supplemental information may include, in addition to a store ID and a product ID, items that comply with the food labeling standards under the Food Labeling Act, but are not included in the basic product information described above, such as the names of ingredients, the place of origin or country of origin of ingredients, additives, the date of manufacture, the amount of nutritional components, and calorie content. However, the product supplemental information is not limited to these, and may include various information related to the product. Furthermore, if the product supplemental information includes a wide variety of items, the product supplemental information may be divided into two or more groups, such as "Group 1" and "Group 2."

[0067] Each item of product supplemental information managed in the product supplemental information DB 124 may be entered directly into the management server 10 by the service provider of the purchase support service, or by staff at the physical store T logging in to the management server 10 using a store account and password from the store system 40, or by staff at the manufacturer accessing the management server 10 from a terminal installed on the manufacturer's side and entering the information.

[0068] Review-related information management unit 115 and review-related information DB 125 The review-related information management unit 115 manages information (hereinafter also referred to as review-related information) relating to user ratings (also referred to as reviews) of the product P using the review-related information DB 125.

[0069] An example of the review-related information managed in the review-related information DB 125 according to this embodiment is shown in Fig. 7. As illustrated in Fig. 7, the review-related information may include items such as ratings (e.g., scores on a 5-point scale) from users who have used the product P, and reviews and comments posted by users about the product P.

[0070] Each item of review-related information managed in the review-related information DB 125 may be automatically collected by the management server 10 from the review site 30 on the network 90 and registered / updated in the review-related information DB 125, or may be entered into the management server 10 by the service provider of the purchase support service, or may be entered by staff of the review site 30 accessing the management server 10.

[0071] Usage-related information management unit 116 and usage-related information DB 126 The usage-related information management unit 116 manages information relating to how to use the product P, how to cook it, and the like (hereinafter also referred to as usage-related information) using a usage-related information DB 126.

[0072] An example of the usage-related information managed in the usage-related information DB 126 according to this embodiment is shown in Fig. 8. As illustrated in Fig. 8, for example, if the product P is a food product, the usage-related information may include items such as the name of a recommended dish and its cooking method.

[0073] Each item of usage-related information managed in the usage-related information DB 126 may be entered into the management server 10 by the service provider of the purchase support service, or may be entered by a chef or user who has a contract with the service provider of the purchase support service accessing the management server 10.

[0074] The above-described product supplemental information, review-related information, and usage-related information are supplemental information about the product P. Therefore, in this description, these are also collectively referred to as additional product information.

[0075] (User terminal 50) Next, the user terminal 50 according to this embodiment will be described in more detail. Fig. 9 is a block diagram showing a schematic configuration example of the user terminal according to this embodiment. As shown in Fig. 9, the user terminal 50 includes a control unit 501, a mobile communication unit 502, a wireless communication unit 503, a camera 504, a sensor unit 505, an IC tag reader 506, a display 507, a microphone 508, a speaker 509, a power supply unit 510, and an interface (I / F) 511.

[0076] The control unit 501 is configured with an information processing unit such as a CPU (Central Processing Unit), and controls each unit and executes predetermined processing.

[0077] The mobile communication unit 502 may be a communication unit for performing mobile communication such as a fourth generation mobile communication system (4G) (including LTE (Long Term Evolution)) or a fifth generation mobile communication system. The mobile communication may include various types of communication such as voice calls and data communication (which may include VoIP (Voice over Internet Protocol)).

[0078] The wireless communication unit 503 may be, for example, a communication unit for performing wireless communication such as Bluetooth (registered trademark), LAN, infrared communication, etc. The earphones 51 for outputting audio to the user U may be, for example, connected to the wireless communication unit 503 so as to be able to communicate with each other.

[0079] The camera 504 may be, for example, an imaging unit that is disposed on the front and / or rear of the user terminal 50 and that captures still images and / or videos of a subject within the angle of view.

[0080] The sensor unit 505 may be, for example, various sensors such as a ToF sensor or an IMU (Inertial Measurement Unit).

[0081] The IC tag reader 506 may be, for example, a data reading unit, such as an RFID reader, for reading data stored in the IC tags 41 and 42 by performing short-range wireless communication with the shelf tag 41 attached to the product shelf R and the product tag 42 attached to the product P.

[0082] The display 507 may be a display unit for displaying various icons, messages, images, etc. to the user U. The display 507 may also have a touch panel function that allows the user U to input operations by touching (also referred to as tapping or clicking) with a fingertip or a dedicated pen, for example.

[0083] The microphone 508 may be, for example, a voice input unit for inputting voice when the user U talks to others or for inputting voice operations on the user terminal 50.

[0084] The speaker 509 may be an audio output unit for outputting audio when the user U talks with others, or for outputting various guidance messages and announcements.

[0085] The interface (I / F) 511 may be an interface for connecting to an external device via a wired connection such as a universal serial bus (USB), etc. The interface 511 may also include an earphone jack for connecting earphones 51 via a wired connection.

[0086] The power supply unit 510 may be a power supply unit for supplying power to each unit of the user terminal 50 and external devices connected to the interface 511 .

[0087] (Example of operation flow) Next, the operation of the product sales system 1 according to this embodiment will be described in detail with reference to the drawings. Note that, hereinafter, an example of the operation of the product sales system 1 will be described, mainly focusing on the operation of the user terminal 50. FIG. 10 is a flowchart showing an example of the operation of the product sales system according to this embodiment. Also, FIGS. 11 to 16 are diagrams showing examples of screens displayed on the user terminal 50 and guidance audio output to the user U each time the operation illustrated in FIG. 10 is performed.

[0088] As shown in FIG. 10, in this operation, when a user U enters a physical store T, a check-in process is first executed (step S101). Details of the check-in process will be described later using FIGS. 17 and 18, etc. In this check-in process, a dedicated app pre-installed in the user terminal 50 is launched in order to receive the purchase support service according to this embodiment. In addition, it is assumed that the user U is wearing earphones 51 connected to the user terminal 50.

[0089] When the check-in process for user U at physical store T is completed, user terminal 50 enters a state of waiting for reading of the IC tag (step S102). In the waiting state, as illustrated in FIG. 11, a waiting screen for reading the IC tag is displayed on display 507 of user terminal 50, and audio guidance such as the procedure for reading the IC tag may be output from earphone 51 (or speaker 509 if earphone 51 is not connected to user terminal 50; the same applies below). Also, in the waiting state, it is determined whether user U will check out (leave) physical store T (NO in step S102 and step S112).

[0090] When an IC tag is read by the IC tag reader 506 (first information acquisition means) of the user terminal 50 (YES in step S102), the user terminal 50 determines whether the read IC tag is a shelf tag 41 (step S103). If it is not a shelf tag 41 (NO in step S103), the operation proceeds to step S105. On the other hand, if it is a shelf tag 41 (YES in step S103), the user terminal 50 provides information to the user U about the category of products P displayed on this product shelf R, etc., based on the product shelf information (see FIG. 4) read from the shelf tag 41 (step S104), and returns to step S102. In the product shelf information, as exemplified in FIG. 12, the products displayed on the product shelf R (hereinafter also referred to as a product shelf information screen) are displayed on the display 507 of the user terminal 50, and audio information such as which category section the product shelf R belongs to and what products are displayed on this product shelf R may be output from the earphones 51.

[0091] The IC tag reader 506 of the user terminal 50 can acquire information from multiple IC tags within a communicable distance. Therefore, when reading an IC tag in step S102, multiple IC tags may be read. Therefore, in this embodiment, for example, the IC tag closest to the user terminal 50 may be identified using a received signal strength indicator (RSSI) function or the like provided in the IC tag reader 506, and the processes from step S102 onward may be performed on the information acquired from this IC tag. As a result, for example, if the IC tag read in step S102 is a shelf tag 41, the user U can identify the shelf R that was guided to in step S104 by spreading their arms and searching for the nearest shelf R among the surrounding shelves R. At this time, the direction (and distance) of the product shelf R based on the position and orientation of the user terminal 50 or the user U may be provided to the user U by voice or the like from the user terminal 50. However, the method of identifying the product shelf R or product P (i.e., IC tag) closest to the user terminal 50 is not limited to the method using RSSI, and may be modified in various ways, such as using a distance measuring sensor in the sensor unit 505.

[0092] Furthermore, since the IC tag reader 506 periodically reads the IC tag (e.g., every few seconds), if a notification is sent to the user U every time the IC tag is read, it may be difficult to provide the user U with accurate information, which may confuse the user U. Therefore, in this embodiment, for a product shelf R that has already been guided to the user once, guidance may not be provided consecutively, or guidance may not be provided again for a certain period of time (e.g., one minute or five minutes). As a result, if the user U approaches a product shelf R by feeling around or the like and there is no guidance for the product shelf R, the user U will know that the product shelf R they have approached is the product shelf R that was previously guided to, and will be able to identify which category of shelf the product shelf R belongs to and what products P are displayed on it from the previous guidance.

[0093] In step S105, the user terminal 50 determines whether the read IC tag is a product tag 42. If it is not a product tag (NO in step S105), the operation returns to step S102. On the other hand, if it is a product tag 42 (YES in step S105), the user terminal 50 provides the user U with basic information about the product P based on the basic product information (see FIG. 5) read from the product tag 42 (step S106), and proceeds to step S107. When providing basic product information, as illustrated in FIG. 13, a basic product information screen is displayed on the display of the user terminal 50, which presents basic information (basic product information) about the product P to the user U, such as the product name, manufacturer name, content volume, price, expiration date, and storage method of the product P, and this basic product information is provided to the user U by voice from the earphones 51 (first voice output means), and operation procedures for receiving more detailed information about the product P and operation procedures for purchasing the product P (adding it to a virtual cart) may also be provided to the user U by voice (second voice output means).

[0094] It is desirable that the positions of buttons (which may be icons) that the user U touches on the product basic information screen (see FIG. 13) and various screens (FIGS. 14 to 18, 20 to 23, etc.) described below are predetermined positions on the display 507. For example, as illustrated in FIG. 13, etc., the display 507 may be divided into five areas: the upper right corner, the upper left corner, the lower right corner, the lower left corner, and the center, and options and functions for inputting operations may be assigned to each area. In the specific example shown in FIG. 13, a button for requesting supplemental information (additional product information) about the product P is assigned to the area in the upper left corner, a button for checking the products in the virtual cart is assigned to the area in the upper right corner, a button for selecting to purchase the product P whose product tag 42 has been read is assigned to the area in the lower right corner, and a button for canceling the purchase of the product P is assigned to the area in the lower left corner.

[0095] Furthermore, step S102 illustrates the case where information stored in IC tag 41 or 42 read by IC card reader 506 is provided to user U (step S104 or S106), but the present invention is not limited to such a configuration. For example, as in the standby screen illustrated in FIG. 14, the user U may be prompted to select whether they currently desire guidance about product shelf R or product P, and if the read IC tag is the desired IC tag (shelf tag 41 or product tag 42), the information read from the IC tag may be provided to user U. In the example shown in FIG. 14, a button for requesting guidance about product shelf R is located on the upper side of display 507, and a button for requesting guidance about product P is located on the lower side, and audio guidance for operating these buttons is output from earphone 51.

[0096] 14, if the user U selects "Product shelf information", the user terminal 50 proceeds to step S104 (i.e., provides product shelf information to the user U) if the IC tag read in step S102 is the shelf tag 41 (YES in step S103), and may return to step S102 without going through step S106 if it is not the shelf tag 41 (i.e., if it is the product tag 42) (NO in step S103). On the other hand, if the user U selects "Product information", the user terminal 50 proceeds to step S106 (i.e., executes the operations of steps S106 to S111 as appropriate) if the IC tag read in step S102 is the product tag 42 (YES in step S105), and may return to step S102 without going through step S104 if it is not the product tag 42 (i.e., if it is the shelf tag 41) (YES in step S103). If the user U has not picked up the product P when "Product Information" is selected, voice guidance such as "Please bring the product closer" may be given. After that, the user U may pick up the product P and bring it closer to the user terminal 50 in accordance with the voice guidance, and the processing from step 106 onwards may be executed.

[0097] 11 and the standby screen illustrated in FIG. 14, audio guidance may be output from the earphone 51 periodically (for example, every five minutes) to the user U. Also, after the IC tag 41 or 42 has been read once (in the example illustrated in FIG. 14, after the IC tag 41 or 42 selected by the user U ("Product Information" or "Product Shelf Information") has been read) (after YES in step S102), reading of the next IC tag 41 or 42 may be permitted after a certain period of time (for example, five minutes) has elapsed.

[0098] In step S107, the user terminal 50 determines whether the user U has requested additional product information about the product P on the product basic information screen illustrated in FIG. 13 (for example, whether the "More Detailed Information" button located in the upper right corner of the product basic information screen is selected). If additional product information is not requested (NO in step S107), the operation proceeds to step S110. On the other hand, if additional product information is requested (YES in step S107), the user terminal 50 accesses the management server 10 via the network 90 (or the LAN 49 and the network 90; the same applies below) and appropriately acquires additional product information (see the product supplemental information illustrated in FIG. 6, the review-related information illustrated in FIG. 7, and the usage-related information illustrated in FIG. 8) managed in association with the product ID included in the basic product information read from the product tag 42 of the product P in step S102 (second information acquisition means) (step S108).

[0099] Then, the user terminal 50 executes an additional product information providing process for providing supplemental information about the product P to the user U based on the additional product information acquired from the management server 50 (step S109), and proceeds to step S110. In providing the additional product information, as exemplified in Fig. 15, buttons are displayed on the display 507 of the user terminal 50 so that the user U can select which item in the additional product information he or she desires a more detailed explanation about, and audio guidance indicating which item is assigned to which position on the display 507 may be output from the earphones 51 (third audio output means).

[0100] In the specific example shown in Fig. 15, a button for selecting an item related to the ingredients of product P is allocated to the area in the upper right corner, a button for selecting an item related to the place of origin or country of origin is allocated to the area in the upper left corner, a button for selecting an item related to user ratings is allocated to the area in the lower right corner, and a button for selecting an item related to recommended cooking methods is allocated to the area in the lower left corner. Also, a button for returning to the previous screen (for example, the basic product information screen shown in Fig. 13) may be allocated to the center of the display 507. When the user U touches each button, additional product information linked to the respective button may be provided to the user U by voice (third voice output means).

[0101] In the management server 10, in response to an inquiry from the user terminal 50, any one of the product supplemental information management unit 114, the review-related information management unit 115, and the usage-related information management unit 116 executes processing based on the product ID notified together with the inquiry and the inquiry content (for example, the item selected by the button in FIG. 15). For example, when an inquiry is made about any item in the product supplemental information of product P (for example, when the button in the upper right corner or the upper left corner in FIG. 15 is selected), the product supplemental information management unit 114 uses the product ID to refer to the product supplemental information DB 124, thereby acquiring the relevant item from the product supplemental information about product P, and transmits this from the communication unit 108 to the user terminal 50 via the network 90. Similarly, when review-related information about product P is inquired (for example, when the button in the lower right corner in Figure 15 is selected), the review-related information management unit 115 obtains review-related information about product P by referring to the review-related information DB 125 using the product ID and transmits this from the communication unit 108 to the user terminal 50 via the network 90, and when usage-related information about product P is inquired (for example, when the button in the lower left corner in Figure 15 is selected), the usage-related information management unit 116 obtains usage-related information about product P by referring to the usage-related information DB 126 using the product ID and transmits this from the communication unit 108 to the user terminal 50 via the network 90.

[0102] Furthermore, in cases where there are more items than the number of buttons that can be set on display 507, such as supplementary product information, taking usability into consideration, it may be possible to provide the user with a hierarchical structure of options, for example, by directing user U to a screen that offers options for other items, with the button in the upper left corner of Figure 15 set as "Other Items."

[0103] In step S110, the user terminal 50 determines whether the user U will purchase the product P whose product tag 42 was read in step S102. For example, when the user U touches the purchase button (a button for selecting "Yes" in response to a question about whether to purchase) set in the lower right corner of the display 507 on the product basic information screen shown in FIG. 13, the user terminal 50 determines that the user U will purchase the product P in question.

[0104] If the user U does not purchase the product P (NO in step S110), the operation returns to step S102. On the other hand, if the user U purchases the product P (YES in step S110), the user terminal 50 notifies the management server 10 of the purchase of the corresponding product P. As a result, the corresponding product P is added to the virtual cart for the user U managed in the virtual cart product management unit 105 of the management server 10, or the number of the corresponding product P is incremented by 1 (step S111). Thereafter, the operation returns to step S102.

[0105] The products in the virtual cart may be managed not only by the management server 10 but also by the user terminal 50. Whether the user U actually purchased the product P (i.e., whether the product P was placed in a real shopping cart or shopping basket) may be determined based on image data acquired by a camera 43 installed in the physical store T.

[0106] In step S112, as mentioned above, it is determined whether the user U checks out (exits) the physical store T. Whether or not to check out may be determined automatically based on, for example, whether the user U has left the building of the physical store T. This determination may be made using location information of the user terminal 50 or image data (specifically, biometric information extracted from the image data) acquired by a camera 43 installed in the physical store T. For example, if the current location of the user terminal 50 moves from inside the physical store T to outside the store, it may be determined that the user U has checked out of the physical store T. Alternatively, it may be determined that the user U has checked out of the physical store T if it is determined from image data acquired by the camera 43 that the user U has moved through an entrance / exit of the physical store T from inside the store to outside the store.

[0107] Alternatively, it may be determined whether the user U checks out from the physical store through a virtual cart product confirmation screen, for example, as illustrated in Fig. 16. This virtual cart product confirmation screen may be displayed on the display 507 when the user U taps the "Virtual Cart" button displayed in the upper left corner of the product basic information screen shown in Fig. 13. Alternatively, it may be determined that the user U has checked out from the physical store T by reading an IC tag installed at the entrance / exit of the physical store T.

[0108] 16, on the virtual cart product confirmation screen, the display 507 of the user terminal 50 displays information such as the product list in the virtual cart, the total price, and the quantity and price of each product, as well as buttons for confirming each product in the virtual cart, a button for instructing to continue shopping, and a button for instructing to proceed to payment (i.e., check out), and the earphones 51 may output an audio announcement regarding the total price of the products in the virtual cart, as well as audio guidance for inputting operations to confirm each product in the virtual cart, to instruct to continue shopping, or to instruct to proceed to payment. Furthermore, when the user U instructs to confirm each product in the virtual cart, the earphones 51 may output the name and quantity (and unit price or subtotal) of each product registered in the virtual cart, and audio guidance for inputting operations.

[0109] In step S112, if the user terminal 50 determines that the user U is checking out (leaving) the physical store T (YES in step S112), it determines whether or not the product P is registered in the virtual cart managed by the management server 10 (step S113). If the product P is not registered in the virtual cart (NO in step S112), the operation proceeds to step S115. On the other hand, if the product P is registered in the virtual cart (YES in step S113), the user terminal 50 requests the management server 10 to settle the payment for the product P in the virtual cart. As a result, a settlement process is executed to pay the fee for the product P in the virtual cart (step S114), and the operation proceeds to step S115. An example of the settlement process will be described in detail later with reference to FIG. 19 etc.

[0110] In step S115, a checkout process is executed for the user U to manage their exit from the physical store T. At the time this checkout process is executed, there are no unpaid items P registered in the virtual cart set for the user U, so the checkout process may reset the virtual cart set for the user U. Then, this operation ends.

[0111] Check-in process flow Next, the check-in process shown in step S101 of Fig. 10 will be described, focusing mainly on the operation of the user terminal 50. Fig. 17 is a flowchart showing a specific example of the check-in process according to this embodiment.

[0112] As shown in FIG. 17, in the check-in process, first, the user U launches a dedicated app that has been pre-installed on the user terminal 50 (step S1011). The launch of the dedicated app may be, for example, a user operation, or may be performed by the user terminal 50 reading an IC tag placed at the entrance / exit of the physical store T. Alternatively, the user U may instruct the user terminal 50 to launch the dedicated app by voice, which launches the dedicated app on the user terminal 50. In this case, a voice assistant function installed in the user terminal 50 may be used as a function that responds to voice input from the user U.

[0113] Next, the physical store T to be entered is set in the launched dedicated app (step S1012). The physical store T to be entered may be identified, for example, by the user U selecting the relevant physical store T from a list of physical stores displayed on the user terminal 50, or the dedicated app may identify the physical store T using the location information of the user terminal 50 and the address of the physical store T. Alternatively, if the user terminal 50 reads an IC tag placed at the entrance / exit of the physical store T when entering the store, information for identifying the physical store T (or store information (see FIG. 3)) may be stored in this IC tag, and the physical store T may be identified using the information read from this IC tag. In this case, the next step S1013 may be omitted.

[0114] Next, the user terminal 50 accesses the management server 10 via the network 90 and acquires store information (see FIG. 3) related to the physical store T set in the dedicated app (step S1013). Next, the user terminal 50 provides the user U with information related to the physical store T to be entered based on the acquired store information (step S1014), and proceeds to step S1015. When providing the information related to the physical store T, as illustrated in FIG. 18, the display 507 of the user terminal 50 displays information such as the name and advertisement of the physical store T to be entered, as well as a button for the user U to select whether or not to check in, and the earphones 51 may output audio guidance for inputting whether or not to check in, along with an audio announcement related to the name and advertisement of the physical store T to be entered.

[0115] In step S1015, the user terminal 50 determines whether the user U has selected to "check in" to the physical store T. If the user U has selected to "not check in" (NO in step S1015), the operation illustrated in Figures 10 and 17 ends. On the other hand, if the user U has selected to "check in" (YES in step S1015), the user terminal 50 activates the virtual cart for the user U at the physical store T by requesting the management server 10 to activate the virtual cart (step S1016), and returns to the operation shown in Figure 10.

[0116] Payment processing flow Next, the payment process shown in step S114 of Fig. 10 will be described, focusing mainly on the operation of the management server 10. Fig. 19 is a flowchart showing a specific example of the payment process according to this embodiment.

[0117] 19, in the payment process, first, the payment request unit 106 of the management server 10 determines whether or not the payment for the user U has been approved (step S1141). The payment approval process for the user U will be described later with reference to FIG.

[0118] If the payment for user U has not been approved (NO in step S1141), this operation proceeds to step S1144. On the other hand, if the payment for user U has been approved (YES in step S1141), the payment request unit 106 requests the payment server 20 to make payment for the total amount of the products managed in the virtual cart product management unit 105 based on the payment information managed in the payment information DB 107 (step S1142), and this operation proceeds to step S1143.

[0119] In step S1143, it is determined whether or not the payment has been completed in the payment server 20. If the payment has been completed (YES in step S1143), the operation proceeds to step S1154. On the other hand, if the payment has not been completed in the payment server 20 (NO in step S1143), the operation proceeds to step S1144.

[0120] In step S1144, the payment request unit 106 notifies the user U that the payment could not be made (also referred to as a payment error) through a dedicated app on the user terminal 50 (step S1144). In the notification of the payment error, as exemplified in Fig. 20, the display 507 of the user terminal 50 displays information such as the existence of unpaid items for which payment could not be made, and the existence of options for resolving the payment error, such as using another payment method or canceling the items, along with buttons for the user U to select from these options, and the earphones 51 may output an audio announcement that the existence of unpaid items for which payment could not be made, along with audio guidance for selecting a method for resolving the payment error.

[0121] If the user U selects to cancel the product in response to the payment error notification in step S1144 (YES in step S1145), the payment request unit 106 instructs the virtual cart product management unit 105 to perform cancellation processing for the product in the virtual cart. In response, the virtual cart product management unit 105 requests the user U to cancel the product in the virtual cart via a dedicated app on the user terminal 50 (step S1146). When requesting a cancellation operation for an item in the virtual cart, as illustrated in Figure 21, the display 507 of the user terminal 50 displays information such as the procedure for reading the item tag 42 of the item P to be canceled and the name of the read item P, as well as buttons for confirming with the user U whether or not to cancel the read item P, a button for returning to the previous screen (see Figure 20), and buttons for confirming the items in the virtual cart and proceeding to make a new payment, and the earphones 51 may output a voice announcement of the procedure for reading the item P to be canceled, the name of the read item P, etc., as well as voice guidance such as the operating procedures for confirming or canceling the cancellation of the read item P, the operating procedures for confirming the item P in the virtual cart and proceeding to make a new payment, and the operating procedures for returning to the previous screen (see Figure 20).

[0122] When the user terminal 50 reads the product tag 42 in response to a request to cancel a product in the virtual cart, the virtual cart product management unit 105 deletes the product P from the virtual cart or decrements the number of registered products P by 1 (step S1147). Next, the virtual cart product management unit 105 determines whether the user U has completed the cancellation operations for all of the products P that the user U wishes to cancel (step S1148). If it is determined that the cancellation operations are complete (step S1148), the process returns to step S1142 and executes the subsequent operations. On the other hand, if it is determined that the cancellation operations are not complete (NO in step S1148), the process returns to step S1146 and the cancellation process for the next product P is executed. Whether the user U has completed the cancellation operations for all of the products P that the user U wishes to cancel may be determined, for example, based on the operation content on the screen illustrated in FIG. 21.

[0123] Furthermore, if the user U selects payment using another payment method in response to the payment error notification in step S1144 (NO in step S1145 and YES in step S1149), options for selecting another payment method are provided to the user U via a dedicated app on the user terminal 50 (step S1150). When providing options for selecting another payment method, for example, audio guidance may be provided to the user U along with a screen with options (buttons) as exemplified in FIGS. 22 and 23. In this case, the app for making payment using another payment method may be linked in advance with the dedicated app according to this embodiment. This allows for smooth payment using another payment method.

[0124] Next, the payment request unit 106 determines whether or not the payment has been completed using another payment method (step S1151), and if it has been completed (YES in step S1151), the operation proceeds to step S1154.

[0125] If neither cancellation of the product nor another payment method is selected in response to the payment error notification in step S1144 (NO in step S1145 and NO in step S1149), or if payment is not completed even when another payment method is used (NO in step S1151), the payment request unit 106 of the management server 10 may notify the store staff via the store system 40 that some action is required for user U (step S1151). Then, if user U is able to check out after the store staff takes action, such as settling the bill at the store register or returning all items in the virtual cart (step S1152), the operation proceeds to step S1154.

[0126] In step S1154, the virtual cart product management unit 105 receives a payment completion notification from the payment request unit 106, resets the virtual cart for user U whose payment has been completed, and permits checkout for user U. After that, the operation returns to the operation shown in FIG.

[0127] Payment approval flow Next, the payment approval process for user U will be described. As described above, in this embodiment, when user U checks out (exits) from physical store T, payment for items in the virtual cart may be automatically executed. To achieve this, in this embodiment, apart from the operation illustrated in FIG. 10, payment approval process for user U may be executed in the background based on image data acquired by a camera 43 installed in physical store T. This payment approval process may utilize, for example, the technology disclosed in Japanese Patent No. 6551765. FIG. 24 is a flowchart showing an example of payment approval process according to this embodiment.

[0128] 24, in this operation, image data acquired by a camera 43 installed in a physical store T is periodically input to the biometric information processing unit 101 of the management server 10 (step S1161). In response to this, the biometric information processing unit 101 identifies the area of ​​a person captured in the input image data, and extracts the biometric information of this person from the identified area (step S1162).

[0129] Next, the authentication processing unit 103 compares the biometric information extracted in step S1062 with the authentication information registered in the authentication information DB 102 (step S1163). If the result of the comparison in step S1163 indicates that the user U is a registered user (YES in step S1164), the authentication processing unit 103 increments the number of comparisons for the user U by 1 (step S1166). On the other hand, if the user is an unregistered user (NO in step S1164), the authentication processing unit 103 performs new user registration for the user U through a dedicated app running on the user terminal 50 (step S1165), and increments the number of comparisons for the new user U by 1 (step S1166). In the new user registration, the biometric information extracted in step S1162 may be registered in the authentication information DB 102 in association with other user information.

[0130] Next, the authentication processing unit 103 determines whether the number of matching attempts updated in step S1166 has reached a preset threshold value (step S1167), and if the threshold value has been reached (YES in step S1167), approves the payment for the user (step S1168). On the other hand, if the threshold value has not been reached (NO in step S1167), the operation returns to step S1161. In this way, by approving the payment for user U based on multiple matching results, it is possible to significantly reduce erroneous approvals.

[0131] Thereafter, it is determined whether or not to end this operation (step S1169), and if it is to be ended (YES in step S1169), this operation ends. On the other hand, if it is not to be ended (NO in step S1169), this operation returns to step S1161, and the subsequent operations are continued.

[0132] In the above operations, some or all of the operations input by the user U by touching the display 507 of the user terminal 50 may be input by voice using the microphone 508 of the user terminal 50. That is, the user U may select a preferred input method between touch input to the display 507 and voice input to the microphone 508 to input the operation. For example, when the user U speaks "Tell me your product review" into the microphone 508 on the screen that provides additional product information as illustrated in FIG. 15, the same operation as when the user U taps the "User Rating" button located in the lower right corner of the display 507 may be executed.

[0133] (Actions and Effects) As described above, according to this embodiment, the user U can easily obtain various pieces of information about the product shelves R and the products P by voice, simply by holding the user terminal 50 over the IC tags 41, 42 attached to the product shelves R or the products P and tapping the buttons arranged on the display 507 in accordance with the voice instructions. This allows even a visually impaired user U to easily and smoothly obtain information that a person without visual impairment would normally pay attention to when selecting a product, as well as more detailed information, thereby enabling visually impaired people to shop safely and smoothly in physical stores.

[0134] Furthermore, by using the purchase support service according to this embodiment, even users without visual impairments (hereinafter also referred to as general users) can appropriately obtain detailed information about the product P through the user terminal 50, which can have effects such as preventing erroneous purchases and promoting sales. For example, by using the purchase support service according to this embodiment, general users can easily check the expiration date (use by) and ingredient names of the product P through the user terminal 50 (for example, the display 507 or the earphones 51), which can prevent the purchase of a product P that has passed its expiration date or a product P that contains an allergen.

[0135] For users U who are not visually impaired, various information and functions may be provided only by displaying them on the display 507. In this case, since the user U can directly and visually check the information displayed on the package of the product P (for example, some or all of the basic product information, such as the product name, price, and best-before (use-by) date), the information not displayed on the package of the product P may be preferentially provided to the user U.

[0136] In this way, the service content for a user U with a visual impairment and a user U without a visual impairment may be switched by, for example, a dedicated app having an operating mode for visually impaired people and an operating mode for general users, and the user U selecting the operating mode.

[0137] Furthermore, the dedicated app according to this embodiment can be used not only by a user U (i.e., a general consumer) who purchases the product P, but also by store staff at a physical store T, contractors who deliver and collect the product P, and the like. That is, by using the dedicated app according to this embodiment, it becomes possible to easily check the expiration date of the product P. As a result, the store staff can identify the product P that is approaching its expiration date by simply holding the user terminal 50 over the product P and provide services such as cash back or affixing a discount sticker. Furthermore, the store staff and collection contractors can identify and collect products that have passed their expiration date by simply holding the user terminal 50 over the product P. As a result, it becomes possible to reduce confirmation errors and the effort required by the store staff and collection contractors, thereby significantly improving work efficiency.

[0138] Furthermore, by allowing store staff to use the dedicated app according to this embodiment, it becomes possible to confirm by voice which product P should be displayed in which position on which shelf R, and therefore even visually impaired people can work as store staff in the physical store T. This also has the effect of promoting employment for visually impaired people.

[0139] When the dedicated app according to this embodiment is intended for store staff or vendors, functions such as purchasing the product P (i.e., registering it in a virtual card) and payment may be omitted. Also, the dedicated app may be configured to have an operation mode for consumers and an operation mode for store staff or vendors, and to be able to switch between the operation modes.

[0140] <Second embodiment> Next, a second embodiment of the present disclosure will be described in detail with reference to the drawings. In the following description, configurations, operations, and effects similar to those of the first embodiment will be cited, and redundant description will be omitted.

[0141] In the first embodiment described above, a case where a virtual cart that registers products P to be purchased by a user U is managed in the management server 10, the payment server 20, the store system 40, or the user terminal 50 is exemplified, but the present invention is not limited to such a configuration. Therefore, in the second embodiment, an example will be described in which payment for a user U is executed without using a virtual cart. Note that in this embodiment, an example will be given in which payment for products to be taken home when leaving a physical store T is executed automatically (hereinafter also referred to as walk-through payment).

[0142] (Product Sales System 2) 25 is a diagram illustrating a schematic configuration example of a product sales system as an information processing system according to the second embodiment of the present disclosure. As shown in Fig. 25, the product sales system 2 according to this embodiment has the same configuration as the product sales system 1 according to the first embodiment (see Fig. 1), except that a checkout gate 241 is added to the physical store T.

[0143] The checkout gate 241 is installed at the entrance / exit of the physical store T and includes an IC tag reader, such as an RFID reader, for reading data stored in the product tag 42 by communicating with the product tag 42 attached to the product P through near-field wireless communication, and a communication unit for transmitting data by connecting to the network 90 and / or the LAN 49. When the user U leaves the physical store T, the checkout gate 241 reads the product tag 42 of the product P carried by the user U. Of the information read from each product tag 42 by the checkout gate 241, at least information on the product ID and quantity of each product (hereinafter also referred to as purchased product information) is transmitted to the management server 10 and compiled in the purchased product compilation unit 201 of the management server 10. The purchased product information may be compiled in the store-side system 40 via the LAN 40 and then transmitted to the management server 10 via the network 90, or may be transmitted directly to the management server 10 via the network 90 (which may include the LAN 49).

[0144] (Management server 210) Fig. 26 is a block diagram showing a more detailed configuration example of the management server according to this embodiment. As shown in Fig. 26, the management server 210 has the same configuration as the management server 10 according to the first embodiment (see Fig. 2), except that the virtual cart product management unit 105 is replaced with a purchase price calculation unit 201.

[0145] The purchase amount tabulation unit 201 tabulates the purchase information acquired at the checkout gate 241 of the physical store T for each user U. The purchase amount tabulation unit 201 also notifies the basic product information management unit 113 of the product ID in the tabulated purchase information, and acquires the unit price (price) of each product P from the basic product information management unit 113. The purchase amount tabulation unit 201 then calculates the total price of the products P that the user U will bring back from the physical store T based on the quantity of each product P and its unit price, and requests the payment request unit 106 to settle the calculated total price. In response, the payment request unit 106 requests the payment server 20 to settle the total price based on the payment information registered as primary for the user U.

[0146] (Example of operation flow) Next, the operation of the product management system 2 according to this embodiment will be described in detail with reference to the drawings. Figure 27 is a flowchart showing an example of the operation of the product sales system according to this embodiment.

[0147] 27, in this operation, when a user U enters a physical store T, check-in processing is executed (step S101) as in the first embodiment. However, in this embodiment, a virtual cart is not used, and therefore step S1016 may be omitted in the operational example of the check-in processing illustrated in FIG.

[0148] When the check-in process for user U to the physical store T is completed, in this embodiment, as in the first embodiment, the reading of the IC tag 41 or 42 is awaited, and a guide to the product shelf R or a guide to the product P is provided to the user U according to the read IC tag 41 or 42 (steps S103 to S109). However, in this embodiment, since the registration of the product P to the virtual cart is omitted, if the guide to the product addition information is not required (NO in step S107), or after the product addition information provision process (step S109) is completed, this operation may return to step S102.

[0149] Also, in the standby state, as in the first embodiment, it is determined whether the user U will check out from the physical store T (step S112). Note that in this embodiment, the determination of whether the user U will check out from the physical store T may be made based on, for example, whether the user U has passed through the checkout gate 241. In this case, whether the user U has passed through the checkout gate 241 in the direction of entering the store or in the direction of leaving the store may be determined using the RSSI function of the IC tag reader of the checkout gate 241, image data acquired by the camera 43, location information of the user terminal 50, etc.

[0150] When the user U checks out from the physical store T (YES in step S112), the IC tag reader at the checkout gate 241 installed in the physical store T reads the product tags 42 of all the products P that the user U is taking home from the physical store T (step S201). The product purchase information that includes at least the product ID and quantity of each product P among the read basic product information is sent to the purchase price calculation unit 201 of the management server 10 and calculated, as described above.

[0151] Next, the purchase price aggregation unit 201 determines from the aggregated purchase product information whether there are any products P that the user U will take home from the physical store T (step S202). If there are no products P to take home (NO in step S202), this operation ends. On the other hand, if there are any products P to take home (YES in step S202), the purchase price aggregation unit 201 acquires the unit price (price) of each product P from the basic product information management unit 113 based on the product ID in the purchase product information, and calculates the total price of the products P that the user U will take home by multiplying the acquired price of each product P by the respective quantity and adding up the results (step S203). The calculated total price is notified to the payment request unit 106.

[0152] In response to this, the payment request unit 106 requests the payment server 20 to make a payment for the notified total amount based on the payment information registered as primary in the payment information DB 107 (step S204). As a result, the payment server 20 executes the payment of the total amount to the user U, and this operation ends.

[0153] In step S204, a determination may be made as to whether or not to approve the payment for user U before making a payment request to the payment server 20. This determination may be made using, for example, the payment approval flow described with reference to Fig. 24 in the first embodiment.

[0154] (Actions and Effects) As described above, according to this embodiment, similarly to the first embodiment, the user U can easily obtain various pieces of information about the product shelves R and the products P by voice, simply by holding the user terminal 50 over the IC tags 41, 42 attached to the product shelves R or the products P and tapping the buttons arranged on the display 507 in accordance with the voice instructions. This allows even a visually impaired user U to easily and smoothly obtain information that a person without visual impairment would normally pay attention to when selecting a product, as well as more detailed information, thereby enabling visually impaired people to shop safely and smoothly in physical stores.

[0155] Furthermore, according to this embodiment, the user U does not have to make any operation to decide whether or not to purchase the product P; simply by taking the product P out of the physical store T, payment for the product P that is being taken home is automatically executed, which not only makes payment smoother but also prevents mistakes such as accidentally forgetting to pay.

[0156] Other configurations, operations, and effects may be similar to those of the first embodiment described above, and therefore detailed description of this embodiment will be omitted.

[0157] <Third embodiment> Next, a third embodiment of the present disclosure will be described in detail with reference to the drawings. In the following description, configurations, operations, and effects similar to those of the first or second embodiment will be referred to, and redundant description will be omitted.

[0158] The first embodiment described above illustrates a case where the product P to be purchased by the user U is identified by reading the product tag 42 attached to the product P with the user terminal 50. In contrast, the third embodiment illustrates a case where the product P to be purchased by the user U is identified by analyzing image data acquired by a camera 43 installed in a physical store T.

[0159] (Product Sales System 3) 28 is a schematic diagram showing an example of a schematic configuration of a product sales system as an information processing system according to the third embodiment of the present disclosure. As shown in Fig. 28, the product sales system 3 according to this embodiment has the same configuration as the product sales system 1 according to the first embodiment (see Fig. 1), except that the management server 10 is replaced with a management server 310 and the shelf tag 41 and the product tag 42 are omitted.

[0160] (Management Server 310) Fig. 29 is a block diagram showing a more detailed configuration example of the management server according to this embodiment. As shown in Fig. 29, the management server 310 has the same configuration as the management server 10 according to the first embodiment (see Fig. 2), except that a user behavior identification unit 301 is newly added.

[0161] The user behavior identification unit 301 may analyze image data acquired by the camera 43 of the physical store T to identify which product shelf R the user U approached and which product P the user U picked up (hereinafter also referred to as taking out the product P).

[0162] (Example of operation flow) Next, the operation of the product management system 3 according to this embodiment will be described in detail with reference to the drawings. Figure 30 is a flowchart showing an example of the operation of the product sales system according to this embodiment.

[0163] As shown in FIG. 30, in this operation, when a user U enters a physical store T, check-in processing is executed in the same manner as in the first embodiment (step S101).

[0164] When the check-in process for user U to physical store T is completed, the user behavior identification unit 301 of the management server 310 identifies user U's behavior by analyzing image data sent continuously or periodically from the store system 40 of physical store T (step S301).

[0165] Next, the user behavior identification unit 301 determines whether the user behavior identified in step S301 is approach to the product shelf R (step S302). If the user behavior is not approach to the product shelf R (NO in step S302), the operation proceeds to step S305.

[0166] On the other hand, if the identified user behavior is approaching a certain product shelf R (YES in step S302), the user behavior identification unit 301 identifies the product shelf R that the user U approached from the analysis results of the image data acquired in step S301 (step S303) and notifies the product shelf ID of the identified product shelf R to the product shelf information management unit 112. For example, the user behavior identification unit 301 may identify the product shelf R that the user U approached from the position of the product shelf R in the image data or the shape and texture of the product shelf R. Also, which camera 43 photographs which product shelf R, i.e., the correspondence between the camera 43 and the shelf ID, may be managed in advance in the store system 40 and / or the management server 110. For example, one or more cameras 43 may be installed for each product shelf R. In this case, when the user U is detected from the camera images acquired by the camera 43 for each product shelf R, it is possible to provide the user U with information about the corresponding product shelf R.

[0167] In response to this, the product shelf information management unit 112 identifies the product shelf information managed in the product shelf information DB 122 for the relevant product shelf R and transmits this to the user terminal 50 (step S304). In response to this, the user terminal 50, as in the first embodiment, informs the user U of the category of the product P displayed on this product shelf R, etc., based on the product shelf information received from the management server 310 (see FIG. 4), and returns to step S301.

[0168] In step S305, the user behavior identification unit 301 determines whether the user behavior identified in step S301 is the removal of a product P. If the user behavior is not the removal of a product P (NO in step S305), the operation proceeds to step S112. On the other hand, if the user behavior is the removal of a certain product P (YES in step S305), the user behavior identification unit 301 identifies the product P that the user U removed from the product shelf R based on the analysis results of the image data acquired in step S301 (step S306) and notifies the product basic information management unit 113 of the product ID of the identified product P. For example, the user behavior identification unit 301 may identify the product P that the user U removed based on the position of the product P in the image data, the shape and texture of the product P, etc. Also, which camera 43 photographs which product P, i.e., the correspondence between the cameras 43 and the products P, may be managed in advance in the store system 40 and / or the management server 110.

[0169] In response to this, the product basic information management unit 113 identifies the product basic information for the corresponding product P managed in the product basic information DB 123 and transmits this to the user terminal 50 (step S307). In response to this, the user terminal 50 provides the user U with basic information about this product P based on the product basic information (see FIG. 5) received from the management server 310, as in the first embodiment (step S106), and proceeds to step S107.

[0170] In steps S107 to S111, similarly to the first embodiment, additional information about the product P is provided to the user U via the user terminal 50 as needed, and the product P is added to the virtual cart in response to a purchase instruction from the user U. Thereafter, the operation returns to step S301.

[0171] In steps S112 to S115, similar to the first embodiment, it is determined whether the user U checks out from the physical store T, and if the user U checks out, the payment process for the product P in the virtual cart is executed. Then, when the virtual cart is reset and checkout is completed, this operation ends.

[0172] (Actions and Effects) As described above, according to this embodiment, the user U can easily obtain various information about the product shelf R and the product P by voice, simply by performing normal shopping actions such as approaching the product shelf R and picking up the product P, and then following the voice prompts and tapping the buttons on the display 507. This allows even a visually impaired user U to easily and smoothly obtain information that a person without vision would normally pay attention to when selecting a product, as well as more detailed information, thereby enabling visually impaired people to shop safely and smoothly in a physical store.

[0173] Other configurations, operations, and effects may be similar to those of the first embodiment described above, and therefore detailed description of this embodiment will be omitted.

[0174] Furthermore, in the above-described embodiments and their variations, examples have been given of information processing systems (i.e., product sales systems 1 to 3) configured mainly for the purpose of enabling automatic or simple payment of purchase prices while supporting shopping by user U at physical store T, but the information processing system according to the present disclosure is not limited to implementation for such purposes. That is, the information processing system according to the present disclosure may be configured for the purpose of supporting users (including staff at each facility who may also be users) in various facilities (including ordinary households as facilities), such as public facilities or public interest facilities such as city halls, libraries, and train stations (or station buildings), other facilities such as restaurants, hotels, and sports gyms, or ordinary homes.

[0175] For example, if the information processing system according to the present disclosure is configured for use in an ordinary household, the expiration date, cooking method, or usage method of the purchased product P may be provided to the user U by voice or the like in response to an operation on the user terminal 50. Furthermore, if the information processing system is configured for use in a library, directions to the Braille corner or various information about a book that the user has picked up may be provided to the user U by voice or the like in response to an operation on the user terminal 50.

[0176] <Hardware configuration> The management servers 10, 210, 310, payment server 20, store system 40, user terminal 50, etc. according to the above-described embodiments and their modifications can be realized by an information processing device 1000 having a configuration such as that shown in FIG. 31. FIG. 31 is a hardware configuration diagram showing an example of the information processing device 1000 that realizes the functions of the management server 10, payment server 20, store system 40, user terminal 50, etc. The information processing device 1000 has a CPU 1100, a ROM (Read Only Memory) 1200, a RAM (Random Access Memory) 1300, a recording device 1400, an input / output interface (I / F) 1500, and a communication unit 1600. The various units of the information processing device 1000 are connected by a bus 1700.

[0177] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1200 or the recording device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1200 or the recording device 1400 into the RAM 1300 and executes processing corresponding to the various programs.

[0178] The ROM 1200 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the information processing device 1000 is started up, programs dependent on the hardware of the information processing device 1000, and the like.

[0179] Recording device 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, recording device 1400 is a recording medium that records programs for executing each operation related to the present disclosure, which are examples of program data.

[0180] The communication unit 1600 is an interface for connecting the information processing device 1000 to an external network 1650 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication unit 1600.

[0181] The input / output I / F 1500 is an interface for connecting the input / output device 1550 and the information processing device 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output I / F 1500. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output I / F 1500. The input / output I / F 1500 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium.

[0182] For example, when the information processing device 1000 functions as the management server 10, the payment server 20, the store system 40, the user terminal 50, etc. according to the above-described embodiments, the CPU 1100 of the information processing device 1000 executes a program loaded onto the RAM 1300 to realize the functions of the management server 10, the payment server 20, the store system 40, the user terminal 50, etc. Also, the recording device 1400 stores programs and the like according to the present disclosure. Note that the CPU 1100 reads and executes program data from the recording device 1400, but as another example, the CPU 1100 may obtain these programs from another device via the external network 1650.

[0183] Although the embodiments and modifications of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments or modifications thereof, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0184] Furthermore, the effects of the embodiments and their modifications described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained. [Explanation of symbols]

[0185] 1, 2, 3 Product Sales System 10, 210, 310 Management Server 20 Payment Server 30 review sites 40 Store system 41 Shelf tag (IC tag) 42 Product tag (IC tag) 43 Camera 49 LAN 50 User Terminals 51 Earphones 90 Network 101 Biometric information processing unit 102 Authentication Information Database (DB) 103 Authentication processing section 104 Check-in processing section 105 Virtual Cart Product Management Section 106 Payment Request Department 107 Payment Information DB 111 Store Information Management Department 112 Product Shelf Information Management Department 113 Basic Product Information Management Department 114 Product Supplementary Information Management Department 115 Review-related Information Management Department 116 Usage-related Information Management Department 121 Store Information DB 122 Product shelf information DB 123 Basic product information DB 124 Product supplementary information DB 125 review related information DB 126 Usage related information DB 201 Purchase price calculation section 241 Checkout Gate 301 User Behavior Identification Unit 501 Control section 502 Mobile Communications Department 503 Radio Communication Department 504 Camera 505 Sensor part 506 IC tag reader 507 Display 508 Mike 509 Speaker 510 Power supply section 511 Interface (I / F) P product R Product shelf T Physical store

Claims

1. 1. An information processing method executed in an information processing system in which a user terminal and a management server are connected via a predetermined network, comprising: An information processing method for outputting information based on a user's behavior within a real facility to the user via voice from the user terminal.

2. The information processing method according to claim 1, further comprising: outputting to the user from the user terminal a voice message indicating how to obtain additional information in response to information based on the user's behavior within the physical facility.

3. The information processing method according to claim 1 , further comprising: outputting the additional information as audio from the user terminal to the user by performing an operation on the user terminal according to the method for obtaining the additional information.

4. The information processing method according to claim 1 , wherein the information based on the user's behavior within the physical facility is identified by data received through short-range communication.

5. The information processing method according to claim 1 , wherein the information based on the user's behavior within the actual facility is identified by analyzing image data acquired by a camera.

Images