Bill settlement device, bill settlement system, and bill settlement method

Abstract

To allow a user to purchase even a commodity not actually existing in an actual shop other than a commodity actually existing in the shop to improve convenience of the user, and similarly to the commodity existing in the shop, allows the user to sufficiently confirm even the commodity not existing in the shop and sufficiently obtain actual feeling of purchasing the commodity.SOLUTION: A virtual commodity selling space control machine controls an MR glass worn by a user to achieve superimposed display of a virtual commodity as a virtual object in a shop as an actual space, and detect an operation of the user who selects the virtual commodity. Then, according to the operation of the user who selects the virtual commodity, a checkout counter 1 controls the MR glass to achieve superimposed display of a virtual commodity 201 on a tray as an actual space so as to be juxtaposed with an actual commodity 211 placed on the tray 41, and performs settlement for the actual commodity and the virtual commodity.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a settlement device that recognizes a product selected by a user on a sales floor and processes the settlement and payment of the price, as well as a settlement system and a settlement method that use the settlement device. [Background technology]

[0002] In retail stores such as convenience stores and supermarkets, a store clerk registers the items a customer wishes to purchase in a POS terminal, after which the POS terminal processes the payment and presents the price of the items to the customer, and the store clerk then performs the accounting (settlement) process by receiving the payment made by the customer. In recent years, various technologies have been proposed to promote product sales using POS terminals.

[0003] A known technique for promoting product sales using such POS terminals involves photographing a customer making a payment at a cash register, obtaining information about the customer's attributes (gender and age group), selecting products to recommend to the customer based on the customer's attributes, and displaying the recommended products on a customer display installed in the POS terminal (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-68713 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional technology, a customer can purchase recommended products by using a touch panel function of a customer display. This allows the customer to purchase the recommended products together with the products that the customer brought to the cash register, and the customer can purchase products that do not exist in the store together with the products that actually exist in the store.

[0006] However, in the conventional technology, only images of recommended products are displayed on the screen, and customers cannot fully check the products they want to purchase. In addition, when deciding to purchase, customers only need to press a purchase button, and customers cannot fully feel as if they were purchasing a product in a real store.

[0007] Therefore, the main object of the present invention is to provide a settlement device, a settlement system, and a settlement method that increase convenience for users by enabling them to purchase products that do not exist in the store in addition to products that actually exist in the store in a real store, and that allow users to fully check the products that do not exist in the store in the same way as the products that actually exist in the store, and that allow them to fully get the feeling of purchasing products in a real store. [Means for solving the problem]

[0008] The settlement device of the present invention is a settlement and payment device that processes settlement and payment for real products and virtual products selected by a user in a store, and includes a main body having a placement unit on which the real product selected by the user on the sales floor is placed, a camera that photographs the real product placed on the placement unit, a control unit that recognizes the target real product based on a product image captured by the camera and processes settlement and payment, and a display unit that displays the settlement result and payment result acquired by the control unit, and the control unit further controls a virtual object display device to recognize the target virtual product in response to a user's operation to select the virtual product as a virtual object superimposed on a predetermined area in the store as a real space, and displays the recognized virtual product on the placement unit as a real space in a state lined up with the real product placed on the placement unit. Through the virtual object display device The display unit displays the virtual products and the real products in a superimposed manner, and then displays a product confirmation screen including a list of recognition results for the purchased products, with the display format changed between the virtual products and the real products, and the user is configured to settle or pay for the purchased products including the virtual products.

[0009] Further, a settlement system of the present invention is a settlement system for performing processes related to settlement and payment for real products and virtual products selected by a user in a store, the settlement device having a placement unit for placing the real product selected by a user on a sales floor, a display unit for photographing the real product placed on the placement unit, recognizing the target real product based on the product image acquired by photographing, performing processes related to settlement and payment, and displaying the settlement result and the payment result; a virtual object display device for superimposing and displaying a virtual object in real space; and a virtual object control device for controlling the virtual object display device to detect a user's operation of selecting the virtual product as a virtual object superimposed in a predetermined area in the store as the real space, and recognize the target virtual product, and the settlement device displays the recognized virtual product on the placement unit as the real space, while being lined up next to the real product placed on the placement unit. Through the virtual object display device The display unit displays the virtual products and the real products in a superimposed manner, and then displays a product confirmation screen including a list of recognition results for the purchased products, with the display format changed between the virtual products and the real products, and the user is configured to settle or pay for the purchased products including the virtual products.

[0010] A settlement method of the present invention is a settlement method for performing processing related to settlement and payment of real products and virtual products selected by a user in a store, in which a virtual object control device controls a virtual object display device that displays a virtual object superimposed on a real space, detects a user's operation of selecting the virtual product as a virtual object superimposed on a predetermined area in the store as the real space, recognizes the target virtual product, and With display unit The settlement device photographs the real product selected by the user at the sales floor and placed on the placement unit, recognizes the target real product based on the product image acquired by photographing, and displays the recognized virtual product on the placement unit as a real space in a state where the recognized virtual product is placed next to the real product placed on the placement unit. Through the virtual object display device The display unit displays the virtual products and the real products in a superimposed manner, and then displays a product confirmation screen including a list of recognition results for the purchased products, with the display format changed between the virtual products and the real products, and the user is configured to settle or pay for the purchased products including the virtual products. Effect of the Invention

[0011] According to the present invention, Users can fully feel as if they are purchasing products that are not actually in stores, just as if they were actually in stores. This allows for increased customer purchasing opportunities while minimizing in-store inventory. [Brief description of the drawings]

[0012] [Figure 1] Overall configuration of an unmanned store system according to the present embodiment. [Diagram 2] Floor plan showing the layout of an unmanned store [Diagram 3] FIG. 1 is an explanatory diagram showing a display state of a virtual product in a virtual product sales area and an operation state of picking up the virtual product. [Figure 4] FIG. 1 is a perspective view showing the exterior of the settlement table 1. [Diagram 5] An explanatory diagram showing the placement of real products on the checkout counter 1 and the display of virtual products. [Figure 6] Block diagram showing the schematic configuration of the checkout counter 1 and MR glass 2 [Figure 7] FIG. 13 is an explanatory diagram showing a screen displayed on the touch panel display 42 of the checkout counter 1. [Figure 8] FIG. 13 is an explanatory diagram showing a screen displayed on the touch panel display 42 of the checkout counter 1. [Figure 9] FIG. 11 is a flow diagram showing the operation procedure of the user terminal 11 during user registration. [Figure 10] Flow diagram showing the operation procedure of checkout counter 1 [Figure 11] Flow diagram showing the operation procedure of checkout counter 1 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A first invention made to solve the above problems is a settlement and payment device that performs processing related to settlement and payment for real products and virtual products selected by a user in a store, the settlement and payment device having a main body having a placement section on which the real product selected by the user on the sales floor is placed, a camera that photographs the real product placed on the placement section, a control section that recognizes the target real product based on a product image acquired by the camera, and performs processing related to settlement and payment, and a display section that displays a settlement result and a payment result acquired by the control section, the control section further controls a virtual object display device to recognize the target virtual product in response to a user's operation to select the virtual product as a virtual object superimposed on a predetermined area in the store as a real space, and displays the recognized virtual product on the placement section as a real space in a state lined up with the real product placed on the placement section. Through the virtual object display device The display unit displays the virtual products and the real products in a superimposed manner, and then displays a product confirmation screen including a list of recognition results for the purchased products, with the display format changed between the virtual products and the real products, and the user is configured to settle or pay for the purchased products including the virtual products.

[0014] According to this, Users can fully feel as if they are purchasing products that are not actually in stores, just as if they were actually in stores. This allows for increased customer purchasing opportunities while minimizing in-store inventory.

[0017] Also, Second The invention is configured such that, when settlement or payment for purchased goods including the virtual goods is completed, the control unit performs processing related to delivery of the actual virtual goods.

[0018] This allows the actual virtual goods purchased by the user to be delivered quickly.

[0019] A third invention is a settlement system for performing a process related to settlement and payment for real products and virtual products selected by a user in a store, the settlement device having a placement unit for placing the real product selected by a user on a sales floor, a display unit for photographing the real product placed on the placement unit, recognizing the target real product based on the product image acquired by photographing, performing a process related to settlement and payment, and displaying a settlement result and a payment result; a virtual object display device for superimposing and displaying a virtual object in a real space; and a virtual object control device for controlling the virtual object display device to detect a user's operation of selecting the virtual product as a virtual object superimposed in a predetermined area in the store as the real space, and recognize the target virtual product, and the settlement device displays the recognized virtual product on the placement unit as the real space, while being lined up next to the real product placed on the placement unit. Through the virtual object display device The display unit displays the virtual products and the real products in a superimposed manner, and then displays a product confirmation screen including a list of recognition results for the purchased products, with the display format changed between the virtual products and the real products, and the user is configured to settle or pay for the purchased products including the virtual products.

[0020] According to this, as in the first invention, Users can fully feel as if they are purchasing products that are not actually in stores, just as if they were actually in stores. This allows for increased customer purchasing opportunities while minimizing in-store inventory.

[0023] Also, Fourth In the invention, the virtual object display device is a wearable device that is worn on the user's head and superimposes and displays the virtual object on the user's real field of vision.

[0024] This allows users to shop for virtual products with a sense of realism that is close to that of the real thing.

[0025] Also, Fifth The invention relates to a virtual object display device, the virtual object display device having a sensor for detecting a user's action. Based on the detection result of the sensor, the hand gesture of the user is recognized, The virtual object control device controls the virtual object based on the hand gesture recognition result. The device is configured to detect the operation of a user selecting an item.

[0026] This allows the user to efficiently select a virtual product. In addition, the device may be configured to allow operations to be performed using a controller or the user's voice.

[0027] A sixth aspect of the present invention is a settlement method for carrying out a process related to settlement and payment for real products and virtual products selected by a user in a store, the method comprising: a virtual object control device controls a virtual object display device that displays virtual objects in a superimposed manner in a real space; detects an operation by a user to select the virtual product as a virtual object superimposed and displayed in a predetermined area in the store as the real space; and recognizes the target virtual product; With display unit The settlement device photographs the real product selected by the user at the sales floor and placed on the placement unit, recognizes the target real product based on the product image acquired by photographing, and displays the recognized virtual product on the placement unit as a real space in a state where the recognized virtual product is placed next to the real product placed on the placement unit. Through the virtual object display device The display unit displays the virtual products and the real products in a superimposed manner, and then displays a product confirmation screen including a list of recognition results for the purchased products, with the display format changed between the virtual products and the real products, and the user is configured to settle or pay for the purchased products including the virtual products.

[0028] According to this, as in the first invention, Users can fully feel as if they are purchasing products that are not actually in stores, just as if they were actually in stores. This allows for increased customer purchasing opportunities while minimizing in-store inventory.

[0031] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0032] FIG. 1 is a diagram showing the overall configuration of an unmanned store system according to this embodiment.

[0033] This unmanned store system (payment settlement system) aims to automate retail stores such as convenience stores and supermarkets, eliminating the need for a clerk to be present at all times to handle payments and accounts.

[0034] This unmanned store system includes a checkout counter 1 (checkout device), MR glasses 2 (virtual object display device), an entrance gate 3, an exit gate 4, a gate controller 5, a virtual product sales floor controller 6 (virtual object control device), and a register 7. The checkout counter 1, MR glasses 2, entrance gate 3, exit gate 4, gate controller 5, virtual product sales floor controller 6, and register 7 are installed in an unmanned store.

[0035] The unmanned store system also includes a user terminal 11, a payment server 12, a user management server 13, a product learning server 14, an authentication server 15, a DB server 16, and a delivery server 17.

[0036] The user terminal 11, payment server 12, user management server 13, product learning server 14, authentication server 15, DB server 16, and delivery server 17 are interconnected with the checkout counter 1, MR glasses 2, gate control machine 5, virtual product sales floor control machine 6, and cash register 7 installed in the unmanned store via a network such as the Internet or a LAN.

[0037] The user management server 13, product learning server 14, authentication server 15, and DB server 16 may be installed within the unmanned store, for example in a back yard adjacent to the sales floor, but they may also be installed at a location away from the unmanned store, for example in the headquarters of the operator of the unmanned store.

[0038] The checkout counter 1 performs processes related to the settlement and payment (payment of the price) of products selected by a user on the sales floor of an unmanned store. In this embodiment, the process related to the settlement involves identifying the products by object recognition from photographed images of the products (product recognition process), and calculating the total amount to be paid from the price (unit price) and quantity of each product (settlement process). In addition, as a process related to the settlement, iris authentication is requested from the authentication server 15, and if the iris authentication is successful, a payment process is requested from the payment server 12 via the user management server 13.

[0039] The MR glasses 2 are wearable devices that are worn on the user's head and display virtual objects superimposed on the user's real field of vision. In this embodiment, the MR glasses 2 superimpose virtual products (virtual 3D objects of products) that do not actually exist in the store and virtual shelves (virtual 3D objects of shelves) that display the virtual products on a virtual product sales floor as a real space in a store. In addition, the MR glasses 2 capture the user's eyeballs to acquire iris information for iris authentication to identify the user.

[0040] The entrance gate 3 restricts users from entering the store. The exit gate 4 restricts users from leaving the store. The gate control device 5 performs processing related to iris authentication to allow users to enter the store, and controls the opening and closing of the entrance gate 3 depending on the iris authentication result. The gate control device 5 also performs processing related to iris authentication to confirm that the user has left the store, and controls the opening and closing of the exit gate 4 depending on the iris authentication result. Note that password authentication may be performed as a countermeasure in case iris authentication fails and users are unable to enter or leave the store.

[0041] The virtual product sales floor controller 6 controls the user's MR glasses 2 to display virtual products and virtual shelves displaying the virtual products in the virtual product sales floor. The virtual product sales floor controller 6 also acquires a hand gesture recognition result from the MR glasses 2, and detects user operations such as an operation to select a virtual product based on the recognition result. The virtual product sales floor controller 6 also registers the virtual product in a shopping list in response to the user's selection operation. Information on the shopping list is provided from the virtual product sales floor controller 6 to the checkout counter 1.

[0042] The register 7 is used by users to perform operations related to user registration (registration of membership information and eye images for iris authentication) that are necessary for users to use this system, and is composed of, for example, a tablet terminal, and has an application for user registration installed.

[0043] The user terminal 11, like the cash register 7, allows the user to perform operations related to user registration (registration of membership information and eye images for iris authentication) necessary for the user to use this system, and also manages purchase history (receipt information).The user terminal 11 is composed of a smartphone, tablet terminal, etc., and has user applications installed.

[0044] The payment server 12 is operated by a payment service provider (such as a credit card company). This payment server 12 executes payment processing for the price of the product purchased by the user in response to a payment request from the checkout counter 1. The payment server 12 may be operated by a payment agency service provider (a payment agency server).

[0045] The user management server 13 functions as a login server that manages user login and performs password authentication. The user management server 13 also functions as a payment and delivery interface server that relays between the checkout counter 1 and the payment server 12 and delivery server 17.

[0046] The product learning server 14 uses machine learning such as deep learning to construct a product recognition engine to be installed on the checkout counter 1. In this product learning server 14, machine learning is performed using product images acquired in advance by photographing the products to be registered as input information and product identification information for each product (such as the product name) as output information, and a database for product recognition is constructed.

[0047] The authentication server 15 includes a management server 21 and a matching server 22. The management server 21 accumulates and manages information such as the names and iris information (iris ID, iris image) of registered users. The matching server 22 executes iris authentication in response to a request for iris authentication from the payment counter 1 and the gate control device 5. In this iris authentication, an eye image of the subject is acquired from the payment counter 1 and the gate control device 5, iris features of the subject are generated from the eye image, and iris matching is performed by comparing the iris features of the subject with those of registered users (registered users) stored in the device, to determine whether the subject is a registered user or not (1:N authentication).

[0048] The DB server 16 accumulates and manages various types of information. Specifically, information such as a payment ID, iris ID, user ID, password, and office code for each user is registered in the database as user management information. In addition, information such as product identification information (product name, product code, etc.) is registered in the database as product master information. In addition, information such as a user ID for each user, and the name and price of a purchased product is registered in the database as purchase log information.

[0049] The delivery server 17 manages the delivery of goods in the delivery department (delivery center). Specifically, similar to the case of EC shopping, it manages the tasks of securing goods in a warehouse, packaging the goods, and requesting a delivery company to deliver the goods. In this embodiment, when the procedure for purchasing a virtual product at the checkout counter 1 (checkout and settlement of the purchased product) is completed, arrangement information instructing the delivery of the actual virtual product is sent from the checkout counter 1 to the delivery server 17 via the user management server 13, and the delivery server 17 performs processing to start the delivery task based on the arrangement information.

[0050] In this embodiment, the checkout counter 1 performs the product recognition process, but the product recognition process may be performed by an external server.

[0051] In addition, in this embodiment, a gate control device 5 is provided and this gate control device 5 controls the entrance gate 3 and the exit gate 4, but the payment counter 1 may also have the function of the gate control device 5.

[0052] In addition, in this embodiment, a virtual product sales floor controller 6 is provided, and this virtual product sales floor controller 6 controls the MR glasses 2 to display virtual products in the virtual product sales floor, but the checkout counter 1 may also have the functions of the virtual product sales floor controller 6.

[0053] In the present embodiment, the MR glasses 2 worn by the user on the head are used as a mixed reality (MR) device that displays virtual products superimposed on real space in the virtual product sales floor, but the MR device is not limited to such a device worn on the user's body. For example, a smartphone may be used as the MR device. In this case, when the smartphone camera is pointed at the virtual product sales floor, a virtual display shelf appears on the image captured by the camera displayed on the smartphone display, and the virtual product may be selected by an operation (touch operation) to select the virtual product displayed on the virtual display shelf. Furthermore, the device that displays virtual products superimposed on real space in the virtual product sales floor can be an AR (Augmented Reality) device in addition to an MR device.

[0054] Next, an unmanned store will be described with reference to Fig. 2, which is a plan view showing the layout of the unmanned store.

[0055] The unmanned store has an entrance / exit, a sales area, a checkout corner, and a registration corner.

[0056] Near the entrance, there are an entrance passage and an exit passage separated by a partition wall. An entrance gate 3 is provided in the entrance passage. Near the entrance passage, there is also an MR glass lending stand 25 on which MR glass 2 to be lent to users is placed. Near the exit passage, there is also an exit gate 4. Near the exit passage, there is also an MR glass return stand 26 on which users return the MR glass 2.

[0057] The sales floor is divided into a real product sales floor and a virtual product sales floor. In the real product sales floor, shelves are installed on which real products (actual products) are displayed. In the virtual product sales floor, virtual products (virtual objects of products) are displayed by the MR glasses 2 worn by the user.

[0058] The checkout corner is located near the exit passage. A plurality of checkout counters 1 are installed at the checkout corner.

[0059] The registration corner is located adjacent to the checkout corner and can be accessed directly from the entrance. A cash register 7 is installed in the registration corner.

[0060] When a user enters the store through the entrance, he or she enters the store entrance passageway and puts on the MR glasses 2 from the MR glasses lending stand 25 on the user's head. At this time, the gate control device 5 acquires iris information from the user's MR glasses 2 and has the authentication server 15 perform iris authentication. If the iris authentication is successful, the store entrance gate 3 opens and the user can enter the sales floor. Note that when the MR glasses 2 detect that the user is wearing them, a login screen is displayed, and when the person is identified by iris authentication, the user's name may be displayed.

[0061] Next, the user enters the sales floor through the store entrance passage, inspects the products displayed in the sales floor, and selects the product to purchase. At this time, in the real product sales floor, the user picks out the real product to be purchased from the real products displayed on the display shelves (picking up a real product). In the virtual product sales floor, a virtual display shelf is displayed by the MR glasses 2. The user selects the virtual product to be purchased from the virtual products displayed on the virtual display shelves (picking up a virtual product).

[0062] Next, the user moves to the checkout corner and performs checkout and settlement operations at the checkout counter 1. At this time, the user places a real product taken from the display shelf in the real product sales floor on the checkout counter 1 to complete the checkout, then iris authentication and password authentication are performed, and if the iris authentication and password authentication are successful, the payment is executed. Also, if the user selects a virtual product in the virtual product sales floor, the checkout and settlement are performed including the virtual product selected by the user.

[0063] Next, the user moves to the exit passage to leave the store. At this time, the gate control device 5 acquires iris information from the user's MR glasses 2 and has the authentication server 15 perform iris authentication. If the iris authentication is successful, the exit gate 4 opens and the user can leave the store through the entrance / exit. The user returns the MR glasses 2 to the MR glasses return stand 26 and then leaves the store through the exit passage.

[0064] In this embodiment, the MR glasses 2 are lent to the user at the store, but the MR glasses 2 that the user regularly uses may be used.

[0065] In addition, in this embodiment, iris authentication is performed based on the iris information of the user acquired by the MR glasses 2 when entering the store, but password authentication may be performed based on a user ID and password. In this case, an input screen may be displayed by the MR glasses 2, and the user ID and password may be input by operating a virtual keyboard.

[0066] Furthermore, this embodiment relates to an unmanned store system and is intended for unmanned stores where no store clerk is always present, but the system (checkout settlement system) disclosed herein allows the user to make the checkout and settlement (accounting) of products purchased without the intervention of a store clerk, and is not necessarily limited to unmanned stores. In other words, it may be intended for manned stores where a store clerk is always present, and may reduce the burden on store clerks in manned stores. In this case, the checkout counter 1 functions as a self-checkout. Also, it is possible for unmanned stores and manned stores to coexist. For example, both unmanned and manned registers may be installed in one store. Also, one store may be divided into an unmanned area and a manned area.

[0067] Next, the display of virtual products in the virtual product sales floor and the operation of picking up the virtual products will be described. Fig. 3 is an explanatory diagram showing the display state of virtual products in the virtual product sales floor and the operation state of picking up the virtual products.

[0068] The virtual product sales floor controller 6 controls the MR glasses 2 worn by the user to display a virtual product 201 in the virtual product sales floor in a state where the virtual product 201 is displayed on a virtual display shelf 202. That is, the virtual product 201, which is a virtual 3D object of the product, and the virtual display shelf 202, which is a virtual 3D object of the display shelf, are superimposed and displayed on the virtual product sales floor as a real space.

[0069] The MR glasses 2 have a function of recognizing hand gestures (hand movements) of a user. The virtual product sales floor controller 6 detects an operation by a hand gesture of the user based on the hand gesture recognition result by the MR glasses 2 (hand tracking).

[0070] Specifically, the user can select the virtual product 201 displayed by the MR glasses 2 by performing a hand gesture operation. For example, the virtual product 201 can be selected by pinching the virtual product 201 with fingers. Also, the virtual product 201 on the extension of the user's finger may be selected by the user's pointing motion. When the user selects the virtual product 201, the virtual product 201 is registered in the shopping list.

[0071] Also, the number of virtual goods 201 to be purchased may be designated by a user's hand gesture operation. For example, the user may select a virtual goods 201 with his / her right hand while designating the number of virtual goods 201 to be purchased with his / her left hand. Specifically, the user may designate the number of virtual goods 201 to be purchased by using fingers to indicate a number (hand signs).

[0072] Also, the virtual products 201 on the virtual display shelves 202 may be replaced by operations using hand gestures by the user. For example, virtual display shelves 202 for each product category may be displayed, and virtual display shelves 202 for another product category may be displayed by a hand gesture by the user. Specifically, the product categories on the virtual display shelves 202 may be sequentially changed, for example, from drinks to snacks to instant foods (cup noodles, etc.), by a gesture of the user sweeping his or her hand (flip gesture). This allows a wide variety of virtual products 201 to be displayed even in a small store with limited space, without having to secure a large virtual product sales floor.

[0073] Furthermore, the virtual products 201 on the virtual display shelf 202 may be recommended products selected according to the attributes (gender, age, etc.) and purchase history of the user.

[0074] 3, a virtual display shelf 202 on which virtual products 201 are displayed is displayed as a virtual object in the virtual product sales floor, but a display method other than a display shelf is also possible, for example, a display method in which the virtual products 201 are placed on a wagon (display stand) or a rotating stand, etc. Furthermore, various display methods are possible, such as a display method unique to the virtual space that is not possible in the real space, for example, a display method in which the virtual products 201 rotate while floating in the air.

[0075] In addition, the example shown in FIG. 3 is based on the premise that the store is a convenience store or supermarket, and a virtual display shelf 202 for rice is displayed as an example of a product category, but the store is not limited to such types of stores, and may be, for example, an apparel store.

[0076] In addition, in this embodiment, the virtual product is displayed by the MR glasses 2 in a virtual product sales floor set up in a store, but the virtual product may be displayed in an area other than the sales floor in the store. For example, while a user is performing a checkout operation at the checkout counter 1, the virtual product may be displayed by the MR glasses 2 around the checkout counter 1. In this case, a product similar to a display shelf installed next to a cash register counter in a real store (a display shelf next to the cash register) may be displayed as a virtual product. Also, the virtual product to be displayed may be selected based on information of the user (attributes, purchase history, etc.) and information of the product the user is about to purchase.

[0077] In addition, in this embodiment, the sales floor is divided into a real product sales floor and a virtual product sales floor (see Figure 2), but the sales floor may not be divided into a real product sales floor and a virtual product sales floor, and virtual products may be displayed by MR glasses 2, for example, in an aisle between shelves displaying real products.

[0078] In general, in EC shopping, customers often search for a desired product (a product they want) and then purchase the product. Some EC shopping sites suggest popular products that match the desired product, but it is difficult to realize shopping with the same feeling as browsing in a store. According to this embodiment, in addition to recognized real products, virtual products that do not actually exist in the store can be selected and made the target of payment in the same feeling as browsing real products in a normal store (looking around at products displayed on shelves), so that it is possible to promote customer purchasing opportunities while minimizing store inventory.

[0079] Furthermore, according to this embodiment, product display (shelf allocation) can be made more flexible; for example, the concepts of freezer shelves, refrigerated shelves, and room temperature shelves are eliminated, making it possible to display room temperature products next to refrigerated products, making it possible to arrange products in innovative layouts that would never be imagined in a normal store that only handles physical products.

[0080] Next, the adjustment counter 1 will be described. Fig. 4 is a perspective view showing the appearance of the adjustment counter 1. Fig. 4(A) shows the entire adjustment counter 1, and Fig. 4(B) shows the upper wall portion 34 as viewed obliquely from below.

[0081] As shown in Fig. 4(A), the main body 31 of the settlement counter 1 comprises a box-shaped portion 32, a top plate portion 33, an upper wall portion 34, and a rear wall portion 35. The top plate portion 33 is provided on the upper side of the box-shaped portion 32. The rear wall portion 35 is provided in a state in which it protrudes upward from the rear side of the box-shaped portion 32. The upper wall portion 34 is provided in a state in which it protrudes forward from the upper end of the rear wall portion 35 like a canopy.

[0082] The top panel 33 is provided with a tray 41 (placement section) on which the customer places the products they wish to purchase (products they have selected at the sales floor). The customer simply arranges the products on the tray 41, and the products are identified by object recognition, and the payment is calculated based on the unit price of the products. The tray 41 is formed in the top panel 33 in a dish-like recess, allowing the customer to easily grasp the area in which to place the products.

[0083] Additionally, a touch panel display 42 is provided on the top panel 33. The product recognition result, i.e., the product to be checked out, is displayed on this touch panel display 42, and if there is no error in the product recognition result, operations related to iris authentication and password authentication can be performed. Also, if there is an error in the product recognition result, operations to correct the product to be checked out can be performed.

[0084] The box-shaped section 32 is provided with a first storage section 46 (shelf section) with an open front side, and a second storage section 48 with a front side closed by a door 47. The first storage section 46 stores accessories for the product. These accessories are provided to the user free of charge and can be freely taken home by the user. Specifically, these accessories include shopping bags and cutlery (spoons, forks, etc.). The second storage section 48 stores a controller 49 (PC) that controls the touch panel display 42 and the like.

[0085] A display 45 is provided on the rear wall 35. This display 45 functions as digital signage, and constantly displays content such as store guides and product advertisements.

[0086] As shown in Fig. 4(B), a camera 51 is provided on the upper wall portion 34. This camera 51 photographs an item placed on the tray 41 of the top panel portion 33. Three cameras 51 are provided here. The central camera 51 photographs the item placed on the tray 41 from directly above, and the photographed image is used to detect the position of the item placed on the tray 41. The two cameras 51 on either side photograph the item placed on the tray 41 from diagonally above, and the photographed image is used to recognize the item (product name) placed on the tray 41.

[0087] Further, a speaker 53 is provided on the upper wall portion 34. This speaker 53 outputs a voice that responds to the user. For example, a guide voice that guides the user on how to operate the device is output from the speaker 53.

[0088] Next, a description will be given of placement of real products on the checkout counter 1 and display of virtual products. Fig. 5 is an explanatory diagram showing the placement of real products on the checkout counter 1 and the display of virtual products.

[0089] The checkout counter 1 is provided with a tray 41 on which a product to be purchased by a user is placed. The user places a real product 211 that he or she has taken out from a display shelf in the real product sales area on the tray 41, and the payment for the real product 211 is made.

[0090] On the other hand, when a user selects a virtual product 201 in the virtual product sales area (see FIG. 3), the virtual product 201 selected by the user is displayed on the tray 41 of the checkout counter 1 by the user's MR glasses 2. That is, the virtual product 201, which is a virtual 3D object of the product, is superimposed on the tray 41 of the checkout counter 1 as real space. This allows the real product that the user is about to purchase and the virtual product to be viewed together on the tray 41 of the checkout counter 1.

[0091] In this embodiment, in response to an operation of selecting a virtual product 201 displayed by the MR glasses 2 in the virtual product sales floor (see FIG. 3), the selected virtual product is registered in a shopping list, and the virtual product 201 registered in the shopping list is displayed on the tray 41 of the checkout counter 1. At this time, the virtual product sales floor control machine 6 registers the virtual product 201 in the shopping list in response to the user's selection operation, and the checkout counter 1 acquires information on the shopping list from the virtual product sales floor control machine 6 and controls the MR glasses 2 of the user to display the virtual product 201 registered in the shopping list on the tray 41 of the checkout counter 1.

[0092] It is not necessary to display the virtual product at its actual size. For example, if the virtual product is large, it may protrude from the tray 41 if it is displayed at its actual size. Also, if the number of products increases, it may not be possible for all the products to be purchased to fit on the tray 41. In this case, the virtual products may be displayed in a reduced size, making it possible for all the products to be purchased to fit on the tray 41.

[0093] In this manner, in this embodiment, virtual products displayed in the virtual product sales floor can be purchased together with real products displayed in the real product sales floor. For example, bulky or heavy products such as rice, shampoo, and detergent that are troublesome to carry home are displayed as virtual products in the virtual product sales floor and are delivered to the user's home or the like. On the other hand, products that are consumed immediately such as bento, prepared foods, and fresh foods (vegetables, fresh fish, etc.) or products that the user wants to inspect in person such as fresh foods are displayed as real products in the real product sales floor so that the user can take them home on the spot. This can increase the convenience of the user. Also, in this embodiment, since virtual products are displayed in a real store, the user can get the feeling of shopping, unlike a virtual store in EC shopping.

[0094] Next, a description will be given of the schematic configuration of the adjustment counter 1 and the MR glass 2. Fig. 6 is a block diagram showing the schematic configuration of the adjustment counter 1 and the MR glass 2.

[0095] The checkout counter 1 is equipped with a camera 51, a touch panel display 42, a display 45, a speaker 53 (see FIG. 4), as well as a communication unit 61, a storage unit 62, and a control unit 63. The communication unit 61, the storage unit 62, and the control unit 63 constitute a controller 49 (see FIG. 4).

[0096] The communication unit 61 communicates with the user management server 13, the product learning server 14, the authentication server 15, the MR glasses 2, and the virtual product sales floor controller 6 via a network.

[0097] The storage unit 62 stores a program executed by a processor constituting the control unit 63. The storage unit 62 also stores product master information. Specifically, the storage unit 62 stores product identification information (product name, product code, etc.) and information used in the settlement process, i.e., information related to the amount (unit price) of each product.

[0098] The control unit 63 includes a product detection unit 71, a product recognition unit 72, a settlement processing unit 73, an authentication instruction unit 74, a virtual object control unit 75, a payment instruction unit 76, and a delivery instruction unit 77. The control unit 63 is composed of a processor, and each unit of the control unit 63 is realized by causing the processor to execute a program stored in the storage unit 62.

[0099] The commodity detection unit 71 detects that a commodity has been placed on the tray 41 based on an image captured by the camera 51 that captures the tray 41. Furthermore, when a commodity has been placed on the tray 41, the commodity detection unit 71 detects the position of the commodity based on the image captured by the camera 51.

[0100] The commodity recognition unit 72 recognizes the commodity placed on the tray 41 based on the image captured by the camera 51. In this embodiment, an object recognition engine constructed by machine learning such as deep learning is used to acquire commodity identification information (such as the name of the commodity) from the commodity image cut out from the image captured by the camera 51.

[0101] The settlement processing unit 73 settles the price of the products placed on the tray 41 based on the product recognition results by the product recognition unit 72. That is, the price (unit price) of each product placed on the tray 41 is obtained, the prices of each product are tallied, and the total price is calculated (tallying process). Also, in this embodiment, when a user selects a virtual product in the virtual product sales floor, information on a shopping list in which the selected virtual product is registered is provided from the virtual product sales floor control machine 6, and the settlement processing unit 73 settles the price of the virtual product together with the real product placed on the tray 41 based on the information on the shopping list.

[0102] The authentication instruction unit 74 instructs the authentication server 15 to perform authentication in order to identify the user at the time of payment, and also instructs the user management server 13 to perform password authentication. Here, during iris authentication, iris information of the user is acquired from the MR glasses 2 and the iris information is sent to the authentication server 15. During password authentication, the user ID and password entered by the user are sent to the user management server 13. Note that, to strengthen security, two-factor authentication of iris authentication (biometric authentication) and password authentication may be adopted, and both iris authentication and password authentication may be successful as a condition.

[0103] The virtual object control unit 75 controls the user's MR glasses 2 to superimpose and display virtual products (virtual 3D objects of products) on the tray 41 of the checkout counter 1, which is the real space. When paying for purchased products at the checkout counter 1, the checkout counter 1 obtains shopping list information from the virtual product sales floor control machine 6 and controls the user's MR glasses 2 to display the virtual products registered in the shopping list on the tray 41 of the checkout counter 1.

[0104] Here, the virtual product is displayed alongside the real product placed on the tray 41, but in this case, the position of the real product on the tray 41 is detected so that the real product is not hidden by the virtual product when viewed from the user, and the position at which the virtual product is displayed is determined based on the position information. The position of the real product on the tray 41 can be detected based on an image captured by a camera 51 provided on the checkout counter 1, or can also be detected based on the detection result of a sensor 82 of the MR glasses 2.

[0105] The payment instruction unit 76 instructs the payment server 12 to execute a payment process for the amount calculated in the settlement process for the user (person to be paid) identified by the iris authentication and password authentication.

[0106] The delivery instruction unit 77 instructs the delivery department (delivery center) to deliver the actual virtual product to a location (delivery destination) designated by the user (delivery arrangement). Specifically, it generates arrangement information instructing the delivery of the actual virtual product, and transmits the arrangement information to the delivery server 17 via the user management server 13. The arrangement information includes the user's delivery destination information (name and address). Note that the user's delivery destination information is registered in advance based on an input operation by the user.

[0107] For real products, payment is always made at the checkout counter 1, but for virtual products, as with e-commerce shopping, payment can be made outside the store, such as through cash on delivery, where the fee is collected when the product is delivered, or through a transfer procedure at a convenience store or financial institution. Therefore, payment at the checkout counter 1 may be omitted depending on the user's request.

[0108] The MR glasses 2 include a display 81, a sensor 82, a communication unit 84, a storage unit 86, and a control unit 87. The MR glasses 2 may further include a speaker.

[0109] The display 81 displays an MR screen in which a virtual object is superimposed on a real space.

[0110] The sensor 82 detects objects around the device. In this embodiment, the sensor 82 includes an outward-facing camera that captures an image of the hand of a person wearing the device, and an inward-facing camera that captures an image of the user's eyeball for iris authentication.

[0111] The communication unit 84 communicates with the checkout counter 1, the gate controller 5, and the virtual product sales floor controller 6.

[0112] The storage unit 86 stores a program executed by a processor constituting the control unit 87 .

[0113] The control unit 87 includes a body recognition unit 91 and an environment recognition unit 92. The control unit 87 is configured with a processor, and each unit of the control unit 87 is realized by causing the processor to execute a program stored in the storage unit 86.

[0114] The body recognition unit 91 recognizes hand gestures (hand movements) of the user wearing the device based on the detection results of the sensor 82. The virtual product sales floor controller 6 detects user operations, such as an operation to select a virtual product, based on the hand gesture recognition results acquired by the body recognition unit 91 (hand tracking). The virtual product sales floor controller 6 also controls the display of the virtual display shelves by the MR glasses 2 based on the hand gesture recognition results.

[0115] Furthermore, the body recognition unit 91 acquires an eyeball image captured by a camera serving as the sensor 82, and extracts iris information (iris image or iris feature amount) from the eyeball image. The authentication server 15 performs iris authentication based on the iris information of the user acquired by the body recognition unit 91 to identify the user.

[0116] The environment recognition unit 92 recognizes the surrounding environment of the user wearing the device, such as the floor of the store on which the user is standing, or the surface of the tray 41 on the checkout counter 1.

[0117] The virtual merchandise selling floor controller 6 and the checkout counter 1 may have the functions of the body recognition unit 91 and the environment recognition unit 92.

[0118] Next, a description will be given of the screens displayed on the touch panel display 42 of the checkout counter 1. Figures 7 and 8 are explanatory diagrams showing the screens displayed on the touch panel display 42 of the checkout counter 1.

[0119] First, the checkout guide screen shown in Fig. 7(A) is displayed on the touch panel display 42 of the checkout counter 1. On this checkout guide screen, a guide message 101 is displayed encouraging the user to place the product on the tray 41 of the checkout counter 1, and a guide image 102 (such as an illustration) explaining how to place the product. When the user places the product on the tray 41, the checkout counter 1 recognizes the product and processes the payment, and the screen transitions to a purchased product confirmation screen (see Fig. 7(B)).

[0120] The purchase product confirmation screen shown in FIG. 7(B) displays a guide message 104 that prompts the user to confirm the product to be purchased. The purchase product confirmation screen also has a recognition result list display section 105. This recognition result list display section 105 displays product boxes 115 (product display sections) in which the names and prices of products are written side by side. These product boxes 115 relate to both real products placed on the tray 41 by the user and virtual products selected by the user at the virtual product counter. The display form (e.g., color, etc.) of the product boxes 115 relating to real products and the product boxes 115 relating to virtual products may be different.

[0121] This purchase product confirmation screen also has a payment result display section 116. This payment result display section 116 displays the payment result, that is, the total number of products placed on the tray 41 and the total price.

[0122] This purchase item confirmation screen also has a "Checkout" button 117 and a "Cancel Checkout" button 119. When the "Checkout" button 117 is pressed, the settlement result is finalized and the screen moves to authentication processing for payment, where iris authentication is performed to identify the user as the person to whom payment is to be made. On the other hand, when the "Cancel Checkout" button 119 is pressed, the screen transitions to a cancel screen (not shown). The screen also transitions to the cancel screen (not shown) when the item is removed from the tray 41.

[0123] If the iris authentication is successful, the screen transitions to an authentication result confirmation screen (see FIG. 7(C)). On the other hand, if the iris authentication is unsuccessful, an error screen (not shown) is displayed.

[0124] The authentication result confirmation screen shown in FIG. 7(C) displays a message 126 inquiring whether the user's name is correct. This authentication result confirmation screen also has a "Yes" button 127. Operating "Yes" button 127 will transition to the password authentication screen (see FIG. 8(A)). Note that the payment result display section 116 and "Cancel payment" button 119 displayed on the screen of FIG. 7(C) are the same as those on the purchased item confirmation screen (see FIG. 7(B)).

[0125] The password authentication screen shown in Fig. 8(A) is provided with a message 131 prompting the user to input a PIN as a password, an image 132 showing the PIN input status, and a numeric keypad 133. Once the user has input the PIN with a predetermined number of digits, password authentication is executed, and if password authentication is successful, the screen transitions to a payment confirmation screen (see Fig. 8(B)). On the other hand, if password authentication is unsuccessful, the user is prompted to re-enter the password.

[0126] This password authentication screen also has a "Pay" button 135 and a "Back" button 136. If the "Back" button 136 is operated, the screen returns to a state in which no PIN has been entered. The "Pay" button 135 is also grayed out and inoperable.

[0127] On the payment confirmation screen shown in Fig. 8(B), a "Pay" button 135 becomes operable, and when this "Pay" button 135 is operated, the screen transitions to a payment completion screen (see Fig. 8(C)). Note that the settlement result display section 116 and the "Cancel payment" button 119 displayed on the screens of Figs. 8(A) and (B) are the same as those on the purchased item confirmation screen (see Fig. 7(B)).

[0128] Next, an explanation will be given of the operation procedure of user terminal 11 during user registration. Fig. 9 is a flow diagram showing the operation procedure of user terminal 11 during user registration.

[0129] When the application is installed and started for the first time on the user terminal 11, a personal information confirmation screen is displayed first (ST101). On this personal information confirmation screen, a consent form regarding the handling of personal information is displayed. When the user performs an operation to approve the consent form on this personal information confirmation screen, an authentication information input screen is displayed (ST102).

[0130] Next, when the user performs an operation to input a user ID and a password on the authentication information input screen, the user ID and the password are transmitted to the user management server 13 (ST103).

[0131] Next, a personal information input screen is displayed on the user terminal 11 (ST104). On this personal information input screen, the user can input personal information. At this time, the personal information input includes the user's name, attributes (gender, date of birth, etc.), delivery destination information (user's location, etc.), etc. When the user performs an operation to input personal information on this personal information input screen, the personal information including the date of birth, etc. is transmitted to the user management server 13 (ST105).

[0132] Next, an eyeball photographing screen for iris authentication is displayed on the user terminal 11 (ST104). When the user performs an operation to photograph his / her own eyeball on this eyeball photographing screen, the eyeball image is extracted from the photographed image and transmitted to the user management server 13 (ST107).

[0133] At this time, the user management server 13 performs a process of registering the user ID and password acquired from the user terminal 11. Also, the user management server 13 transmits the eyeball image acquired from the user terminal 11 to the authentication server 15, and the authentication server 15 performs a process of registering the eyeball image.

[0134] Next, a credit information input screen is displayed on the user terminal 11 (ST108). When the user performs an operation to input credit information on this credit information input screen, the credit information is transmitted to the payment server 12 (ST109). The payment server 12 performs a process of registering the credit information acquired from the user terminal 11.

[0135] Next, when the user terminal 11 receives a notification of completion of the credit information registration from the settlement server 12, a registration completion screen is displayed (ST110).

[0136] It should be noted that the user can also perform user registration operations at the cash register 7 installed in the store, and the procedure is the same as that for the user terminal 11.

[0137] Next, there will be described the operation procedure of the adjustment counter 1. Figures 10 and 11 are flow charts showing the operation procedure of the adjustment counter 1.

[0138] At the checkout counter 1, first, a checkout guide screen (see FIG. 7(A)) is displayed (ST201). Then, when it is detected that an object has been placed on the tray 41 based on the image captured by the camera 51 (Yes in ST202), the position of the object placed on the tray 41 is detected (ST203). Next, the product that the object placed on the tray 41 is identified (ST204). Next, the price of the product placed on the tray 41 is paid for (ST205). Next, a purchase product confirmation screen (see FIG. 7(B)) is displayed (ST206).

[0139] Next, when the user operates the operation to select cancel on the purchased item confirmation screen (see FIG. 7(B)), specifically, the "Cancel checkout" button 119 ("Cancel" in ST207), the screen transitions to a cancellation screen (not shown) (ST208). Note that here, it may be possible to perform an operation to modify items (change, add, or delete items) in case there is an error in the settlement result due to a failure in item recognition, or in case the user wishes to cancel the purchase of some items.

[0140] On the other hand, when the user approves the purchase and instructs payment, specifically by operating the "Pay" button 117 ("Payment" in ST207), the payment result is finalized and the process moves to authentication processing for payment. That is, iris information is acquired from the MR glasses 2, and a request for personal authentication including the iris information is sent to the authentication server 15 (ST209). At this time, in response to the iris authentication request, the authentication server 15 performs iris authentication based on the iris information acquired from the checkout counter 1, and an iris authentication response including the authentication result is sent to the checkout counter 1.

[0141] Next, the checkout counter 1 receives the iris authentication response from the authentication server 15 (ST210), and if the authentication result included in the iris authentication response is unsuccessful (No in ST211), an error screen (not shown) is displayed (ST212). On the other hand, if the authentication result included in the iris authentication response is successful (Yes in ST211), an authentication result confirmation screen (see FIG. 7(C)) is displayed (ST213).

[0142] If the "Yes" button 127 is operated on this authentication result confirmation screen (see FIG. 7(C)), password authentication is proceeded to, and the password authentication screen (see FIG. 8(A)) is displayed (ST214).

[0143] Next, when the user enters a password (personal identification number) on the password authentication screen (see FIG. 8(A)) at the checkout counter 1, a password authentication request is sent to the user management server 13 (ST215). Upon receiving the password authentication request, the user management server 13 performs password authentication based on the authentication information acquired from the checkout counter 1, and sends a password authentication response including the authentication result to the checkout counter 1.

[0144] Next, the checkout counter 1 receives a password authentication response from the user management server 13 (ST216), and if the authentication result included in the password authentication response indicates failure (No in ST217), an error screen (not shown) is displayed (ST208). On the other hand, if the authentication result included in the password authentication response indicates success (Yes in ST217), a payment confirmation screen (see FIG. 8(B)) is displayed (ST219).

[0145] When the user operates the "Pay" button 135 on this payment confirmation screen (see FIG. 8(B)), a payment request is sent to the payment server 12 via the user management server 13 (ST220). Upon receiving the payment request, the payment server 12 executes a payment process and sends a payment response to the checkout counter 1 via the user management server 13.

[0146] Next, when the checkout counter 1 receives a payment response from the payment server 12 (ST221), a payment completion screen (see FIG. 8(C)) is displayed (ST222). Next, the checkout counter 1 performs a receipt issuing process (ST223). At this time, the receipt information is sent to the user terminal 11 via the user management server 13, and the user terminal 11 stores the received receipt information in its own device.

[0147] In addition, at the checkout counter 1, arrangement information is generated to instruct the delivery of the actual virtual product purchased by the user, and the arrangement information is transmitted to the delivery server 17 via the user management server 13 (delivery arrangement) (ST224).

[0148] In this embodiment, in order to strengthen security, two-factor authentication consisting of iris authentication and password authentication is adopted when paying for purchases at the checkout counter 1, and password authentication is performed even if iris authentication is successful, but password authentication may be omitted and only iris authentication may be used. Also, if iris authentication fails, payment may be made using only password authentication. In this case, the screen may transition to a screen where the user selects or inputs their user ID, and then transition to a password authentication screen in ST214 (see FIG. 8(A)).

[0149] In addition, in this embodiment, iris authentication is performed, but face authentication may be performed instead of iris authentication. In this case, a camera that captures the face of the user may be provided at the checkout counter 1, and face authentication may be performed based on a face image extracted from an image captured by the camera. Furthermore, authentication other than such biometric authentication (iris authentication or face authentication), such as card authentication using an IC card, may also be performed.

[0150] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0151] The settlement device, settlement system, and settlement method of the present invention increase convenience for users by allowing them to purchase products that are not actually present in the store in addition to products that are actually present in the store, and have the effect of allowing users to fully check products that are not actually present in the store in the same way as products that are actually present in the store, and to fully feel as if they have purchased the products.They are useful as a settlement device that recognizes the products selected by a user on the sales floor and processes the settlement and payment of the price, as well as a settlement system and settlement method using a settlement device. [Explanation of symbols]

[0152] 1. Payment counter (payment device) 2. MR glasses (virtual object display device) 3. Entry gate 4. Exit gate 5. Gate control machine 6 Virtual product sales floor controller (virtual object controller) 17 Distribution Server 31 Main unit 41 Tray (Placement part) 42 Touch panel display (display unit) 51 Camera 63 Control Unit 71 Product detection unit 72 Product recognition department 73 Payment Processing Section 74 Authentication Instruction Section 75 Virtual Object Control Unit 76 Payment Instructions Department 201 Virtual Goods 202 Virtual Shelf 211 Actual Product

Claims

1. A settlement device that processes settlement and payment for real products and virtual products selected by a user in a store, A main body having a placement section on which the actual product selected by the user at the sales floor is placed; A camera that photographs the actual product placed on the placement section; A control unit that recognizes the target actual product based on the product image captured by the camera and processes payment and settlement; a display unit that displays the settlement result and the payment result acquired by the control unit, The control unit further A virtual object display device is controlled to recognize the target virtual product in response to a user's operation of selecting the virtual product as a virtual object superimposed on a predetermined area of ​​the store as a real space, and the recognized virtual product is displayed next to the real product placed on the placement unit. The recognized virtual product is superimposed and displayed on the placement unit as a real space via the virtual object display device; Next, a product confirmation screen including a list of recognition results of the purchased products in different display formats for the virtual products and the real products is displayed on the display unit, and settlement or payment for the purchased products including the virtual products is performed.

2. The control unit is 2. The settlement device according to claim 1, wherein, when settlement or payment for purchased goods including the virtual goods is completed, processing for delivery of the actual virtual goods is performed.

3. A settlement system that processes settlement and payment for real products and virtual products selected by a user in a store, a settlement device having a placement unit on which the actual product selected by a user at the sales floor is placed, the settlement device photographs the actual product placed on the placement unit, recognizes the actual product based on the product image obtained by photographing, performs settlement and payment related processing, and displays the settlement result and payment result; a virtual object display device that displays a virtual object in a superimposed manner on a real space; a virtual object control device that controls the virtual object display device to detect a user's operation of selecting the virtual product as a virtual object superimposed and displayed in a predetermined area of ​​a store as a real space, and recognizes the target virtual product; The settlement device includes: a virtual object display device that displays the recognized virtual product in a superimposed manner on the placement unit as a real space in a state where the recognized virtual product is arranged next to the real product placed on the placement unit; Next, a product confirmation screen including a list of recognition results of the purchased products in different display formats for the virtual products and the real products is displayed on the display unit, and settlement or payment for the purchased products including the virtual products is performed.

4. The virtual object display device includes:

4. The settlement system according to claim 3, which is a wearable device that is worn on the user's head and displays the virtual object in a superimposed manner on the user's real field of vision.

5. The virtual object display device includes: The device is provided with a sensor for detecting a user's movement, and recognizes the user's hand gestures based on the detection result of the sensor; The virtual object control device includes: The settlement system according to claim 3 , further comprising a step of detecting a user's operation of selecting the virtual product based on the hand gesture recognition result.

6. A settlement method for processing a settlement and payment for real products and virtual products selected by a user in a store, comprising: A virtual object control device, controlling a virtual object display device that displays a virtual object superimposed on a real space, detecting an operation by a user to select the virtual product as a virtual object superimposed on a predetermined area of ​​the store as the real space, and recognizing the target virtual product; A settlement device having a display unit, Taking a picture of the actual product that the user has selected at the sales floor and placed on the placement unit, and recognizing the actual product based on the product image obtained by taking the picture; a virtual object display device that displays the recognized virtual product in a superimposed manner on the placement unit as a real space in a state where the recognized virtual product is arranged next to the real product placed on the placement unit; Next, a product confirmation screen including a list of recognition results of the purchased products in different display formats for the virtual product and the real product is displayed on the display unit, and settlement or payment for the purchased products including the virtual product is performed.

Images