Information processing apparatus, method for controlling information processing apparatus, and program

Abstract

To provide a more suitable system related to management and sales of commodities.SOLUTION: An information processing apparatus has: detection means that detects a phenomenon in which a commodity is taken out from a display place where commodities are displayed; and control means that, when the detection means detects the phenomenon in which the commodity is taken out from the display place, in the case of a first state in which the display place and a purchase planning person or an electronic apparatus operated by the purchase planning person are associated with each other, controls not to output predetermined notification, and in the case of a second state in which the display place and the purchase planning person or the electronic apparatus owned by the purchase planning person are not associated with each other, to output the predetermined notification.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a system that enables the management and sale of products with few or no store employees.

Background Art

[0002] Conventionally, sales systems that reduce the labor costs required for product sales, such as vending machines and unmanned stores, have been proposed.

[0003] In Patent Document 1, a sales system (product management device) is proposed in which a plurality of products each attached with an RFID tag are arranged on a product shelf equipped with a weight sensor. When a change in the weight of the product shelf is detected, a product placed on the product shelf with a changed weight and having a changed RFID tag reception intensity is identified as a physically moved product (a product that a purchaser may have picked up). At the time of settlement, the total amount is calculated by reading the RFID tags attached to the products in the shopping basket by an RFID reader and displayed on the display unit of the product management device. Then, settlement is performed when the purchaser inserts money into the payment amount reading unit.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] <> The system described in Patent Document 1 was such that in an unmanned store, for example, an RFID tag was attached to each product, a display unit for displaying the total amount was installed, and a device for settlement was installed. However, in a sales system with few or no store employees, since there is little or no guidance by store employees, there was room for improvement in the purchaser's ability to shop appropriately.

[0006] In view of at least one of the above problems, the present invention aims to provide a more suitable system for managing and selling goods. [Means for solving the problem]

[0007] To solve the above problems, the information processing apparatus of the present invention is: A detection means for detecting the phenomenon of a product being removed from the display area where the product is displayed, When the detection means detects that a product is removed from the display area, If the display location and the prospective buyer or the electronic device operated by the prospective buyer are associated in the first state, the prescribed notification will not be output. Control means to output the predetermined notification if the display location and the prospective buyer or the electronic device owned by the prospective buyer are not associated in a second state. It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, a more suitable system for managing and selling goods can be provided. [Brief explanation of the drawing]

[0009] [Figure 1] This is a system configuration diagram of the sales system in this embodiment. [Figure 2] This is a hardware configuration diagram of each device included in the sales system of this embodiment. [Figure 3] This is a processing flowchart for the purchaser terminal 100a. [Figure 4] This is a processing flowchart for server device 100b. [Figure 5] This diagram illustrates the information managed by server device 100b. [Figure 6] This is a processing flowchart for the Edge PC100c. [Figure 7]This is a sequence diagram of an example of processing in the sales system of this embodiment. [Figure 8] This is an example of the screen display on the purchaser's terminal 100a. [Modes for carrying out the invention]

[0010] The system described in Patent Document 1 requires, for example, attaching RFID tags to each product, installing a display unit to show the total amount, and installing a payment device, which increases the cost and installation space required for setting up product shelves. In other words, there was room for improvement in the introduction cost and / or installation space of the sales system. To reduce costs and space, it is conceivable to use user terminals owned by each purchaser, but in that case, it is preferable that the functions used on the user terminals be more versatile so that more users can use them. Furthermore, in unmanned or minimally staffed sales systems, there is little to no guidance from staff, so there was also room for improvement in enabling purchasers to shop appropriately, easily, and casually.

[0011] In this embodiment, the objective is to provide a technology that offers a more suitable system for managing and selling products, in view of at least one of the above-mentioned problems.

[0012] Preferred embodiments of the present invention will be described below with reference to the drawings.

[0013] Figure 1 shows an example of the system configuration of the information processing system (sales system) of this embodiment. The sales system of this embodiment is an unmanned sales system that can sell without the need for sales staff. The sales system of this embodiment includes a buyer terminal 100a (electronic device), a server device 100b (information processing device), and an edge PC 100c (terminal device as an information processing device). The buyer terminal 100a can communicate with the server device 100b via the internet or directly. Similarly, the edge PC 100c can also communicate with the server device 100b via the internet or directly. On the other hand, the buyer terminal 100a and the edge PC 100c do not need to communicate directly.

[0014] One or more (at least one) products, including product 131, are placed (displayed) on shelf 130, which is a product shelf serving as an example of a product display location (including the place where the products are placed). Furthermore, a QR code (trademark) 160 containing the shelf ID (identification information) of shelf 130 is displayed on shelf 130 itself or near shelf 130. In this embodiment, shelf 130 does not have partitions such as doors, lids, or covers to separate it from the customer, allowing the customer to directly take out the product without opening a door. This lowers the physical and psychological barriers for the customer to pick up the product, making it easier for the customer to make a purchase and leading to increased sales for the seller. Note that this embodiment, described later, is still applicable even if doors, lids, or covers are provided. Also, this embodiment, described later, is still applicable even if store employees are present for sales promotion purposes.

[0015] As a sensor for acquiring information for detecting that a product displayed on shelf 130 has been taken out or returned, camera 140 is installed as a device included in the sales system of the present embodiment. Camera 140 is installed so as to have a shooting range in which the state of taking in and out of products from shelf 130 can be shot by camera 140. Edge PC 100c is associated with shelf 130 and is connected to camera 140 by wired or wireless communication so as to be able to acquire the image shot by camera 140. In the present embodiment, an example of processing by associating an edge PC and a product shelf on a one-to-one basis will be described, but a plurality of product shelves may be associated with one edge PC for processing.

[0016] Edge PC 100c analyzes the captured image (video) transmitted from camera 140 and detects that a product has been taken out or returned from shelf 130, etc. Then, information indicating that a product has been taken out or returned is transmitted to server device 100b.

[0017] Purchaser 120 acquires a shelf ID by shooting QR code 160 with a purchaser terminal 100a such as a smartphone held by the purchaser. Then, the shelf ID and the identification information (user ID) of purchaser 120 are transmitted to server device 100b.

[0018] In the server device 100b, based on the shelf ID and user ID received from the purchaser terminal 100a, the user ID of the purchaser 120 and the shelf ID of the shelf 130 are associated (that is, the purchaser 120 and the shelf 130 are associated). Hereinafter, the process of associating (associating and storing) the user ID and the shelf ID is called the check-in process, and the associated state is called the check-in state (checked-in state). When the check-in process is completed, the server device 100b notifies the purchaser terminal 100a and the edge PC 100b that the check-in has been completed. In this state, the purchaser 120 can take out the products placed on the shelf 130. When the purchaser 120 picks up and takes out the products placed on the shelf 130, the edge PC 100c detects them based on the images captured by the camera 140 and transmits the detection results to the server device 100b. The server identifies the taken-out products and adds the taken-out products to the virtual cart (shopping cart. Hereinafter, simply referred to as the cart) of the purchaser 120 associated with the shelf 130. The state of the cart is transmitted from the server device 100b to the purchaser terminal 100a and displayed on the purchaser terminal 100a, so that the purchaser 120 can confirm that the products picked up by himself / herself are in the cart. Then, when the user operates the purchaser terminal 100a to perform the settlement process, the server device 100b calls the settlement service 170 on the Internet to perform the settlement. In this way, the purchaser 120 can purchase products through a simple and convenient process of reading the QR code 160 on the purchaser terminal 100a, picking up the products, and performing the settlement operation.

[0019] FIG. 2(a) shows an example of the configuration of the purchaser terminal 100a as an example of the device to which the present invention is applicable. The purchaser terminal 100a is an electronic device held by a user who is a purchaser, and can be configured using a smartphone (mobile phone terminal), a tablet PC, or the like.

[0020] In Figure 2, the CPU 101a, memory 102a, non-volatile memory 103a, image processing unit 104a, display 105a, operation unit 106a, recording medium interface 107a, external interface 109a, and communication interface 110a are connected to the internal bus 150a. Each component connected to the internal bus 150a is configured to exchange data with each other via the internal bus 150a.

[0021] Memory 102a consists of, for example, RAM (volatile memory using semiconductor elements). CPU 101a controls various parts of the purchaser terminal 100a using memory 102a as work memory, according to a program stored in, for example, non-volatile memory 103a. Non-volatile memory 103a stores image data, audio data, other data, and various programs for the operation of CPU 101a. Non-volatile memory 103a consists of, for example, flash memory or ROM.

[0022] The image processing unit 104a performs various image processing operations on images stored in the non-volatile memory 103a or recording medium 108a, images acquired via the external I / F 109a or communication I / F 110a, and captured images, based on the control of the CPU 101a. The image processing operations performed by the image processing unit 104a include A / D conversion, D / A conversion, image data encoding, compression, decoding, resizing, noise reduction, and color conversion. The image processing unit 104a may be composed of dedicated circuit blocks for performing specific image processing operations. Furthermore, depending on the type of image processing, the CPU 101a may perform image processing according to a program without using the image processing unit 104a.

[0023] The display 105a displays images and GUI (Graphical User Interface) screens based on the control of the CPU 101a. The CPU 101a generates display control signals according to the program and controls various parts of the purchaser terminal 100a to generate video signals for display on the display 105a and output them to the display 105a. The display 105 displays images based on the output video signals. The purchaser terminal 100a itself is configured to have an interface for outputting video signals for display on the display 105, and the display 105a may be an external monitor (such as a television).

[0024] The operation unit 106a is an input device for receiving user input. The operation unit 1006a includes a character information input device such as a keyboard, a pointing device such as a mouse or touch panel 70a, buttons, dials, joysticks, touch sensors, and touchpads. The touch panel 70a is an input device that is superimposed on the display 105a to form a planar structure and outputs coordinate information corresponding to the position of contact.

[0025] The recording medium interface 107a is capable of receiving recording media 108a such as memory cards, and, based on the control of the CPU 101a, reads data from the received recording media 108a and writes data to the recording media 108a. The external interface 109a is an interface for connecting to external devices via wired cables or wirelessly and for inputting and outputting video and audio signals. The communication interface 110a is an interface for communicating with external devices and the Internet 111 to send and receive various data such as files and commands.

[0026] The camera unit 112a is a camera unit composed of an image sensor (imaging sensor) such as a CCD or CMOS element that converts optical images into electrical signals. The camera unit 112a includes a lens group (photographic lens) including a zoom lens and a focus lens, a shutter with an aperture function, an image sensor, an A / D converter that converts analog signals output from the image sensor into digital signals, and a barrier that covers the imaging system to prevent dirt and damage. The image processing unit 104a performs resizing processing such as predetermined pixel interpolation and reduction, as well as color conversion processing, on the data captured and acquired by the camera unit 112a. Based on the calculation results obtained by the image processing unit 104a, the CPU 101a performs exposure control, distance measurement control, and AWB (auto white balance) processing. The image data for display, captured by the camera unit 112a and processed by the image processing unit 104a, is displayed on the display 105a. The digital signal captured by the camera unit 112a, converted once to digital by the A / D converter and stored in the memory 102a, is converted to analog by the D / A converter and sequentially transferred to the display 105a for display, enabling live view display (LV display). Live view can be displayed in the still image shooting standby state, the video shooting standby state, and during video recording, and the captured subject image is displayed in near real time. The CPU 101a controls the camera unit 112a and the image processing unit 104a to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB processing in response to shooting preparation instructions based on user operations made on the operation unit 106a. In response to the shooting instruction, the CPU 101a controls the start of a series of shooting operations (main shooting), from the main exposure and reading the signal from the image sensor, to the image processing unit 104a processing the captured image to generate an image file and recording it on the recording medium 108a. Shooting instructions can be made by user operations on the operation unit 106a. The camera unit 112a is capable of capturing both still images and videos.

[0027] Figure 2(b) shows an example of the configuration of a server device 100b, which is an example of a device to which the present invention can be applied. The server device 100b is an information processing device that can communicate with the purchaser terminal 100a and the edge PC 100c, and can be configured using a personal computer (PC) or the like. The configuration of the server device 100b, indicated by reference numerals 101b to 104b, 106b to 110b, and 150b, is the same as the configuration of the purchaser terminal 100a indicated by reference numerals 101a to 104a, 106a to 110a, and 150a as described above. Therefore, a detailed explanation is omitted. In addition, although the server device 100b is described as not having a display and a camera unit, a device having a display and a camera unit may be used as the server device 100b.

[0028] Figure 2(c) shows an example configuration of an edge PC 100c, which is an example of a device to which the present invention can be applied. The edge PC 100c is a terminal device (information processing device) that can communicate with the server device 100b, and can be configured using a personal computer (PC) or the like. The configuration of the edge PC 100c, indicated by reference numerals 101c to 104c, 106c to 110c, and 150c, is the same as the configuration of the aforementioned purchaser terminal 100a, indicated by reference numerals 101a to 104a, 106a to 110a, and 150a. Therefore, a detailed explanation is omitted. Furthermore, the edge PC 100c will be described as not having a display and a camera unit. That is, it can be applied to a device that consists only of the desktop PC body, and since it can be configured without having at least one of the display and camera units, costs and installation space can be reduced. However, a device having a display and a camera unit may also be used as the edge PC 100c. The audio output unit 113c is an output unit that transmits an audio signal to a speaker via wired or wireless connection to produce sound, or it is the speaker itself. The notification sound or voice message output from the audio output unit 113c can notify (make known) the purchaser 120 of various guidance and warning information.

[0029] Figure 3 shows a flowchart of the sales system processing on the buyer terminal 100a. This processing is achieved by loading the program recorded in the non-volatile memory 103a into memory 102a and executing it on the CPU 101a. When a buyer 120 launches an application program capable of reading a QR code on the buyer terminal 100a, or launches an application program for this system (hereinafter simply referred to as the purchase app), the processing shown in Figure 3 begins in order to purchase an item from shelf 130.

[0030] In S301, the CPU 101a displays the LV image captured by the camera unit 112a on the QR code reading screen. Figure 8(a) shows an example of the QR code reading screen displayed on the display 105a (touch panel 70a) by the purchase application. Message 801 is a guidance message prompting the purchaser 120 to have the purchaser terminal 100a read the QR code 160 (frame the image so that the QR code 160 is within the shooting range). For example, it may say, "Please read the QR code posted on the product shelf." Frame 802 is a guide frame that serves as a reference for framing the image so that the QR code is within the shooting range. The cart icon 803 is a touch icon that accepts instructions to transition to the cart screen. The cart icon 803 is displayed when there are items in the cart corresponding to the purchaser 120's user ID, and not displayed otherwise. In other words, it is not displayed at initial startup. Although not shown in the diagram, the QR code reading screen displays the LV image captured by the camera unit 112a, and the message 801, frame 802, and cart icon 803 are superimposed on the LV image.

[0031] In S302, the CPU 101a refers to the cart information stored in the non-volatile memory 103a and determines whether or not there are items in the cart. If it determines that there are items in the cart, it proceeds to S303; otherwise (the cart is empty, or the initial state where cart information has not been generated), it proceeds to S304.

[0032] In S304, CPU 101a determines whether or not there was an instruction to transition to the cart screen. If there was an instruction to transition to the cart screen, i.e., a touch operation was performed on the cart icon 803, the process proceeds to S330; otherwise, the process proceeds to S304.

[0033] In S304, the CPU 101a determines whether the QR code was captured and whether the information contained in the QR code was obtained. If the QR code 160 is within the capture range because the purchaser 120 framed the purchaser terminal 100a, and the QR code 160 is obtained from the captured image as an LV image, and the information contained in the QR code 160 is obtained by analysis, the process proceeds to S310. Otherwise, the process proceeds to S305.

[0034] In S305, CPU 101a determines whether or not a termination event has occurred (for example, closing the QR code reading screen). If no termination event has occurred, the process returns to S302 and is repeated. If a termination event has occurred, the process shown in Figure 3 is terminated.

[0035] In S310, CPU 101a determines whether the information in the QR code acquired in S304 includes the shelf ID. If the information in the QR code does not include the shelf ID, it means that the captured QR code is unrelated to the sales system of this embodiment. If the information includes the shelf ID, the process proceeds to S312; otherwise, the process proceeds to S311 to display an error or to perform a different process (a process based on the QR code) from this system.

[0036] In S312, CPU 101a determines whether an operation to execute the process indicated by the information in the QR code acquired in S304 has occurred. For example, if the scanned QR code contains shelf ID information, the display 105a displays the message "Do you want to check in to shelf number 67890?" along with the options "Yes" and "No". "Shelf number 67890" is just an example; any identification information that allows the user to identify which shelf is being checked in to is acceptable. If "Yes" is selected, it is determined that an operation to execute the process indicated by the information in the QR code has occurred, and the process proceeds to S313. If "No" is selected, the process returns to S302. Note that QR code 160 may also contain information to launch a purchase application or access a web application on the internet. In that case, QR code 160 is read with a general-purpose QR code reader, and if a decision instruction is received, the process from S313 onwards is performed according to the purchase application or web application. Alternatively, the process in S312 may be omitted, and if it is determined that the read QR code contains the shelf ID (S310Yes), the process in S313 may be performed without waiting for an operation from the purchaser 120 (an operation to give a decision instruction).

[0037] In S313, the CPU 101a sends check-in request information to the server device 100b, which includes the shelf ID obtained from the scanned QR code and a user ID to identify the purchaser 120. The shelf ID is display location information to identify the shelf 130 where the product is displayed. In this embodiment, the user ID is the login ID for the purchase application (either one that is stored in advance or one that the purchaser 120 entered when the purchase application was launched). The user ID is not limited to this; other information may be used as long as it can distinguish the purchaser 120 who is checking in to shelf 130 from other users. For example, the SSID of the purchaser terminal 100a, account information for other linked services such as SNS, or login account information for the purchaser terminal 100a may be used as the user ID. In other words, the user ID is user information to identify the purchaser terminal 100a or the purchaser 120. When the buyer terminal 100a sends the shelf ID and user ID to the server device 100b, the server device 100b attempts to check in, and then sends back information indicating whether the check-in was successful.

[0038] In this embodiment, the method for obtaining the shelf ID on the purchaser terminal 100a is described as taking a picture of a two-dimensional code, a QR code 160, but this is not the only method. The shelf ID may also be obtained by taking a picture of a one-dimensional code such as a barcode, or by taking a picture of the shelf ID displayed as a string and obtaining it through character recognition from the image. Alternatively, the shelf ID may be obtained by the user inputting the shelf ID displayed as a string on the shelf ID input screen shown on the display 105a of the purchaser terminal 100a. Or, the shelf ID may be obtained based on the location information of the purchaser terminal 100a obtained from a location information acquisition unit (not shown), such as a GPS positioning unit of the purchaser terminal 100a. The shelf ID may also be obtained by combining at least two of the following: information obtained from the captured image, information obtained through user input, and location information.

[0039] In S314, CPU 101a determines whether or not a check-in failure notification has been received from server device 100b indicating that check-in was not possible. For example, if a user other than buyer 120 has already checked into shelf 130, buyer 120 will not be able to check into shelf 130. If a check-in failure notification is received, proceed to S315; otherwise, proceed to 316.

[0040] In S315, the CPU 101a displays check-in error information on the display 105a indicating that check-in is currently unavailable. For example, it displays a message such as, "You cannot check in because someone else is using it. Please check in after the previous user has finished using it." In other words, it displays information indicating that another user is checking in, and / or information prompting the user to send a check-in request again after some time. If the shelf 130 had a door, it would be possible to determine whether another buyer is in the middle of a purchase by the open / closed state of the door (i.e., if another person has the door open, it is clear that that person has priority), but in this embodiment, there is no door. Therefore, it can be difficult to tell from the appearance of the shelf 130 whether there are other users. For example, even if another person is near the shelf 130, it may be impossible to determine whether that person has not decided whether to purchase, is there for a different reason, or is in the middle of a purchase. However, as in this embodiment, by displaying a check-in error, the buyer 120 can recognize that they should wait because another user is in the middle of a purchase. Furthermore, as in this embodiment, by preventing other users from checking in when one purchaser has already checked in, it is possible to prevent confusion of purchasers by making it impossible to determine which user took the product from shelf 130.

[0041] In S316, CPU 101a determines whether or not a check-in completion notification has been received from server device 100b indicating that the check-in was successful. If a check-in completion notification is received, proceed to S317; otherwise, return to 314.

[0042] In S317, CPU101a determines whether or not there are items in the cart, similar to S302. If there are items in the cart, proceed to S330; otherwise, proceed to S320.

[0043] In S320, CPU 101a displays an empty cart on display 105a. An example of the empty cart display is shown in Figure 8(b).

[0044] Display item 810 is the screen title portion indicating that the currently displayed screen is the cart display screen, which shows whether there are items in the cart. In this embodiment, it displays "Items in cart".

[0045] Guidance 811 is a guide display that prompts the user, the buyer 120, to take the product from the shelf 130. In the example in Figure 8(b), the string of characters (message) "Please take the product from the shelf" is displayed. In the sales system of this embodiment, the product shelf 130 does not have a door or cover. Therefore, it is difficult for the buyer 120 (user) to determine from the appearance whether or not it is OK to take the product from the shelf 130. Especially in an unmanned store, there is a possibility that the buyer may hesitate to take the product without permission. In contrast, by displaying guidance 811, the buyer 120 can recognize that it is OK to take the product from the shelf 130, and can take the product and shop with peace of mind. Furthermore, by displaying guidance 811 on the display 105a of the buyer terminal 100a, it is possible to notify the buyer that it is OK to take the product without relying on voice notification from the voice output unit 113c of the edge PC 100c. Therefore, even in environments where voice notification is difficult (environments that should be quiet, or conversely, environments with a lot of noise), it is possible to reliably notify the buyer 120 that it is OK to take the product. Furthermore, since there is no need to connect a display to the edge PC 100c or install a notification monitor near the shelf 130, installation costs and installation space increases are kept to a minimum, and the purchaser 120 can be reliably notified that the product is ready to be taken. In other words, the sales system of this embodiment becomes easier for users to use when purchasing products. Note that the guidance 811 is not limited to the illustrated example as long as it prompts the purchaser 120 to take the product from the shelf 130. For example, it could be a message with a different string of characters, an icon display, an animation display, video playback, or guidance via voice output from the purchaser terminal 100a.

[0046] Display item 812 shows the total price of the items in the cart. In the example in Figure 8(b), since there are no items in the cart, it displays a price of 0 yen.

[0047] The purchase instruction icon 813 is a touch icon (display item) used to instruct the user to purchase items in their cart and proceed with payment. In the example in Figure 8(b), there are no items in the cart and nothing can be purchased, so it is grayed out (inactive) and does not accept any input when touched. When items are added to the cart, the graying is removed, the icon becomes active, and it becomes possible to input via touch.

[0048] The Cancel Instruction icon 814 is a touch icon (display item) that accepts a command to cancel a purchase action (shopping).

[0049] Shelf Move Item 815 is a touch icon (display item) used to instruct the user to check into a different shelf than the one they are currently checking into.

[0050] In S321, CPU 101a determines whether or not a cancellation operation (touching the cancellation instruction icon 814) has occurred. If a cancellation instruction has occurred, the process proceeds to S322; otherwise, it proceeds to S325.

[0051] In S322, the CPU 101a sends a abort instruction to the server device 100b indicating that an abort operation has been performed.

[0052] In S323, CPU 101a determines whether or not it has received a checkout completion notification from server device 100b. If it has received a checkout completion notification, it proceeds to S324; otherwise, it waits in S323 for the checkout completion notification to be received.

[0053] In S324, the CPU 101a clears (erases) the cart information held in memory 102a and displays a checkout message on the display 105a. It also erases (clears) the cart information held in memory 102a. Figure 8(d) shows an example of the checkout display. The checkout display shows a message 830 such as "Thank you for using our service" to indicate that the purchase process has been completed. After the checkout display is shown for a predetermined time (for example, 5 seconds), the system returns to the LV display (QR code reading screen) in S301.

[0054] In S325, it is determined whether or not a shelf movement operation (touching shelf movement item 815) has occurred. If a shelf movement operation has occurred, proceed to S326; otherwise, proceed to S328.

[0055] In S326, the CPU 101a sends a shelf movement instruction to the server device 100b, indicating that a shelf movement operation has been performed.

[0056] In S327, CPU 101a determines whether or not it has received a checkout completion notification from server device 100b. If it has received a checkout completion notification, it returns to the LV display (QR code reading screen) in S301 to read the QR codes of other shelves. Otherwise, it waits in S323 for a checkout completion notification to be received.

[0057] In S328, the CPU 101a determines whether or not it has received a cart information update notification from the server device 100b. If an update notification is received, it stores the information indicating the latest cart status included in the received update notification in memory 102a (i.e., updates the cart information stored in memory 102a), and proceeds to S317. If no update notification is received, it proceeds to S321.

[0058] In this embodiment, an example was described in which touch operations on the purchase instruction icon 813 are not accepted (ignored) when an empty cart is displayed, but this is not the only example. For example, if a touch operation on the purchase instruction icon 813 is detected while an empty cart is displayed, a display prompting the user to take the product from the shelf again (a second display different from the display in S320) may be shown. For example, the guidance 811 may be highlighted in a different display format than S320, such as by flashing or changing the color, or a display prompting the user to take the product from the shelf may be shown with a display different from the guidance 811. In particular, it is good to display that it is OK to take the product from the shelf before payment. If the purchase instruction icon 813 is operated when there are no products in the cart, that is, when no products have been taken out, the buyer 120 may mistakenly believe that they cannot take the product from the shelf until payment is made. Therefore, by displaying a display prompting the user to take the product from the shelf again, the buyer's misunderstanding can be resolved and a smooth purchase operation can be performed.

[0059] In S330, CPU 101a displays the shopping cart information when there are items in the display unit. An example of the shopping cart display with items in the display unit is shown in Figure 8(c). The same reference numerals are used for the same display items as in Figure 8(b), and their explanations are omitted.

[0060] The product list 820 is information that includes product information contained in the update notification received from the server device 100b, and is a list of products in the cart based on the latest cart status information held in memory 102a. Each row of the product list 820 represents information for one product and includes columns 821 to 824. Column 821 is a display item that shows the appearance of the product in each row, and displays a photo of the product that was taken out, or an icon that represents the type of product that was taken out (for example, an icon resembling a plastic bottle for beverages, or an icon resembling food for food). When the type of product is represented by a photo, it displays an image that was previously stored in the non-volatile memory 103a of the buyer terminal 100a as data from the purchase application, or an image received from the server device 100b. Column 822 represents the name (product name) of the product in each row, and column 823 represents the price of the product in each row. Column 824 is a display item for accepting modification instructions for the product in each row. In the example in Figure 8(c), the purchase instruction icon 813 is displayed in a touch-enabled format (active display) because there is an item in the cart and it is available for purchase. On the other hand, the cancel instruction icon 814 is displayed in a grayed-out state. This is because, in this case, there is an item in the cart, and the purchase cannot be canceled unless the item in the cart is returned to shelf 130. However, in this embodiment, touch operation to the cancel instruction icon 814 is accepted even when it is grayed out. However, this is not the only option, and touch operation to the cancel instruction icon 814 may be disabled if there is an item in the cart.

[0061] In S331, the CPU 101a determines whether or not a correction operation has occurred. A correction operation is an operation in which a display item shown in column 824 of any row is touched, followed by completion of the operation on the correction content input screen (a screen for correcting the product in the touched row), and confirmation of the correction content. The correction content input screen is, for example, a screen that displays a list of products currently for sale on shelf 130 and accepts the selection of any product from the list. By accepting product correction operations in this way, if the product determined by product identification in image analysis happens to be different from the product actually picked up, the purchaser 120 can correct it to the correct content.

[0062] In S332, the CPU 101a sends information indicating the corrected product, which is the corrected information entered on the correction input screen, to the server device 100b.

[0063] In S333, CPU 101a waits for an update notification from server device 100b. When an update notification is received, it updates the cart information held in memory 102a based on the notification and proceeds to S317. As a result, the cart display on display 105a also reflects the correction.

[0064] In S334, CPU 101a determines whether or not a cancel operation (touching the cancel instruction icon 814) has occurred. If a cancel instruction has occurred, the process proceeds to S335; otherwise, it proceeds to S340.

[0065] In S335, the CPU 101a sends a abort instruction to the server device 100b indicating that an abort operation has been performed.

[0066] In S336, the CPU 101a displays a return message on the display 105a. The return message prompts the buyer 120 to return the item they took from shelf 130 to shelf 130, and displays a string of characters (message) such as "Please return the item to its original position." This allows the buyer 120 to realize that they cannot cancel their purchase unless they return the item they took to shelf 130, and they attempt to return the item they are holding to shelf 130. As a result, when the item in the cart (the item taken by the buyer 120) is returned to shelf 130, the camera 140 detects that the item has been returned from the video footage of the return, and information that the item has been returned is sent from the edge PC 100c to the server device 100b. Upon receiving the information that the item has been returned, the server device 100b updates the cart information it holds and sends an update notification to the buyer terminal 100a.

[0067] In S337, CPU 101a determines whether or not it has received an update notification from server device 100b. If an update notification is received, it updates the cart information in memory 102a with the latest cart information contained in the received update notification and proceeds to S338. If an update notification cannot be received, it waits for an update notification to be received in S337 and does not proceed to the checkout display.

[0068] In S338, CPU 101a determines whether there are items in the cart based on the latest cart information updated based on the update notification received in S337. If there are no items in the cart, it means that all the items that buyer 120 took from shelf 130 have been returned to shelf 130. If there are no items in the cart, proceed to S339; otherwise (if there are still items in the cart), proceed to S336.

[0069] Here, if no update notification is received in S337, or if it is determined in S338 that there are items in the cart, it means that the buyer 120 has not yet returned the items they took from shelf 130 to shelf 130. Therefore, if no update notification is received in S337 or if it is determined in S338 that there are items in the cart after a cancellation instruction is given in S334, the return message displayed in S336 will continue to be displayed to encourage the buyer 120 to return the items for sure. Furthermore, if no update notification is received in S337 or if it is determined in S338 that there are items in the cart after the return message has been displayed, the return message may be emphasized in stages. For example, the return message may be made to flash after a predetermined time has elapsed, or a warning sound may be emitted.

[0070] In S339, CPU 101a displays the checkout information, similar to S324. It also clears the cart information stored in memory 102a. This allows the buyer 120 to correctly recognize that they have returned the retrieved items to shelf 130, that the purchase has been successfully canceled, and that there is no problem in leaving without making a payment.

[0071] In S340, CPU 101a determines whether a purchase instruction operation (touching the purchase instruction icon 813) has occurred. A purchase instruction operation is an operation to make a payment and is a type of payment operation. If a purchase instruction operation has occurred, the process proceeds to S341; otherwise, it proceeds to S344.

[0072] In S341, the CPU 101a sends a purchase instruction notification to the server device 100b indicating that the purchase instruction operation has been performed.

[0073] In S342, the CPU 101a performs payment processing in conjunction with the server device 100b. For example, a payment screen for credit card payments or an electronic money payment screen is displayed, and payment (settlement, billing) is made for the items in the cart according to the user's actions on these payment screens (payment operations on the touch panel 70a). Once payment is complete, the process proceeds to S339. The checkout display in S339 allows the buyer 120 to recognize that payment has been successfully completed and that they may leave the area with the items they have taken from the shelf 130 (purchased items).

[0074] In S344, CPU 101a determines whether or not a shelf movement operation (touching shelf movement item 815) has occurred. If a shelf movement operation has occurred, the process proceeds to S326; otherwise, it proceeds to S345.

[0075] In S345, the CPU 101a determines whether or not it has received a cart information update notification from the server device 100b. If an update notification is received, it stores the information indicating the latest cart status included in the received update notification in memory 102a (i.e., updates the cart information stored in memory 102a), and proceeds to S317. If no update notification is received, it proceeds to S331.

[0076] Figure 4 shows a flowchart of the processing related to the sales system in the server device 100b. This processing is achieved by loading the program recorded in the non-volatile memory 103a into memory 102a and executing it on the CPU 101a. This processing is started manually at the timing when sales by the sales system in this embodiment begin (such as the start time of business hours), or it is started automatically when a predetermined time (such as the time of disclosure of business hours) arrives.

[0077] In S401, CPU 101b initializes various variables and information (cart information) held in memory 102b. Non-volatile memory 103b stores the IDs of multiple product shelves managed by server device 100b. All IDs of these managed product shelves are read, check-in status management information is created in memory 102b, and then the status of all shelves is set to "not checked in." An example of check-in status management information is shown in Figure 5(a). The management information stores the ID of the managed shelf, its check-in status, and, if checked in, the ID of the user who checked in (CI). In S401, the check-in status is set to "not checked in" for all shelves, and there are no CI users for all shelves.

[0078] In S402, CPU 101b determines whether or not it has received the shelf ID transmitted from the buyer terminal 100a. This is the same determination made in S313 of Figure 3 above, regarding whether or not it has received the shelf ID and user ID transmitted from the buyer terminal 100a. If the shelf ID has been received, the process proceeds to S402; otherwise, it proceeds to S407.

[0079] In S403, the system refers to the check-in status management information stored in memory 102b and determines whether the shelf with the received shelf ID is already being checked in (is in a checked-in state). If it is in a checked-in state, the system proceeds to S404; otherwise (it is not in a checked-in state), the system proceeds to S405.

[0080] In S404, a check-in failure notification is sent to the purchaser terminal 100a, the source of the shelf ID received in S402, indicating that check-in is not possible. The check-in status management information held in memory 102b is not updated. In other words, in this case, no check-in process is performed for the received shelf ID. The check-in failure notification sent in S404 is received in S314 in Figure 3, and a check-in error is displayed in S315.

[0081] In S405, CPU 101b performs the check-in process. Specifically, in the management information of the check-in status held in memory 102b, for the shelf with the shelf ID received in S402, the check-in status is set to "checked in," and the user ID received along with the shelf ID is associated with the CI user and stored. In other words, the check-in process is an association process that associates the shelf ID (display location information) with the user ID (user information). An example of the management information in this case is shown in Figure 5(a). For the shelf with shelf ID "678 user ID 90," the check-in status is set to "checked in," and the CI user ID "12345" is associated with it and stored.

[0082] Furthermore, in S404, if the cart information of the received CI user has not been generated, the cart information of the CI user is generated in memory 102b. Figure 5(b) shows an example of cart information for user ID "12345". Cart information is generated for each user ID. The cart information stores the user ID, the shelf ID of the item checked in, the product name (or product identification information), price, quantity, subtotal, the shelf ID from which the item was taken, and the total price of all items in the cart, all associated with each other. The example in Figure 5(a) shows the state immediately after a user checks in to shelf ID "67890", and there are no items in the cart.

[0083] Note that if the cart information for the corresponding user ID is already stored in memory 102b among the shelf ID and user ID received in S402, new cart information will not be generated. In this case, the shelf ID currently checked in in the existing cart information for the corresponding user ID will be updated with the received shelf ID. This case is when there is a shelf move instruction from the first shelf, and after checking out from the first shelf without payment, the user checks in to another shelf (the second shelf). After checking in to the second shelf, if an item is taken from the second shelf, the taken item will be added to the cart. Figure 5(c) shows the cart information when items A and B are taken from shelf ID "67890" (the first shelf), a shelf move instruction is given, and the user checks in to shelf ID "67891" (the second shelf) and item C is taken. In addition to items A and B taken from the first shelf, item C taken from the second shelf is also recorded, and the total price is the sum of items A, B, and C.

[0084] In S406, the CPU 101b sends a check-in completion notification to both the purchaser terminal 100a and the edge PC 100c, indicating that check-in has been completed. The check-in completion notification sent here is received by the purchaser terminal 100a at S316 in Figure 3 above, and by the edge PC 100c at S603 in Figure 6, which will be described later.

[0085] In S407, CPU101b determines whether or not it has received the shelf ID and product recognition score transmitted from edge PC100c. If it has received them, it proceeds to S408; otherwise, it proceeds to S411.

[0086] In S408, the CPU 101b determines what product was picked up / returned from the shelf based on the product recognition score received from the edge PC 100c. In this embodiment, the final product determination is performed on the server device 100b rather than the edge PC 100c. This has the effect of allowing product determination to be performed simply by reflecting information on the server device 100b without using advanced or cutting-edge determination methods such as machine learning or reflecting new product information on the edge PC. In other words, it is possible to reduce the increased installation costs associated with the advanced functionality of the edge PC 100c and the management burden such as updating the edge PC 100c. The product recognition score received from the edge PC may be information that has been estimated to some extent (inference score), or it may be information such as shape, color, and size parameters before the product is estimated. By receiving a product recognition score that has been parameterized to some extent, rather than the image itself captured by the camera 140, the amount of communication between the edge PC 100c and the server device 100b is suppressed, enabling high-speed determination and response.

[0087] It should be noted that this is not limited to sending a product recognition score from the edge PC 100c to the server device 100b. For example, the edge PC 100c may not perform any processing to determine what the product is, but instead send the image itself (video or still image) captured by the camera 140 in the scene where it is presumed that a product was taken out or returned. In this case, the server device 100b will determine the product based on the image captured by the received camera 140. Conversely, the edge PC 100c may determine that a product was taken out or returned and what the product in question is, and send only the determination result to the server device 100b. In this case, processing S408 is omitted.

[0088] In S409, CPU 101b updates the cart information held in memory 102b based on the product determined in S408. That is, if S408 determines that the product has been removed from the shelf, the information of the product determined in S408 is added to the cart information of the user ID associated with the received shelf ID. Conversely, if S408 determines that the product has been returned to the shelf, the product determined in S408 is deleted from the cart information of the user currently checking in, specifically from the cart information where the received shelf ID is recorded as the source of removal. Figure 5(b) shows an example of cart information when, starting from a state where product A is originally in the cart information, S407 receives the shelf ID "67890" and product recognition score, and S408 determines that product B has been removed from the shelf. Information of product B has been added in addition to product A.

[0089] In S410, the CPU 101b sends an update notification to the buyer terminal 100a indicating that the cart information has been updated. The update notification also includes the latest cart information stored in memory 102b. That is, if an item is taken from the product shelf 130, the cart information including the information of the retrieved item, as determined in S408, is also sent.

[0090] In S411, the CPU 101b determines whether or not it has received a correction instruction (the correction content sent from the purchaser terminal 100a in S332 of Figure 3 above). If it determines that a correction instruction has been received, it proceeds to S409 and updates the cart information held in memory 102b based on the correction content received as a correction instruction. If no correction instruction has been received, it proceeds to S412.

[0091] In S412, CPU 101b determines whether or not it has received a cancellation instruction sent from the purchaser terminal 100a. If a cancellation instruction has been received, it proceeds to S413; otherwise, it proceeds to S417.

[0092] In S413, CPU101b determines whether or not there are items in the shopping cart information of the user ID of the user who issued the cancellation instruction in S412. If there are items, proceed to S414; otherwise, proceed to S419.

[0093] The processing in S414-S416 is the same as the processing in S408-S410. As a result, when the product is removed from the cart of the user who issued the cancellation order, S413 will be determined as No, and the cancellation process with checkout (processing in S419-S422) will be performed. As long as there are still products in the cart of the user who issued the cancellation order, S413 will not be determined as No, and therefore the cancellation process with checkout will not be performed.

[0094] In S417, the CPU 101b determines whether or not it has received the payment instruction sent from the purchaser terminal 100a (the payment instruction sent from the purchaser terminal 100a in S341 of Figure 3 above). If it determines that the payment instruction has been received, it proceeds to S418; otherwise, it proceeds to S423.

[0095] In S418, CPU 101b calls payment service 170 and performs payment processing based on payment service 170.

[0096] In S419, CPU 101b clears (deletes) the cart information held in memory 102b for the user ID for which a cancellation instruction was received in S412 or for the user ID for which a payment instruction was received in S417.

[0097] In S420, CPU101b performs the checkout process. Specifically, CPU101b updates the status of the product shelves checked in by the user ID that received the cancellation instruction in S412 or the user ID that received the payment instruction in S417 to the checkout state. That is, for the corresponding shelf ID in the check-in status management information (Figure 5(a)), the status is set to checkout state and the CI user is deleted. This deletes the association between the shelf and the user (purchaser).

[0098] In S421, CPU101b sends a checkout completion notification to the buyer terminal 100a and edge PC 100c indicating that checkout has been completed.

[0099] In S422, the CPU 101b determines whether or not it has received a shelf relocation instruction sent from the purchaser terminal 100a (the shelf relocation instruction sent from the purchaser terminal 100a in S328 of Figure 3 above). If it determines that it has received a shelf relocation instruction, it proceeds to S423; otherwise, it proceeds to S424.

[0100] In S423, CPU101b performs the checkout process, similar to S420 mentioned above.

[0101] In S424, CPU101b determines whether or not there has been a termination event for the process shown in Figure 4. If there is a termination event such as the end of business hours, a termination instruction from the user, or power off, the process in Figure 4 is terminated. If there is no termination event, the process proceeds to S402 and is repeated.

[0102] Figure 6 shows a flowchart of the processing related to the sales system on the edge PC 100c. This processing is achieved by loading the program recorded in the non-volatile memory 103c into memory 102c, and then executing it on the CPU 101c. This processing is started manually at the timing when sales by the sales system in this embodiment begin (such as the start time of business hours), or it is started automatically when a predetermined time (such as the time of disclosure of business hours) arrives. Figure 6 shows the processing related to an individual shelf, and in this embodiment, it will be explained as the processing related to shelf 130 (shelf ID: 67890).

[0103] In S601, the CPU 101c stores the state information stored in memory 102c or non-volatile memory 103c as the state of shelf 130, setting it to information corresponding to non-check-in. That is, it sets it to a non-check-in state.

[0104] In S602, CPU101c acquires video from camera140. The video acquired here is assumed to be a video showing the product dispensing area of ​​shelf130. If a sensor other than camera140 is used to detect product dispensing / returning, the process will be to obtain the output value from that sensor.

[0105] In S603, the CPU 101c determines whether or not it has received a check-in completion notification sent from the server device 100b. If it has received the check-in completion notification, it proceeds to S604; otherwise, it proceeds to S606.

[0106] In the S604, the CPU 101c stores and manages the state information of shelf 130, which is stored in memory 102c or non-volatile memory 103c, as information corresponding to the check-in. That is, it sets it to the check-in state (a state in which shelf 130 is associated with a prospective buyer).

[0107] In S605, the CPU 101c outputs a voice message from the voice output unit 113c indicating that it is now OK to take the product from shelf 130. For example, it may play a sound effect such as "ping" or a voice message such as "Welcome" to notify the user that they have been able to check in to shelf 130 (shelf 130 has responded) after scanning the QR code 160 with the purchaser terminal 100a. Alternatively, it may play a voice message (voice guidance) prompting the purchaser 120 to take the product from the shelf, such as "Please take the product from the shelf" or "After taking the product from the shelf, please make the payment with your terminal," or a voice message explaining how to shop. In other words, it notifies the user that their status has changed from a non-checked-in state to a checked-in state, and / or prompts them to take the product from shelf 130.

[0108] In S606, the CPU 101c determines, based on the video footage acquired from the camera 140, whether an item has been taken from the shelf 130 or returned to the shelf 130. That is, it detects whether an item has been taken from its display location based on the image of the shelf 130 captured by the camera 140. For example, based on the image, it detects that an item has been taken if a hand enters from a reference position in front of the shelf 130, the subject of the area where the item was placed changes, and the hand leaves the reference position. If an item has been taken out or returned, the process proceeds to S607; otherwise, it proceeds to S610. In this embodiment, an example of detecting the removal / return of products based on images captured by camera 140 has been described, but the detection method is not limited to this. For example, a weight sensor that detects the weight of items displayed on a shelf, such as a shelf board of shelf 130, may be attached to the display location, and the removal / return of products may be detected by the change in weight based on the output from the weight sensor. Alternatively, an RFID (radio frequency identifier) ​​tag, such as an IC tag, may be attached to each product, and the removal / return of products may be detected by wireless communication with this tag. Furthermore, an infrared sensor may be provided to detect when a product moves beyond the reference line in front of shelf 130 (the dispensing opening), when a product disappears from its previous location, or when a product is placed in a location where there was no product before. The removal / return of products may be detected based on the output from the infrared sensor indicating such detection. One of the multiple detection methods may be adopted, or multiple methods may be combined to detect the removal / return of products.

[0109] In S607, the CPU 101c refers to the state information stored in memory 102c or non-volatile memory 103c as the state of shelf 130 and determines whether it is in a checked-in state or not. If it is in a checked-in state, proceed to S608; otherwise, proceed to S609.

[0110] In S608, the CPU 101c transmits to the server device 100b the shelf ID of the managed shelf 130 (67890 in this embodiment), the fact that an item was taken out or returned in S606, and the product recognition score of the item for which the take-out or return phenomenon was detected. In other words, it transmits the detection details regarding the detection of an item being taken out or returned. The product recognition score is information output as a result of inference processing, for example, by inputting an image captured by the camera 140 into a trained model trained by machine learning. The product recognition score can also be said to be the result of scoring candidate products by inferring the product from its appearance and shape based on the image captured by the camera 140.

[0111] In S609, the CPU 101c issues an error notification (error output) via voice from the voice output unit 113c. In this case, the edge PC 100c does not send the product score to the server device 100b. By issuing an error notification without communicating with the server device 100b, the error notification can be issued more quickly, allowing the buyer 120 to recognize the action as an error sooner. In this case, the product score may also be sent to the server device 100b to manage what product was taken out without checking it in, but the timing of the transmission is after the error notification. That is, at the time of outputting the error notification, the error notification is output without sending the detection result to the server device 100b. Furthermore, the trigger for outputting the error notification may not be the detection of a product being taken out as described in S606, but rather, for example, the detection of a hand entering the product shelf 130. By outputting an error notification with this trigger, the buyer 120 can be recognized as an errorful action at an earlier stage, potentially preventing the product from being taken out. This phenomenon is one in which an item may be removed, and it is a preparatory action for removing an item. Thus, by detecting the preparatory action for removing an item as a phenomenon in which an item is removed, the same processing as when an item has been removed is detected. In this case, the object of detection is the phenomenon of an item being removed, and upon detection of this phenomenon, it is processed as if an item has been removed. The same applies to returns; not only the phenomenon of an item being returned, but also the preparatory action for the return of an item as a phenomenon in which an item is returned is detected, and the same processing as when an item has been returned is detected. In this case, the object of detection is the phenomenon of an item being returned, and upon detection of this phenomenon, it is processed as if an item has been returned.

[0112] The error notification is automatically canceled when the item taken without being checked in is returned to shelf 130, or when a predetermined time (e.g., 1 minute) has elapsed. Even if the taken item is not returned, the error notification is output for a predetermined time and then automatically canceled. This prevents unauthorized operation from causing the error notification to persist for an extended period on an unattended shelf 130, thus preventing other customers from missing out on purchasing opportunities.

[0113] Error notifications are, for example, voice messages or sound effects that prompt action to check in, such as "Please check in before taking the product" or "Please scan the QR code displayed on the shelf with your device before taking the product." Alternatively, messages such as "Please return the product to the shelf and then scan the QR code on the shelf" may be used to prompt the user to return the product to its original position. Error notifications are output as sounds that can be heard by the person who has taken or is attempting to take a product from the product shelf 130. This error notification allows the purchaser 120 (prospective purchaser) to realize if they have taken a product when it is not in a condition to be taken and to return it. In other words, the sales system of this embodiment can be used appropriately.

[0114] In S610, the CPU 101c determines whether or not it has received a checkout completion notification sent from the server device 100b. If a checkout completion notification has been received, the process proceeds to S611; otherwise, it proceeds to S613.

[0115] In the S611, the CPU 101c stores and manages the status information of shelf 130 in memory 102c or non-volatile memory 103c, setting it to information corresponding to non-check-in. That is, it checks out and sets it to non-check-in (a state where shelf 130 is not associated with a prospective buyer).

[0116] In the S612, the CPU 101c outputs audio from the audio output unit 113c in conjunction with checkout from the shelf 130. For example, it may play a sound effect such as "Piro-rin" or an audio message such as "Thank you for using our service."

[0117] In S613, CPU101c determines whether or not there has been a termination event for the process shown in Figure 6. If there is a termination event such as the end of business hours, a termination instruction from the user, or power off, the process in Figure 6 is terminated. If there is no termination event, the process proceeds to S02 and is repeated.

[0118] Figure 7 shows an example of a sales processing sequence in the sales system of this embodiment, which is realized through the coordination of the processing of the buyer terminal 100a described in Figure 3, the processing of the server device 100b described in Figure 4, and the processing of the edge PC 100c described in Figure 6. The corresponding steps of the processing described above in Figures 3, 4, and 6 are shown in parentheses.

[0119] In the S700, the buyer terminal 100a opens the QR code reading screen, and with the LV image displayed, the buyer 120 points the camera unit 112a of the buyer terminal 100a towards the QR code 160.

[0120] In S701, the camera unit 112a of the purchaser terminal 100a captures and reads the QR code 160 (S304 Yes).

[0121] In S702, the shelf ID and user ID are transmitted from the purchaser terminal 100a to the server device 100b (S313).

[0122] In S703, the check-in process is performed on server device 100b (S405).

[0123] In S704, a check-in completion notification is sent from the server device 100b to the edge PC 100c (S406). In response, in S705, the edge PC performs a check-in (S604) and outputs the audio guide (S605).

[0124] In S706, a check-in completion notification is sent from the server device 100b to the buyer terminal 100a (S406), and in response, in S707, an empty cart is displayed on the buyer terminal 100a (S320).

[0125] In S708, customer 120 takes an item from shelf 130.

[0126] In S709, the edge PC 100c detects that an item has been removed (S606Yes) and sends the shelf ID and item recognition score to the server device 100b (S608).

[0127] In S710, the server device 100b performs product detection (S408) and updates the products in the shopping cart (S409).

[0128] In S711, the server device 100b sends a notification of updated shopping cart information to the buyer terminal 100a (S410).

[0129] In S712, the shopping cart display on the buyer terminal 100a is updated. In this example, the shopping cart display will now show the item that was removed in S708 as part of the cart display (S330).

[0130] In S713, the purchaser 120 performs a cancellation operation by touching the cancellation instruction icon 814.

[0131] In S714, in response to the cancellation operation, the purchaser terminal 100a sends a cancellation instruction to the server device 100b (S335).

[0132] In S715, since there is an item in the cart, a return message is displayed on the buyer terminal 100a (S336).

[0133] In S716, the buyer 120, upon seeing the return message, returns the item they took out in S708 to shelf 130.

[0134] In S717, the edge PC 100c detects that an item has been returned based on the image captured by the camera 140 (S606), and transmits the shelf ID and the returned item recognition score to the server device 100b (S608).

[0135] In S718, the server device 100b determines (identifies) the returned product based on the received shelf ID and product recognition score (S415), and updates the products in the cart (S416). In other words, it deletes the returned product from the products that were in the cart.

[0136] In S719, a notification of updated cart information is sent from the server device 100b to the buyer terminal 100a (S416).

[0137] In S720, since the item retrieved in S708 has been returned, the cart is now empty (S413No), so the checkout process is performed in response to the cancellation instruction sent in S714 (S420).

[0138] In S721, the server device 100b sends a checkout completion notification to the edge PC 100c (S421). In S722, the edge PC 100c changes the status of shelf 130 to unchecked in in response to the checkout completion notification (S611).

[0139] In S722, the server device 100b sends a checkout completion notification to the edge PC 100c (S421). In S724, in response to the checkout completion notification, the checkout screen is displayed on the purchaser terminal 100a (S339), and then the LV is displayed on the QR code reading screen.

[0140] In another case, if there are items in the cart and the buyer 120 touches the purchase instruction icon 813 on S730 to perform a purchase instruction operation, S731 sends a payment instruction notification from the buyer terminal 100a to the server device 100b.

[0141] In S732, the server device 100b processes the payment using the payment service 170 in response to the payment instruction notification. After payment is made, the items in the cart are cleared, and the checkout process is performed in S733.

[0142] In S734, a checkout completion notification is sent from the server device 100b to the edge PC 100c, and in S735, Edge PC100c will be marked as not checked in when shelf 130 is in a non-checked-in state.

[0143] In S736, the server device 100b sends a checkout completion notification to the buyer terminal 100a. In SS724, in response to the checkout completion notification, the buyer terminal 100a displays the checkout screen and then displays the LV on the QR code reading screen (S301).

[0144] According to the embodiment described above, a more suitable system for managing and selling products can be provided.

[0145] Furthermore, as is clear from the sequence diagram in Figure 7, the sales system of this embodiment does not require direct communication between the purchaser terminal 100a and the edge PC 100c. In other words, in this embodiment, the purchaser terminal 100a and the edge PC 100c do not establish direct communication. Therefore, even devices that do not have the function for direct communication between the purchaser terminal 100a and the edge PC 100c can be used as the purchaser terminal 100a. Functions for direct communication between the purchaser terminal 100a and the edge PC 100c include, for example, near-field communication functions such as NFC (Near Field Communication) and Bluetooth (Trademark). However, some inexpensive electronic devices such as smartphones and mobile phone terminals have a camera but do not have NFC or Bluetooth functionality. In the sales system of this embodiment, even electronic devices that do not have NFC or Bluetooth functionality can be used as the purchaser terminal 100a to make purchases. In other words, it is possible to prevent hindering the purchasing opportunities of users of electronic devices without near-field communication functionality and to prevent limiting purchases to only users who own electronic devices with near-field communication functionality. In other words, it becomes possible for a wider range of users to purchase the product, contributing to an increase in sales opportunities. That is, it becomes possible to provide a sales system that is accessible to more users.

[0146] Furthermore, since the purchaser terminal 100a and the edge PC 100c do not need to communicate directly, there is no need to install equipment such as short-range wireless communication transceivers connected to the edge PC 100c near the product shelf 130. Therefore, an increase in installation space and installation costs can be kept to a minimum.

[0147] Furthermore, when attempting to communicate between the buyer terminal 100a and the edge PC 100c, one possible communication method would be to connect via NFC and then hand over to Bluetooth. In this case, NFC has a short effective communication range of about 10 cm, and there is a constraint on the installation location, as the NFC transceiver must be placed within the user's reach. Also, when using Bluetooth, there is a disadvantage in that if the buyer terminal 100a is connected to another device via Bluetooth, the original connection may have to be disconnected. For example, it may not be possible to make a purchase while listening to music with Bluetooth earphones. With the sales system of this embodiment, there is no need for the buyer terminal 100a and the edge PC 100c to communicate directly, so these problems are also solved.

[0148] Furthermore, in this embodiment of the sales system, when a buyer 120 purchases a product placed on the shelf 130, there is no need for the buyer 120 to perform any operations or inputs to the edge PC 100c. Therefore, there is no need to install a device on the shelf 130 to receive operations or inputs, thus reducing the increase in sales space and installation costs. In addition, since payment is also performed by processing between the buyer terminal 100a and the server device 100b, there is no need to install devices for handling cash or payment devices such as credit card readers, thus reducing installation space and costs.

[0149] In the above embodiment, a product shelf 130 was given as an example of a place to display products, but it is not limited to this, and the display location may be any place other than a shelf from which a purchaser can take out the products.

[0150] Furthermore, the above-described embodiment is also applicable to cases such as replenishing inventory instead of selling products. In other words, this embodiment is applicable to product management systems, including sales systems and inventory management systems. In the case of inventory management, it can be applied by replacing the above-described buyer 120 with an inventory replenisher, replacing the detection of returning a product with the detection of replenishing inventory, and replacing the detection of removing a product with the detection of removing a product in order to remove it from display. That is, the inventory replenisher checks in to the shelf 130 by reading the QR code 160 with a terminal owned by the inventory replenisher. Inventory replenishment and replacement of displayed products can be performed while checked in, and operations performed while not checked in may be treated as errors and various notifications as described above may be issued.

[0151] Furthermore, the various controls described above, which are performed by CPUs 101a, 101b, and 101c, may each be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0152] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is also possible to combine each embodiment as appropriate.

[0153] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having one or more computers (or CPUs, MPUs, etc.) in that system or device read and execute the program code. In this case, the program and the storage media storing the program constitute the present invention. [Explanation of Symbols]

[0154] 100a: Purchaser terminal 100b: Server device 100c: Edge PC 130: Shelf

Claims

1. An acquisition means for acquiring user identification information; a display control means for controlling an electronic device to display information about the product in association with a user related to the identification information in response to the product being removed from the display location; An information processing system comprising:

2. The information processing system as described in Claim 1, characterized in that it is further characterized by being provided with an output means for controlling the output of error information when a product is removed from the display location without the acquisition means having acquired the user's identification information.

3. The information processing system described in Claim 2, characterized in that the output means cancels the output of the error information when the product is returned to its display location or a predetermined time has elapsed.

4. The information processing system described in Claim 2 or 3, characterized in that the output means outputs, as the error information, information that prompts the user to take action to obtain identification information or information that prompts the user to return the product to its display location.

5. An information processing system as described in any one of claims 1 to 4, characterized in that when the acquisition means has acquired a user's identification information, it restricts the acquisition of identification information of a user other than the user.

6. When the product is returned to its display location, 6. The information processing system according to claim 1, wherein the display control means controls the electronic device to display, for products returned to the display area, a quantity reduced by the number of products returned.

7. An information processing system as described in any one of claims 1 to 6, characterized in that it is provided with a notification means for notifying an electronic device of information regarding a product that has been removed from or returned to the display area.

8. A management means for managing display locations in association with user identification information; a notification means for notifying a user associated with identification information associated with a display location of information related to the product when the product is removed from the display location; An information processing system comprising:

9. The information processing system described in Claim 8, characterized in that the management means manages display location information for identifying a display location received from an electronic device and identification information for identifying the electronic device or a user of the electronic device in association with each other.

10. The information processing system described in Claim 9, characterized in that the display location information is information obtained by imaging a one-dimensional code or a two-dimensional code with an imaging means provided in the electronic device.

11. The notification means: If the display location is in a first state in which the identification information of the user is associated with the display location, a notification is issued to prompt the user to remove the product from the display location; An information processing system as described in any one of claims 8 to 10, characterized in that when a product is removed from the display location in a second state in which the display location is not associated with the user's identification information, error information is notified.

12. The information processing system according to claim 11, characterized in that the notification means notifies information regarding the product when the product is removed from the display location in the first state.

13. An information processing system as described in any one of claims 1 to 12, characterized in that the information regarding the product includes at least one of the name of the product taken out from the display area, a photo of the product, an icon indicating the type of product, the price of the product, and the number of products taken out.

14. An information processing system as described in any one of claims 1 to 13, further comprising a detection means for detecting the phenomenon of a product being removed from or returned to the display area.

15. The information processing system according to claim 14, characterized in that the detection means performs detection based on at least one of an image taken of the display location, a change in weight of the display location, output from an infrared sensor related to the display location, and a result of communication with an IC tag attached to the product.

16. An acquisition step of acquiring user identification information; a display control step of controlling an electronic device to display information about the product in association with the user related to the identification information in response to the product being removed from the display location; A method for controlling an information processing system comprising:

17. A management step of associating and managing a display location with user identification information; a notification step of notifying a user associated with identification information associated with the display location of information related to the product when the product is removed from the display location; A method for controlling an information processing system comprising:

18. A program for causing at least one computer to function as each of the means of an information processing system described in any one of claims 1 to 15.

Images