Information processing device, control method for information processing device, and program

Abstract

To provide a more suitable system related to management and sales of commodities.SOLUTION: An information processing apparatus has: detection means that detects a phenomenon in which a commodity is taken out from a display place where commodities are displayed; and control means that, when the detection means detects the phenomenon in which the commodity is taken out from the display place, in the case of a first state in which the display place and a purchase planning person or an electronic apparatus operated by the purchase planning person are associated with each other, controls not to output predetermined notification, and in the case of a second state in which the display place and the purchase planning person or the electronic apparatus owned by the purchase planning person are not associated with each other, to output the predetermined notification.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a system that allows for the management and sale of merchandise with no staff or a small number of staff. [Background technology]

[0002] Conventionally, sales systems that reduce labor costs required for selling products, such as vending machines and unmanned stores, have been proposed.

[0003] Patent Document 1 proposes a sales system (product management device) in which multiple products, each with an RFID tag attached, are placed on shelves equipped with weight sensors. When a change in the weight of a product shelf is detected, products that were placed on the shelf where the weight change occurred and whose RFID tag reception strength has changed are identified as products that have physically moved (products that may have been picked up by the purchaser). When paying, an RFID reader reads the RFID tags attached to the products in the shopping cart to calculate the total amount and display it on the display unit of the product management device. The purchaser then inserts money into the payment amount reader to complete the payment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-160042 Summary of the Invention [Problem to be solved by the invention]

[0005] The system described in Patent Document 1 is an unmanned store that, for example, attaches an RFID tag to each product, installs a display that shows the total price, and installs a device for payment. However, in an unmanned or reduced-staff sales system, there is little or no guidance from a staff member, so there is room for improvement in terms of helping customers shop appropriately.

[0006] In view of at least one of the above problems, an object of the present invention is to provide a more suitable system for managing and selling products. [Means for solving the problem]

[0007] In order to solve the above problems, the information processing device of the present invention comprises: a detection means for detecting a phenomenon in which a product is removed from a display location where the product is displayed; When the detection means detects that a product is being removed from the display area, If the display location is in a first state in which the prospective purchaser or the electronic device operated by the prospective purchaser is associated with the display location, a predetermined notification is not output; a control means for controlling to output the predetermined notification when the display location is in a second state in which the display location is not associated with the prospective purchaser or the electronic device owned by the prospective purchaser; The present invention is characterized by having the following. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a more suitable system for managing and selling products. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a system configuration diagram of a sales system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a hardware configuration diagram of each device included in the sales system of the present embodiment. [Figure 3] 10 is a flowchart of a process performed by a purchaser terminal 100a. [Figure 4] 10 is a flowchart of a process performed by the server device 100b. [Figure 5] FIG. 10 is a diagram illustrating information managed by a server device 100b. [Figure 6] 10 is a flowchart of a process performed by the edge PC 100c. [Figure 7]FIG. 10 is a sequence diagram of an example of processing in the sales system of this embodiment. [Figure 8] 10 is an example of a screen display on a purchaser terminal 100a. DETAILED DESCRIPTION OF THE INVENTION

[0010] The system described in Patent Document 1 required, for example, attaching an RFID tag to each product, installing a display unit showing the total price, and installing a payment device, which increased the cost and installation space required to install shelves. In other words, there was room for improvement in the implementation cost and / or installation space of the sales system. While it is possible to use user terminals owned by each purchaser to reduce costs and space, in that case, it is preferable that the functions used on the user terminals be more versatile so that more users can use the system. Furthermore, in unmanned or minimally staffed sales systems, there is little or no staff guidance, so there is room for improvement in terms of enabling purchasers to shop appropriately, simply, or casually.

[0011] In view of at least one of the above problems, the present embodiment aims to provide a technology for providing a more suitable system for managing and selling products.

[0012] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0013] FIG. 1 shows an example of the system configuration of an information processing system (sales system) according to this embodiment. The sales system according to this embodiment is an unmanned sales system that allows sales without the need for a salesperson. The sales system according to this embodiment includes a purchaser terminal 100a (electronic device), a server device 100b (information processing device), and an edge PC 100c (terminal device serving as an information processing device). The purchaser terminal 100a can communicate with the server device 100b via the Internet or directly. The edge PC 100c can also communicate with the server device 100b via the Internet or directly. However, direct communication between the purchaser terminal 100a and the edge PC 100c is not required.

[0014] One or more (at least one) products, including product 131, are placed (displayed) on shelf 130, which is a product shelf serving as an example of a product display location (including a location where products are placed). A QR code (trademark) 160 containing identification information (shelf ID) for shelf 130 is displayed on or near shelf 130 itself. In this embodiment, shelf 130 is not provided with a partition member, such as a door, lid, or cover, separating it from a purchaser, allowing purchasers to directly pick up products by hand without having to open a door. This lowers the physical and psychological barriers for purchasers to pick up products, allowing purchasers to make purchases more easily and leading to increased sales for sellers. The present embodiment, described below, is applicable even when a door, lid, cover, or the like is provided. The present embodiment, described below, is also applicable even when a store clerk is present for sales promotion or the like.

[0015] A camera 140 is installed as a device included in the sales system of this embodiment, and serves as a sensor that acquires information for detecting when an item displayed on the shelf 130 has been removed or returned. The camera 140 is installed so that its imaging range allows it to capture images of items being taken in and out of the shelf 130. The edge PC 100c is associated with the shelf 130, and is connected to the camera 140 by wire or wireless communication so that images captured by the camera 140 can be acquired. In this embodiment, an example will be described in which the edge PC and the product shelf are processed in a one-to-one association, but multiple product shelves may also be processed in association with one edge PC.

[0016] The edge PC 100c analyzes the captured image (video) transmitted from the camera 140 and detects whether a product has been taken out or returned from the shelf 130. The edge PC 100c then transmits information that the product has been taken out or returned to the server device 100b.

[0017] The purchaser 120 acquires the shelf ID by photographing the QR code 160 with the purchaser terminal 100a, such as a smartphone, owned by the purchaser 120. The purchaser 120 then transmits the shelf ID and the identification information (user ID) of the purchaser 120 to the server device 100b.

[0018] Based on the shelf ID and user ID received from the purchaser terminal 100a, the server device 100b associates the user ID of the purchaser 120 with the shelf ID of the shelf 130 (i.e., associates the purchaser 120 with the shelf 130). Hereinafter, the process of associating (storing in association) the user ID with the shelf ID will be referred to as the check-in process, and the associated state will be referred to as the check-in state (checked-in state). When the check-in process is complete, the server device 100b notifies the purchaser terminal 100a and the edge PC 100b that check-in is complete. In this state, the purchaser 120 is ready to take out the product placed on the shelf 130. When the purchaser 120 picks up and takes out the product placed on the shelf 130, the edge PC 100c detects this based on the image captured by the camera 140 and transmits the detection result to the server device 100b. The server identifies the product that has been picked up and adds it to the virtual cart (shopping basket, hereinafter simply referred to as the cart) of the purchaser 120 associated with the shelf 130. The status of the cart is sent from the server device 100b to the purchaser terminal 100a and displayed on the purchaser terminal 100a, so the purchaser 120 can confirm that the product that he or she picked up is in the cart. Then, when the user operates the purchaser terminal 100a to perform payment processing, the server device 100b calls a payment service 170 on the Internet to perform payment. In this way, the purchaser 120 can purchase the product through the easy and simple process of reading the QR code 160 with the purchaser terminal 100a, picking up the product, and performing payment.

[0019] 2(a) shows an example of the configuration of a purchaser terminal 100a as an example of a device to which the present invention can be applied. The purchaser terminal 100a is an electronic device held by a user who is a purchaser, and can be configured using a smartphone (mobile phone terminal), a tablet PC, or the like.

[0020] 2, a CPU 101a, a memory 102a, a nonvolatile memory 103a, an image processing unit 104a, a display 105a, an operation unit 106a, a recording medium I / F 107a, an external I / F 109a, and a communication I / F 110a are connected to an internal bus 150a. The units connected to the internal bus 150a are capable of exchanging data with each other via the internal bus 150a.

[0021] The memory 102a is, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 101a controls each part of the purchaser terminal 100a using the memory 102a as a work memory in accordance with a program stored in, for example, the nonvolatile memory 103a. The nonvolatile memory 103a stores image data, audio data, other data, various programs for the operation of the CPU 101a, etc. The nonvolatile memory 103a is, for example, a flash memory or a ROM.

[0022] Under the control of the CPU 101a, the image processing unit 104a performs various image processing on images stored in the nonvolatile memory 103a or the recording medium 108a, images acquired via the external I / F 109a or the communication I / F 110a, captured images, etc. The image processing performed by the image processing unit 104a includes A / D conversion processing, D / A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, etc. The image processing unit 104a may be configured with a circuit block dedicated to performing specific image processing. Furthermore, depending on the type of image processing, the CPU 101a may perform image processing according to a program without using the image processing unit 104a.

[0023] Display 105a displays images, GUI screens constituting a GUI (Graphical User Interface), and the like under the control of CPU 101a. CPU 101a generates display control signals in accordance with a program, and controls each part of purchaser terminal 100a to generate video signals to be displayed on display 105a and output them to display 105a. Display 105 displays video based on the output video signals. Note that purchaser terminal 100a itself is limited to having an interface for outputting video signals to be displayed on display 105, and display 105a may be configured as an external monitor (such as a television).

[0024] The operation unit 106a is an input device for accepting user operations. The operation unit 106a includes a character information input device such as a keyboard, a pointing device such as a mouse or touch panel 70a, a button, a dial, a joystick, a touch sensor, a touch pad, etc. The touch panel 70a is an input device that is configured as a plane overlaid on the display 105a and outputs coordinate information according to the position where it is touched.

[0025] The recording medium I / F 107a allows a recording medium 108a such as a memory card to be attached, and reads data from and writes data to the attached recording medium 108a under the control of the CPU 101a. The external I / F 109a is an interface for connecting to an external device via a wired cable or wirelessly and inputting and outputting video signals and audio signals. The communication I / F 110a is an interface for communicating with external devices, the Internet 111, etc., and sending and receiving various data such as files and commands.

[0026] The camera unit 112a is a camera unit including an image sensor (image sensor) such as a CCD or CMOS element that converts an optical image into an electrical signal. The camera unit 112a includes a lens group (taking lenses) including a zoom lens and a focus lens, a shutter with an aperture function, the image sensor, an A / D converter that converts the analog signal output from the image sensor into a digital signal, and a barrier that covers the image sensor to prevent contamination and damage. The image processing unit 104a performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the data captured by the camera unit 112a. The CPU 101a performs exposure control, distance measurement control, and AWB (auto white balance) processing based on the calculation results obtained by the image processing unit 104a. The display 105a displays image data captured by the camera unit 112a and processed by the image processing unit 104a. A digital signal captured by the camera unit 112a, A / D converted by an A / D converter, and stored in the memory 102a is converted to analog by a D / A converter and sequentially transferred to and displayed on the display 105a, thereby enabling live view display (LV display). The live view can be displayed in a still image capture standby state, a video capture standby state, and during video recording, displaying the captured subject image in nearly real time. The CPU 101a controls the camera unit 112a and the image processing unit 104a to start operations such as AF (autofocus) processing, AE (autoexposure) processing, and AWB processing in response to a capture preparation instruction based on a user operation performed on the operation unit 106a. In response to a capture instruction, the CPU 101a controls the start of a series of capture processing (main capture) operations, including main exposure, reading a signal from the image capture element, image processing the captured image by the image processing unit 104a, generating an image file, and recording the image on the recording medium 108a. A capture instruction can be given by a user operation performed on the operation unit 106a. The camera unit 112a is capable of taking still images and moving images.

[0027] FIG. 2(b) shows an example of the configuration of a server device 100b as an example of a device to which the present invention can be applied. The server device 100b is an information processing device capable of communicating with the purchaser terminal 100a and the edge PC 100c, and can be configured using a personal computer (PC) or the like. The components of the server device 100b, indicated by the reference numerals 101b to 104b, 106b to 110b, and 150b, are the same as the components of the purchaser terminal 100a indicated by the reference numerals 101a to 104a, 106a to 110a, and 150a. Therefore, detailed description thereof will be omitted. Furthermore, although the server device 100b will be described as not having a display or a camera unit, a device having a display and a camera unit may also be used as the server device 100b.

[0028] FIG. 2(c) shows an example of the configuration of an edge PC 100c as an example of a device to which the present invention can be applied. The edge PC 100c is a terminal device (information processing device) capable of communicating with the server device 100b and can be configured using a personal computer (PC) or the like. The components of the edge PC 100c, indicated by the reference numerals 101c to 104c, 106c to 110c, and 150c, are similar to the components of the purchaser terminal 100a indicated by the reference numerals 101a to 104a, 106a to 110a, and 150a. Therefore, detailed description will be omitted. The edge PC 100c will be described as not having a display or a camera unit. In other words, the edge PC 100c can be applied to a device that only has the main body of a desktop PC, and can be configured without at least one of a display and a camera unit, thereby reducing costs and installation space. However, a device with a display and a camera unit may also be used as the edge PC 100c. The audio output unit 113c is an output unit that transmits an audio signal to a speaker via wired or wireless communication to produce a sound, or is the speaker itself. The notification sound or audio message output from the audio output unit 113c can notify (make the purchaser 120 aware of) information such as various guidances and warnings.

[0029] 3 shows a flowchart of the processing related to the sales system in the purchaser terminal 100a. This processing is realized by loading a program recorded in the non-volatile memory 103a into the memory 102a and executing it by the CPU 101a. The processing in FIG. 3 starts when the purchaser 120 launches an application program capable of reading a QR code on the purchaser terminal 100a or an application program for this system (hereinafter simply referred to as the purchase application) in order to purchase an item on the shelf 130.

[0030] In S301, the CPU 101a displays an LV image captured by the camera unit 112a on a QR code reading screen. FIG. 8(a) shows an example of a QR code reading screen displayed on the display 105a (touch panel 70a) by the purchasing app. A message 801 is a guidance message that prompts the purchaser 120 to have the purchaser terminal 100a read the QR code 160 (frame the screen so that the QR code 160 is within the image capturing range). For example, it may say, "Please allow me to read the QR code attached to the product shelf." A frame 802 is a guide frame that serves as a guide for framing the QR code so that it is within the image capturing range. A cart icon 803 is a touch icon for receiving an instruction to transition to the cart screen. The cart icon 803 is displayed when a product is in the cart corresponding to the user ID of the purchaser 120, and is not displayed otherwise. In other words, it is not displayed at initial startup. Although not shown in the figure, the LV image captured by the camera unit 112a is displayed on the QR code reading screen, and the message 801, frame 802, and cart icon 803 are displayed superimposed on the LV image.

[0031] In S302, the CPU 101a refers to the cart information recorded in the nonvolatile memory 103a and determines whether or not there is a product in the cart. If it is determined that there is a product in the cart, the process proceeds to S303, and if not (the cart is empty or is in the initial state where no cart information has been generated), the process proceeds to S304.

[0032] In S304, the CPU 101a determines whether or not an instruction to transition to the cart screen has been given. If an instruction to transition to the cart screen has been given, that is, if a touch operation has been performed on the cart icon 803, the process proceeds to S330; otherwise, the process proceeds to S304.

[0033] In S304, the CPU 101a determines whether the QR code has been photographed and the information contained in the QR code has been acquired. If the purchaser 120 has framed the purchaser terminal 100a so that the QR code 160 falls within the photographing range, and the QR code 160 is acquired from the image photographed as an LV image, analyzed, and the information contained in the QR code 160 has been acquired, the process proceeds to S310. If not, the process proceeds to S305.

[0034] In S305, the CPU 101a determines whether or not an end event (for example, an operation to close the QR code reading screen) has occurred. If no end event has occurred, the process returns to S302 and repeats, and if an end event has occurred, the process in FIG. 3 ends.

[0035] In S310, the CPU 101a determines whether or not the information in the QR code acquired in S304 includes a shelf ID. If the information in the QR code does not include a shelf ID, this is the case when the captured QR code is a QR code that is not related to the sales system of this embodiment. If the information in the QR code includes a shelf ID, the process proceeds to S312, and if the information does not include a shelf ID, the process proceeds to S311, where an error is displayed, or a process separate from this system (processing based on the QR code) is performed.

[0036] In S312, the CPU 101a determines whether an operation to determine the execution of the process indicated by the QR code information acquired in S304 has been performed. For example, if the captured QR code contains shelf ID information, the display 105a displays a message asking, "Do you want to check in to shelf No. 67890?" along with the options "Yes" or "No." "67890" is just an example, and any identification information that allows the user to identify the shelf to be checked in may be used. If "Yes" is selected, it is determined that an operation to determine the execution of the process indicated by the QR code information has been performed, and the process proceeds to S313. If "No" is selected, the process returns to S302. Note that the QR code 160 may contain information for launching a purchasing app or accessing a web app on the Internet. In that case, the QR code 160 is read with a general-purpose QR code reader, and if a confirmation instruction is received, the process from S313 onward is performed according to the purchasing app or web app. Furthermore, the process of S312 may be omitted, and if it is determined that the read QR code contains a shelf ID (Yes in S310), the process of S313 may be performed without waiting for an operation from the purchaser 120 (operation to instruct decision).

[0037] In S313, the CPU 101a transmits check-in request information to the server device 100b, including the shelf ID acquired from the read QR code and a user ID for identifying the purchaser 120. The shelf ID is display location information for identifying the shelf 130, which is the display location. In this embodiment, the user ID is a login ID to the purchasing app (pre-stored or entered by the purchaser 120 when launching the purchasing app). The user ID is not limited to this, and other information may be used as long as it can identify the purchaser 120 who has checked in to the shelf 130 from other users. For example, the SSID of the purchaser terminal 100a, account information for other linked services such as SNS, or login account information for the purchaser terminal 100a may be used as the user ID. In other words, the user ID is user information for identifying the purchaser terminal 100a or the purchaser 120. When the purchaser terminal 100a transmits the shelf ID and user ID to the server device 100b, the server device 100b attempts to check in, and the server device 100b returns information indicating whether or not the check-in was successful.

[0038] In this embodiment, the method of acquiring a shelf ID at the purchaser terminal 100a has been described as a method of capturing an image of the QR code 160, which is a two-dimensional code, but this is not limiting. The shelf ID may also be acquired by capturing an image of a one-dimensional code such as a barcode, or by capturing an image of a shelf ID displayed as a character string and performing character recognition from the image. The shelf ID displayed as a character string may also be acquired by the user manually entering the information on a shelf ID input screen displayed on the display 105a of the purchaser terminal 100a. Alternatively, the shelf ID may be acquired based on location information of the purchaser terminal 100a acquired from a location information acquisition unit (not shown), such as a GPS positioning unit, of the purchaser terminal 100a. The shelf ID may also be acquired by combining at least two of the information acquired from the captured image, information acquired by manually entering the information, and location information.

[0039] In S314, the CPU 101a determines whether or not a check-in impossible notice has been received from the server device 100b, indicating that check-in was not possible. For example, if a user other than the purchaser 120 has already checked in to the shelf 130, the purchaser 120 cannot check in to the shelf 130. If a check-in impossible notice has been received, the process proceeds to S315; otherwise, the process proceeds to S316.

[0040] In S315, the CPU 101a displays check-in error information on the display 105a, indicating that check-in is currently unavailable. For example, a message such as "Check-in is unavailable because another user is using the shelf. Please check in after the previous user has finished using the shelf" is displayed. That is, information indicating that another user has checked in and / or information urging the user to wait and submit a check-in request again is displayed. If the shelf 130 has a door or the like, it would be possible to determine whether another purchaser is in the middle of making a purchase based on the door's open / closed state (i.e., if another person has the door open, that person has priority). However, in this embodiment, no door is provided. Therefore, it may be difficult to determine from the exterior of the shelf 130 whether another user is present. For example, even if another person is near the shelf 130, it may be difficult to determine whether that person has not yet decided whether to make a purchase, is there for a reason other than a purchase, or is in the middle of a purchase. However, by displaying a check-in error, as in this embodiment, the purchaser 120 can recognize that another user is in the middle of a purchase and should wait. Furthermore, as in this embodiment, by preventing other users from checking in when one purchaser is checked in, it is possible to prevent the situation where it becomes impossible to determine which user has taken an item from shelf 130 (causing confusion among purchasers).

[0041] In S316, the CPU 101a determines whether or not a check-in completion notice has been received from the server device 100b, indicating that check-in has been successful. If a check-in completion notice has been received, the process proceeds to S317; if not, the process returns to S314.

[0042] In S317, the CPU 101a determines whether or not there is a product in the cart, similarly to S302. If there is a product in the cart, the process proceeds to S330, and if not, the process proceeds to S320.

[0043] In S320, the CPU 101a displays an empty cart on the display 105a. An example of the empty cart display is shown in FIG.

[0044] Display item 810 is a screen title indicating that the currently displayed screen is a cart display screen that shows whether there are any products in the cart. In this embodiment, "Products in cart" is displayed.

[0045] The guidance 811 is a guide display that prompts the user, the purchaser 120, to remove a product from the shelf 130. In the example of FIG. 8(b), a character string (message) saying "Please remove the product from the shelf" is displayed. In the sales system of this embodiment, the shelf 130 does not have a door or cover. Therefore, it is difficult for the purchaser 120 (user) to determine from the outside whether the shelf 130 is in a state where it is OK to remove a product. Particularly in an unmanned store, the purchaser 120 may be hesitant to remove a product without permission. In contrast, by displaying the guidance 811, the purchaser 120 can recognize that it is OK to remove a product from the shelf 130, and can remove the product and continue shopping with peace of mind. Furthermore, by displaying the guidance 811 on the display 105a of the purchaser terminal 100a, it is possible to notify the purchaser 120 that it is OK to remove the product without relying on a voice notification from the audio output unit 113c of the edge PC 100c. Therefore, even in an environment where voice notification is difficult (a quiet environment or, conversely, a noisy environment), it is possible to reliably notify the purchaser 120 that it is OK to remove the product. Furthermore, since there is no need to connect a display to the edge PC 100c or to install a notification monitor near the shelf 130, the installation cost and space can be reduced, and the purchaser 120 can be reliably notified that the product is ready to be removed. In other words, the sales system of this embodiment can be more easily used by users who purchase products. Note that the guidance 811 is not limited to the example shown in the figure, as long as it encourages the purchaser 120 to remove the product from the shelf 130. For example, the guidance may be a message using a different character string, an icon display, an animation display, video playback, or audio output from the purchaser terminal 100a.

[0046] Display item 812 shows the total price of the items in the cart. In the example of Fig. 8(b), since there are no items in the cart, the total price is displayed as 0 yen.

[0047] The purchase instruction icon 813 is a touch icon (display item) for issuing an instruction to purchase an item in the cart and make payment. In the example of FIG. 8(b), since there are no items in the cart and nothing can be purchased, the icon is grayed out (displayed inactive) and does not accept any operation even if touched. When an item is added to the cart, the grayed-out icon is removed, the icon becomes active, and it becomes possible to accept operation by touch.

[0048] The cancel instruction icon 814 is a touch icon (display item) that accepts an instruction to cancel the purchase action (shopping).

[0049] The shelf transfer item 815 is a touch icon (display item) for issuing an instruction to check in to a shelf other than the shelf to which the user is currently checked in.

[0050] In S321, the CPU 101a determines whether or not a stop operation has been performed (a touch on the stop instruction icon 814). If a stop instruction has been performed, the process proceeds to S322, and if not, the process proceeds to S325.

[0051] In S322, the CPU 101a transmits a cancel instruction indicating that a cancel operation has been performed to the server apparatus 100b.

[0052] In S323, the CPU 101a determines whether or not a check-out completion notice has been received from the server apparatus 100b. If a check-out completion notice has been received, the process proceeds to S324; if not, the process waits in S323 for the check-out completion notice to be received.

[0053] In S324, the CPU 101a clears (deletes) the cart information stored in the memory 102a and displays a checkout message on the display 105a. The cart information stored in the memory 102a is also deleted (cleared). FIG. 8(d) shows an example of the checkout message. The checkout message displays a message 830 such as "Thank you for your patronage," indicating that the purchase (shopping) process has been completed. After the checkout message is displayed for a predetermined time (for example, 5 seconds), the display returns to the LV display (QR code reading screen) of S301.

[0054] In S325, it is determined whether or not a shelf moving operation (touch on the shelf moving item 815) has been performed. If a shelf moving operation has been performed, the process proceeds to S326, and if not, the process proceeds to S328.

[0055] In S326, the CPU 101a transmits a shelf movement instruction indicating that a shelf movement operation has been performed to the server device 100b.

[0056] In S327, the CPU 101a determines whether or not a check-out completion notification has been received from the server device 100b. If a check-out completion notification has been received, the process returns to the LV display (QR code reading screen) in S301 to read the QR code of another shelf. If not, the process waits in S323 for reception of the check-out completion notification.

[0057] In S328, the CPU 101a determines whether or not an update notification of the cart information has been received from the server device 100b. If an update notification has been received, the CPU 101a stores information indicating the latest cart status included in the received update notification in the memory 102a (i.e., updates the cart information stored in the memory 102a), and proceeds to S317. If an update notification has not been received, the CPU 101a proceeds to S321.

[0058] In the present embodiment, an example has been described in which a touch operation on the purchase instruction icon 813 is not accepted (ignored) when an empty cart is displayed, but this is not limiting. For example, if a touch operation on the purchase instruction icon 813 is detected while an empty cart is displayed, a display urging the customer to remove a product from the shelf (a second display different from the display in S320) may be displayed. For example, the guidance 811 may be highlighted in a different display format from S320, such as by flashing or changing its color, or a display different from the guidance 811 may be displayed to urge the customer to remove a product from the shelf. In particular, it is preferable to display a message indicating that the customer may remove a product from the shelf before payment. If the purchase instruction icon 813 is operated when there are no products in the cart, i.e., when no products have been removed, the customer 120 may mistakenly believe that the customer must make a payment before removing a product from the shelf. Therefore, by displaying a display urging the customer to remove a product from the shelf, the customer's mistaken belief can be resolved, allowing the customer to make a smooth purchase.

[0059] In S330, the CPU 101a displays a cart when there are products on the display unit. An example of a cart display when there are products is shown in Fig. 8(c). The same display items as in Fig. 8(b) are assigned the same reference numerals and descriptions thereof will be omitted.

[0060] The product list 820 is a list of products in the cart based on the cart information indicating the latest cart status stored in the memory 102a, including product information included in the update notification received from the server device 100b. The product list 820 displays information about one product per row and includes columns 821 to 824. Column 821 is a display element that indicates the appearance of the product in each row, and displays a photo of the selected product or an icon representing the type of the selected product (for example, an icon resembling a plastic bottle as shown in the figure for beverages, or an icon resembling food for food). When the type of product is represented by a photo, an image pre-stored in the non-volatile memory 103a of the purchaser terminal 100a as data for the purchasing application or an image received from the server device 100b is displayed. Column 822 displays the name (product name) of the product in each row, and column 823 displays the price of the product in each row. Column 824 is a display item for receiving correction instructions for the product in each row. In the example of FIG. 8(c), the purchase instruction icon 813 is displayed in a display form that can accept touch operations (active display) because there is a product in the cart and it can be purchased. On the other hand, the cancel instruction icon 814 is displayed grayed out. This is because in this case, there is a product in the cart, and the purchase cannot be canceled completely unless the product in the cart is returned to the shelf 130. However, in this embodiment, touch operations to the cancel instruction icon 814 are accepted even when it is grayed out. However, this is not limited to this, and if there is a product in the cart, touch operations to the cancel instruction icon 814 may not be accepted.

[0061] In S331, the CPU 101a determines whether or not a correction operation has been performed. A correction operation is an operation in which a display item displayed in column 824 of any row is touched, and then an operation is completed on the correction content input screen (a screen for correcting the product in the touched row) that is displayed thereafter, and the correction content is confirmed. The correction content input screen is, for example, a screen that displays a list of products on sale that are placed on the shelf 130 and accepts the selection of any product from the list. In this way, by accepting a product correction operation, in the unlikely event that the product determined by product determination using image analysis is different from the product that the purchaser 120 actually took out, the purchaser 120 can correct the content to the correct one.

[0062] In S332, the CPU 101a transmits information indicating the corrected product, which is the correction content input on the correction content input screen, to the server device 100b.

[0063] In S333, the CPU 101a waits for an update notification from the server device 100b, and upon receiving the update notification, updates the cart information stored in the memory 102a based on the notification and proceeds to S317. In this way, the cart display on the display 105a also reflects the modification.

[0064] In S334, the CPU 101a determines whether or not a stop operation has been performed (a touch on the stop instruction icon 814). If a stop instruction has been performed, the process proceeds to S335, and if not, the process proceeds to S340.

[0065] In S335, the CPU 101a transmits a cancel instruction indicating that a cancel operation has been performed to the server apparatus 100b.

[0066] In S336, the CPU 101a displays a return message on the display 105a. The return message is a display that prompts the purchaser 120 to return the product that he or she removed from the shelf 130 to the shelf 130, and for example, a character string (message) such as "Please return the product to its original position" is displayed. This enables the purchaser 120 to recognize that he or she cannot stop (cancel) the shopping unless he or she returns the removed product to the shelf 130, and he or she attempts to return the product that he or she is holding to the shelf 130. As a result, when the product in the cart (the product that the purchaser 120 removed) is returned to the shelf 130, the camera 140 detects that the product has been returned from the video of the return, and the edge PC 100c transmits information that the product has been returned to the server device 100b. When the server device 100b receives the information that the product has been returned, it updates the cart information stored in the server device 100b and transmits the information as an update notification to the purchaser terminal 100a.

[0067] In S337, the CPU 101a determines whether an update notification has been received from the server device 100b. If an update notification has been received, the cart information in the memory 102a is updated with the latest cart information included in the received update notification, and the process proceeds to S338. If an update notification has not been received, the process waits for reception of the update notification in S337, and does not proceed to the checkout display.

[0068] In S338, the CPU 101a determines whether there are any products in the cart based on the latest cart information updated based on the update notification received in S337. If there are no products in the cart, this is because all of the products that the purchaser 120 took out from the shelf 130 have been returned to the shelf 130. If there are no products in the cart, the process proceeds to S339; if not (if there are still products in the cart), the process proceeds to S336.

[0069] Here, if an update notification is not received in S337 or if it is determined in S338 that there is an item in the cart, this means that the purchaser 120 has not yet returned the item that was removed from the shelf 130 to the shelf 130. Therefore, if an update notification is not received in S337 after a cancel instruction is given in S334, or if it is determined in S338 that there is an item in the cart, the return message displayed in S336 continues to be displayed to encourage the purchaser 120 to return the item safely. Furthermore, if an update notification is not received in S337 after the return message is displayed, or if it is determined in S338 that there is an item in the cart, the return message may be emphasized in stages. For example, the return message may be displayed flashing after a predetermined time has elapsed, or a warning sound may be emitted, etc.

[0070] In S339, the CPU 101a displays a checkout message in the same manner as in S324. The CPU 101a also erases (clears) the cart information stored in the memory 102a. This allows the purchaser 120 to correctly recognize that the removed product has been returned to the shelf 130, and that the purchase has been correctly canceled, so that it is okay to leave without making a payment.

[0071] In S340, the CPU 101a determines whether or not a purchase instruction operation (touch on the purchase instruction icon 813) has been performed. The purchase instruction operation is an operation for making a payment, and is a type of payment operation. If a purchase instruction operation has been performed, the process proceeds to S341; if not, the process proceeds to S344.

[0072] In S341, the CPU 101a transmits a purchase instruction notification indicating that a purchase instruction operation has been performed to the server device 100b.

[0073] In S342, the CPU 101a performs payment processing in cooperation with the server device 100b. For example, a payment screen using a credit cart or a payment screen using electronic money is displayed, and payment (payment, billing) for the items in the cart is made in response to operations on these payment screens (payment operations on the touch panel 70a). When payment is complete, the process proceeds to S339. The checkout display in S339 lets the purchaser 120 know that payment has been successfully completed and that he or she may leave the premises with the items (purchased items) taken from the shelf 130.

[0074] In S344, the CPU 101a determines whether or not a shelf moving operation (touch on the shelf moving item 815) has been performed. If a shelf moving operation has been performed, the process proceeds to S326, and if not, the process proceeds to S345.

[0075] In S345, the CPU 101a determines whether or not an update notification of the cart information has been received from the server device 100b. If an update notification has been received, the CPU 101a stores information indicating the latest cart status included in the received update notification in the memory 102a (i.e., updates the cart information stored in the memory 102a), and proceeds to S317. If an update notification has not been received, the CPU 101a proceeds to S331.

[0076] 4 shows a flowchart of processing related to the sales system in server device 100b. This processing is realized by loading a program recorded in nonvolatile memory 103a into memory 102a and executing it with CPU 101a. This processing is started manually when sales by the sales system of this embodiment start (such as the start time of business hours), or automatically when a predetermined time (such as the start time of business hours) arrives.

[0077] In S401, the CPU 101b initializes various variables, information (cart information), etc. stored in the memory 102b. The non-volatile memory 103b also stores the IDs of multiple product shelves managed by the server device 100b. The IDs of all of the managed product shelves are read, and check-in status management information is created in the memory 102b, after which the status of all shelves is set to not checked in. An example of check-in status management information is shown in Figure 5(a). The management information stores the ID of the shelf to be managed, the check-in status, and, if the status is checked in, the ID of the user who has checked in (CI), in association with each other. In S401, all check-in statuses are set to not checked in, and there are no CI users for all shelves.

[0078] In S402, the CPU 101b determines whether or not the shelf ID transmitted from the purchaser terminal 100a has been received. This is a determination as to whether or not the shelf ID and user ID transmitted from the purchaser terminal 100a in S313 of Fig. 3 have been received. If the shelf ID has been received, the process proceeds to S402; if not, the process proceeds to S407.

[0079] In S403, the check-in status management information stored in memory 102b is referenced to determine whether or not the shelf with the received shelf ID is already checked in (in a checked-in state). If it is in a checked-in state, the process proceeds to S404, and if not (if it is in a non-checked-in state), the process proceeds to S405.

[0080] In S404, a check-in impossible notice indicating that check-in is not possible is sent to the purchaser terminal 100a that sent the shelf ID received in S402. Also, the check-in status management information stored in memory 102b is not updated. In other words, in this case, check-in processing is not performed using the received shelf ID. In this way, the check-in impossible notice sent in S404 is received in S314 in Figure 3 described above, and a check-in error is displayed in S315.

[0081] In S405, the CPU 101b performs check-in processing. Specifically, in the check-in status management information held in the memory 102b, the check-in status for the shelf with the shelf ID received in S402 is set to "checked in," and the user ID received together with the shelf ID is stored in association with the CI user. In other words, the check-in processing is an association process that associates the shelf ID (display location information) with the user ID (user information). An example of the management information in this case is shown in FIG. 5(a). For the shelf with shelf ID "678 user ID 90," the check-in status is set to "checked in," and the CI user ID "12345" is stored in association.

[0082] Furthermore, in S404, if cart information for the CI user who received the message has not been generated, cart information for the CI user is generated in memory 102b. Figure 5(b) shows an example of cart information for user ID "12345." Cart information is information generated for each user ID. The cart information stores the user ID, the checked-in shelf ID, the product name (or product identification information) of the product taken from the product shelf (product in the cart), price, quantity, subtotal, the shelf ID from which the product was taken, and the total price of all products in the cart, all in association with each other. The example in Figure 5(a) is an example of the state immediately after the user checks in to the shelf with shelf ID "67890," and there are no products in the cart.

[0083] Note that if the cart information for the corresponding user ID from the shelf ID and user ID received in S402 is already stored in memory 102b, new cart information is not generated. In this case, the checked-in shelf ID in the existing cart information for the corresponding user ID is updated with the received shelf ID. In this case, a shelf transfer instruction is received from the first shelf, and after checking out from the first shelf without making a payment, the user checks in to another shelf (second shelf). If an item is removed from the second shelf after checking in to the second shelf, the removed item is added to the cart. Figure 5(c) shows cart information for the case where an instruction to transfer items is received after items A and B are removed from the shelf with shelf ID "67890" (first shelf), and item C is removed after checking in to the shelf with shelf ID "67891" (second shelf). In addition to items A and B removed from the first shelf, item C removed from the second shelf is recorded, and the total price is the sum of items A, B, and C.

[0084] In S406, the CPU 101b transmits a check-in completion notification indicating that check-in has been completed to both the purchaser terminal 100a and the edge PC 100c. The transmitted check-in completion notification is received by the purchaser terminal 100a in S316 of Fig. 3 described above, and is received by the edge PC 100c in S603 of Fig. 6 described below.

[0085] In S407, the CPU 101b determines whether or not the shelf ID and product recognition score transmitted from the edge PC 100c have been received. If received, the process proceeds to S408, and if not, the process proceeds to S411.

[0086] In S408, the CPU 101b determines the identity of the product that has been removed or returned from the shelf based on the product recognition score received from the edge PC 100c. In this embodiment, the final product determination is performed on the server device 100b side, not the edge PC 100c. This has the advantage of enabling product determination using sophisticated or cutting-edge determination methods such as machine learning, or by simply updating the server device 100b without updating the edge PC. This reduces the increased installation costs associated with the sophisticated functionality of the edge PC 100c and the management burden of updating the edge PC 100c. The product recognition score received from the edge PC may be information (inference score) resulting from a certain degree of product estimation, or may be information such as parameters of shape, color, and size prior to product estimation. Receiving a product recognition score that has been parameterized to some extent, rather than the image captured by the camera 140, reduces the amount of communication between the edge PC 100c and the server device 100b, enabling high-speed determination and response.

[0087] Note that the transmission of the product recognition score from the edge PC 100c to the server device 100b is not limited to this. For example, the edge PC 100c may transmit the image (video or still image) itself captured by the camera 140 of the scene where it is estimated that the product was taken out / put back, without performing processing to determine what the product is. In this case, the server device 100b may determine the product based on the received image captured by the camera 140. Conversely, the edge PC 100c may determine that a product has been taken out / put back and what the product is, and transmit only the determination result to the server device 100b. In this case, the processing of S408 is omitted.

[0088] In S409, CPU 101b updates the cart information held in memory 102b based on the product determined in S408. That is, if it is determined in S408 that the product has been removed from the shelf, information about the product determined in S408 is added to the cart information for the user ID associated with the received shelf ID. Conversely, if it is determined in S408 that the product has been returned to the shelf, the product determined in S408 is deleted from the cart information of the checked-in user in which the received shelf ID is recorded as the removal source. Figure 5(b) shows an example of cart information in a case where product A was originally included in the cart information, but the shelf ID and product recognition score for shelf ID "67890" were received in S407, and it was determined in S408 that product B had been removed from the shelf. In addition to product A, information about product B has been added.

[0089] In S410, the CPU 101b transmits an update notification to the purchaser terminal 100a indicating that the cart information has been updated. The update notification also includes the latest cart information stored in the memory 102b. In other words, if a product has been removed from the product shelf 130, the CPU 101b also transmits cart information including the information on the removed product determined in S408.

[0090] In S411, the CPU 101b determines whether or not a correction instruction sent from the purchaser terminal 100a (the correction content sent from the purchaser terminal 100a in S332 of FIG. 3 described above) has been received. If it is determined that a correction instruction has been received, the process proceeds to S409, where the cart information stored in the memory 102b is updated based on the correction content received as the correction instruction. If a correction instruction has not been received, the process proceeds to S412.

[0091] In S412, the CPU 101b determines whether or not a stop instruction transmitted from the purchaser terminal 100a has been received. If a stop instruction has been received, the process proceeds to S413; if not, the process proceeds to S417.

[0092] In S413, the CPU 101b determines whether or not the cart information of the user ID of the user who issued the cancellation instruction received in S412 contains any products. If the cart information contains any products, the process proceeds to S414; if not, the process proceeds to S419.

[0093] The processing of S414 to S416 is the same as the processing of S408 to S410. If there are no more items in the cart of the user who issued the cancel instruction, the result of S413 is determined to be No, and the cancel processing accompanied by checkout processing (the processing of S419 to S422) is performed. If there are still items in the cart of the user who issued the cancel instruction, the result of S413 is not determined to be No, and the cancel processing accompanied by checkout processing is not performed.

[0094] In S417, the CPU 101b determines whether or not a payment instruction sent from the purchaser terminal 100a (the payment instruction sent from the purchaser terminal 100a in S341 of FIG. 3 described above) has been received. If it is determined that a payment instruction has been received, the process proceeds to S418; otherwise, the process proceeds to S423.

[0095] In S418, the CPU 101b calls the payment service 170 and performs payment processing based on the payment service 170.

[0096] In S419, the CPU 101b clears (deletes) the cart information stored in the memory 102b for the user ID for which the cancellation instruction was accepted in S412 or the user ID for which the payment instruction was accepted in S417.

[0097] In S420, the CPU 101b performs check-out processing. Specifically, the CPU 101b updates the status of the product shelf that was checked in by the user ID for which the cancellation instruction was received in S412 or the user ID for which the payment instruction was received in S417 to checked-out status. In other words, the status of the relevant shelf ID in the check-in status management information (FIG. 5(a)) is changed to checked-out status, and the CI user is deleted. This deletes the association between the shelf and the user (purchaser).

[0098] In S421, the CPU 101b transmits a check-out completion notice indicating that the check-out has been completed to the purchaser terminal 100a and the edge PC 100c.

[0099] In S422, the CPU 101b determines whether or not a shelf movement instruction transmitted from the purchaser terminal 100a (the shelf movement instruction transmitted from the purchaser terminal 100a in S328 of FIG. 3 described above) has been received. If it is determined that a shelf movement instruction has been received, the process proceeds to S423; otherwise, the process proceeds to S424.

[0100] In S423, the CPU 101b performs the check-out process in the same manner as in S420 described above.

[0101] In S424, the CPU 101b determines whether or not an end event has occurred for the processing in Fig. 4. If an end event has occurred, such as the end of business hours, an end instruction from the user, or power off, the processing in Fig. 4 is ended, and if no end event has occurred, the CPU 101b proceeds to S402 and repeats the processing.

[0102] Figure 6 shows a flowchart of processing related to the sales system in edge PC 100c. This processing is realized by loading a program recorded in non-volatile memory 103c into memory 102c and executing it with CPU 101c. This processing is started manually when sales by the sales system in this embodiment begin (such as the start time of business hours), or is started automatically when a predetermined time (such as the start time of business hours) arrives. Figure 6 shows processing related to an individual shelf, and in this embodiment, the processing will be described as processing related to shelf 130 (shelf ID: 67890).

[0103] In S601, the CPU 101c sets the state information stored in the memory 102c or the nonvolatile memory 103c as the state of the shelf 130 to information corresponding to non-check-in and stores it. That is, it sets the state to the non-check-in state.

[0104] In S602, the CPU 101c acquires video from the camera 140. The video acquired here is assumed to be a video showing the product removal opening of the shelf 130. If a sensor other than the camera 140 is used to detect the removal / return of a product, the process acquires an output value from that sensor.

[0105] In S603, the CPU 101c determines whether or not a check-in completion notice transmitted from the server apparatus 100b has been received. If a check-in completion notice has been received, the process proceeds to S604; if not, the process proceeds to S606.

[0106] In S604, the CPU 101c sets and stores the state information stored and managed in the memory 102c or the nonvolatile memory 103c as the state of the shelf 130 as information corresponding to check-in. That is, it sets the state to the checked-in state (a state in which the shelf 130 and the prospective purchaser are associated with each other).

[0107] In S605, the CPU 101c causes the audio output unit 113c to output audio indicating that it is now OK to remove products from the shelf 130. For example, a sound effect such as "ding" or an audio message such as "Welcome" may be output to notify the user that they have successfully checked in to the shelf 130 (the shelf 130 has responded) in response to reading the QR code 160 with the purchaser terminal 100a. Alternatively, audio (audio guidance) encouraging the purchaser 120 to remove products from the shelf or audio explaining how to shop may be output, such as "Please remove the products from the shelf" or "After removing the products from the shelf, please make payment using your terminal." In other words, the CPU 101c notifies the user of a change in status from a non-checked-in status to a checked-in status, and / or prompts the user to remove products from the shelf 130.

[0108] In S606, CPU 101c determines whether a product has been removed from shelf 130 or whether a product has been returned (put back) to shelf 130, based on the image acquired from camera 140. That is, based on an image of product shelf 130 captured by camera 140, it detects whether a product has been removed from its display location. For example, based on the image, it detects that a hand has entered from a reference position in front of product shelf 130, the subject at the location where the product was placed has changed, and the hand has been removed from the reference position, as a product removal phenomenon. If a removal or return phenomenon is detected, it is determined that a removal or return has occurred, and the process proceeds to S607, and if not, the process proceeds to S610. In the present embodiment, an example in which the removal / return of a product is detected based on the video captured by the camera 140 has been described, but the detection method is not limited to this. For example, a weight sensor that detects the weight of an item displayed in a display area, such as a shelf of the shelf 130, may be attached to the display area, and the removal / return of a product may be detected based on a change in weight based on the output of the weight sensor. Alternatively, an RFID (radio frequency identifier) ​​tag such as an IC tag may be attached to each product, and the removal / return of a product may be detected through wireless communication with the tag. Furthermore, an infrared sensor may be provided, and the infrared sensor may detect when a product moves beyond a reference line in front of the shelf 130 (removal opening), when a product is removed from its previous location, when a product is placed in a location where there was no product, or the like. The removal / return of a product may be detected based on the output from the infrared sensor indicating such detection. One of several detection methods may be employed, or a combination of several methods may be used to detect the removal / return of a product.

[0109] In S607, the CPU 101c refers to the state information stored in the memory 102c or the nonvolatile memory 103c as the state of the shelf 130, and determines whether or not the shelf 130 is in a checked-in state. If the shelf 130 is in a checked-in state, the process proceeds to S608, and if not, the process proceeds to S609.

[0110] In S608, the CPU 101c transmits to the server device 100b the shelf ID (67890 in this embodiment) of the shelf 130 to be managed, information indicating that a product has been removed or returned in S606, and the product recognition score of the product for which the removal or return was detected. In other words, the CPU 101c transmits the detection details regarding the detection of the removal / return of the product. The product recognition score is, for example, information output as a result of inputting an image captured by the camera 140 into a trained model trained by machine learning and performing an inference process. The product recognition score can also be considered as a result of scoring candidate products by inferring the product from its appearance and shape based on the image captured by the camera 140.

[0111] In S609, the CPU 101c issues an error notification (error output) by voice from the voice output unit 113c. In this case, the edge PC 100c does not transmit the product score to the server device 100b. By issuing the error notification without communicating with the server device 100b, the error notification can be issued more quickly, allowing the purchaser 120 to recognize that the behavior is an error more quickly. Note that in this case, the product score may also be transmitted to the server device 100b to manage the identity of the product removed without checking in, but the timing of transmission is after the timing of the error notification. In other words, the error notification is output without transmitting the detection result to the server device 100b. Furthermore, the trigger for outputting the error notification may not be the detection of the removal of a product as described in S606, but may be, for example, the detection of a phenomenon in which a hand has entered the product shelf 130. Outputting the error notification using this trigger allows the purchaser 120 to recognize that the behavior is an error earlier, potentially preventing the product from being removed. This phenomenon is a phenomenon in which a product may be removed, and is a preparatory action for removing a product. In this way, by detecting the preparatory action for removing a product as a phenomenon in which a product is removed, the same processing as when the removal of a product is detected may be performed. In this case, the object of detection is the phenomenon of removing a product, and the detection of this phenomenon is processed as if the removal of a product has occurred. The same applies to returns, and by detecting not only the phenomenon in which a product is returned, but also the preparatory action for returning a product as a phenomenon in which a product is returned, the same processing as when the return of a product is detected may be performed. In this case, the object of detection is the phenomenon in which a product is returned, and the detection of this phenomenon is processed as if the return of a product has occurred.

[0112] The error notification is automatically canceled when the product taken out without being checked in is returned to the shelf 130 or when a predetermined time (for example, one minute) has passed. By automatically canceling the error notification after outputting it for a predetermined time even if the taken product is not returned, it is possible to prevent fraudulent operations from causing the error notification to continue for a long time on an unmanned shelf 130, which could result in the loss of another customer's purchasing opportunity.

[0113] The error notification may be, for example, a voice message or sound effect that prompts the user to check in, such as "Please check in before removing the product" or "Please scan the QR code displayed on the shelf with your device before removing the product." Alternatively, a message such as "Please return the product to the shelf and then scan the QR code on the shelf" may be output to prompt the user to return the product to its original position. The error notification is output as a sound that can be heard by a person who has removed or is attempting to remove an item from the product shelf 130. This error notification allows the purchaser 120 (prospective purchaser) to realize that they have removed an item when it is not in a suitable condition and return it to its original position. In other words, the sales system of this embodiment can be used appropriately.

[0114] In S610, the CPU 101c determines whether or not a check-out completion notice has been received from the server apparatus 100b. If a check-out completion notice has been received, the process proceeds to S611; if not, the process proceeds to S613.

[0115] In S611, the CPU 101c sets and stores the status information stored and managed in the memory 102c or the nonvolatile memory 103c as the status of the shelf 130 to information corresponding to non-check-in. That is, the status is checked out and non-checked in (a state in which the shelf 130 and the prospective purchaser are not associated).

[0116] In S612, the CPU 101c outputs, from the audio output unit 113c, audio accompanying checkout from the shelf 130. For example, a sound effect such as "ringing" or an audio message such as "Thank you for using our service" may be output.

[0117] In S613, the CPU 101c determines whether or not an end event has occurred for the processing in Fig. 6. If an end event has occurred, such as the end of business hours, an end instruction from the user, or power off, the processing in Fig. 6 is ended, and if no end event has occurred, the processing proceeds to S02 and the processing is repeated.

[0118] Fig. 7 shows an example of a processing sequence related to sales in the sales system of this embodiment, which is realized by cooperation between the processing of the purchaser terminal 100a described in Fig. 3, the processing of the server device 100b described in Fig. 4, and the processing of the edge PC 100c described in Fig. 6. The corresponding steps of the processing described above in Figs. 3, 4, and 6 are shown in parentheses.

[0119] In S700, the QR code reading screen is opened on the purchaser terminal 100a, and the purchaser 120 points the camera unit 112a of the purchaser terminal 100a at the QR code 160 while the LV image is displayed.

[0120] In S701, the camera unit 112a of the purchaser terminal 100a photographs and reads the QR code 160 (S304 Yes).

[0121] In S702, the shelf ID and the user ID are transmitted from the purchaser terminal 100a to the server device 100b (S313).

[0122] In S703, the server device 100b performs check-in processing (S405).

[0123] In S704, the server device 100b transmits a check-in completion notification to the edge PC 100c (S406), and in response to this, in S705, the edge PC performs check-in (S604) and outputs a voice guide (S605).

[0124] In S706, the server device 100b transmits a check-in completion notice to the purchaser terminal 100a (S406), and in response, in S707, an empty cart display is performed on the purchaser terminal 100a (S320).

[0125] In S708, the purchaser 120 removes the product from the shelf 130.

[0126] In S709, the edge PC 100c detects that the product has been removed (S606 Yes), and transmits the shelf ID and the product recognition score to the server device 100b (S608).

[0127] In S710, the server device 100b performs product evaluation (S408) and updates the products in the cart (S409).

[0128] In S711, the server device 100b transmits a notice of update of the cart information to the purchaser terminal 100a (S410).

[0129] In S712, the cart display is updated on the purchaser terminal 100a. In this example, in this state, the cart display shows that the product taken out in S708 has been added to the cart (S330).

[0130] In S713, the purchaser 120 performs a cancel operation by touching the cancel instruction icon 814.

[0131] In S714, in response to the cancellation operation, a cancellation instruction is sent from the purchaser terminal 100a to the server device 100b (S335).

[0132] In S715, since the product is in the cart, a return message is displayed on the purchaser terminal 100a (S336).

[0133] In S716, the purchaser 120, having seen the return message, returns (returns) the product taken out in S708 to the shelf 130.

[0134] In S717, the edge PC 100c detects that the product has been returned based on the image captured by the camera 140 (S606), and transmits the shelf ID and the returned product recognition score to the server device 100b (S608).

[0135] In S718, the server device 100b determines (identifies) the returned product based on the received shelf ID and product recognition score (S415), and updates the products in the cart (S416). That is, the returned product is deleted from the products in the cart.

[0136] In S719, the server device 100b transmits a notice of update of the cart information to the purchaser terminal 100a (S416).

[0137] In S720, since the product taken out in S708 has been returned and there are no products left in the cart (No in S413), the checkout process is performed in accordance with the cancellation instruction sent in S714 (S420).

[0138] In S721, the server device 100b transmits a check-out completion notification to the edge PC 100c (S421). In S722, the edge PC 100c changes the status of the shelf 130 to non-checked-in in response to the check-out completion notification (S611).

[0139] In S722, the server device 100b transmits a checkout completion notice to the edge PC 100c (S421). In S724, in response to the checkout completion notice, the checkout screen is displayed on the purchaser terminal 100a (S339), and then the LV is displayed on the QR code reading screen.

[0140] In another case, when there is a product in the cart, if the purchaser 120 performs a purchase instruction operation by touching the purchase instruction icon 813 in S730, a payment instruction notification is sent from the purchaser terminal 100a to the server device 100b in S731.

[0141] In S732, in response to the notification of the payment instruction, the server device 100b performs payment processing using the payment service 170. Once payment is made, the items in the cart are cleared, and in S733, checkout processing is performed.

[0142] In S734, the server device 100b sends a check-out completion notice to the edge PC 100c, and in S735 The edge PC 100c sets the status of the shelf 130 to not checked in.

[0143] In S736, the server device 100b transmits a checkout completion notice to the purchaser terminal 100a. In SS724, in response to the checkout completion notice, the checkout screen is displayed on the purchaser terminal 100a, and then the LV is displayed on the QR code reading screen (S301).

[0144] According to the present embodiment described above, it is possible to provide a more suitable system for managing and selling products.

[0145] Furthermore, as is clear from the sequence diagram of FIG. 7 , the sales system of this embodiment does not require direct communication between the purchaser terminal 100a and the edge PC 100c. That is, in this embodiment, the purchaser terminal 100a and the edge PC 100c do not establish direct communication. Therefore, a device that does not have the function for direct communication between the purchaser terminal 100a and the edge PC 100c can also be used as the purchaser terminal 100a. Functions for direct communication between the purchaser terminal 100a and the edge PC 100c include, for example, near-field communication (NFC) and Bluetooth (trademark). However, some electronic devices, such as inexpensive smartphones and mobile phones, have a camera unit but do not have NFC or Bluetooth functionality. In the sales system of this embodiment, even electronic devices without NFC or Bluetooth functionality can be used as the purchaser terminal 100a to perform shopping. That is, it is possible to prevent a situation in which users of electronic devices without near-field communication functionality are denied purchasing opportunities and purchasers are limited to users who own electronic devices with near-field communication functionality. In other words, a wider range of users can purchase the product, which contributes to an increase in sales opportunities. That is, a sales system that can be used by a greater number of users can be provided.

[0146] Furthermore, since there is no need for direct communication between the purchaser terminal 100a and the edge PC 100c, there is no need to install devices such as a short-range wireless communication transceiver connected to the edge PC 100c near the product shelf 130. This also helps to prevent increases in installation space and installation costs.

[0147] Furthermore, when attempting to communicate between the purchaser terminal 100a and the edge PC 100c, a possible communication method is to establish an NFC connection and then hand over to Bluetooth. In this case, the effective communication distance of NFC is short, approximately 10 cm, which places restrictions on installation location, requiring the NFC transceiver to be installed within reach of the user. Furthermore, when Bluetooth is used, there is a disadvantage in that if the purchaser terminal 100a has established a Bluetooth connection with another device, the original connection may need to be terminated. For example, this may result in being unable to make a purchase while listening to music through Bluetooth earphones. The sales system of this embodiment solves this problem because there is no need for direct communication between the purchaser terminal 100a and the edge PC 100c.

[0148] Furthermore, in the sales system of this embodiment, the purchaser 120 does not need to perform any operation or input on the edge PC 100c when purchasing a product placed on the shelf 130. Therefore, there is no need to install a device for accepting operations or input on the shelf 130, which can prevent an increase in sales space and installation costs. Furthermore, because payment is also performed by processing between the purchaser terminal 100a and the server device 100b, there is no need to install payment devices such as a device for receiving cash or a device for reading credit cards, which can reduce installation space and installation costs.

[0149] In the above embodiment, the product shelf 130 is used as an example of a product display location, but this is not limited to this, and the display location may be a location other than a shelf as long as the purchaser can take out the product.

[0150] The above-described embodiment can also be applied to cases such as replenishing inventory instead of selling products. That is, the present embodiment can be applied to product management systems including sales systems and inventory management systems. In the case of inventory management, the above-described purchaser 120 can be replaced with a stock replenisher, and the detection of returning a product can be replaced with the detection of replenishing inventory, and the detection of removing a product can be replaced with the detection of removing a product to remove it from display. That is, the stock replenisher checks in to the shelf 130 by reading the QR code 160 with a terminal owned by the stock replenisher. Once checked in, stock replenishment, replacement of displayed products, etc. can be performed, and any operations performed without checking in can be treated as errors, and the various notifications described above can be issued.

[0151] The various controls described above as being performed by CPUs 101a, 101b, and 101c may each be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0152] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0153] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and one or more computers (or CPUs, MPUs, etc.) in the system or device read and execute the program code. In this case, the program and the storage media storing the program constitute the present invention. [Explanation of symbols]

[0154] 100a: Purchaser terminal 100b: Server device 100c: Edge PC 130: Shelf

Claims

1. A control means for controlling execution of a process for associating display location information relating to a display location received from an electronic device with user identification information; an output means for outputting product information based on the content of detection of the phenomenon of the product being removed from the display location to the electronic device corresponding to the user identification information; Equipped with When the output means receives other user identification information in a first state in which the display location information and the user identification information are associated with each other, the output means controls the output means to output a notification to the electronic device associated with the other user identification information that the association process cannot be executed, in order to limit the user associated with the display location. An information processing system characterized by:

2. The information processing system described in Claim 1, characterized in that the output means is controlled to output error information when a product is removed from the display location in a second state in which the display location information and the user identification information are not associated.

3. 3. The information processing system according to claim 2, wherein the output means cancels the output of the error information when the product is returned to its display location or when a predetermined time has elapsed.

4. The information processing system according to claim 2 or 3, characterized in that the output means outputs, as the error information, information encouraging the user to take action to obtain user identification information or information encouraging the user to return the product to its display location.

5. An information processing system described in any one of claims 1 to 4, characterized in that it is provided with a display control means that associates the product information with the user identification information and controls the electronic device to display it.

6. The information processing system described in Claim 5, characterized in that the display control means controls the electronic device to display product information relating to the products returned to the display location in a number reduced by the number of products returned, when the products are returned to the display location.

7. An information processing system described in any one of claims 1 to 6, characterized in that the output means outputs product information relating to products that have been removed from or returned to the display location to the electronic device.

8. 8. The information processing system according to claim 1, wherein the display location information is information obtained by capturing an image of a one-dimensional code or a two-dimensional code with an imaging means provided in the electronic device.

9. The output means If the display location information is in a first state in which the user's identification information is associated with the display location information, a notification is sent to prompt the user to remove the product from the display location; An information processing system as described in any one of claims 1 to 8, characterized in that when a product is removed from the display location in a second state in which the display location information and the user's identification information are not associated, control is performed to output error information.

10. An information processing system as claimed in any one of claims 1 to 9, characterized in that the product information includes at least one of the name of the product taken out from the display area, a photo of the product, an icon indicating the product type, the price of the product, and the number of products taken out.

11. 11. The information processing system according to claim 1, further comprising a detection means for detecting a phenomenon in which a product is removed from or returned to the display area.

12. The information processing system described in claim 11, characterized in that the detection means performs detection based on at least one of an image taken of the display location, a change in weight of the display location, output from an infrared sensor related to the display location, and the results of communication with an IC tag attached to the product.

13. A control method for an information processing system, comprising: a control step of controlling a control means of the information processing system to execute a process of associating display location information relating to the display location received from the electronic device with user identification information; an output step in which an output means of the information processing system outputs product information based on the detection content of the phenomenon of the product being removed from the display location to the electronic device corresponding to the user identification information; Equipped with The output step is a control step in which, in a first state in which the display location information and the user identification information are associated with each other, when another user identification information is received, a notification that the associating process cannot be executed is output to an electronic device associated with the other user identification information in order to limit users associated with the display location. A method for controlling an information processing system, comprising:

14. A program for causing at least one computer to function as each of the means of the information processing system according to any one of claims 1 to 12.

Images