Information processing systems, methods for controlling information processing systems, electronic devices, programs

Abstract

To provide more effective information about products. [Solution] A detection means for detecting a return phenomenon, which is a phenomenon corresponding to the return of a product to the display location where the product was displayed, When the detection means detects the return-related phenomenon, the control means controls the electronic device to output information regarding the return-related product that is subject to return in the detected phenomenon. It has.

Description

Technical Field

[0001] The present invention relates to a system for guiding at the sales site of products.

Background Art

[0002] Conventionally, sales systems that reduce the labor costs required for product sales, such as vending machines and unmanned stores, have been proposed.

[0003] In Patent Document 1, when an action indicating that a customer has reached for a first product displayed on a product shelf is detected, and when the time counted from that point reaches a predetermined time, it is proposed to execute a product presentation targeting the product for which the hand was reached. Thereby, it is disclosed that only customers who are presumed to be interested in front of the product shelf but are hesitating whether to purchase are targeted for product appeal, enhancing the effectiveness of the product presentation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Even with the method of product presentation described in Patent Document 1, it is not always linked to product purchase, and there is room for improvement in product presentation, promotion, and appeal.

[0006] In view of the above problems, an object of the present invention is to provide a mechanism that can more effectively provide guidance regarding products.

Means for Solving the Problems

[0007] To solve the above problems, the information processing system of the present invention A detection means for detecting a return phenomenon, which is a phenomenon corresponding to the return of a product to the display location where the product was displayed, When the detection means detects the return-related phenomenon, the control means controls the electronic device to output information regarding the return-related product that is subject to return in the detected phenomenon. It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, product information can be provided more effectively. [Brief explanation of the drawing]

[0009] [Figure 1] This is a system configuration diagram of the sales system in this embodiment. [Figure 2] This is a hardware configuration diagram of each device included in the sales system of this embodiment. [Figure 3] This is a processing flowchart for the purchaser terminal 100a. [Figure 4] This is a processing flowchart for server device 100b. [Figure 5] This diagram illustrates the information managed by server device 100b. [Figure 6] This is a processing flowchart for the Edge PC100c. [Figure 7] This is a flowchart of the signage control process. [Figure 8] This is an example of the screen display on the purchaser's terminal 100a. [Figure 9] This is an example of a response display on the signage device 100d. [Figure 10] This is another example of a response displayed on the signage device 100d. [Modes for carrying out the invention]

[0010] The system described in Patent Document 1 requires, for example, attaching RFID tags to each product, installing a display unit to show the total amount, and installing a payment device, which increases the cost and installation space required for setting up product shelves. In other words, there was room for improvement in the introduction cost and / or installation space of the sales system. To reduce costs and space, it is conceivable to use user terminals owned by each purchaser, but in that case, it is preferable that the functions used on the user terminals be more versatile so that more users can use them. Furthermore, in unmanned or minimally staffed sales systems, there is little to no guidance from staff, so there was also room for improvement in enabling purchasers to shop appropriately, easily, and casually.

[0011] In this embodiment, the objective is to provide a technology that offers a more suitable system for managing and selling products, in view of at least one of the above-mentioned problems.

[0012] Preferred embodiments of the present invention will be described below with reference to the drawings.

[0013] Figure 1 shows an example of the system configuration of the information processing system (sales system) of this embodiment. The sales system of this embodiment is an unmanned sales system that can sell without the need for sales staff. The sales system of this embodiment includes a customer terminal 100a (electronic device), a server device 100b (information processing device), an edge PC 100c (terminal device as an information processing device), and a signage device 100d. The customer terminal 100a can communicate with the server device 100b via the internet or directly. The edge PC 100c can also communicate with the server device 100b via the internet or directly. On the other hand, the customer terminal 100a and the edge PC 100c do not need to communicate directly. The signage device 100d is installed near the shelf 130 (display area) and can display images (still images, videos) output from the edge PC 100c and output (speak) audio from the edge PC 100c. Additionally, it accepts voice and operation input from users (purchasers 120) and inputs them into the Edge PC 100c.

[0014] One or more (at least one) products, including product 131, are placed (displayed) on shelf 130, which is a product shelf serving as an example of a product display location (including the place where the products are placed). Furthermore, a QR code (trademark) 160 containing the shelf ID (identification information) of shelf 130 is displayed on shelf 130 itself or near shelf 130. In this embodiment, shelf 130 does not have partitions such as doors, lids, or covers to separate it from the customer, allowing the customer to directly take out the product without opening a door. This lowers the physical and psychological barriers for the customer to pick up the product, making it easier for the customer to make a purchase and leading to increased sales for the seller. Note that this embodiment, described later, is still applicable even if doors, lids, or covers are provided. Also, this embodiment, described later, is still applicable even if store employees are present for sales promotion purposes.

[0015] As a sensor for acquiring information for detecting that a product displayed on shelf 130 has been taken out or returned, camera 140 is installed as a device included in the sales system of the present embodiment. Camera 140 is installed so as to have a shooting range in which the state of taking in and out of products from shelf 130 can be photographed by camera 140. Edge PC 100c is associated with shelf 130 and is connected to camera 140 by wired or wireless communication so as to be able to acquire the image photographed by camera 140. In the present embodiment, an example of processing by associating an edge PC and a product shelf on a one-to-one basis will be described, but a plurality of product shelves may be associated with and processed for one edge PC.

[0016] Edge PC 100c analyzes the captured image (video) transmitted from camera 140 and detects that a product has been taken out or returned from shelf 130. Then, information indicating that the product has been taken out or returned is transmitted to server device 100b.

[0017] Purchaser 120 acquires the shelf ID by photographing QR code 160 with a purchaser terminal such as a smartphone held by the purchaser. Then, the shelf ID and the identification information (user ID) of purchaser 120 are transmitted to server device 100b.

[0018] The server device 100b associates the user ID of buyer 120 with the shelf ID of shelf 130 based on the shelf ID and user ID received from the buyer terminal 100a (i.e., associates buyer 120 with shelf 130). Hereinafter, the process of associating (remembering) the user ID and shelf ID will be referred to as the check-in process, and the associated state will be referred to as the check-in state. Once the check-in process is complete, the server device 100b notifies the buyer terminal 100a and the edge PC 100b that the check-in is complete. At this point, buyer 120 is ready to take the product from shelf 130. When buyer 120 picks up the product from shelf 130, the edge PC 100c detects this based on the image captured by the camera 140 and sends the detection result to the server device 100b. The server identifies the retrieved item and adds it to the virtual shopping cart (hereinafter simply referred to as "cart") of the buyer 120, which is associated with the shelf 130. The status of the cart is transmitted from the server device 100b to the buyer terminal 100a and displayed on the buyer terminal 100a, so that the buyer 120 can confirm that the item they picked up is in the cart. When the user operates the buyer terminal 100a to perform the payment process, the server device 100b calls the internet payment service 170 to complete the payment. In this way, the buyer 120 can purchase items through a simple and easy process: scanning the QR code 160 with the buyer terminal 100a, picking up the item, and performing the payment operation.

[0019] Figure 2(a) shows an example of the configuration of a purchaser terminal 100a, which is an example of a device to which the present invention can be applied. The purchaser terminal 100a is an electronic device owned by the user who is the purchaser, and can be configured using a smartphone (mobile phone terminal), a tablet PC, or the like.

[0020] In Figure 2, the CPU 101a, memory 102a, non-volatile memory 103a, image processing unit 104a, display 105a, operation unit 106a, recording medium interface 107a, external interface 109a, and communication interface 110a are connected to the internal bus 150a. Each component connected to the internal bus 150a is configured to exchange data with each other via the internal bus 150a.

[0021] Memory 102a consists of, for example, RAM (volatile memory using semiconductor elements). CPU 101a controls various parts of the purchaser terminal 100a using memory 102a as work memory, according to a program stored in, for example, non-volatile memory 103a. Non-volatile memory 103a stores image data, audio data, other data, and various programs for the operation of CPU 101a. Non-volatile memory 103a consists of, for example, flash memory or ROM.

[0022] The image processing unit 104a performs various image processing operations on images stored in the non-volatile memory 103a or recording medium 108a, images acquired via the external I / F 109a or communication I / F 110a, and captured images, based on the control of the CPU 101a. The image processing operations performed by the image processing unit 104a include A / D conversion, D / A conversion, image data encoding, compression, decoding, resizing, noise reduction, and color conversion. The image processing unit 104a may be composed of dedicated circuit blocks for performing specific image processing operations. Furthermore, depending on the type of image processing, the CPU 101a may perform image processing according to a program without using the image processing unit 104a.

[0023] The display 105a displays images and GUI (Graphical User Interface) screens based on the control of the CPU 101a. The CPU 101a generates display control signals according to the program and controls various parts of the purchaser terminal 100a to generate video signals for display on the display 105a and output them to the display 105a. The display 105 displays images based on the output video signals. The purchaser terminal 100a itself is configured to have an interface for outputting video signals for display on the display 105, and the display 105a may be an external monitor (such as a television).

[0024] The operation unit 106a is an input device for receiving user input. The operation unit 1006a includes a character information input device such as a keyboard, a pointing device such as a mouse or touch panel 70a, buttons, dials, joysticks, touch sensors, and touchpads. The touch panel 70a is an input device that is superimposed on the display 105a to form a planar structure and outputs coordinate information corresponding to the position of contact.

[0025] The recording medium interface 107a is capable of receiving recording media 108a such as memory cards, and, based on the control of the CPU 101a, reads data from the received recording media 108a and writes data to the recording media 108a. The external interface 109a is an interface for connecting to external devices via wired cables or wirelessly and for inputting and outputting video and audio signals. The communication interface 110a is an interface for communicating with external devices and the Internet 111 to send and receive various data such as files and commands.

[0026] The camera unit 112a is a camera unit composed of an image sensor (imaging sensor) such as a CCD or CMOS element that converts optical images into electrical signals. The camera unit 112a includes a lens group (photographic lens) including a zoom lens and a focus lens, a shutter with an aperture function, an image sensor, an A / D converter that converts analog signals output from the image sensor into digital signals, and a barrier that covers the imaging system to prevent dirt and damage. The image processing unit 104a performs resizing processing such as predetermined pixel interpolation and reduction, as well as color conversion processing, on the data captured and acquired by the camera unit 112a. Based on the calculation results obtained by the image processing unit 104a, the CPU 101a performs exposure control, distance measurement control, and AWB (auto white balance) processing. The image data for display, captured by the camera unit 112a and processed by the image processing unit 104a, is displayed on the display 105a. The digital signal captured by the camera unit 112a, converted once to digital by the A / D converter and stored in the memory 102a, is converted to analog by the D / A converter and sequentially transferred to the display 105a for display, enabling live view display (LV display). Live view can be displayed in the still image shooting standby state, the video shooting standby state, and during video recording, and the captured subject image is displayed in near real time. The CPU 101a controls the camera unit 112a and the image processing unit 104a to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB processing in response to shooting preparation instructions based on user operations made on the operation unit 106a. In response to the shooting instruction, the CPU 101a controls the start of a series of shooting operations (main shooting), from the main exposure and reading the signal from the image sensor, to the image processing unit 104a processing the captured image to generate an image file and recording it on the recording medium 108a. Shooting instructions can be made by user operations on the operation unit 106a. The camera unit 112a is capable of capturing both still images and videos.

[0027] Figure 2(b) shows an example of the configuration of a server device 100b, which is an example of a device to which the present invention can be applied. The server device 100b is an information processing device that can communicate with the purchaser terminal 100a and the edge PC 100c, and can be configured using a personal computer (PC) or the like. The configuration of the server device 100b, indicated by reference numerals 101b to 104b, 106b to 110b, and 150b, is the same as the configuration of the purchaser terminal 100a indicated by reference numerals 101a to 104a, 106a to 110a, and 150a as described above. Therefore, a detailed explanation is omitted. In addition, although the server device 100b is described as not having a display and a camera unit, a device having a display and a camera unit may be used as the server device 100b.

[0028] Figure 2(c) shows an example configuration of an edge PC 100c, which is an example of a device to which the present invention can be applied. The edge PC 100c is a terminal device (information processing device) that can communicate with the server device 100b, and can be configured using a personal computer (PC) or the like. The configuration of the edge PC 100c, indicated by reference numerals 101c to 104c, 106c to 110c, and 150c, is the same as the configuration of the aforementioned purchaser terminal 100a, indicated by reference numerals 101a to 104a, 106a to 110a, and 150a. Therefore, a detailed explanation is omitted. Furthermore, the edge PC 100c will be described as not having a display and a camera unit. That is, it can be applied to a device that consists only of the desktop PC body, and since it can be configured without having at least one of the display and camera units, costs and installation space can be reduced. However, a device having a display and a camera unit may also be used as the edge PC 100c. The audio output unit 113c is an output unit that transmits an audio signal to a speaker via wired or wireless connection to produce sound, or it is the speaker itself. The notification sound or voice message output from the audio output unit 113c can notify (make known) the purchaser 120 of various guidance and warning information.

[0029] Figure 2(d) shows an example of the configuration of a signage device 100d as an example of a device to which the present invention can be applied. The signage device 100d is a terminal device (information processing device, electronic device) that can communicate with the edge PC 100c, and can be configured using a tablet PC or the like. The configuration of the signage device 100d, indicated by reference numerals 101d to 106d, 109d, 110d, and 150d, is the same as the configuration of the aforementioned purchaser terminal 100a, indicated by reference numerals 101a to 106a, 109a, 110a, and 150a. Therefore, a detailed explanation is omitted. The audio output unit 113d is an output unit that transmits an audio signal to a speaker via wired or wireless communication to produce sound, or the speaker itself. The display 105d is installed in a position visible to the purchaser 120 standing in front of the shelf 130, and the sound output from the audio output unit 113d is positioned so that it can be heard by the purchaser 120 standing in front of the shelf 130. In other words, the signage device 100d is installed in a position where customers near the display area can recognize the content output from the signage device 100d. The touch panel 70d (TP70d) is an input device that is superimposed on the display 105d to form a planar structure and outputs coordinate information corresponding to the position of contact. The voice input unit 114d is a sound collection unit for collecting words spoken by the purchaser 120 (customer), and is, for example, a microphone.

[0030] Figure 3 shows a flowchart of the sales system processing on the buyer terminal 100a. This processing is achieved by loading the program recorded in the non-volatile memory 103a into memory 102a and executing it on the CPU 101a. When a buyer 120 launches an application program capable of reading a QR code on the buyer terminal 100a, or launches an application program for this system (hereinafter simply referred to as the purchase app), the processing shown in Figure 3 begins in order to purchase an item from shelf 130.

[0031] In S301, the CPU 101a displays the LV image captured by the camera unit 112a on the QR code reading screen. Figure 8(a) shows an example of the QR code reading screen displayed on the display 105a (touch panel 70a) by the purchase application. Message 801 is a guidance message prompting the purchaser 120 to have the purchaser terminal 100a read the QR code 160 (frame the image so that the QR code 160 is within the shooting range). For example, it may say, "Please read the QR code posted on the product shelf." Frame 802 is a guide frame that serves as a reference for framing the image so that the QR code is within the shooting range. The cart icon 803 is a touch icon that accepts instructions to transition to the cart screen. The cart icon 803 is displayed when there are items in the cart corresponding to the purchaser 120's user ID, and not displayed otherwise. In other words, it is not displayed at initial startup. Although not shown in the diagram, the QR code reading screen displays the LV image captured by the camera unit 112a, and the message 801, frame 802, and cart icon 803 are superimposed on the LV image.

[0032] In S302, the CPU 101a refers to the cart information stored in the non-volatile memory 103a and determines whether or not there are items in the cart. If it determines that there are items in the cart, it proceeds to S303; otherwise (the cart is empty, or the initial state where cart information has not been generated), it proceeds to S304.

[0033] In S304, CPU 101a determines whether or not there was an instruction to transition to the cart screen. If there was an instruction to transition to the cart screen, i.e., a touch operation was performed on the cart icon 803, the process proceeds to S330; otherwise, the process proceeds to S304.

[0034] In S304, the CPU 101a determines whether the QR code was captured and whether the information contained in the QR code was obtained. If the QR code 160 is within the capture range because the purchaser 120 framed the purchaser terminal 100a, and the QR code 160 is obtained from the captured image as an LV image, and the information contained in the QR code 160 is obtained by analysis, the process proceeds to S310. Otherwise, the process proceeds to S305.

[0035] In S305, CPU 101a determines whether or not a termination event has occurred (for example, closing the QR code reading screen). If no termination event has occurred, the process returns to S302 and is repeated. If a termination event has occurred, the process shown in Figure 3 is terminated.

[0036] In S310, CPU 101a determines whether the information in the QR code acquired in S304 includes the shelf ID. If the information in the QR code does not include the shelf ID, it means that the captured QR code is unrelated to the sales system of this embodiment. If the information includes the shelf ID, the process proceeds to S312; otherwise, the process proceeds to S311 to display an error or to perform a different process (a process based on the QR code) from this system.

[0037] In S312, CPU 101a determines whether an operation to execute the process indicated by the information in the QR code acquired in S304 has occurred. For example, if the scanned QR code contains shelf ID information, the display 105a displays the message "Do you want to check in to shelf number 67890?" along with the options "Yes" and "No". "Shelf number 67890" is just an example; any identification information that allows the user to identify which shelf is being checked in to is acceptable. If "Yes" is selected, it is determined that an operation to execute the process indicated by the information in the QR code has occurred, and the process proceeds to S313. If "No" is selected, the process returns to S302. Note that QR code 160 may also contain information to launch a purchase application or access a web application on the internet. In that case, QR code 160 is read with a general-purpose QR code reader, and if a decision instruction is received, the process from S313 onwards is performed according to the purchase application or web application. Alternatively, the process in S312 may be omitted, and if it is determined that the read QR code contains the shelf ID (S310Yes), the process in S313 may be performed without waiting for an operation from the purchaser 120 (an operation to give a decision instruction).

[0038] In S313, the CPU 101a sends check-in request information to the server device 100b, which includes the shelf ID obtained from the scanned QR code and a user ID to identify the purchaser 120. The shelf ID is display location information to identify the shelf 130 where the product is displayed. In this embodiment, the user ID is the login ID for the purchase application (either one that is stored in advance or one that the purchaser 120 entered when the purchase application was launched). The user ID is not limited to this; other information may be used as long as it can distinguish the purchaser 120 who is checking in to shelf 130 from other users. For example, the SSID of the purchaser terminal 100a, account information for other linked services such as SNS, or login account information for the purchaser terminal 100a may be used as the user ID. In other words, the user ID is user information to identify the purchaser terminal 100a or the purchaser 120. When the buyer terminal 100a sends the shelf ID and user ID to the server device 100b, the server device 100b attempts to check in, and then sends back information indicating whether the check-in was successful.

[0039] In this embodiment, the method for obtaining the shelf ID on the purchaser terminal 100a is described as taking a picture of a two-dimensional code, a QR code 160, but this is not the only method. The shelf ID may also be obtained by taking a picture of a one-dimensional code such as a barcode, or by taking a picture of the shelf ID displayed as a string and obtaining it through character recognition from the image. Alternatively, the shelf ID may be obtained by the user inputting the shelf ID displayed as a string on the shelf ID input screen shown on the display 105a of the purchaser terminal 100a. Or, the shelf ID may be obtained based on the location information of the purchaser terminal 100a obtained from a location information acquisition unit (not shown), such as a GPS positioning unit of the purchaser terminal 100a. The shelf ID may also be obtained by combining at least two of the following: information obtained from the captured image, information obtained through user input, and location information.

[0040] In S314, CPU 101a determines whether or not a check-in failure notification has been received from server device 100b indicating that check-in was not possible. For example, if a user other than buyer 120 has already checked into shelf 130, buyer 120 will not be able to check into shelf 130. If a check-in failure notification is received, proceed to S315; otherwise, proceed to 316.

[0041] In S315, the CPU 101a displays check-in error information on the display 105a indicating that check-in is currently unavailable. For example, it displays a message such as, "You cannot check in because someone else is using it. Please check in after the previous user has finished using it." In other words, it displays information indicating that another user is checking in, and / or information prompting the user to send a check-in request again after some time. If the shelf 130 had a door, it would be possible to determine whether another buyer is in the middle of a purchase by the open / closed state of the door (i.e., if another person has the door open, it is clear that that person has priority), but in this embodiment, there is no door. Therefore, it can be difficult to tell from the appearance of the shelf 130 whether there are other users. For example, even if another person is near the shelf 130, it may be impossible to determine whether that person has not decided whether to purchase, is there for a different reason, or is in the middle of a purchase. However, as in this embodiment, by displaying a check-in error, the buyer 120 can recognize that they should wait because another user is in the middle of a purchase. Furthermore, as in this embodiment, by preventing other users from checking in when one purchaser has already checked in, it is possible to prevent confusion of purchasers by making it impossible to determine which user took the product from shelf 130.

[0042] In S316, CPU 101a determines whether or not it has received a check-in completion notification from server device 100b indicating that the check-in was successful. If a check-in completion notification is received, proceed to S317; otherwise, return to S314.

[0043] In S317, CPU101a determines whether or not there are items in the cart, similar to S302. If there are items in the cart, proceed to S330; otherwise, proceed to S320.

[0044] In S320, CPU 101a displays an empty cart on display 105a. An example of the empty cart display is shown in Figure 8(b).

[0045] Display item 810 is the screen title portion indicating that the currently displayed screen is the cart display screen, which shows whether there are items in the cart. In this embodiment, it displays "Items in cart".

[0046] Guidance 811 is a guide display that prompts the user, the buyer 120, to take the product from the shelf 130. In the example in Figure 8(b), the string of characters (message) "Please take the product from the shelf" is displayed. In the sales system of this embodiment, the product shelf 130 does not have a door or cover. Therefore, it is difficult for the buyer 120 (user) to determine from the appearance whether or not it is OK to take the product from the shelf 130. Especially in an unmanned store, there is a possibility that the buyer may hesitate to take the product without permission. In contrast, by displaying guidance 811, the buyer 120 can recognize that it is OK to take the product from the shelf 130, and can take the product and shop with peace of mind. Furthermore, by displaying guidance 811 on the display 105a of the buyer terminal 100a, it is possible to notify the buyer that it is OK to take the product without relying on voice notification from the voice output unit 113c of the edge PC 100c. Therefore, even in environments where voice notification is difficult (environments that should be quiet, or conversely, environments with a lot of noise), it is possible to reliably notify the buyer 120 that it is OK to take the product. Furthermore, since there is no need to connect a display to the edge PC 100c or install a notification monitor near the shelf 130, installation costs and installation space increases are kept to a minimum, and the purchaser 120 can be reliably notified that the product is ready to be taken. In other words, the sales system of this embodiment becomes easier for users to use when purchasing products. Note that the guidance 811 is not limited to the illustrated example as long as it prompts the purchaser 120 to take the product from the shelf 130. For example, it could be a message with a different string of characters, an icon display, an animation display, video playback, or guidance via voice output from the purchaser terminal 100a.

[0047] Display item 812 shows the total price of the items in the cart. In the example in Figure 8(b), since there are no items in the cart, it displays a price of 0 yen.

[0048] The purchase instruction icon 813 is a touch icon (display item) used to instruct the user to purchase items in their cart and proceed with payment. In the example in Figure 8(b), there are no items in the cart and nothing can be purchased, so it is grayed out (inactive) and does not accept any input when touched. When items are added to the cart, the graying is removed, the icon becomes active, and it becomes possible to input via touch.

[0049] The Cancel Instruction icon 814 is a touch icon (display item) that accepts a command to cancel a purchase action (shopping).

[0050] Shelf Move Item 815 is a touch icon (display item) used to instruct the user to check into a different shelf than the one they are currently checking into.

[0051] In S321, CPU 101a determines whether or not a cancellation operation (touching the cancellation instruction icon 814) has occurred. If a cancellation instruction has occurred, the process proceeds to S322; otherwise, it proceeds to S325.

[0052] In S322, the CPU 101a sends a abort instruction to the server device 100b indicating that an abort operation has been performed.

[0053] In S323, CPU 101a determines whether or not it has received a checkout completion notification from server device 100b. If it has received a checkout completion notification, it proceeds to S324; otherwise, it waits in S323 for the checkout completion notification to be received.

[0054] In S324, the CPU 101a clears (erases) the cart information held in memory 102a and displays a checkout message on the display 105a. It also erases (clears) the cart information held in memory 102a. Figure 8(d) shows an example of the checkout display. The checkout display shows a message 830 such as "Thank you for using our service" to indicate that the purchase process has been completed. After the checkout display is shown for a predetermined time (for example, 5 seconds), the system returns to the LV display (QR code reading screen) in S301.

[0055] In S325, it is determined whether or not a shelf movement operation (touching shelf movement item 815) has occurred. If a shelf movement operation has occurred, proceed to S326; otherwise, proceed to S328.

[0056] In S326, the CPU 101a sends a shelf movement instruction to the server device 100b, indicating that a shelf movement operation has been performed.

[0057] In S327, CPU 101a determines whether or not it has received a checkout completion notification from server device 100b. If it has received a checkout completion notification, it returns to the LV display (QR code reading screen) in S301 to read the QR codes of other shelves. Otherwise, it waits in S323 for a checkout completion notification to be received.

[0058] In S328, the CPU 101a determines whether or not it has received a cart information update notification from the server device 100b. If an update notification is received, it stores the information indicating the latest cart status included in the received update notification in memory 102a (i.e., updates the cart information stored in memory 102a), and proceeds to S317. If no update notification is received, it proceeds to S321.

[0059] In this embodiment, an example was described in which touch operations on the purchase instruction icon 813 are not accepted (ignored) when an empty cart is displayed, but this is not the only example. For example, if a touch operation on the purchase instruction icon 813 is detected while an empty cart is displayed, a display prompting the user to take the product from the shelf again (a second display different from the display in S320) may be shown. For example, the guidance 811 may be highlighted in a different display format than S320, such as by flashing or changing the color, or a display prompting the user to take the product from the shelf may be shown with a display different from the guidance 811. In particular, it is good to display that it is OK to take the product from the shelf before payment. If the purchase instruction icon 813 is operated when there are no products in the cart, that is, when no products have been taken out, the buyer 120 may mistakenly believe that they cannot take the product from the shelf until payment is made. Therefore, by displaying a display prompting the user to take the product from the shelf again, the user's misunderstanding can be resolved and a smooth purchase operation can be performed.

[0060] In S330, CPU 101a displays the shopping cart information when there are items in the display unit. An example of the shopping cart display with items in the display unit is shown in Figure 8(c). The same reference numerals are used for the same display items as in Figure 8(b), and their explanations are omitted.

[0061] The product list 820 is information that includes product information contained in the update notification received from the server device 100b, and is a list of products in the cart based on the latest cart status information held in memory 102a. Each row of the product list 820 represents information for one product and includes columns 821 to 824. Column 821 is a display item that shows the appearance of the product in each row, and displays a photo of the product that was taken out, or an icon that represents the type of product that was taken out (for example, an icon resembling a plastic bottle for beverages, or an icon resembling food for food). When the type of product is represented by a photo, it displays an image that was previously stored in the non-volatile memory 103a of the buyer terminal 100a as data from the purchase application, or an image received from the server device 100b. Column 822 represents the name (product name) of the product in each row, and column 823 represents the price of the product in each row. Column 824 is a display item for accepting modification instructions for the product in each row. In the example in Figure 8(c), the purchase instruction icon 813 is displayed in a touch-enabled format (active display) because there is an item in the cart and it is available for purchase. On the other hand, the cancel instruction icon 814 is displayed in a grayed-out state. This is because, in this case, there is an item in the cart, and the purchase cannot be canceled unless the item in the cart is returned to shelf 130. However, in this embodiment, touch operation to the cancel instruction icon 814 is accepted even when it is grayed out. However, this is not the only option, and touch operation to the cancel instruction icon 814 may be disabled if there is an item in the cart.

[0062] In S331, CPU 101a determines whether or not a correction operation has occurred. A correction operation is an operation in which a display item shown in column 824 of any row is touched, followed by completion of operations on the correction content input screen (a screen for correcting the product in the touched row), and confirmation of the correction content. The correction content input screen is, for example, a screen that displays a list of products currently for sale on shelf 130 and accepts the selection of any product from the list. By accepting product correction operations in this way, if the product determined by product identification in image analysis happens to be different from the product actually picked up, the purchaser 120 can correct it to the correct content.

[0063] In S332, the CPU 101a sends information indicating the corrected product, which is the corrected information entered on the correction input screen, to the server device 100b.

[0064] In S333, CPU 101a waits for an update notification from server device 100b. When an update notification is received, it updates the cart information held in memory 102a based on the notification and proceeds to S317. As a result, the cart display on display 105a also reflects the correction.

[0065] In S334, CPU 101a determines whether or not a cancel operation (touching the cancel instruction icon 814) has occurred. If a cancel instruction has occurred, the process proceeds to S335; otherwise, it proceeds to S340.

[0066] In S335, the CPU 101a sends a abort instruction to the server device 100b indicating that an abort operation has been performed.

[0067] In S336, the CPU 101a displays a return message on the display 105a. The return message prompts the buyer 120 to return the item they took from shelf 130 to shelf 130, and displays a string of characters (message) such as "Please return the item to its original position." This allows the buyer 120 to realize that they cannot cancel their purchase unless they return the item they took to shelf 130, and they attempt to return the item they are holding to shelf 130. As a result, when the item in the cart (the item taken by the buyer 120) is returned to shelf 130, the camera 140 detects that the item has been returned from the video footage of the return, and information that the item has been returned is sent from the edge PC 100c to the server device 100b. Upon receiving the information that the item has been returned, the server device 100b updates the cart information it holds and sends an update notification to the buyer terminal 100a.

[0068] In S337, CPU 101a determines whether or not it has received an update notification from server device 100b. If an update notification is received, it updates the cart information in memory 102a with the latest cart information contained in the received update notification and proceeds to S338. If an update notification cannot be received, it waits for an update notification to be received in S337 and does not proceed to the checkout display.

[0069] In S338, CPU 101a determines whether there are items in the cart based on the latest cart information updated based on the update notification received in S337. If there are no items in the cart, it means that all the items that buyer 120 took from shelf 130 have been returned to shelf 130. If there are no items in the cart, proceed to S339; otherwise (if there are still items in the cart), proceed to S336.

[0070] Here, if no update notification is received in S337, or if it is determined in S338 that there are items in the cart, it means that the buyer 120 has not yet returned the items they took from shelf 130 to shelf 130. Therefore, if no update notification is received in S337 or if it is determined in S338 that there are items in the cart after a cancellation instruction is given in S334, the return message displayed in S336 will continue to be displayed to encourage the buyer 120 to return the items for sure. Furthermore, if no update notification is received in S337 or if it is determined in S338 that there are items in the cart after the return message has been displayed, the return message may be emphasized in stages. For example, the return message may be made to flash after a predetermined time has elapsed, or a warning sound may be emitted.

[0071] In S339, CPU 101a displays the checkout information, similar to S324. It also clears the cart information stored in memory 102a. This allows the buyer 120 to correctly recognize that they have returned the retrieved items to shelf 130, that the purchase has been successfully canceled, and that there is no problem in leaving without making a payment.

[0072] In S340, CPU 101a determines whether a purchase instruction operation (touching the purchase instruction icon 813) has occurred. A purchase instruction operation is an operation to make a payment and is a type of payment operation. If a purchase instruction operation has occurred, proceed to S341; otherwise, proceed to S344.

[0073] In S341, the CPU 101a sends a purchase instruction notification to the server device 100b indicating that the purchase instruction operation has been performed.

[0074] In S342, the CPU 101a performs payment processing in conjunction with the server device 100b. For example, a payment screen for credit card payments or an electronic money payment screen is displayed, and payment (settlement, billing) is made for the items in the cart according to the user's actions on these payment screens (payment operations on the touch panel 70a). Once payment is complete, the process proceeds to S339. The checkout display in S339 allows the buyer 120 to recognize that payment has been successfully completed and that they may leave the area with the items they have taken from the shelf 130 (purchased items).

[0075] In S344, CPU 101a determines whether or not a shelf movement operation (touching shelf movement item 815) has occurred. If a shelf movement operation has occurred, the process proceeds to S326; otherwise, it proceeds to S345.

[0076] In S345, the CPU 101a determines whether or not it has received a cart information update notification from the server device 100b. If an update notification is received, it stores the information indicating the latest cart status included in the received update notification in memory 102a (i.e., updates the cart information stored in memory 102a), and proceeds to S317. If no update notification is received, it proceeds to S331.

[0077] Figure 4 shows a flowchart of the processing related to the sales system in the server device 100b. This processing is achieved by loading the program recorded in the non-volatile memory 103a into memory 102a and executing it on the CPU 101a. This processing is started manually at the timing when sales by the sales system in this embodiment begin (such as the start time of business hours), or it is started automatically when a predetermined time (such as the time of disclosure of business hours) arrives.

[0078] In S401, CPU 101b initializes various variables and information (cart information) held in memory 102b. Non-volatile memory 103b stores the IDs of multiple product shelves managed by server device 100b. All IDs of these managed product shelves are read, check-in status management information is created in memory 102b, and then the status of all shelves is set to "not checked in." An example of check-in status management information is shown in Figure 5(a). The management information stores the ID of the managed shelf, its check-in status, and, if checked in, the ID of the user who checked in (CI). In S401, the check-in status is set to "not checked in" for all shelves, and there are no CI users for all shelves.

[0079] In S402, CPU 101b determines whether or not it has received the shelf ID transmitted from the buyer terminal 100a. This is the same determination made in S313 of Figure 3 above, regarding whether or not it has received the shelf ID and user ID transmitted from the buyer terminal 100a. If the shelf ID has been received, the process proceeds to S402; otherwise, it proceeds to S407.

[0080] In S403, the system refers to the check-in status management information stored in memory 102b and determines whether the shelf with the received shelf ID is already being checked in (is in a checked-in state). If it is in a checked-in state, the system proceeds to S404; otherwise (it is not in a checked-in state), the system proceeds to S405.

[0081] In S404, a check-in failure notification is sent to the purchaser terminal 100a, the source of the shelf ID received in S402, indicating that check-in is not possible. The check-in status management information held in memory 102b is not updated. In other words, in this case, no check-in process is performed for the received shelf ID. The check-in failure notification sent in S404 is received in S314 in Figure 3, and a check-in error is displayed in S315.

[0082] In S405, CPU 101b performs the check-in process. Specifically, in the management information of the check-in status held in memory 102b, for the shelf with the shelf ID received in S402, the check-in status is set to "checked in," and the user ID received along with the shelf ID is associated with the CI user and stored. In other words, the check-in process is an association process that associates the shelf ID (display location information) with the user ID (user information). An example of the management information in this case is shown in Figure 5(a). For the shelf with shelf ID "678 user ID 90," the check-in status is set to "checked in," and the CI user ID "12345" is associated with it and stored.

[0083] Furthermore, in S405, if the received CI user's cart information has not been generated, the CI user's cart information is generated in memory 102b. Figure 5(b) shows an example of cart information for user ID "12345". Cart information is generated for each user ID. The cart information stores the user ID, the shelf ID of the item checked in, the product name (or product identification information), price, quantity, subtotal, the shelf ID from which the item was taken (items in the cart), and the total price of all items in the cart, all associated with each other. The example in Figure 5(a) shows the state immediately after a user checks in to shelf ID "67890", and there are no items in the cart.

[0084] Note that if the cart information for the corresponding user ID is already stored in memory 102b among the shelf ID and user ID received in S402, new cart information will not be generated. In this case, the shelf ID currently checked in in the existing cart information for the corresponding user ID will be updated with the received shelf ID. This case is when there is a shelf move instruction from the first shelf, and after checking out from the first shelf without payment, the user checks in to another shelf (the second shelf). After checking in to the second shelf, if an item is taken from the second shelf, the taken item will be added to the cart. Figure 5(c) shows the cart information when items A and B are taken from shelf ID "67890" (the first shelf), a shelf move instruction is given, and the user checks in to shelf ID "67891" (the second shelf) and item C is taken. In addition to items A and B taken from the first shelf, item C taken from the second shelf is also recorded, and the total price is the sum of items A, B, and C.

[0085] In S406, the CPU 101b sends a check-in completion notification to both the purchaser terminal 100a and the edge PC 100c, indicating that check-in has been completed. The check-in completion notification sent here is received by the purchaser terminal 100a at S316 in Figure 3 above, and by the edge PC 100c at S603 in Figure 6, which will be described later.

[0086] In S407, CPU101b determines whether it has received the shelf ID and product recognition score transmitted from edge PC100c, as well as whether it has received a retrieval notification or a return notification. If it has received these notifications, it proceeds to S408; otherwise, it proceeds to S411.

[0087] In S408, the CPU 101b determines what product was picked up / returned from the shelf based on the product recognition score received from the edge PC 100c. In this embodiment, the final product determination is performed on the server device 100b rather than the edge PC 100c. This has the effect of allowing product determination to be performed simply by reflecting information on the server device 100b without using advanced or cutting-edge determination methods such as machine learning or reflecting new product information on the edge PC. In other words, it is possible to reduce the increased installation costs associated with the advanced functionality of the edge PC 100c and the management burden such as updating the edge PC 100c. The product recognition score received from the edge PC may be information that has been estimated to some extent (inference score), or it may be information such as shape, color, and size parameters before the product is estimated. By receiving a product recognition score that has been parameterized to some extent, rather than the image itself captured by the camera 140, the amount of communication between the edge PC 100c and the server device 100b is suppressed, enabling high-speed determination and response.

[0088] It should be noted that this is not limited to sending a product recognition score from the edge PC 100c to the server device 100b. For example, the edge PC 100c may not perform any processing to determine what the product is, but instead send the image itself (video or still image) captured by the camera 140 in the scene where it is presumed that a product was taken out or returned. In this case, the server device 100b will determine the product based on the image captured by the received camera 140. Conversely, the edge PC 100c may determine that a product was taken out or returned and what the product in question is, and send only the determination result to the server device 100b. In this case, processing S408 is omitted.

[0089] In S409, CPU 101b updates the cart information held in memory 102b based on the product determined in S408. That is, if S408 determines that the product has been removed from the shelf, the information of the product determined in S408 is added to the cart information of the user ID associated with the received shelf ID. Conversely, if S408 determines that the product has been returned to the shelf, the product determined in S408 is deleted from the cart information of the user currently checking in, specifically from the cart information where the received shelf ID is recorded as the source of removal. Figure 5(b) shows an example of cart information when, starting from a state where product A is originally in the cart information, S407 receives the shelf ID "67890" and product recognition score, and S408 determines that product B has been removed from the shelf. Information of product B has been added in addition to product A.

[0090] In S410, the CPU 101b sends an update notification to the buyer terminal 100a indicating that the cart information has been updated. The update notification also includes the latest cart information stored in memory 102b. That is, if an item is taken from the product shelf 130, the cart information including the information of the retrieved item, as determined in S408, is also sent. After processing in S410, the process proceeds to S430.

[0091] In the S430, CPU 101b performs signage control processing. Details of the signage control processing will be described later using Figure 7.

[0092] In S411, the CPU 101b determines whether or not it has received a correction instruction (the correction content sent from the purchaser terminal 100a in S332 of Figure 3 above). If it determines that a correction instruction has been received, it proceeds to S439 and updates the cart information held in memory 102b based on the correction content received as a correction instruction. Then, in S440, it sends an update notification to the purchaser terminal 100a indicating that the cart information has been updated. If no correction instruction has been received, it proceeds to S412.

[0093] In S412, CPU 101b determines whether or not it has received a cancellation instruction sent from the purchaser terminal 100a. If a cancellation instruction has been received, it proceeds to S413; otherwise, it proceeds to S417.

[0094] In S413, CPU101b determines whether or not there are items in the shopping cart information of the user ID of the user who issued the cancellation instruction in S412. If there are items, proceed to S414; otherwise, proceed to S419.

[0095] The processing in S414-S416 is the same as the processing in S408-S410. As a result, when the product is removed from the cart of the user who issued the cancellation order, S413 will be determined as No, and the cancellation process with checkout (processing in S419-S422) will be performed. As long as there are still products in the cart of the user who issued the cancellation order, S413 will not be determined as No, and therefore the cancellation process with checkout will not be performed.

[0096] In S417, the CPU 101b determines whether or not it has received the payment instruction sent from the purchaser terminal 100a (the payment instruction sent from the purchaser terminal 100a in S341 of Figure 3 above). If it determines that the payment instruction has been received, it proceeds to S418; otherwise, it proceeds to S423.

[0097] In S418, CPU 101b calls payment service 170 and performs payment processing based on payment service 170.

[0098] In S419, CPU 101b clears (deletes) the cart information held in memory 102b for the user ID for which a cancellation instruction was received in S412 or for the user ID for which a payment instruction was received in S417.

[0099] In S420, CPU101b performs the checkout process. Specifically, CPU101b updates the status of the product shelves checked in by the user ID that received the cancellation instruction in S412 or the user ID that received the payment instruction in S417 to the checkout state. That is, for the corresponding shelf ID in the check-in status management information (Figure 5(a)), the status is set to checkout state and the CI user is deleted. This deletes the association between the shelf and the user (purchaser).

[0100] In S421, CPU101b sends a checkout completion notification to the buyer terminal 100a and edge PC 100c indicating that checkout has been completed.

[0101] In S422, the CPU 101b determines whether or not it has received a shelf relocation instruction sent from the purchaser terminal 100a (the shelf relocation instruction sent from the purchaser terminal 100a in S328 of Figure 3 above). If it determines that it has received a shelf relocation instruction, it proceeds to S423; otherwise, it proceeds to S424.

[0102] In S423, CPU101b performs the checkout process, similar to S420 mentioned above.

[0103] In S424, CPU101b determines whether or not there has been a termination event for the process shown in Figure 4. If there is a termination event such as the end of business hours, a termination instruction from the user, or power off, the process in Figure 4 is terminated. If there is no termination event, the process proceeds to S402 and is repeated.

[0104] Figure 6 shows a flowchart of the processing related to the sales system on the edge PC 100c. This processing is achieved by loading the program recorded in the non-volatile memory 103c into memory 102c, and then executing it on the CPU 101c. This processing is started manually at the timing when sales by the sales system in this embodiment begin (such as the start time of business hours), or it is started automatically when a predetermined time (such as the time of disclosure of business hours) arrives. Figure 6 shows the processing related to an individual shelf, and in this embodiment, it will be explained as the processing related to shelf 130 (shelf ID: 67890).

[0105] In S601, the CPU 101c stores the state information stored in memory 102c or non-volatile memory 103c as the state of shelf 130, setting it to information corresponding to non-check-in. That is, it sets it to a non-check-in state.

[0106] In S602, CPU101c acquires video from camera140. The video acquired here is assumed to be a video showing the product dispensing area of ​​shelf130. If a sensor other than camera140 is used to detect product dispensing / returning, the process will be to obtain the output value from that sensor.

[0107] In S603, the CPU 101c determines whether or not it has received a check-in completion notification sent from the server device 100b. If it has received the check-in completion notification, it proceeds to S604; otherwise, it proceeds to S606.

[0108] In the S604, the CPU 101c stores and manages the state information of shelf 130, which is stored in memory 102c or non-volatile memory 103c, as information corresponding to the check-in. That is, it sets it to the check-in state (a state in which shelf 130 is associated with a prospective buyer).

[0109] In S605, the CPU 101c outputs a voice message from the voice output unit 113c indicating that it is now OK to take the product from shelf 130. For example, it may play a sound effect such as "ping" or a voice message such as "Welcome" to notify the user that they have been able to check in to shelf 130 (shelf 130 has responded) after scanning the QR code 160 with the purchaser terminal 100a. Alternatively, it may play a voice message (voice guidance) prompting the purchaser 120 to take the product from the shelf, such as "Please take the product from the shelf" or "After taking the product from the shelf, please make the payment with your terminal," or a voice message explaining how to shop. In other words, it notifies the user that their status has changed from a non-checked-in state to a checked-in state, and / or prompts them to take the product from shelf 130.

[0110] In S606, the CPU 101c determines, based on the video footage acquired from the camera 140, whether an item has been taken from the shelf 130 or returned to the shelf 130. That is, it detects whether an item has been taken from its display location based on the image of the shelf 130 captured by the camera 140. For example, based on the image, it detects that an item has been taken if a hand enters from a reference position in front of the shelf 130, the subject of the area where the item was placed changes, and the hand leaves the reference position. If an item has been taken out or returned, the process proceeds to S607; otherwise, it proceeds to S610. In this embodiment, an example of detecting the removal / return of products based on images captured by camera 140 has been described, but the detection method is not limited to this. For example, a weight sensor that detects the weight of items displayed on a shelf, such as a shelf board of shelf 130, may be attached to the display location, and the removal / return of products may be detected by the change in weight based on the output from the weight sensor. Alternatively, an RFID (radio frequency identifier) ​​tag, such as an IC tag, may be attached to each product, and the removal / return of products may be detected by wireless communication with this tag. Furthermore, an infrared sensor may be provided to detect when a product moves beyond the reference line in front of shelf 130 (the dispensing opening), when a product disappears from its previous location, or when a product is placed in a location where there was no product before. The removal / return of products may be detected based on the output from the infrared sensor indicating such detection. One of the multiple detection methods may be adopted, or multiple methods may be combined to detect the removal / return of products.

[0111] In S607, the CPU 101c refers to the state information stored in memory 102c or non-volatile memory 103c as the state of shelf 130 and determines whether it is in a checked-in state or not. If it is in a checked-in state, proceed to S608; otherwise, proceed to S609.

[0112] In S608, the CPU 101c transmits to the server device 100b the shelf ID of the managed shelf 130 (67890 in this embodiment), the fact that an item was taken out or returned in S606 (take-out notification or return notification), and the product recognition score of the item for which the take-out or return phenomenon was detected. In other words, it transmits the detection details regarding the detection of item take-out / return. The product recognition score is information output as a result of inference processing, for example, by inputting an image captured by the camera 140 into a trained model trained by machine learning. The product recognition score can also be said to be the result of scoring candidate products by inferring the product from its appearance and shape based on the image captured by the camera 140.

[0113] In S609, the CPU 101c issues an error notification (error output) via voice from the voice output unit 113c. In this case, the edge PC 100c does not send the product score to the server device 100b. By issuing an error notification without communicating with the server device 100b, the error notification can be issued more quickly, allowing the buyer 120 to recognize the action as an error sooner. In this case, the product score may also be sent to the server device 100b to manage what product was taken out without checking it in, but the timing of the transmission is after the error notification. That is, at the time of outputting the error notification, the error notification is output without sending the detection result to the server device 100b. Furthermore, the trigger for outputting the error notification may not be the detection of a product being taken out as described in S606, but rather, for example, the detection of a hand entering the product shelf 130. By outputting an error notification with this trigger, the buyer 120 can be recognized as an errorful action at an earlier stage, potentially preventing the product from being taken out. This phenomenon is one in which an item may be removed, and it is a preparatory action for removing an item. Thus, by detecting the preparatory action for removing an item as a phenomenon in which an item is removed, the same processing as when an item has been removed is detected. In this case, the object of detection is the phenomenon of an item being removed, and upon detection of this phenomenon, it is processed as if an item has been removed. The same applies to returns; not only the phenomenon of an item being returned, but also the preparatory action for the return of an item as a phenomenon in which an item is returned is detected, and the same processing as when an item has been returned is detected. In this case, the object of detection is the phenomenon of an item being returned, and upon detection of this phenomenon, it is processed as if an item has been returned.

[0114] The error notification is automatically canceled when the item taken without being checked in is returned to shelf 130, or when a predetermined time (e.g., 1 minute) has elapsed. Even if the taken item is not returned, the error notification is output for a predetermined time and then automatically canceled. This prevents unauthorized operation from causing the error notification to persist for an extended period on an unattended shelf 130, thus preventing other customers from missing out on purchasing opportunities.

[0115] Error notifications are, for example, voice messages or sound effects that prompt action to check in, such as "Please check in before taking the product" or "Please scan the QR code displayed on the shelf with your device before taking the product." Alternatively, messages such as "Please return the product to the shelf and then scan the QR code on the shelf" may be used to prompt the user to return the product to its original position. Error notifications are output as sounds that can be heard by the person who has taken or is attempting to take a product from the product shelf 130. This error notification allows the purchaser 120 (prospective purchaser) to realize if they have taken a product when it is not in a condition to be taken and to return it. In other words, the sales system of this embodiment can be used appropriately.

[0116] In S610, the CPU 101c determines whether or not it has received a checkout completion notification sent from the server device 100b. If a checkout completion notification has been received, the process proceeds to S611; otherwise, it proceeds to S613.

[0117] In the S611, the CPU 101c stores and manages the status information of shelf 130 in memory 102c or non-volatile memory 103c, setting it to information corresponding to non-check-in. That is, it checks out and sets it to non-check-in (a state where shelf 130 is not associated with a prospective buyer).

[0118] In the S612, the CPU 101c outputs audio from the audio output unit 113c in conjunction with checkout from the shelf 130. For example, it may play a sound effect such as "Piro-rin" or an audio message such as "Thank you for using our service."

[0119] In S613, the CPU 101c determines whether or not it has received a signage control instruction from the server device 100b. If a signage control instruction has been received, the process proceeds to S614; otherwise, it proceeds to S615. The signage control instruction received here consists of various response instructions and various response contents (advertising data including image and audio data) output (transmitted) in S708, S709, or S711 of Figure 7, which will be described later. In S614, the CPU 101c outputs a response to the signage device 100d corresponding to the instruction received in S614, and controls the signage device 100d to output a response. That is, it outputs data (image and audio) to the signage device 100d via the external I / F 109c to output the response instructed by the server device 100b.

[0120] In S615, CPU101c determines whether or not there has been a termination event for the process shown in Figure 6. If there is a termination event such as the end of business hours, a termination instruction from the user, or power off, the process in Figure 6 is terminated. If there is no termination event, the process proceeds to S02 and is repeated.

[0121] Figure 7 shows a detailed flowchart of the signage control process described in S430 of Figure 4. This process is executed in response to the detection of a phenomenon indicating the removal of an item from shelf 130 and the identification of the item to be removed, or in response to the detection of a phenomenon indicating the return of an item to shelf 130 and the identification of the item to be returned.

[0122] In S701, the system control unit 50 determines whether a phenomenon indicating the removal of an item or the return of an item has been detected. More specifically, it determines whether the notification received in S407 was an removal notification or a return notification. If it was an removal notification, it proceeds to S702; if it was a return notification, it proceeds to S710.

[0123] In S702, the system control unit 50 resets and starts the timer T. Timer T is, for example, 6 seconds. Until this timer T expires, the retrieval response regarding the retrieved product (which is output in S708 or S709) described later will not be output. For customers who are not hesitant about purchasing, it may be annoying to have an advertisement displayed every time they retrieve an item, so the retrieval response described later will not be displayed until 6 seconds have passed, which is assumed to indicate that the customer is still undecided about purchasing. However, to the extent that it is not annoying, information such as what the retrieved product is may be displayed on the display 105d before the timer T expires. Alternatively, instead of setting up S702 to S705, the processing from S706 onwards may be performed immediately after S701 determines that a retrieval has occurred, so that a response is given as soon as the item is retrieved.

[0124] In S703, the system control unit 50 determines whether it has received the shelf ID, product recognition score, and retrieval or return notification transmitted from the edge PC 100c. If it has received them, it proceeds to S704; otherwise, it proceeds to S705. In S704, the system control unit 50 stops the timer T and proceeds to S408 in Figure 4 above.

[0125] In S705, the system control unit 50 determines whether or not timer T has expired. If timer T has not expired, the process proceeds to S703; if timer T has expired, the process proceeds to S706. In this way, for example, if product B is taken out immediately after product A is taken out, S703 determines Yes before timer T, which started in response to the taking out of product A, expires, and the process in S706 is not performed. In other words, the response to taking out product A (the response output in S708 or S709) is not output from the signage device 100d. In this case, since the buyer 120 takes out product B immediately after taking out product A, it is assumed that the buyer intends to purchase product A without hesitation. And in this case, since the advertisement based on the response to taking out product A is not displayed, the possibility that the buyer 120 will find it bothersome can be reduced.

[0126] In S706, the system control unit 50 retrieves advertising information for the product to be retrieved (the retrieved product) from the non-volatile memory 103b or from a storage means that stores advertising information and is communicable to an external server. The storage means that stores advertising information, for example, stores a database of advertising information, and information such as limited-time campaign information, new product and discontinued product information, and store-specific inventory and prices are updated and stored therein.

[0127] In S707, the system control unit 50 determines whether the advertising information obtained as a result of the search in S706 contains a video. If a video is included, the process proceeds to S708; otherwise, it proceeds to S709.

[0128] In S708, the system control unit 50 controls the signage device 100d to output a video-accompanied retrieval response related to the product to be retrieved, based on the advertising information obtained as a result of the search in S706. More specifically, it sends an instruction to output a video-accompanied retrieval response (signage control instruction) and the content of the video-accompanied retrieval response (video, still images, product information, related product information, price information, campaign information, etc.) to the edge PC 100c.

[0129] Figure 9(a) shows an example of displaying a video-accompanied pick-up response on the display 105d of the signage device 100d. In area 901, the video advertisement obtained as a result of the search in S706 is played and displayed. The audio of the video being played is output from the audio output unit 113d of the signage device 100d. Since the video takes time to watch, it may not be possible for customers moving around the store to stop and watch it. However, users who have picked up and are holding a product are more likely to be standing still and are likely to be interested in that product. Therefore, a video played at the moment a product is picked up from the shelf is more likely to be watched by the purchaser 120 and is an effective advertisement. In area 902, various product information about the product that was picked up is displayed. The product information displayed in area 902 shall include at least one of the following pieces of information: product name, product image (video or still image), product regular price, product sale price (discounted price from the regular price), and currently applicable campaign information. Area 902 displays product descriptions, selling points, features, efficacy, effects, usage instructions, recommended usage scenarios, ingredients, designated ingredients, allergy-related information, and halal information (information regarding whether or not it contains haram). Furthermore, electronic coupons (discount coupons, point redemption coupons) related to the product being retrieved may also be displayed. The product information displayed in area 902 includes at least one piece of more detailed information not displayed on the product's packaging, or information that is difficult to read due to small print on the product's packaging. Area 903 displays related products that are different from the product being retrieved. It also displays discount information for purchasing related products together with the product being retrieved. Furthermore, area 904 displays the current location corresponding to the shelf location of the product being retrieved, and the sales area (display shelf) of the related products displayed in area 903. The example shown is a store layout (map), but guidance using text such as shelf numbers and which shelf it is on may also be used. Displaying area 904 saves customers the trouble of searching for related products, reducing factors (psychological barriers) that discourage them from purchasing related products. Therefore, it contributes to increased sales of related products.

[0130] In S709, the system control unit 50 controls the signage device 100d to output a non-video retrieval response for the product targeted for retrieval, based on the advertising information obtained as a result of the search in S706. More specifically, it sends an instruction to output a non-video retrieval response (signage control instruction) and the content of the non-video retrieval response (still image, product information, related product information, price information, campaign information, etc.) to the edge PC 100c. The non-video retrieval response, for example, displays a still image of the advertisement in area 901 of Figure 9(a).

[0131] On the other hand, if a phenomenon indicating a return is detected, in S710, the system control unit 50 determines whether the returned item is the same as the item targeted by the displayed retrieval response (the item targeted for retrieval that caused the retrieval response). If the returned item is the same as the item targeted by the displayed retrieval response, the process proceeds to S711. Otherwise, i.e., if a different item from the displayed retrieval response is returned, the process in Figure 7 is terminated without issuing a return response. Also, if the retrieval response described in S708 or S709 above is not displayed on the signage device 100d, S710 determines No and terminates the process in Figure 7 without issuing a return response.

[0132] In S711, the system control unit 50 controls the signage device 100d to output a return response regarding the returned product (returned product, product subject to return). More specifically, it sends an instruction to output the return response (signage control instruction) and the content of the return response (video, still image, product information, related product information, price information, campaign information, etc.) to the edge PC 100c.

[0133] Figure 9(b) shows an example of the display of a return response on the display 105d of the signage device 100d. The return response will have at least partially different display content from the retrieval response. The audio output unit 113d of the signage device 100d will also output audio corresponding to the return response. In Figure 9(b), the video displayed in area 901 will continue to play until the end without stopping, even if a phenomenon corresponding to a return is detected. This is because stopping the video midway would degrade the quality of the advertisement, and there is a possibility that other customers besides the one who made the return may be watching. However, this is not the only option; the video displayed in area 901 may be stopped, a still image advertisement quickly conveying the product's features may be displayed, and an audio message such as "Please wait a moment" may be played to attract the customer's attention.

[0134] Area 902 in Figure 9(b) displays different content from area 901 in Figure 9(a). Area 902 in Figure 9(a) displays "Items you are currently holding," while area 902 in Figure 9(b) displays "Items you were holding," indicating that this is a response to the act of returning an item. It also displays limited-time information 902a (information about the duration of a limited-time event), such as "Until 10 / 31" for the discount campaign period of the items to be returned. This motivates the customer (purchaser 120) to purchase the item immediately rather than later, encouraging early purchase and retrieval. Furthermore, coupon information 902b is displayed, indicating that further price reductions (in this example, points) are possible on top of the price displayed in Figure 9(a). While the coupon information 902b in Figure 9(b) is displayed as point redemption as a price reduction, it could also be displayed as a price discount. This motivates customers to purchase the returned items, encouraging them to pick them up again and make another purchase.

[0135] In area 905 of Figure 9(b), information about other products of the same type as the product to be returned is displayed as related products to the product to be returned. For example, area 905 displays products that are more expensive than the product to be returned (similar products one rank higher) and products that are cheaper (similar products that are more affordable). In this way, if the reason the buyer 120 returned the product to be returned was dissatisfaction with its quality, it is possible to suggest a higher quality (higher price) product, thereby contributing to the purchase of a product that satisfies the customer. Conversely, if the reason the buyer 120 returned the product to be returned was dissatisfaction with its price, it is possible to suggest a lower price product, thereby contributing to the purchase of a product that satisfies the customer. Furthermore, among products of the same type as the product to be returned, the most popular product, the product with the highest sales volume, and products with many positive reactions such as "likes" and word-of-mouth on the internet may also be displayed. For the products displayed in area 905, information indicating the display location is also displayed, such as display location information 905a and 905b, saving the customer the trouble of searching for the product if they are interested. Although examples of indicating the display location using strings as display location information 905a and 905b have been explained, it is also possible to show it on a map (store layout) like the one displayed in area 904.

[0136] Although an example has been described in which a return response is output when "Yes" is determined in S710, this is not the only example. The process in S710 may be omitted, and a return response may be output even if the retrieval response described in S708 or S709 above is not displayed on the signage device 100d. Furthermore, the process in S710 may be omitted, and if the signage device 100d displays a retrieval response for a product different from the product to be returned, the system may be controlled to either stop the retrieval response or output a return response for the product to be returned in parallel. In this case, the displayed content may be as shown in Figure 9(b), or not limited to that, it may be as shown in Figure 9(a). By doing so, it is possible to output an effective advertisement that will make customers who have retrieved a product but returned it (purchaser 120), i.e., users who hesitated to purchase, more likely to purchase. Furthermore, the system may be designed to not process the retrieval response (S708, S709) and only process the return response (S711). This approach prevents ads from being displayed every time an item is retrieved, thus reducing the likelihood that users (buyers 120) will find it annoying. Furthermore, by reducing the frequency of ad displays—only showing them when the user has considered purchasing an item but then decided against it, such as when they retrieve an item and then return it—the guidance can be made more memorable and effective for the user (buyer 120).

[0137] Alternatively, the return response could be a notification thanking the customer for taking the time to retrieve the item and considering purchasing it, regardless of the item being returned. For example, the return response could display or voice a message such as, "Thank you for your consideration. We hope you will consider us again in the future." Doing so is expected to improve customer satisfaction with the store and contribute to long-term sales growth.

[0138] Thus, in this embodiment, a retrieval response is output in response to the retrieval of an item, and a return response is output in response to the return of an item. Therefore, more effective announcements (product information) can be made in accordance with the customer's actions.

[0139] Figure 10 illustrates another example of a retrieval response. Instead of S706-S709, the system may be controlled to display a retrieval response as shown in Figure 10 on the display 105d of the signage device 100d at this time. The CPU 101b controls the signage device 100d to activate its dialogue functions (voice dialogue function, screen-operated dialogue function, virtual assistant function) in response to the detection of a phenomenon corresponding to the retrieval of an item. In response to the activation of the dialogue function, a screen like that shown in Figure 10 is displayed on the display 105d (touch panel 70d) of the signage device 100d. The avatar 1001 is a character that mimics the person being interacted with. It provides information about the product and acts like a store clerk. The avatar 1001 may be an animation or still image of a character such as a person or animal, or it may be a photograph or video. The speech bubble 1002 displays the item to be retrieved and a message indicating that the retrieval has been recognized ("Please take it"). It also displays a message informing the customer that information about the product and the store is available ("We will guide you"). This allows the purchaser 120 to recognize that the content displayed on the display unit 105d is directed at them. The speech bubble 1003 displays a message prompting input via voice or operation of the control unit (screen touch in this example) because the dialogue function has been activated. The content displayed in speech bubbles 1002 and 1003 may also be output (read aloud) by voice from the voice output unit 113d. Options 1004 to 1006 are touch operation items that accept user input for dialogue via screen touch. When Figure 10 is displayed and the user speaks, it is picked up by the voice input unit 114d and input as voice input. The voice input is transmitted to the server device 100b via the edge PC 100 (or not) through the external I / F 109d or communication I / F 110d. The server device 100b analyzes the received audio input, performs a response generation process to generate response information according to the analysis results, and transmits the generated response information to the signage device 100d via (or not via) the edge PC 100c.The signage device 100d displays information based on response information transmitted from the server device 100b on the display 105d and outputs it from the audio output unit 113d. For example, when the screen shown in Figure 10 is displayed, if the user says, "Tell me the ingredients of this product," then information regarding the ingredients of the product being retrieved (e.g., "Kirameki Shampoo") will be displayed and output aloud accordingly. For example, content such as "Ingredients," "Specified Ingredients," and "Halal Information" in area 902 of Figure 9(a) will be displayed and read aloud. Similarly, it is possible to output detailed information about the product being retrieved, information about related products, and information unrelated to the product being retrieved (such as other products or store information) in response to user inquiries.

[0140] Furthermore, the activation of the dialogue function in the signage device 100d and the display shown in Figure 10 may be performed not only as a response to retrieval, but also as a return response (processing S711 in Figure 7) in response to the detection of a phenomenon corresponding to the return of a product (return response phenomenon). When the dialogue function is displayed in response to a return response phenomenon, information indicating that the return response phenomenon has been recognized and the product to be returned will be output (displayed and voiced).

[0141] In this embodiment, an example has been described in which a signage control instruction is sent from the server device 100b to the edge PC 100c, and in response, the edge PC 100c outputs the content of the response to the signage device 100d. However, the system is not limited to this. That is, the signage control instruction may be sent from the server device 100b to the signage device 100d without going through the edge PC 100c, causing the signage device 100d to perform various responses. Also, although the processing in Figure 7 has been described as being performed by the server device 100b, the system is not limited to this. That is, the edge PC 100c may identify the product that is to be retrieved or returned, and the signage control processing in Figure 7 may be performed on the edge PC 100c without going through the server device 100b. In this case, when the edge PC 100c detects that a product has been retrieved or returned (S606Yes), the edge PC 100c performs the processing to identify the product that is to be retrieved or returned, and then the CPU 101c of the edge PC 100c executes the processing in Figure 7. In this case, the output destination for the various responses from S708, S709, and S711 is the signage device 100d. Alternatively, the output destination for the various responses from S708, S709, and S711 can be the display (not shown, to be installed in a position visible to the purchaser 120) and the audio output unit 113c of the edge PC 100c. This allows the edge PC 100c and the signage device 100d to be implemented as a single device, rather than being separate devices.

[0142] Furthermore, the signage control process shown in Figure 7 is applicable to systems that do not perform check-in / check-out control. Moreover, it is applicable not only to purchase systems using the purchaser terminal 100a, but also to stores where purchases are settled at manned or self-service registers. In other words, it is applicable to systems that omit at least one of the purchaser terminal 100a, server device 100b, payment service 170, and QR code 160 shown in Figure 1.

[0143] As described above, this embodiment allows for more effective guidance regarding products. Conventionally, advertising such as digital signage in stores has been ineffective in attracting customers. For example, even if a video is used to appeal to customers moving around the store, they are not guaranteed to stop and watch the video advertisement. In contrast, this embodiment controls the system to display product information, as explained in Figures 9 and 10, to customers who have taken out or returned a product. Taking out or returning a product indicates that the customer has at least shown interest in that product. Moreover, there is a high probability that they have stopped to take out or return the product. By displaying information about the product that the customer is interested in to such a customer who has stopped and shown interest in the product, the likelihood of the customer seeing the information increases. In other words, this embodiment allows for more effective recognition of product information by customers, contributing to increased product sales.

[0144] In the above embodiment, a product shelf 130 was given as an example of a place to display products, but it is not limited to this, and the display location may be any place other than a shelf from which a purchaser can take out the products.

[0145] Furthermore, the various controls described above, which are performed by CPUs 101a, 101b, and 101c, may each be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0146] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is also possible to combine each embodiment as appropriate.

[0147] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having one or more computers (or CPUs, MPUs, etc.) in that system or device read and execute the program code. In this case, the program and the storage media storing the program constitute the present invention. [Explanation of symbols]

[0148] 100a: Purchaser terminal 100b: Server device 100c: Edge PC 130: Shelf

Claims

1. A detection means for detecting at least one of the following: a withdrawal phenomenon, which is a phenomenon corresponding to an item being taken out of a location where an item is placed; and a return phenomenon, which is a phenomenon corresponding to an item being returned to the location where the item is placed. A control means that controls the detection means to initiate a dialogue with the item to be retrieved when it detects the retrieval phenomenon, and controls the detection means to initiate a dialogue with the item to be returned when it detects the return phenomenon, An information processing system characterized by having the following features.

2. The information processing system according to claim 1, characterized in that when the aforementioned dialogue is initiated, the electronic device outputs at least one of the following: a character modeled after the dialogue partner, information indicating the item to be retrieved or the item to be returned, a statement that a phenomenon corresponding to retrieval or return has been detected, a statement that guidance can be provided to the customer, and a voice prompt or a prompt to operate the control unit.

3. The character representing the person being spoken to is a character that represents a person or an animal. The information processing system according to claim 2, characterized in that when a character resembling the conversation partner is output from the electronic device in response to the commencement of the aforementioned conversation, a video or still image relating to the said character is output.

4. The information processing system according to any one of claims 1 to 3, characterized in that the control means controls the system to initiate a dialogue with respect to the retrieved item or the returned item in response to a voice input to an electronic device or operation of an operation unit by a user who has retrieved or returned an item from the placement location while the dialogue has been initiated.

5. The information processing system according to claim 4, characterized in that the operation relates to at least one of the following requests: a request for a detailed description of an item to be retrieved or an item to be returned; a request for information about an item related to an item to be retrieved or an item to be returned; a request for information about another item; or a request for store information.

6. The information processing system according to claim 4, characterized in that the control means controls the display means of the electronic device to display predetermined information in response to the start of the dialogue.

7. The information processing system according to claim 4, characterized in that the control means controls the output of predetermined information from the audio output means of the electronic device in response to the start of the dialogue.

8. Association means for performing a process of associating location information for identifying the aforementioned location with user identification information, When the item is associated by the association means and the retrieval response phenomenon is detected, an output means outputs information relating to the item to be retrieved that is the subject of the retrieval response phenomenon to an electronic device corresponding to the user identification information, An information processing system according to any one of claims 1 to 7, characterized by comprising the above.

9. The information processing system according to claim 8, characterized in that the output means is controlled to output a notification to the electronic device corresponding to the different user identification information that the association process by the association means cannot be performed when it receives user identification information different from the user identification information while the system is associated by the association means.

10. comprising a means for identifying the product to be returned, The information processing system according to any one of claims 1 to 9, characterized in that the control means controls the system to initiate a dialogue relating to the return target product identified by the identification means.

11. comprising a means for identifying the product to be returned, The information processing system according to any one of claims 1 to 9, characterized in that the control means controls the system so as not to initiate a dialogue with respect to the item to be returned when the item to be returned and the item to be retrieved are different.

12. The information processing system according to any one of claims 1 to 11, characterized in that the article is a product and the location is a display location for displaying the article.

13. A method for controlling an information processing system, The detection means of the information processing system includes a detection step of detecting at least one of the following: a withdrawal response phenomenon, which is a phenomenon corresponding to an item being taken out of a location where an item is placed; and a return response phenomenon, which is a phenomenon corresponding to an item being returned to the location where the item is placed. A control step in which the control means of the information processing system controls to start a dialogue with respect to the item to be retrieved that is the subject of the retrieval phenomenon when the retrieval phenomenon is detected in the detection step, and controls to start a dialogue with respect to the item to be returned that is the subject of the return phenomenon when the return phenomenon is detected in the detection step, A control method for an information processing system, characterized by having the following features.

14. An electronic device included in an information processing system having detection means for detecting at least one of a retrieval phenomenon, which is a phenomenon corresponding to an item being taken out of a location where an item is placed, and a return phenomenon, which is a phenomenon corresponding to an item being returned to the location where the item is placed, Control means that, when the detection means detects the retrieval response phenomenon, initiates a dialogue with the item to be retrieved that is the subject of the retrieval response phenomenon, and when the detection means detects the return response phenomenon, initiates a dialogue with the item to be returned that is the subject of the return response phenomenon. An electronic device characterized by having the following features.

15. A program for causing at least one computer to function as one of the means of an information processing system described in any one of claims 1 to 12.

Images