System

Abstract

To provide a checking device to ensure the registration of articles to be purchased through operation by a customer.SOLUTION: A checking device according to an embodiment includes a receiving unit, a camera, an object detection unit, and a checking unit. The receiving unit receives, from an electronic device, article information about an article that a customer intends to purchase, which is stored in the electronic device. The camera captures an image. The object detection unit detects objects corresponding to the article from the image. The checking unit checks for a match between the articles within the range of the image captured by the camera and the article corresponding to the article information based on the article information and the object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiment of the present invention relates to an electronic device, a system, and a program. [Background technology]

[0002] In retail stores such as supermarkets, checkout systems that allow customers to perform part of the accounting process have been introduced to improve the efficiency of accounting processes using POS (Point Of Sales). For example, a self-checkout system equipped with a self-checkout register (self-checkout terminal) has been introduced. The self-checkout register performs a registration process in which a customer operates the register to read the product code (usually a barcode) of the product, registers the product, generates payment information for the payment process, and performs a payment process in which the customer pays the price based on the payment information generated by the registration process.

[0003] In recent years, personal checkout systems have been developed that register products that customers plan to purchase while they are shopping around the store. In this system, it is necessary that the products that are added to the cart or basket as purchases match the registered products. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2016-532932 Summary of the Invention [Problem to be solved by the invention]

[0005] According to an embodiment, the electronic device includes a receiving means and a transmitting means. The receiving means receives product information related to a product that the customer plans to purchase. The transmitting means transmits a notification of no storage required to another electronic device when the product information indicates a specific product that does not need to be stored in a storage unit for storing the product that the customer plans to purchase. [Means for solving the problem]

[0006] According to an embodiment, the product data processing system has a display unit, a display means, a storage means, a first output means, a recognition means, and a second output means. The display means displays the product information acquired by the product information acquisition means on the display unit. The storage means stores the product information acquired by the product information acquisition means in the storage unit. When a preset predetermined time has elapsed since the product information was stored in the storage unit, the first output means outputs to the display unit a first guide to the customer encouraging the customer to store the product in a storage unit for storing the product that the customer plans to purchase. The recognition means recognizes the product stored in the storage unit. The second output means outputs to the display unit a second guide to the customer encouraging the customer to acquire the product information by the product information acquisition means when product information corresponding to the product recognized by the recognition means is not stored in the storage unit. [Brief description of the drawings]

[0007] [Figure 1] 1 is a system configuration diagram illustrating an overall configuration of a system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an electronic device according to an embodiment. [Diagram 3] 1A and 1B are diagrams showing examples of use of a clip terminal according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing the appearance of the clip terminal according to the embodiment. [Diagram 5] 5 is a flowchart illustrating the operation of the electronic device according to the embodiment. [Figure 6] 5 is a flowchart for explaining the operation of a clip terminal in the embodiment. [Figure 7] 5 is a flowchart for explaining the operation of a clip terminal in the embodiment. [Figure 8] FIG. 4 is a diagram showing an example of a product checklist in the present embodiment. [Figure 9]FIG. 4 is a diagram showing an example of a message according to the embodiment. [Figure 10] 4A and 4B are diagrams showing examples of images captured by a camera in the present embodiment. [Figure 11] 11 is a diagram showing an example of an object corresponding to a commodity detected from the image shown in FIG. 10. [Figure 12] 10 is a flowchart showing a registered position determination process for determining whether confirmation is necessary based on a registered position in the present embodiment. [Figure 13] FIG. 1 is a diagram showing an example of the arrangement of product shelves and the positions of carts in a store. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the present embodiment will be described with reference to the drawings. Fig. 1 is a system configuration diagram showing the overall configuration of the system of this embodiment. In the system shown in Fig. 1, a system 10 installed at a headquarters and a store system 12 installed at each of a plurality of stores are connected via a network 14 including a dedicated line or the Internet.

[0009] The system 10 provided at the headquarters includes, for example, a headquarters system 10A, a customer system 10B, a payment system 10C, and a sales promotion system 10D, which are interconnected via a network 10E such as a LAN (Local Area Network).

[0010] The headquarters system 10A executes processes to control the store systems installed in each store. The customer system 10B manages customers who use the store system to purchase products, etc. The payment system 10C executes processes (personal authentication, credit card payment, etc.) for payment processing using credit cards, etc. by customers in the store system. The sales promotion system 10D manages the provision of information such as product advertisements for sales promotion and in-store guides to customers (electronic devices used by customers) through the store system.

[0011] The store system 12 installed in the store is realized as, for example, a POS (Point Of Sales) system. The store system 12 includes, for example, a store server 20A, a clip terminal management server 20B, a plurality of accounting machines 20C, a plurality of POS terminals 20D, and the like, which are connected to each other via a network 20E such as a LAN.

[0012] The store server 20A controls the entire store system 12. The store server 20A stores data related to products sold in the store, such as product codes, product names, unit prices, weights, product positions (product shelf positions) in the store, and product images for each product. In the registration process, the store server 20A receives and stores product data corresponding to product codes read by scanning the products, and can generate payment information (including price, product, quantity, etc.) for payment processing. The clip terminal management server 20B manages electronic devices 50 (clip terminals described later) used to check whether products (purchased products) registered by a customer using the electronic device 30 as products to be purchased match products placed in a cart 60 or a basket as products to be purchased. The checkout machine 20C executes payment processing for products registered by a customer using the electronic device 30. The POS terminal 20D executes registration processing for products purchased by a customer using the store clerk, payment processing for products registered by the registration processing, and the like.

[0013] A plurality of access points (base stations) 20F are connected to a network 20E wired within the store. The access points 20F communicate wirelessly with various electronic devices used within the store, for example, via wireless LAN. The electronic devices used within the store include, for example, electronic devices 30 used by customers, electronic devices 40 used by store attendants, and electronic devices 50 (clip terminals 50 described below) used while attached to, for example, a cart 60.

[0014] The electronic devices 30, 40 may be, for example, a smartphone or a tablet personal computer (PC). The electronic device 30, in cooperation with the electronic device 50 (clip terminal), executes a registration process for a product to be purchased and an individual check process for checking whether the registered product (purchased product) matches the product placed in the cart 60 or basket as a purchase, based on an application program (including a check program) for using the service provided by the store system 12. The electronic device 40 executes a check process for enabling a store clerk (attendant) to easily check whether the product information registered by the registration process matches the product placed in the cart 60, based on a check program for checking the registration process executed by the electronic device 30.

[0015] The electronic device 30 and the electronic device 50 (clip terminal 50) may communicate with each other via the clip terminal management server 20B (network 20E), or may directly communicate with each other by short-distance wireless communication. Furthermore, they may be connected to each other by a communication cable for wired communication.

[0016] Fig. 2 is a block diagram showing the configuration of electronic devices 30, 50 in this embodiment. The electronic device 30 shown in Fig. 2 will be described as being realized by, for example, a smartphone. The electronic device 50 is configured as a clip terminal 50 dedicated to executing individual check processing in the store system 12. Hereinafter, the clip terminal 50 will be described. The clip terminal 50 can also be realized by installing a check program in a general-purpose device (computer) equipped with a camera, such as a smartphone.

[0017] The electronic device 30 includes a processor 30A, a memory 30B, a storage device 30C, a touch panel 30D, a camera 30E, a communication module 30F, and a battery 30G. The processor 30A is a circuit that controls each part to realize various processes of the electronic device 30 by executing a program recorded in the memory 30B.

[0018] The memory 30B records the programs executed by the processor 30A as well as data used by the processor 30A to execute various processes. The programs recorded in the memory 30B include, for example, a registration processing program that executes a registration process for a product to be purchased, and a check program that executes an individual check process to check whether the registered product (purchased product) matches the product placed in the cart 60 or basket as a product to be purchased. The data stored in the memory 30B includes payment information generated based on the product code read in the registration process, product information on the product registered by the registration process, and the like.

[0019] The storage device 30C is realized by a hard disk drive (HDD) device or a solid state drive (SSD) device, and records data used by the processor 30A when performing various processing, or data generated by processing by the processor 30A.

[0020] The touch panel 30D is a device on which a customer performs an input operation, and includes a display device and a touch sensor. The touch panel 30D displays any screen, such as a GUI (Graphical User Interface) screen, associated with the processing of the processor 30A. Camera 30E captures images under the control of processor 30A.

[0021] The communication module 30F controls communication with external electronic devices under the control of the processor 30A. The communication module 30F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the clip terminal 50.

[0022] The clip terminal 50 includes a processor 50A, a memory 50B, a storage device 50C, a touch panel 50D, a camera 50E, a communication module 50F, a battery 50G, an I / O interface 50H, and a reader / writer 50J.

[0023] The processor 50A is a circuit that controls each unit to realize various processes as the clip terminal 50 by executing a program recorded in the memory 50B.

[0024] The memory 50B records programs executed by the processor 50A as well as data used by the processor 50A to execute various processes. The programs recorded in the memory 50B include a check program that realizes an individual check process to check whether a registered product (purchased product) matches a product placed in the cart 60 or a basket as a purchase candidate. The check program includes an image processing program that realizes image processing to detect an object corresponding to the product from an image captured by the camera 50E and detect that the product has been placed in the cart 60 and that the product has been removed from the cart 60. The data stored in the memory 50B includes various data such as product information registered by the registration process of the electronic device 30, object management data (see FIG. 11) that manages objects detected by image processing, and a product checklist (see FIG. 8) for checking products placed in the cart 60.

[0025] The storage device 50C is realized by a hard disk device (HDD (Hard disk drive)) or an SSD (solid state drive) device, and records data used by the processor 50A when performing various processing, or data generated by processing by the processor 50A.

[0026] The touch panel 50D is a device on which a customer performs an input operation, and includes a display device and a touch sensor. The touch panel 50D displays any screen, such as a GUI (Graphical User Interface) screen, associated with the processing of the processor 50A. Camera 50E captures images under the control of processor 50A.

[0027] The communication module 50F controls communication with external electronic devices under the control of the processor 50A. The communication module 50F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the electronic device 30.

[0028] The I / O interface 50H is an interface for inputting signals from external devices, and is connected to, for example, a weight sensor 601 (a weighing device, a pressure sensor, a weighing scale, etc.) attached to the basket bottom 60C of the cart 60. The weight sensor 601 detects the weight of a product placed (placed) in the cart 60 and outputs a detection signal.

[0029] The reader / writer 50J communicates with RF (Radio Frequency) tags attached to each product sold in the store and to product shelves in the store. The communication range of the reader / writer 50J is, for example, a range in which it can communicate with the RF tags of products placed in the cart 60, or a range in which it can communicate with RF tags attached to product shelves by moving the cart 60 in the store. The reader / writer 50J can be used to detect when a product has been placed in or removed from the cart 60, or can be used to detect the position in the store of the cart 60 to which the clip terminal 50 is attached.

[0030] FIG. 3 is a diagram showing an example of how the clip terminal 50 of this embodiment is used, and FIG. 4 is a diagram showing the external appearance of the clip terminal 50 of this embodiment.

[0031] The clip terminal 50 is used, for example, by being attached to an upper frame 60B of a cart 60 used in a store. As shown in FIG. 4, the clip terminal 50 is provided with a clip 60k, which can be attached and detached as desired. Therefore, it is not necessary to attach the clip terminal 50 to all carts 60 prepared in advance in the store, and it is sufficient to attach the clip terminal 50 as necessary only to carts 60 used by customers who use the individual check processing service using the clip terminal 50. Therefore, in order to realize the individual check processing service, large-scale investments such as remodeling the store or replacing the cart 60 are not required, and it is possible to realize it at low cost.

[0032] In the example shown in FIG. 3, the clip terminal 50 is attached to the upper frame 60B on the handle 60A side of the cart 60. When attaching the clip terminal 50, the camera 50E provided on the clip terminal 50 is directed toward the basket bottom 60C, and the shooting range is adjusted so that at least all of the products placed in the cart 60 can be photographed. The shooting range of the camera 50E is preferably set to include the upper frame 60B, so that the area inside and outside the basket can be clearly distinguished in the image photographed by the camera 50E. Furthermore, it is desirable that the shooting range of the camera 50E is set to be capable of photographing, for example, the movement of the customer's arm (hand) so that the action of the customer putting in and taking out the product can be detected.

[0033] 4, the clip terminal 50 is attached to the upper frame 60B so that the display surface of the touch panel 50D faces the customer. This makes it easy to check the images captured by the camera 50E and information (messages, etc.) showing the check results of the individual check process using the captured images.

[0034] Next, the operation of the electronic device 30 and the clip terminal 50 (electronic device) used in the store system 12 of this embodiment will be described. Fig. 5 is a flowchart for explaining the operation of the electronic device 30 in this embodiment. Figs. 6 and 7 are flowcharts for explaining the operation of the clip terminal 50 in this embodiment.

[0035] When a customer performs registration processing using the electronic device 30, the customer starts an application program (check program) for using the services provided by the store system 12. The processor 30A of the electronic device 30 then executes the registration processing and individual check processing described below based on the application.

[0036] First, the electronic device 30 executes a login process for connecting to the store system 12 (Act 10). In the login process, the customer operates the camera 30E to read (take an image) a barcode containing a membership code unique to the member, which is attached to, for example, a membership card. The processor 30A extracts the membership code from the barcode read by the camera 30E and transmits it to the clip terminal management server 20B. The clip terminal management server 20B performs member authentication by comparing the membership code received from the electronic device 30 with membership data managed in the customer system 10B.

[0037] When the electronic device 30 is notified by the clip terminal management server 20B that the electronic device 30 has been authenticated as a member, the electronic device 30 displays a selection screen for services available at the store. The services available at the store include registration processing for a planned purchase, display of flyer information, display of sales product information, display of store information (map), and use of the clip terminal 50. By using the clip terminal 50, for example, if a registration processing has not been performed for a product placed in the cart 60, a message is output from the clip terminal 50. This allows the customer to prevent forgetting to register a product to be purchased. In addition, since the clip terminal 50 checks the products to be purchased while the customer is shopping around the store, the burden of the confirmation work by the attendant (store clerk) after the customer has completed the registration processing can be reduced.

[0038] When a customer uses the clip terminal 50, the store system 12 may encourage the customer to use the clip terminal 50 by providing the customer with a benefit (such as points) (by storing data indicating the benefit to the customer in the customer system 10B).

[0039] When the clip terminal 50 is used, the electronic device 30 and the clip terminal 50 are set in a state in which they can communicate with each other. For example, the clip terminal 50 is attached to a cart 60 used by a customer (or the customer uses a cart 60 with the clip terminal 50 attached in advance), and a link is set between the electronic device 30 and the clip terminal 50 of the customer. For example, the electronic device 30 and the clip terminal 50 are paired so that they can communicate with each other by short-distance wireless communication. Alternatively, the unique code of the clip terminal 50 is read by the electronic device 30 and transmitted from the electronic device 30 to the clip terminal management server 20B. The clip terminal management server 20B identifies the electronic device 30 and the clip terminal 50 that cooperate to execute the individual check process, and enables communication between the electronic device 30 and the clip terminal 50 via the clip terminal management server 20B (network 20E). Note that other methods can be used for connecting the electronic device 30 and the clip terminal 50.

[0040] When the processor 30A of the electronic device 30 is connected to the clip terminal 50 (Act 11), the processor 30A puts the electronic device 30 in a state in which it can execute a registration process for a commodity. When the processor 50A of the clip terminal 50 is connected to the electronic device 30 (Act 40), the processor 50A starts a commodity detection process. The commodity detection process is a process for detecting an object corresponding to a commodity from an image captured by the camera 50E. The processor 50A detects an object corresponding to a commodity, etc. from an image captured by the camera 50E by using existing image processing, and stores the position of each object. For example, when the processor 50A detects an object moving from an area corresponding to the outside of the cart (outside the cart) to the inside of the cart (inside the cart), the processor 50A determines that the customer has performed an action to put in or take out a commodity. Based on the change in the object present in the cart before and after the detection of this action, the processor 50A detects that a commodity has been put in or taken out. Even when the processor 50A detects a customer's action, if the number of objects corresponding to the commodity in the cart has not changed, the processor 50A determines that, for example, the customer has rearranged the commodity in the cart. The processor 50A stores the position of the rearranged object. The processor 50A stores objects detected from the images captured by the camera 50E in a product checklist.

[0041] Fig. 8 is a diagram showing an example of a product checklist in this embodiment. As shown in Fig. 8, the product checklist stores data (e.g., D1 to D4) in which data on objects (detected objects) detected from an image captured by camera 50E, data on product information (registered products) registered by a registration process in electronic device 30, and data showing the results of checking whether the registered products match the products corresponding to the detected objects (check results "OK" or "NG").

[0042] When a customer is shopping around the store, the customer uses the camera 30E of the electronic device 30 to scan the barcode attached to the product to register the product (Act 12). The processor 30A detects the barcode from the image captured by the camera 30E and extracts the product code from the barcode. The processor 30A acquires product information corresponding to the product code from the store server 20A via, for example, the clip terminal management server 20B (network 20E) and displays it on the touch panel 30D. That is, the product name, price, product image, etc. of the product read by the product scan are displayed for the customer to confirm. Here, when the customer performs an instruction operation to register the product (for example, touching a product registration button displayed on the touch panel 30D) (Act 13, Yes), the processor 30A stores the product read by the product scan as a registered product to be purchased (Act 14). The processor 30A also transmits information on the registered product to the clip terminal 50 (Act 15).

[0043] On the other hand, when the clip terminal 50 receives the information of the registered product from the electronic device 30 (Act 42, Yes), it stores the information of the registered product in the product checklist (Act 43). When the processor 50A refers to the information of the registered product and detects that the product is a large product, that is, when it is determined that the product does not need to be (cannot be) added to the cart 60, it sets a skip flag in the product checklist (Act 45). In the example shown in FIG. 8, a skip flag is set in association with the registered product "product C" as data D2. For the registered product "product C" for which the skip flag is set, the processor 50A excludes it from the targets for checking whether the registered product matches the product added to the cart 60. In addition, the processor 50A transmits a product-adding-unnecessary notification to the electronic device 30 (Act 57).

[0044] When the processor 30A of the electronic device 30 receives a notification that no product needs to be added from the clip terminal 50 (Act 18, Yes), the processor 30A displays on the touch panel 30D a message notifying the customer that no product needs to be added to the cart 60, together with registered product information (product name, product image, price, etc.). The message notifying that no product needs to be added may be displayed not only on the electronic device 30, but also on the touch panel 50D of the clip terminal 50.

[0045] On the other hand, if the registered product is not a large product, the processor 50A acquires product dictionary data for identifying the registered product by image processing from the clip terminal management server 20B through the network 20E, for example. For example, if the processor 50A does not store product dictionary data corresponding to the registered product, it outputs a transmission request for the product dictionary data corresponding to the registered product to the clip terminal management server 20B. In response to the transmission request from the clip terminal 50, the clip terminal management server 20B transmits the product dictionary data of the corresponding product to the clip terminal 50. Note that, when the electronic device 30 and the clip terminal 50 communicate with each other via the clip terminal management server 20B, the clip terminal management server 20B may transmit the product dictionary data of the corresponding product to the clip terminal 50 together with the information of the registered product when it receives the information of the registered product from the electronic device 30. Also, the product dictionary data may be stored in the clip terminal 50 in advance.

[0046] In addition, when the registered product is stored in the product checklist, the processor 50A starts measuring the elapsed time from when the registered product was stored. The processor 50A determines whether the elapsed time has elapsed a preset time until it is confirmed that the product corresponding to the registered product has been put into the cart 60 (Act 46).

[0047] In many cases, a customer handles a product that he or she plans to purchase by a series of actions, such as picking up the product from a shelf or the like, scanning the product using the electronic device 30, and then placing the product in the cart 60. Therefore, if a preset time has passed since the product was scanned (storage of the registered product in the clip terminal 50), the customer may have forgotten to put the registered product in the cart 60, and this is notified to the customer. If the processor 50A determines that the preset time has passed for the registered product (Act 46, Yes), it transmits a product placement request to the electronic device 30 (Act 47).

[0048] When the processor 30A of the electronic device 30 receives the product putting request from the clip terminal 50 (Act 18, Yes), the processor 30A displays, on the touch panel 30D, information on the registered product (product name, product image, price, etc.) and a product putting request message urging the customer to put the product in the cart 60 (Act 19). For example, as shown in FIG. 9(A), a message "Please put the registered product in the cart." may be displayed. The product putting request message may be displayed not only on the electronic device 30, but also on the touch panel 50D of the clip terminal 50. When there are multiple products targeted by the product putting request message, the information on the multiple target products may be displayed in a list form so that the target products can be recognized reliably by the customer.

[0049] On the other hand, when the processor 50A of the clip terminal 50 detects that the customer has performed an action to put in or take out a product by image processing of the image captured by the camera 50E, the processor 50A detects that the product has been put in or taken out based on the change in the objects present in the cart before and after the detection of this action. The processor 50A not only detects the action of putting in or taking out a product based on the image, but can also determine whether the product has been put in or taken out by combining this action with the change in weight detected by the weight sensor 601 at the time the action is detected.

[0050] An example of an image captured by the camera 50E is shown in Fig. 10. The example shown in Fig. 10 shows a state in which three products A, B, and E have been placed in the cart 60.

[0051] The processor 50A detects the position of the upper frame 60B of the cart 60 and the area of ​​the basket bottom 60C from the image captured by the camera 50E. Because the clip terminal 50 is fixed to the cart 60, the areas of the upper frame 60B and the basket bottom 60C in the image do not generally change during shopping. When the processor 50A detects an object moving, for example, from an area equivalent to the outside of the cart (outside the cart) to the inside of the cart (inside the cart) based on the position of the upper frame 60B, it determines that the customer has made an action to put in or take out a product.

[0052] Furthermore, the processor 50A detects objects equivalent to products or the like from the images captured by the camera 50E using existing image processing, and stores the positions of the respective objects as object management data.

[0053] Fig. 11 shows examples of objects OJ-A, OJ-B, and OJ-E corresponding to products detected from the image shown in Fig. 10. Processor 50A detects that a product has been put in or taken out based on the change in the objects present in the cart before and after the customer puts in or takes out the product.

[0054] When an object corresponding to a product is detected from the image (Act 48, Yes), the processor 50A refers to the product checklist and determines whether a registered product with a check result of "NG" is stored. That is, it determines whether there is a product that has been registered by the electronic device 30 but has not been confirmed to have been added to the cart 60. Here, if a registered product with a check result of "NG" is stored (Act 49, Yes), the processor 50A executes a recognition process for the object detected from the image using product dictionary data corresponding to the registered product with a check result of "NG" (Act 52). The product dictionary data is created so that a product can be recognized based on features such as a barcode attached to the product, or the design and characters of the package.

[0055] Here, based on the result of the recognition process using the product dictionary data, it is determined whether the object detected from the image is a registered product corresponding to the product dictionary data. For example, if the characteristics of the image represented by the object and the characteristics indicated by the product dictionary data show a similarity exceeding a preset reference value, it is determined that the object is a registered product corresponding to the product dictionary data.

[0056] If it is determined that the product is a registered product (Act 53, Yes), the processor 50A stores the object detected from the image as a detected object in association with the registered product previously stored in the product checklist and corresponding to the product dictionary data used in the recognition process (Act 54). Note that data indicating the similarity (similarity) in the recognition process may be stored as data related to the detected object. This allows the attendant to be prompted to check the product before the payment process, even if the product is determined to be a registered product corresponding to the product dictionary data, because the similarity is low.

[0057] Furthermore, the processor 50A sets the check result corresponding to the pair of the registered product and the detected object to "OK" (Act 55), and stores that it has been confirmed that the registered product has been put into the cart 60. In this case, the processor 50A transmits a product putting confirmation to the electronic device 30 (Act 56).

[0058] When the electronic device 30 receives the confirmation of putting the product from the clip terminal 50 (Act 20, Yes), if a message requesting putting the product is displayed, the electronic device 30 erases the message from the touch panel 30D (Act 28). This makes it possible to notify the customer that the clip terminal 50 has confirmed that the registered product has been put into the cart 60.

[0059] In addition, when there are multiple registered products with a check result of "NG" in the product checklist, the processor 50A executes a recognition process on the object using the product dictionary data of each registered product. In addition, for an object that is not determined to be a registered product by the recognition process (Act 53, No), the processor 50A stores the product as an unregistered detected object with a check result of "NG" in the product checklist (Act 50). In addition, when an object is detected from the image (Act 48, Yes), if a registered product with a check result of "NG" is not stored (Act 49, No), the processor 50A similarly stores the product as an unregistered detected object with a check result of "NG" in the product checklist (Act 50).

[0060] 8 indicates that product C has been registered in the electronic device 30, but the check result is set to "NG," indicating that it has not been confirmed that the product has been added to the cart 60. Data D4 indicates that unregistered products E and F have been added to the cart 60 earlier, and thus have been detected as objects E and F, respectively (check result "NG").

[0061] When the processor 50A stores a detected object with an unregistered product (check result "NG") in the product checklist (Act 50), the processor 50A transmits a product registration request to the electronic device 30 (Act 51). When the processor 30A of the electronic device 30 receives the product registration request from the clip terminal 50 (Act 21, Yes), the processor 30A displays a product registration request message on the touch panel 30D to prompt the customer to register the product. For example, as shown in FIG. 9(B), the processor 50A displays a product registration request message stating "There is an unregistered product in the cart. Please register the product." The product registration request message may be displayed not only on the electronic device 30 but also on the touch panel 50D of the clip terminal 50 (Act 22).

[0062] As described above, after the product is registered, the product registration request message displayed on the touch panel 50D is erased (Acts 20, 28) when it is confirmed that the product has been placed in the cart 60 based on recognition processing using the product dictionary data of the registered product (Acts 49-51).

[0063] On the other hand, when returning a product that has been registered as a purchase candidate, the customer uses the camera 30E of the electronic device 30 to scan the barcode attached to the product to be returned, in the same manner as when registering the product (Act 12). The processor 30A detects the barcode from the image captured by the camera 30E and extracts the product code from the barcode. The processor 30A displays the product information (product name, price, product image, etc.) corresponding to the product code on the touch panel 30D, and allows the customer to confirm the product to be returned. When the customer performs an instruction operation to cancel the product (for example, touching a product cancellation button displayed on the touch panel 30D) (Act 13, No), the processor 30A deletes the product read by the product scan from the purchase candidate (Act 16). The processor 30A also transmits a registration cancellation notice to the clip terminal 50 together with information on the product to be canceled (returned) (Act 17).

[0064] When the clip terminal 50 receives a registration cancellation notification from the electronic device 30 (Act 60, Yes), it identifies the corresponding registered product from the product checklist based on the information of the product to be cancelled (Act 61), and deletes the information of the registered product (Act 62). The processor 50A transmits a cancellation completion notification to the electronic device 30 (Act 63). When the electronic device 30 receives the cancellation completion notification (Act 25, Yes), and if a product placement request message is displayed, it erases the product placement request message (Act 28). For example, as described above, the electronic device 30 displays the product placement request message when a set time has passed without the product being placed in the cart 60 after the product registration process. When the customer performs a product cancellation operation for the product that has been registered, the electronic device 30 erases the product placement request message for the corresponding product.

[0065] When a customer removes an item from the cart 60, the processor 50A of the clip terminal 50 determines that the customer has taken an action to put in or take out the item based on the image captured by the camera 50E, and detects an object corresponding to the removed item by existing image processing. As in the case of detecting an object when an item is put in, it is also possible to determine that an item has been removed by combining this with a change in weight detected by the weight sensor 601.

[0066] When the processor 50A detects that an object (goods) has been removed from the cart 60 (Act 64, Yes), it determines whether a registered good corresponding to the detected object is stored in the good checklist (check result: "OK"). If the corresponding registered good is not stored (Act 65, No), the processor 50A deletes the detected object corresponding to the removed good from the good checklist (Act 68).

[0067] On the other hand, if the corresponding registered product is stored (Act 65, Yes), the processor 50A transmits a product cancellation request to the electronic device 30 (Act 66). In addition, the processor 50A sets the check result of the detection object corresponding to the taken-out product to “NG” (Act 67).

[0068] When the processor 30A of the electronic device 30 receives a product cancellation request from the clip terminal 50 (Act 23, Yes), the processor 30A displays a product cancellation request message on the touch panel 30D to prompt the customer to cancel the product (Act 24). For example, as shown in Fig. 9(C), the processor 30A displays a product cancellation request message stating "The product has been removed from the cart. If you wish to return the product, please perform a return operation." Note that the product cancellation request message may be displayed not only on the electronic device 30, but also on the touch panel 50D of the clip terminal 50.

[0069] When a product is removed from the cart 60, the data stored in the product checklist is updated immediately, but the data may be updated after a predetermined set time (e.g., 10 seconds) has elapsed, assuming that there is a high possibility of canceling (returning) the product. In this case, the product cancellation request message may be displayed on the electronic device 30 immediately in response to the removal of the product, or after the set time has elapsed.

[0070] In this way, the customer can easily check by a message displayed on the electronic device 30 if he / she forgets to register a product he / she plans to purchase or puts a product in the cart 60 by attaching the clip terminal 50 to the cart 60 and shopping around while performing registration processing using the electronic device 30.

[0071] When the customer completes the registration process for the product they plan to purchase, they instruct the end of product registration, for example by operating the subtotal button. When the customer instructs the end of product registration (Act 26, Yes), the processor 30A of the electronic device 30 displays a confirmation screen (product list) of the registered products and prompts the customer to specify a payment method (credit, cash, direct debit, electronic money, etc.). The processor 30A generates payment information based on the information of the products registered by the registration process.

[0072] Furthermore, the processor 30A of the electronic device 30 transmits a notification of the end of the registration process to the clip terminal 50 (Act 27). When the processor 50A of the clip terminal 50 receives the notification of the end of the registration process from the electronic device 30 (Act 69, Yes), it transmits product checklist data to the electronic device 30 (Act 70). This data includes, for example, data indicating the similarity when the detected object is recognized. After transmitting the product checklist data, the processor 50A clears the product checklist data and terminates the process (Act 71).

[0073] In addition, when an instruction to end the registration process is given, if any data stored in the product checklist has a check result of "NG", a message prompting the customer to check again is displayed on the touch panel 30D of the electronic device 30 (touch panel 50D of the clip terminal 50).

[0074] When a final confirmation instruction is given by the customer, the processor 30A of the electronic device 30 displays on the touch panel 30D a code (a barcode or a two-dimensional code) including payment information and data included in the product checklist that prompts the attendant to make a confirmation.

[0075] The attendant uses the camera of the electronic device 40 to scan the code displayed on the touch panel 30D of the electronic device 30. The electronic device 40 displays a confirmation screen according to the payment information included in the code and the data that needs to be confirmed. For example, the electronic device 40 displays a list of products to be purchased that the customer has registered (including product names, prices, subtotals, etc.) and the number of products. This allows the attendant to check whether the products placed in the cart 60 match the registered products. Furthermore, the electronic device 40 clearly displays products corresponding to detected objects that have a low similarity in the recognition process of the clip terminal 50, for example, based on the data that needs to be confirmed. This allows the attendant to easily recognize products that particularly need to be confirmed, reducing the burden of the confirmation work.

[0076] In the above description, an example is described in which image processing in the clip terminal 50 is used to check whether a registered product matches a product placed in the cart 60. However, the weight of the product detected by the weight sensor 601 may also be used to check the product. For example, the data on the product provided by the store server 20A includes data on the weight of the product, so that a check can be made based on whether the weight increased when the product is placed in the cart 60 matches the weight of the registered product within an allowable range. In particular, for weighed products whose prices vary depending on their weight, it is effective to use the weight data detected by the weight sensor 601 in order to perform a correct check. For example, it is possible to check fraud such as placing a product heavier than the same type of registered product in the cart 60 after a registration process is performed using a light-weight product. The product checklist stores data indicating that confirmation is required when there is a weighed product and when the difference between the weight of the registered product and the weight when it is placed in the cart 60 is greater than a set value. This allows the attendant to recognize that confirmation is particularly required, as described above.

[0077] Furthermore, in the above description, it is described that the product registered by the registration process is checked for a match with the product placed in the cart 60, but it is also possible to check the registration of other products using the clip terminal 50. For example, it is also possible to determine that confirmation is necessary when the difference between the position in the store at the time of product registration in the electronic device 30 or the time when the product (detected object) is detected by the clip terminal 50 and the position (product position) of the product shelf in the store where the target product is placed is greater than a preset determination value.

[0078] 12 is a flowchart showing a registration position determination process for determining whether or not confirmation is necessary based on the registration position in this embodiment. The processor 50A of the clip terminal 50 executes the registration position determination process based on a check program. The registration position determination process is executed in parallel with the individual check process described above.

[0079] The processor 50A detects the current location (Act 82) when it receives product information of a registered product from the electronic device 30 (Act 80, Yes), or when it is determined which registered product the detected object corresponds to (check result: "OK") (Act 81, Yes). The current location can be detected, for example, by communicating with an RF tag attached to each product shelf in the store using the reader / writer 50J.

[0080] 13 is a diagram showing an example of the arrangement of product shelves 70-1 to 70-5 in a store and the position of a cart 60. The clip terminal 50 attached to the cart 60 can determine that it is near the product shelf 70-5 when it can communicate with an RF tag 72-5 attached to the product shelf 70-5 through communication within a limited range using the reader / writer 50J. Note that the current position of the cart 60 (clip terminal 50) can be detected not only by a method using the reader / writer 50J (RF tag), but also by a method such as the difference in radio wave intensity between the communication module 50F and a plurality of access points 20F, or image processing for detecting the cart 60 in an image taken inside the store.

[0081] When the processor 50A detects the current location, it acquires data indicating the product location (product shelf location) where the registered product is sold next, for example, through the store server 20A or the electronic device 30 (Act 83). The processor 50A judges whether the difference between the current location and the product location (product shelf location) is greater than a preset judgment value (Act 84). For example, if the product 74 arranged on the product shelf 70-1 shown in FIG. 13 is registered or placed in the cart 60 at a location close to the product shelf 70-5, it is judged that the customer has performed an unusual action and that confirmation is required (Act 85, Yes). The clip terminal 50 stores confirmation data that is associated with the corresponding registered product on the product checklist and data corresponding to the detected object (Act 86). This allows the attendant to recognize that confirmation is particularly required, as described above.

[0082] According to at least one of the embodiments described above, by using the clip terminal 50, a match between the product information registered by the registration process by the electronic device 30 and the product corresponding to the detection object corresponding to the product put into or taken out of the cart 60 detected by the clip terminal 50 is checked. Depending on the check result, a message prompting the customer for confirmation can be displayed as appropriate, so that the customer can reliably register the product to be purchased by operation.

[0083] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0084] Furthermore, the processes described in the above-mentioned embodiments can be written as a program that can be executed by a computer onto a recording medium such as a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and provided to various devices. Also, it is possible to transmit the programs via a communication medium and provide them to various devices. The computer reads the program recorded on the recording medium, or receives the program via the communication medium, and executes the above-mentioned processes by having its operation controlled by the program. [Explanation of symbols]

[0085] 12...store system, 20A...store server, 20B...clip terminal management server, 30, 40...electronic device, 50...clip terminal (electronic device), 50A...processor, 50B...memory, 50C...storage device, 50D...touch panel, 50E...camera, 50F...communication module, 50J...reader / writer, 60...cart, 601...weight sensor.

Claims

1. A receiving means for receiving product information related to a product that a customer plans to purchase; a transmitting means for transmitting a notification of no storage to other electronic devices when the product information indicates a specific product that does not need to be stored in a storage unit for storing the product that the customer plans to purchase; An electronic device having the

2. The electronic device according to claim 1 , wherein when the product information indicates the specific product, a message indicating that storage is unnecessary is displayed.

3. and means for setting data indicating that, when the product information indicates the specific product, the product corresponding to the product information is excluded from the target for determining whether the product is stored in the storage unit; 2. The electronic device according to claim 1, wherein, when the data is set, it is not determined whether the product corresponding to the product information is stored in the storage unit.

4. In a system having a first electronic device and a second electronic device, The first electronic device includes: A receiving means for receiving product information related to a product that a customer plans to purchase; a transmitting means for transmitting a notification of no storage to the second electronic device when the product information indicates a specific product that does not need to be stored in a storage unit for storing the product that the customer plans to purchase, The second electronic device includes: A system having a display means for displaying a message indicating that storage is not required when the storage-notification is received.

5. The second electronic device includes: a storage means for storing product information relating to a product that a customer plans to purchase in response to an operation by the customer; 5. The system according to claim 4, wherein said display means displays said message together with said product information stored in said storage means.

6. Computer, A receiving means for receiving product information related to a product that a customer plans to purchase; A program for functioning as a transmitting means for transmitting a no-storage notification to other electronic devices when the product information indicates a specific product that does not need to be stored in a storage unit for storing the product that the customer plans to purchase.

Images