Checkout device and program for checking number of articles
The checkout device uses a camera and processor to count products in real-time, addressing the issue of unregistered items by providing accurate detection and alerting operators, thus ensuring complete payment processing.
Patent Information
- Application Number
- JP2024152263
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-12-14
AI Technical Summary
Conventional checkout devices struggle to reliably detect unregistered products due to operator errors, whether intentional or unintentional, leading to incomplete payment collection.
A checkout device equipped with a camera that captures images of a basket containing products to be purchased, a processor that counts the number of products based on these images, and an alert system to notify operators of any unregistered items.
Accurately detects and alerts operators to unregistered products, ensuring complete payment processing by verifying the number of registered items against the number taken out of the basket.
Smart Images

Figure 2024161216000001_ABST
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to a checkout device and a program for checking the number of items. [Background technology]
[0002] A checkout device is a device that registers information on all products that a customer wishes to purchase and performs accounting processing to settle the price of the registered products. Prior to accounting processing, the checkout device must perform a registration process to register information on all products that are to be purchased. Conventional checkout devices perform a registration process to read information on products presented in a reading area by an operator's operation and register the read product information. With such checkout devices, if the operator's operation is not performed reliably, information on some products may not be registered.
[0003] If a checkout process is performed when there are unregistered products, the checkout device cannot collect the payment for all of the products to be purchased. Conventionally, there is technology that detects the presence of unregistered products at a checkout device by detecting suspicious behavior of an operator from surveillance video. However, it is not easy to reliably detect unregistered products from the behavior of an operator detected from surveillance video. Furthermore, the non-registration of products at a checkout device is not limited to fraudulent operations by an operator, but can also occur unintentionally by the operator. For this reason, there is a demand for a checkout device that can reliably detect the presence of unregistered products. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2011-141644 A Summary of the Invention [Problem to be solved by the invention]
[0005] An object of the present invention is to provide a checkout device and a program for checking the number of items that can detect the presence of an unregistered item. [Means for solving the problem]
[0006] According to an embodiment, the checkout device includes an interface and a processor. The interface acquires an image captured by a camera having an image capturing range including an area where a basket containing products to be purchased is placed. The processor detects the number of products removed from the basket based on the image captured by the camera acquired by the interface, and detects whether the number of products registered in the checkout device is appropriate for the number of products removed from the basket. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a checkout system including a checkout device according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the checkout device according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating the flow of registration processing and transaction processing at the checkout terminal according to the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of a process for checking the number of products in the checkout device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of an alert displayed on the display of the checkout terminal according to the first embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of a hand detection process in the checkout device according to the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a process for checking the number of products in the checkout device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, the first and second embodiments will be described with reference to the drawings. (First embodiment) First, the first embodiment will be described. FIG. 1 is a block diagram showing an example of the configuration of a checkout system including a checkout device 1 according to this embodiment. 1, the checkout terminal 1 executes a registration process in which a customer registers (inputs) information on products to be purchased, and a checkout process in which the customer pays for the products to be purchased. In the following explanation, the checkout terminal 1 executes the registration process and the checkout process as the checkout process. However, the checkout terminal 1 may also execute the registration process and transmit the result of the registration process to a device that executes the checkout process.
[0009] The checkout terminal 1 executes registration processing and accounting processing for products to be purchased by a customer in response to an operation by an operator. In the checkout terminal 1, the operator who performs the operations for the registration processing and accounting processing may be the customer himself or a staff member such as a store clerk. In other words, the checkout terminal 1 may be a self-checkout terminal (self-register) that performs registration processing and accounting processing for products to be purchased by operation by the customer himself. The checkout terminal 1 may also be a terminal that performs registration processing and accounting processing for products to be purchased by operation by a staff member such as a store clerk.
[0010] The work table 2 is used for placing products and the like so that an operator can carry out tasks (operations) associated with the registration process and the transaction process at the checkout device 1. The work table 2 has a first product area Ra in which products before the registration process are placed, and a second product area Rb in which products after the registration process are placed. In the configuration example shown in FIG. 1, the first product area Ra and the second product area Rb are provided on both sides of the checkout device 1. The first product area Ra is a product placement area for placing products before the registration process. Also, the second product placement table 12 is used for placing products after the registration process.
[0011] In the first embodiment, a fixture 3 is installed in the first product area Ra. The fixture 3 holds a shopping basket (hereinafter also simply referred to as a basket) containing products to be purchased (products before registration) in a predetermined position. The fixture 3 may be any device that fixes (holds) the basket in a predetermined position, and is not limited to a specific configuration.
[0012] The camera 4 captures images for monitoring the actions of an operator operating the checkout device 1. The camera 4 is connected to the checkout device 1. The camera 4 supplies the captured images to the checkout device 1. In the first embodiment, the camera 4 is installed so as to capture an image of a capture range including a basket held in a predetermined position by the fixture 3 installed in the first product area Ra. The camera 4 also captures an image of the opening (frontage) of the basket held by the fixture 3 from above.
[0013] Next, the configuration of the checkout device 1 according to the first embodiment will be described. FIG. 2 is a block diagram showing an example of the configuration of the checkout device 1 according to the first embodiment. The checkout device 1 shown in Fig. 2 has a main body 10 that houses a control unit including each component of a control system. The control unit housed in the main body 10 is realized by, for example, a computer. The main body 10 is provided with a display 21, a card reader / writer (RW) 22, a printer 23, a speaker 24, a scanner 25, a hand scanner 26, and a cash processing machine 27. Furthermore, a camera 4 is connected to the main body 10.
[0014] The display 21 is a touch panel including a display device 211 and a touch sensor 212. The display device 211 displays operation guides, icons (touch keys) selectable by a customer, or warning messages as alerts. The display device 211 is, for example, an LCD (Liquid Crystal Display). The touch sensor 212 detects a touch position of a person on the display screen (operation screen) of the display device 211. The touch sensor 212 sends information indicating the detected touch position to a processor, which will be described later, serving as a processing unit.
[0015] For example, the display device 211 displays a GUI (Graphical User Interface) for inputting a selection instruction such as a payment method (cash, card) and various information by a customer's touch operation. The display device 211 also displays a guidance screen, an information input screen, a product registration screen, a settlement screen, and the like. The guidance screen is a screen that displays guidance (messages) to inform the customer of the operation method. The information input screen is, for example, an operation screen for inputting information on a product that does not have a barcode. The product registration screen is a screen that displays information on a registered product. The product registration screen displays, for example, information on a product identified by a barcode read by the scanner 25 or the hand scanner 26 and a product specified by key operation. The settlement screen is a screen that displays the total price of the products to be purchased (registered products), the amount to be deposited, the amount of change, and the like in the accounting process in which the customer pays for the products.
[0016] The card reader / writer (RW) 22 processes cards. The card reader / writer 22 has a card insertion slot. The card reader / writer 22 has a function of reading data recorded on a card and a function of writing data to the card. The card reader / writer 49 may be a magnetic card that records magnetic information, or may be a contact or non-contact IC card. The card reader / writer 22 may also have a mechanism for processing multiple types of cards.
[0017] For example, the card reader / writer 22 processes cards such as credit cards or membership cards for payment that are inserted into a card insertion slot. Cards for payment (settlement) include credit cards, debit cards, electronic money cards, prepaid cards, etc. Furthermore, membership cards include member's cards that record member information, or point cards that record points that can be used as a service.
[0018] The printer 23 prints receipts. The printer 23 has a receipt issuing port. The printer 23 issues receipts by printing various characters, images, and the like on receipt paper. The printer 23 may be, for example, a thermal printer or a dot-matrix printer. For example, when a transaction process is completed, the printer 23 creates a receipt on which the transaction details (transaction details) are printed and ejects the created receipt from the receipt issuing port. The speaker 24 outputs sounds such as a warning sound as an alert or a voice message.
[0019] Scanner 25 and hand scanner 26 are for reading information that identifies a product. In this embodiment, scanner 25 and hand scanner 26 are for reading barcodes attached to products as information that identifies the products. Scanner 25 optically reads an image of a barcode of a product held by a customer so that the barcode attached to the product faces the scanner. Hand scanner 26 is handheld and operated by a customer, and optically reads an image of the barcode by being brought close to the barcode attached to the product. Scanner 25 and hand scanner 26 may be any scanner that reads information that identifies a product, and may be any scanner that reads information other than a barcode, for example.
[0020] The cash processing machine 27 processes cash. The cash processing machine 27 has a banknote input port, a banknote discharge port, a coin input port, and a coin discharge port. The cash processing machine 27 processes banknotes inserted into the banknote input port 27a. The cash processing machine 27 discharges banknotes as change from the banknote discharge port 27b. The cash processing machine 27 accepts and processes coins inserted into the coin input port 27c. The cash processing machine 27 discharges coins as change from the coin discharge port 27d.
[0021] The camera 4 is provided separately from the main body 10 of the checkout device 1, and is connected to a control unit provided within the main body 10. However, the camera 4 may be provided integrally with the main body 10 as one device constituting the checkout device 1. The camera 4 captures an image of a shooting range including a first product area Ra in which a product before registration processing (unregistered) is placed.
[0022] In the first embodiment, the camera 4 is installed so as to photograph, from above, the opening of a basket containing pre-registration products fixed by a fixture 3 installed in the first product area Ra. That is, the shooting range of the camera 4 is set so as to photograph the entire inside of the basket placed in a predetermined position defined by the fixture 3. The image (video) captured by the camera 4 is supplied to a control unit provided in the main body 10 of the checkout device 1.
[0023] 2, the control unit provided in the main body 10 of the checkout terminal 1 has a processor 40, a read-only memory (ROM) 41, a random-access memory (RAM) 42, a data memory 43, and various interfaces (IF) 44 to 52. The display 21, the card reader / writer 22, the printer 23, the speaker 24, the scanner 25, the hand scanner 26, and the cash processing machine 27 are connected to the processor 40 in the control unit via various interfaces (IF) 44 to 50.
[0024] Processor 40 executes various processes by executing programs. Processor 40 is, for example, a CPU (central processing unit). Processor 40 realizes various processing functions by executing programs stored in ROM 41 or data memory 43. For example, processor 40 executes a registration process for registering (inputting) product information of a product in a registration mode by executing a program for product registration (input) processing. Processor 40 executes a payment process for settling the price of the product in a payment mode by executing a program for payment processing.
[0025] Moreover, processor 40 executes a program for checking the number of items, thereby executing a process for checking the number of items, which will be described later. Moreover, processor 40 executes a program for detecting hands, thereby executing a process for detecting hands, which will be described later. Note that programs such as the program for checking the number of items and the program for detecting hands may be executed by a computer connected to checkout device 1.
[0026] The ROM 41 is a non-volatile memory that stores programs and data. For example, the ROM 41 stores programs such as an operating system, middleware, and applications. The ROM 41 may also store data that the processor 40 refers to when executing various processes. The RAM 42 stores working data. The RAM 42 is a memory used as a so-called work area. The RAM 42 appropriately stores data referenced when the processor 40 executes various processes, data used temporarily, and the like.
[0027] The data memory 43 is a memory that stores data. The data memory 43 is a rewritable non-volatile memory. The data memory 43 is composed of, for example, an EEPROM (registered trademark) (electric erasable programmable read-only memory), a HDD (hard disk drive), or an SSD (solid state drive). The data memory 43 stores data used when the processor 40 executes various processes, or data generated by the processes by the processor 40. The data memory 43 may also store various programs executed by the processor 40. For example, the data memory 43 may store a program for registration processing or a program for accounting processing.
[0028] Furthermore, the RAM 42 or the data memory 43 stores information about the products registered in the registration process (registered products). For example, the RAM 42 or the data memory 43 may record information about the registered products as list-format data (registered product list). The RAM 42 or the data memory 43 also stores images captured by the camera 28. The data memory 43 may store, for example, video data in which timestamp data indicating the time of capture is added to the images captured by the camera 28.
[0029] Processor 40 connects to scanner 25 via scanner IF 44. Processor 40 acquires an image (scanned image) read by scanner 25 via scanner IF 44. Processor 40 also connects to hand scanner 26 via scanner IF 45. Processor 40 acquires an image (scanned image) read by hand scanner 26 via scanner IF 45. Processor 40 decodes a barcode included in the scanned image of scanner 25 or hand scanner 26 to identify a product indicated by the barcode.
[0030] The processor 40 is connected to the display 21 via the display IF 46. The processor 40 controls the display content displayed by the display device 211 of the display 21 via the display IF 46. The processor 40 also acquires information indicating a touch position detected by the touch sensor 212 of the display 21 via the display IF 46. For example, the processor 40 displays a message on the display 21 as an alert or indicator to notify the user that there is a product that has not yet been registered (unregistered). The processor 40 also detects an input to each icon (touch key) displayed on the display 21 using the information indicating the touch position.
[0031] The processor 40 is connected to the card reader / writer 22 via the card IF 47. The processor 40 controls processing on the card by the card reader / writer 22 via the card IF 47. For example, the processor 40 executes a credit card settlement process processed by the card reader / writer 22.
[0032] The processor 40 is connected to the printer 23 via the printer IF 48. The processor 40 controls the printer 23 via the printer IF 48. For example, the processor 40 causes the printer 23 to issue a receipt indicating the result of the transaction.
[0033] The processor 40 is connected to the speaker 24 via the audio IF 49. The processor 40 controls the sound output from the speaker 24 via the audio IF 49. For example, the processor 40 outputs a sound (buzzer) from the speaker 24 as an alert to notify the user that a product that has not yet been registered is present.
[0034] Processor 40 is connected to the cash processing machine 27 via the deposit / withdrawal IF 50. Processor 40 controls the cash processing machine 27 via the deposit / withdrawal IF 50. For example, processor 40 counts the cash deposited by a customer by the cash processing machine 27 and acquires information indicating the deposited amount counted by the cash processing machine 27. Processor 40 also controls the cash processing machine 27 to dispense change calculated from the deposited amount and the total amount of registered products (product prices).
[0035] The processor 40 is connected to the camera 4 via the camera IF 51. The processor 40 acquires images captured by the camera 4 via the camera IF 51. For example, during the registration process, the processor 40 acquires images including an image of the opening of the basket set in the fixture 3 in the first product area Ra captured by the camera 4. The processor 40 also acquires images including an image of the opening of the basket set in the fixture 3 in the first product area Ra captured by the camera 4 continuously (for example, at a predetermined cycle) during the registration process.
[0036] The processor 40 connects to an external device via the communication IF 52. For example, the processor 40 connects to an attendant terminal that monitors the operation status of the checkout device via the communication IF 53. The processor 40 transmits information indicating the operation status to the attendant terminal via the communication IF 52. For example, the processor 40 may transmit an alert to the attendant terminal when an error in the registration process is detected in the process described below.
[0037] The processor 40 may also be connected to a computer such as a management server via the communication IF 52. For example, the processor 40 transmits information about products registered in the registration process at the checkout device 1 to the management server via the communication IF 52. The processor 40 may also appropriately acquire information about products from a store computer with which it communicates via the communication IF 52.
[0038] Next, an operation of the checkout device 1 configured as above according to a first embodiment will be described. FIG. 3 is a flowchart for explaining an example of the operation of a checkout process including a registration process and a transaction process by the checkout device 1 according to the first embodiment. First, a customer approaches the checkout device 1 with the products to be purchased. The customer or a staff member places the products to be purchased (products before registration) in a predetermined position. In the first embodiment, the customer or a staff member sets a basket containing the products to be purchased in a fixture 3 installed in the first product area Ra. As a result, the basket (shopping basket) containing the products before registration is set in the predetermined position defined by the fixture 3 in the first product area Ra.
[0039] The processor 40 of the checkout device 1 receives an instruction to start a registration process to register product information in a standby state (ACT11). The processor 40 detects the start of the registration process in response to an instruction input by an operator (the customer himself or an attendant). After the operator places a basket containing the product to be registered in the fixture 3, the operator inputs an instruction to start a registration process to register product information. For example, the processor 40 detects the start of the registration process by detecting that the operator has touched a touch key displayed on the display 21 that instructs the start of the registration process.
[0040] The processor 40 may also detect the start of the registration process by detecting that the basket has been set on the fixture 3 installed in the first product area Ra. The processor 40 may also recognize the movement of the operator from an image captured by the camera 4, and detect the start of the registration process according to the recognized movement of the operator.
[0041] When processor 40 of checkout device 1 detects the start of the registration process (ACT11, YES), it executes the registration process to register (input) product information (e.g., information including identification information such as a product code) of the product to be purchased (ACT12). In the registration process (registration mode), the operator holds the product removed from the basket over the reading position of scanner 25. Processor 40 causes scanner 25 to read the product information attached to the product in response to the operator's operation. For example, scanner 25 reads a barcode indicating the product information attached to the product. Processor 40 acquires the product information read by scanner 25. When processor 40 acquires the information read using scanner 25, it registers the product information, such as product identification information, included in the acquired information.
[0042] In addition, in the registration process, the processor 40 may register product information of a product designated by the operator on the display 21. For example, the processor 40 registers product information of a product designated by the operator operating a touch key displayed on the display 21. In such a case, the operator performs an operation (operation) to remove the product from the basket after designating the product with the touch key or immediately before designating the product with the touch key.
[0043] The checkout device 1 may be configured to allow products registered (input) using the hand scanner 26 to be outside the basket set in the fixture 3. When such an operation is assumed, for example, the processor 40 stores information indicating the products registered using the hand scanner 26 in a memory such as the RAM 42. This may allow the processor 40 to count the number of products registered using the hand scanner 26.
[0044] Processor 40 receives an instruction to end the registration process (or proceed to checkout) while the registration process is in progress (ACT13). For example, when an operator (the customer or a staff member) ends the registration process, he or she touches a touch key that is displayed on display 21 and indicates the end of the registration process (proceed to checkout). In this case, processor 40 detects the end of the registration process by detecting that the operator has touched the touch key that is displayed on display 21 and indicates the end of the registration process.
[0045] Processor 40 may also detect the end of the registration process by detecting that there are no products in the basket from the image captured by camera 4. Processor 40 may also recognize the movement of the operator from the image captured by camera 4, and detect the end of the registration process based on the recognized movement of the operator.
[0046] When the end of the registration process is detected (ACT13, YES), the processor 40 judges whether the number of products registered in the registration process is appropriate (ACT14). In parallel with the above-mentioned registration process, the processor 40 performs a product number check process to determine the appropriateness of the number of registered products. The product number check process is executed as a process shown in FIG. 4, which will be described later. The processor 40 judges whether the product number is appropriate based on the result of the product number check process.
[0047] If it is determined that the number of registered items is not appropriate (ACT14, NO), processor 40 stops the checkout process without proceeding to the transaction process. In this case, processor 40 outputs an alert. For example, processor 40 displays an alert on display 21 indicating that there are unregistered items. If an alert is output, processor 40 may resume the checkout process by having the operator input confirmation of the registered details, by an attendant input confirmation, or by registering additional items (redoing the registration process), etc.
[0048] If it is determined that the number of registered items is appropriate (ACT14, YES), processor 40 proceeds to checkout processing. Once the process proceeds to checkout processing, processor 40 executes checkout processing to settle the total amount of all registered items (ACT15). For example, processor 40 identifies the price of each registered item and calculates the total amount of each item. Processor 40 displays the calculated total amount on display 21 and performs processing to collect the total amount from the customer. Once the total amount of the items has been collected, processor 40 ends the checkout processing for the registered items.
[0049] Next, a process for checking the number of products in the checkout device 1 according to the first embodiment will be described. FIG. 4 is a flowchart for explaining an example of the operation of the checkout device 1 according to the first embodiment for checking the number of products. The processor 40 performs a product quantity check process as shown in Fig. 4 in parallel with the process shown in Fig. 3, which includes the registration process described above. The processor 40 waits for a basket to be set on the fixture 3 in the first product area Ra (ACT21), and for the start of the registration process to be detected as the product quantity check process (ACT22).
[0050] When the processor 40 detects the start of the registration process (ACT22, YES), it acquires an image captured by the camera 4 via the camera IF 51 (ACT23). In the first embodiment, the camera 4 captures an image of a shooting range including the opening of the basket set in the fixture 3. The position of the fixture 3 (the position where the fixture 3 holds the basket) is fixed in the first product area Ra. If the basket has a predetermined shape, the position of the basket opening (image area) in the image captured by the camera 4 is identified based on the position of the fixture 3 and the shape of the basket.
[0051] If the camera 4 is set to capture an image of a capture range including the opening of the cage under certain capture conditions, the image of the opening of the cage will appear in a predetermined area in the image captured by the camera 4. That is, the processor 40 can extract the image of the predetermined area in the image captured by the camera 4 as the image of the opening of the cage.
[0052] When processor 40 acquires an image captured by camera 4, it executes hand detection processing based on the image captured by camera 4. The hand detection processing is processing for detecting the ingress and ingress of the operator's hand into and out of the basket. Here, the hand refers to a part of the hand, such as the palm, that performs the action (task) of grasping and removing an item from the basket. In the hand detection processing, processor 40 detects that the operator has removed an item from the basket based on a change in the state of the ingress and ingress of the hand into and out of the basket.
[0053] In addition, in the hand detection process, the processor 40 performs a process of counting the number of products that the operator has taken out of the basket. As a result of the hand detection process, the processor 40 obtains information indicating the number of products that have been taken out of the basket. The hand detection process will be described in detail later with reference to FIG. 6.
[0054] During the registration process, processor 40 detects an instruction to end the registration process (or move to checkout process) as described above (ACT25). Processor 40 repeatedly executes the processes of ACT23 and ACT24 until the product registration process is completed (ACT25, NO).
[0055] When the end of the registration process is detected (ACT25, YES), the processor 40 counts the number of products registered in the registration process (ACT26). For example, the processor 40 provides a registration number counter in a memory such as the RAM 42 to count the number of registered products (registration number). The processor 40 counts up the registration number counter every time product information is registered (input) in response to the operation of the operator. In this case, the processor 40 can count the number of registered products based on the value of the registration number counter when the registration process is completed. After counting the number of registered products, the processor 40 determines whether the number of registered products is appropriate (ACT27).
[0056] For example, processor 40 determines whether the number of registered products is valid based on whether the number of registered products matches the number of products removed from the basket obtained as a result of the hand detection process. This allows processor 40 to check whether all products removed from the basket have been registered based on the number of registered products.
[0057] However, it is assumed that an item registered (input) using hand scanner 26 may be outside the basket. For this reason, processor 40 may determine whether the number of items is appropriate by taking into account the number of items registered using hand scanner 26. For example, processor 40 counts the number of items registered using hand scanner 26. In this way, processor 40 can determine the appropriateness of the number of items by taking into account the number of items registered using hand scanner 26.
[0058] For example, processor 40 may determine whether the number of products is valid when the number obtained by subtracting the number of products registered using hand scanner 26 from the number of all registered products matches the number of products taken out. This determination is effective when all products registered using hand scanner 26 are outside the basket. Processor 40 may also determine that the number of registered products is valid when the number obtained by subtracting the number of products registered using hand scanner 26 from the number of all registered products is equal to or greater than the number of products taken out. This determination is effective when there is a mixture of cases where products are registered using hand scanner 26 outside the basket and cases where products are taken out of the basket.
[0059] If it is determined that the number of registered products is appropriate (ACT27, YES), the processor 40 ends the process of checking the number of products. According to the operation example shown in Fig. 3, after determining that the number of products is appropriate, the processor 40 proceeds to the checkout process for settling the price of the registered products.
[0060] Furthermore, when it is determined that the number of registered products is not appropriate (ACT27, NO), the processor 40 outputs an alert (ACT28). For example, the processor 40 displays, as an alert output, an alert on the display 21 indicating that the number of registered products is not appropriate (or that there are unregistered products).
[0061] Furthermore, the processor 40 may be configured to issue an alert to the effect that the number of products is inappropriate to an attendant terminal, which is an external device communicating via the communication IF 52. This allows the attendant terminal, which is used by an observer to monitor the status of the checkout device, to issue an alert indicating that the registered number of products is inappropriate (or that unregistered products exist).
[0062] FIG. 5 is a diagram showing an example of an alert displayed on the display 21 of the checkout terminal 1. As shown in FIG. In the example shown in Fig. 5, in response to an instruction to output an alert from processor 40, display 21 displays a message on display 21 indicating that there is an unregistered product. After displaying an alert as shown in Fig. 5, processor 40 may prompt an attendant (monitor) to take action, or may prompt an operator to confirm the registered products. In the former case, processor 40 may proceed to checkout after an attendant takes action (attendant operation). In the latter case, processor 40 may proceed to checkout after an instruction to confirm by the operator or after additional registration of the unregistered product is performed.
[0063] The processor 40 may output an alert by notifying a mobile terminal carried by the monitor (administrator). For example, the processor 40 may send an email containing the alert to a specific email address (e.g., the email address of the monitor). Furthermore, the processor 40 may light up a display (warning light) provided on the checkout device 1 to notify the alert.
[0064] Next, a hand detection process in the checkout device 1 according to the first embodiment will be described. FIG. 6 is a flowchart for explaining an example of the operation of the hand detection process in the checkout device 1 according to the first embodiment. First, the processor 40 determines whether the operator's hand is located inside the basket based on an image captured by the camera 4 (ACT31). The processor 40 stores information indicating the hand position in the RAM 42. The processor 40 detects whether the hand is located outside or inside the basket by a hand exit detection process or a hand entry detection process described below. Furthermore, when the registration process is started, the processor 40 detects whether the operator's hand is located inside or outside the basket based on an image captured by the camera 4. Furthermore, when the registration process is started, the processor 40 may determine that the operator's hand is located outside the basket as an initial state.
[0065] When it is determined that the operator's hand is in the basket (ACT31, YES), the processor 40 executes a hand exit detection process to detect that the hand has exited from the basket (ACT32). The processor 40 detects that the hand has exited from the basket based on an image of the basket opening extracted from an image captured by the camera 4. For example, the processor 40 detects that the operator's hand has exited from the basket based on the amount of change in the image of the basket opening.
[0066] Here, the processor 40 stores in a memory such as the RAM 42 an image of the basket opening extracted from the image captured by the camera 4. In the first embodiment, the basket is set in the fixture 3. Therefore, the processor 40 can specify the position of the basket opening as a fixed value based on the position where the fixture 3 holds the basket. In addition, since the processor 40 can specify the position of the basket opening as a fixed value, the image of the basket opening can be easily extracted from the image captured by the camera 4.
[0067] A specific example of the hand exit detection process is the following method. For example, the processor 40 calculates a displacement vector indicating the displacement of a feature point position in a current image relative to a previous image for an image of the opening of the basket captured by the camera 4. The processor 40 judges whether or not the norm value of the calculated maximum displacement vector is below a threshold value for exit detection. If the norm value of the calculated maximum displacement vector is below the threshold value for exit detection, the processor 40 detects that a hand has exited the basket.
[0068] However, the method of detecting the exit of the hand is not limited to the above-mentioned method. For example, the processor 40 may determine whether or not the maximum change in pixel value in the image of the opening of the basket captured by the camera 4 in time series is below a threshold. In this case, the processor 40 may detect that the hand has exited the basket when the maximum change in pixel value in the image of the opening of the basket is below a threshold.
[0069] When it is detected that the hand has left the basket (ACT33, YES), the processor 40 determines that the operator's hand has removed a product from the basket. The processor 40 counts the number of products removed from the basket (ACT34). For example, the processor 40 provides a removal number counter in a memory such as the RAM 42 for counting the number of products removed from the basket (removal number). The processor 40 increments the number of products removed (adds 1) by counting up the removal number counter every time it detects that the hand has left the basket.
[0070] When it is detected that the hand has left the cage (ACT33, YES), the processor 40 records information indicating that the operator's hand is outside the cage in a memory such as the RAM 42. If it is not detected that the hand has left the basket (ACT33, NO), the processor 40 determines that the position of the operator's hand relative to the basket has not changed and ends the hand detection process.
[0071] Furthermore, when it is determined that the operator's hand is outside the basket (ACT31, NO), the processor 40 executes a hand entry detection process to detect that the hand has entered the basket (ACT36). The processor 40 detects the entry of the hand based on an image of the basket opening extracted from an image captured by the camera 4. For example, the processor 40 detects the entry of the hand based on the amount of change in the image of the basket opening extracted from the image captured by the camera 4 and stored in the RAM 42 or the like.
[0072] For example, the processor 40 calculates a displacement vector indicating the displacement of the feature point position between the previous image and the current image as the amount of change in the image of the basket opening. The processor 40 judges whether or not the norm value of the calculated maximum displacement vector exceeds the threshold value for entry detection. The processor 40 detects that the hand has entered the basket when the norm value of the calculated maximum displacement vector exceeds the threshold value for entry detection.
[0073] However, the method of detecting the entry of a hand is not limited to the above-mentioned method. For example, the processor 40 may determine whether or not the maximum amount of change in pixel values in the images of the opening of the basket captured in time series by the camera 4 exceeds a threshold value. In this case, the processor 40 may detect that a hand has entered the basket when the maximum amount of change in pixel values in the images of the opening of the basket exceeds a threshold value.
[0074] When it is detected that the hand has entered from inside the basket (ACT37, YES), the processor 40 records information indicating that the operator's hand is in the basket in a memory such as the RAM 42. When it is not detected that the hand has entered from inside the basket (ACT37, NO), the processor 40 determines that the position of the operator's hand relative to the basket has not changed and ends the hand detection process.
[0075] Processor 40 executes hand detection processing as shown in Fig. 6 every time it acquires an image captured by camera 4 until the registration processing is completed. As a result of the hand detection processing, processor 40 can count the number of products (number of items taken) that the operator has taken out of the basket from the start to the end of the registration processing.
[0076] As described above, the checkout device according to the first embodiment detects that products have been removed from a basket set in a predetermined position based on an image captured by a camera. If the number of registered products is not appropriate for the number of products removed from the basket, the checkout device outputs an alert to the effect that there are unregistered products. This allows the checkout device to detect that an item removed from the basket has not been properly registered, and to notify the customer that an unregistered item exists.
[0077] The checkout device according to the first embodiment acquires an image captured by a camera of an imaging range including the opening of a basket that is set in a fixed position by a fixture. The checkout device acquires an image of the basket opening based on the fixed position defined by the fixture from the image captured by the camera. This allows the checkout device to reliably acquire an image of the basket opening from the image captured by the camera and perform hand detection processing with high accuracy. As a result, the checkout device can accurately count the number of items removed from the basket and accurately detect unregistered items.
[0078] Second embodiment Next, a second embodiment will be described. The checkout device 1 according to the second embodiment has a similar hardware configuration to the checkout device described in the first embodiment. That is, the checkout device 1 according to the second embodiment is a device having the configuration as shown in Fig. 2 described in the first embodiment. For this reason, a detailed description of the hardware configuration of the checkout device 1 according to the second embodiment will be omitted.
[0079] The checkout device 1 according to the second embodiment differs from the first embodiment in that it detects that a pre-registration product has been removed from an image of a basket or shopping cart (hereinafter, cart) placed at an arbitrary position. That is, the checkout device 1 according to the second embodiment is used in a system in which a basket containing pre-registration products is placed at an arbitrary position, or a cart containing pre-registration products is stopped at an arbitrary position. For example, the checkout device 1 according to the second embodiment is applied to a system in which the fixture 3 is omitted from the exemplary configuration shown in FIG. 1.
[0080] 1, if the fixture 3 is omitted, the basket containing the product before registration processing is placed at an arbitrary position in the first product area Ra. When the basket is placed at an arbitrary position in the first product area Ra, the camera 4 according to the second embodiment is installed so as to capture an image of a range including the entire first product area Ra.
[0081] Furthermore, the checkout device 1 according to the second embodiment may detect that a pre-registration product placed in a cart stopped at a predetermined stopping position (cart stopping position) has been removed. Here, the cart is assumed to have a basket section (basket) in which to place the pre-registration product. The basket section of the cart may be one in which products are directly placed, or may be a shopping basket placed on the cart.
[0082] Furthermore, the cart stopping position where the cart containing the product before the registration process is placed (stopped) is provided, for example, in an area included in the first product area Ra or an area adjacent to the first product area Ra. The checkout device 1 according to the second embodiment starts the registration process in a state where the product before the registration process is placed in the basket part of the cart stopped at a predetermined stopping position (cart stopping position).
[0083] The checkout device according to the second embodiment can also check products that an operator takes out of the basket of a cart stopped at the cart stopping position. When checking products taken out of the basket of a cart, the camera 4 according to the second embodiment is installed so as to capture an image of a shooting range including an opening of the basket of the cart stopped at the cart stopping position. In the following description, the opening of the basket corresponds to the opening of the basket of the cart stopped at the cart stopping position.
[0084] Next, a process for checking the number of products in the checkout device 1 according to the second embodiment will be described. FIG. 7 is a flowchart for explaining an example of the operation of the checkout device 1 according to the second embodiment for checking the number of products. In the second embodiment, the processor 40 also executes the registration process and the transaction process as shown in Fig. 3 described in the first embodiment. The processor 40 also executes the product quantity check process as shown in Fig. 7 in parallel with the process as shown in Fig. 3.
[0085] After the basket is placed in the first product area Ra (ACT121), the processor 40 enters a state of waiting for the start of the registration process (ACT122). When detecting removal of a product from the cart, the processor 40 accepts the start of the registration process after the cart is stopped at a predetermined cart stopping position. When detecting removal of a product from the cart, the camera 4 captures an image of a shooting range including the opening of a basket part (basket or shopping basket placed on the cart) of the cart stopped at the cart stopping position.
[0086] When the processor 40 detects the start of the registration process (ACT122, YES), it acquires an image captured by the camera 4 via the camera IF 51 (ACT123). Here, it is assumed that the camera 4 captures an image of a capturing range including the first product area Ra, in order to detect the removal of a product from a basket placed in the first product area Ra. The basket is placed at an arbitrary position in the first product area Ra. Therefore, the processor 40 detects the basket (image of the basket) based on the image captured by the camera 4 (ACT131).
[0087] For example, the processor 40 detects an image of the basket from the pixel values of the pixels constituting the image captured by the camera 4 based on a threshold value set based on the color information of the basket used. In this case, the processor 40 specifies the image area of the basket opening from the image detected as the image of the basket. However, the processor 40 is not limited to the image processing described above, and may detect the basket by applying a known image processing technique. For example, the processor 40 may detect the image of the basket from the image captured by the camera 4 using an image processing technique such as template matching.
[0088] Processor 40 extracts an image of the basket opening from the image captured by camera 4 according to the detected basket position. The image of the basket opening (the image of the basket opening captured by camera 4) is stored in a memory that stores images, such as RAM 42. Processor 40 executes hand detection processing based on the image of the basket opening captured by camera 4 and stored in RAM 42.
[0089] The hand detection process is a process for detecting the operator's hand moving in and out of the basket, similar to the process described in the first embodiment. For example, the processor 40 executes the hand detection process as shown in Fig. 6 described in the first embodiment. By the hand detection process as described above, the processor 40 obtains information indicating the number of products (taken number) detected as having been taken out of the basket by the operator.
[0090] During the registration process, the processor 40 receives an instruction to end the registration process (or to proceed to the checkout process) as described above (ACT125). The processor 40 repeatedly executes the processes of ACT123 and ACT124 until the product registration process is completed (ACT125, NO).
[0091] When the end of the registration process is detected (ACT125, YES), the processor 40 counts the number of products registered in the registration process (ACT126). After counting the number of registered products, the processor 40 determines whether the number of registered products is appropriate (ACT127).
[0092] The processor 40 determines whether the number of registered products is valid based on whether the number of registered products matches the number of products taken out of the basket obtained as a result of the hand detection process. This allows the processor 40 to check whether all products taken out of the basket have been registered based on the number of registered products.
[0093] However, as described in the first embodiment, it may be assumed that the product registered (input) using hand scanner 26 is outside the basket. In this case, processor 40 may determine whether the number of products is appropriate, taking into account the number of products registered using hand scanner 26, as described in the first embodiment.
[0094] If it is determined that the number of registered products is appropriate (ACT127, YES), the processor 40 ends the process of checking the number of products. According to the operation example shown in Fig. 3, after determining that the number of products is appropriate, the processor 40 proceeds to the checkout process for settling the price of the registered products.
[0095] When it is determined that the registered number of products is not appropriate (ACT127, NO), the processor 40 outputs an alert (ACT128). For example, the processor 40 displays an alert indicating that the registered number of products is not appropriate on the display 21. The processor 40 may also output an alert indicating that the number of products is not appropriate to an attendant terminal as an external device communicating via the communication IF.
[0096] As described in the first embodiment, the processor 40 may output an alert by sending an alert to a mobile terminal carried by the monitor (administrator). For example, the processor 40 may send an email containing the alert to a specific email address (e.g., the email address of the monitor). Furthermore, the processor 40 may light up a display (warning light) provided on the checkout device 1 to notify the alert.
[0097] As described above, the checkout device according to the second embodiment detects a basket (or cart) placed in any position based on an image captured by the camera. The checkout device extracts an image of the basket opening from the image captured by the camera 4 based on the detected basket. The checkout device detects that items have been removed based on the image of the basket opening captured by the camera 4. If the number of items removed from the basket is not appropriate compared to the number of registered items, the checkout device outputs an alert to the effect that there are unregistered items. This allows the checkout device to detect that the product removed from the basket has not been properly registered, and to notify the customer that there is an unregistered product.
[0098] The checkout device according to the second embodiment can detect the basket from the image captured by the camera, and extract an image of the basket opening from the detected basket position. This allows the checkout device according to the second embodiment to obtain an image of the basket opening even when the basket is placed in an arbitrary position, and can perform hand detection processing. As a result, the checkout device can accurately count the number of items removed from the basket, and can accurately detect unregistered items.
[0099] The first and second embodiments described above may also be applied to a checkout device dedicated to registration processing that does not execute transaction processing. The checkout device dedicated to registration processing may transmit information about the products registered by the registration process to a transaction device or transaction system that executes transaction processing. Even in a checkout device dedicated to registration processing, it is possible to detect the presence of unregistered products by executing a process to check the number of products, including the hand detection process described above.
[0100] Furthermore, in the first and second embodiments, the case where the function for implementing the present invention is pre-recorded in the device has been described, but this is not limited thereto, and the same function may be downloaded from a network to the device, or a recording medium having the same function stored therein may be installed in the device. The recording medium may be in any form as long as it is capable of storing a program, such as a CD-ROM, and is readable by the device. The function thus obtained by pre-installation or downloading may cooperate with an OS (operating system) in the device to realize the function.
[0101] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0102] 1...checkout device, 2...work table, Ra...first product area, 3...fixture, 4...camera, 21...display, 40...processor, 51...camera interface.
Claims
1. A checkout device comprising: an interface for acquiring an image captured by a camera including an area in which a basket containing products to be purchased is placed; a processor that detects the number of products removed from the basket based on the image captured by the camera and acquired through the interface, and detects whether the number of products registered in the checkout device is appropriate for the number of products removed from the basket; A checkout device having a
2. the processor outputs an alert when the number of products registered in the checkout device is not valid with respect to the number of products removed from the basket. The checkout device according to claim 1 .
3. The processor detects the entry and exit of a person's hand into and from the basket from the image captured by the camera, and detects the number of products taken out of the basket based on the number of times the person's hand enters and exits the basket. The checkout device according to claim 1 .
4. The interface acquires an image captured by a camera including, in its imaging range, the cage held in a predetermined position by a fixture installed in a predetermined position, The processor extracts an image of an opening of the basket from the image captured by the camera based on an installation position of the fixing device, and detects the person's hand entering and leaving the basket in the image of the opening of the basket. The checkout device according to claim 3 .
5. The processor detects the basket from the image captured by the camera, extracts an image of an opening of the basket, and detects the person's hand entering and leaving the basket in the image of the opening of the basket. The checkout device according to claim 3 .
6. On the computer, Acquire an image captured by a camera including an area in which a basket containing the product to be purchased is placed; Detecting the number of products taken out of the basket based on the image captured by the camera; and detecting whether the number of products registered in the checkout device is appropriate for the number of products taken out of the basket. A program to check the number of items to execute this.
Citation Information
Patent Citations
Settlement device, registration settlement device and information processing program
JP2019096035A
Sham registration injustice determination device, control method for the same, and program
JP2011141644A
