Checkout device and product quantity checking program

Abstract

To provide a checkout device that can detect the presence of unregistered articles and a program for checking the number of articles.SOLUTION: A checkout device according to an embodiment has an interface and a processor. The interface acquires images taken by a camera whose shooting range includes an area where a basket containing articles to be purchased is placed. The processor detects the number of articles removed from the basket based on the images taken by the camera acquired by the interface, and detects whether the number of articles registered in the checkout device is reasonable with respect to the number of articles removed from the basket.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to a checkout device and a program for checking the number of items.

Background Art

[0002] A checkout device is a device that registers information on all items to be purchased by a customer and performs an accounting process for settling the price of the registered items. Prior to the accounting process, the checkout device needs to perform a registration process of registering information on all items to be purchased. A conventional checkout device reads information on an item presented in a reading area by an operator's operation and performs a registration process of registering the read item information. In such a checkout device, if the operator's operation is not performed properly, information on some items may not be registered.

[0003] If the accounting process is executed with unregistered items, the checkout device cannot receive the price of all items to be purchased. Conventionally, there is a technique for detecting that there are unregistered items in a checkout device by detecting suspicious behavior of an operator from surveillance video. However, it is not easy to reliably detect unregistered items based on the behavior of the operator detected from the surveillance video. In addition, unregistration of items in a checkout device may occur not only due to an illegal operation by an operator but also accidentally by the operator. For this reason, there is a demand for a checkout device that can reliably detect the presence of unregistered items.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the present invention is to provide a checkout device capable of detecting the presence of unregistered products and a product quantity checking program.

Means for Solving the Problem

[0006] According to an embodiment, the checkout device has an interface and Scanner and a processor. The interface acquires an image captured by a camera whose shooting range includes the area where the basket containing the products to be purchased is placed. The scanner acquires product information from an image having information about the product attached to the product to be purchased. The processor detects whether the image captured by the camera acquired by the interface The number of times of detecting the entry and exit of a person's hand into and from the basket is regarded as the number of products taken out of the basket is appropriate for the number of products taken out of the basket. The number of products counted from the product information acquired by the scanner

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Best Mode for Carrying Out the Invention

[0008] Hereinafter, the first and second embodiments will be described with reference to the drawings. (First Embodiment) First, the first embodiment will be described. FIG. 1 is a block diagram showing a configuration example of a checkout system including a checkout device 1 according to the present embodiment. In the configuration example shown in FIG. 1, the checkout device 1 executes a registration process for registering (inputting) information on products to be purchased by a customer and an accounting process for settling the price of the products to be purchased. In the following description, the checkout device 1 shall execute the registration process and the accounting process as checkout processes. However, the checkout device 1 may perform the registration process and transmit the result of the registration process to a device that performs the accounting process.

[0009] The checkout device 1 executes a registration process and an accounting process for products to be purchased by a customer in response to an operator's operation. In the checkout device 1, the operator who performs operations for the registration process and the accounting process may be the customer himself / herself or a staff member such as a store clerk. That is, the checkout device 1 may be a self-checkout device (self-checkout register) that performs a registration process and an accounting process for products to be purchased by the operation of the customer himself / herself. Also, the checkout device 1 may be a device that performs a registration process and an accounting process for products to be purchased by the operation of a staff member such as a store clerk.

[0010] The workbench 2 is for an operator to place goods and the like in order to perform operations (manipulations) associated with the registration process and accounting process in the checkout device 1. The workbench 2 has a first product area Ra where products before the registration process are placed and a second product area Rb where products after the registration process are placed. In the configuration example shown in FIG. 1, the first product area Ra and the second product area Rb are provided on both sides of the checkout device 1. The first product area Ra is a product placement area for placing products before the registration process. Also, the second product placement table 12 is for placing products after the registration process.

[0011] In the first embodiment, a fixture 3 is installed in the first product area Ra. The fixture 3 holds a shopping basket (hereinafter, also simply referred to as a basket) containing products to be purchased (products before registration) at a predetermined position. The fixture 3 only needs to fix (hold) the basket at a predetermined position and is not limited to a specific configuration.

[0012] The camera 4 takes an image for monitoring the operations of an operator operating the checkout device 1 and the like. The camera 4 is connected to the checkout device 1. The camera 4 supplies the captured image to the checkout device 1. In the first embodiment, the camera 4 is installed so as to capture a shooting range including the basket held at a predetermined position by the fixture 3 installed in the first product area Ra. Also, the camera 4 captures the opening (frontage) of the basket held by the fixture 3 from above.

[0013] Next, the configuration of the checkout device 1 according to the first embodiment will be described. FIG. 2 is a block diagram showing a configuration example of the checkout device 1 according to the first embodiment. The checkout device 1 shown in FIG. 2 has a main body 10 that stores a control unit including each component of the control system. The control unit stored in the main body 10 is realized by, for example, a computer. The main body 10 is provided with a display 21, a card reader / writer (RW) 22, a printer 23, a speaker 24, a scanner 25, a hand scanner 26, and a cash processor 27. Further, a camera 4 is connected to the main body 10.

[0014] The display 21 is a touch panel including a display device 211 and a touch sensor 212. The display device 211 displays operation guidance, icons (touch keys) selectable by the customer, or warning messages as alerts. The display device 211 is, for example, an LCD (Liquid Crystal Display). The touch sensor 212 detects the touch position of a person on the display screen (operation screen) of the display device 211. The touch sensor 212 sends information indicating the detected touch position to a processor, which will be described later, as a processing unit.

[0015] For example, the display device 211 displays a GUI (Graphical User Interface) for inputting selection instructions such as payment methods (cash, card) and various information by the customer's touch operation. The display device 211 also displays a guidance screen, an information input screen, a product registration screen, a settlement screen, etc. The guidance screen is a screen that displays guidance (message) to inform the customer of the operation method. The information input screen is, for example, an operation screen for inputting information of products without barcodes. The product registration screen is a screen that displays information about the registered products. On the product registration screen, for example, information about products identified by barcodes read by the scanner 25 or the hand scanner 26, and products specified by key operations, etc. are displayed. The settlement screen is a screen that displays the total amount of the products to be purchased (registered products), the deposit amount, and the change amount, etc. in the accounting process where the customer pays for the products.

[0016] The card reader / writer (RW) 22 processes cards. The card reader / writer 22 has a card insertion slot. The card reader / writer 22 has a function of reading data recorded on the card and a function of writing data to the card. The card reader / writer 49 may be a magnetic card that records magnetic information, or a contact-type or non-contact-type IC card. Also, the card reader / writer 22 may have a mechanism for processing multiple types of cards.

[0017] For example, the card reader / writer 22 processes a card such as a credit card or membership card for settlement inserted into the card insertion slot. Examples of cards for settlement (payment) include credit cards, debit cards, electronic money cards, prepaid cards, etc. Examples of membership cards include member cards recording member information, or point cards recording points available for use as a service, etc.

[0018] The printer 23 prints a receipt. The printer 23 has a receipt issuing port. The printer 23 issues a receipt by printing various character strings, images, etc. on receipt paper. The printer 23 is realized by, for example, a thermal printer or a dot impact printer. For example, when the accounting process is completed, the printer 23 creates a receipt printed with the processing content (accounting content) and discharges the created receipt from the receipt issuing port. The speaker 24 outputs sounds such as a warning sound for alerting or voice as a message.

[0019] The scanner 25 and the hand scanner 26 read information for identifying a product. In the present embodiment, the scanner 25 and the hand scanner 26 are assumed to read a barcode attached to the product as information for identifying the product. The scanner 25 optically reads an image of the barcode of the product held by the customer so that the barcode attached to the product faces it. The hand scanner 26 is operated manually by the customer and optically reads an image of the barcode by approaching the barcode attached to the product. Note that the scanner 25 and the hand scanner 26 only need to be able to read information for recognizing the product, and for example, they may be able to read information other than barcodes.

[0020] The cash processor 27 processes cash. The cash processor 27 has a banknote insertion slot, a banknote ejection slot, a coin insertion slot, and a coin ejection slot, etc. The cash processor 27 processes the banknotes inserted into the banknote insertion slot 27a. The cash processor 27 ejects the banknotes as change from the banknote ejection slot 27b. The cash processor 27 accepts and processes the coins inserted into the coin insertion slot 27c. The cash processor 27 ejects the coins as change from the coin ejection slot 27d.

[0021] The camera 4 is provided separately from the main body 10 of the checkout device 1 and is connected to the control unit provided in the main body 10. However, the camera 4 may be provided integrally with the main body 10 as one device constituting the checkout device 1. The camera 4 captures an image of a shooting range including the first product area Ra where the products before the registration process (unregistered) are placed.

[0022] In the first embodiment, the camera 4 is installed so as to capture the opening of the basket containing the products before the registration process, which is fixed by the fixture 3 installed in the first product area Ra, from above. That is, the shooting range of the camera 4 is set so as to capture the entire inside of the basket placed at the predetermined position defined by the fixture 3. The image (video) captured by the camera 4 is supplied to the control unit provided in the main body 10 of the checkout device 1.

[0023] In the configuration example shown in FIG. 2, the control unit provided in the main body 10 of the checkout device 1 has a processor 40, a ROM (read-only memory) 41, a RAM (random-access memory) 42, a data memory 43, and various interfaces (IFs) 44 to 52. The display 21, the card reader / writer 22, the printer 23, the speaker 24, the scanner 25, the hand scanner 26, and the cash processor 27 are connected to the processor 40 in the control unit via various interfaces (IFs) 44 to 50.

[0024] Processor 40 executes various processes by executing programs. Processor 40 is, for example, a CPU (central processing unit). Processor 40 realizes various processing functions by executing programs stored in ROM 41 or data memory 43. For example, processor 40 executes a registration (input) process for registering (inputting) product information of a product as a registration mode by executing a program for product registration (input) processing. Processor 40 executes an accounting process for settling the price of a product as an accounting mode by executing a program for accounting processing.

[0025] Also, processor 40 executes a product quantity check process described later by executing a program for product quantity check processing. Also, processor 40 executes a hand detection process described later by executing a program for hand detection processing. Note that programs such as the program for product quantity check processing and the program for hand detection processing may be executed by a computer connected to the checkout device 1.

[0026] ROM 41 is a non-volatile memory that stores programs and data. For example, ROM 41 stores programs such as an operating system, middleware, and applications. Also, ROM 41 may store data that processor 40 refers to when executing various processes. RAM 42 stores working data. RAM 42 is a memory used as a so-called work area. RAM 42 appropriately stores data that processor 40 refers to when executing various processes, or data that is temporarily used.

[0027] The data memory 43 is a memory for storing data. The data memory 43 is a rewritable non-volatile memory. The data memory 43 is composed of, for example, EEPROM (registered trademark) (electric erasable programmable read-only memory), HDD (hard disk drive), or SSD (solid state drive). The data memory 43 stores data used when the processor 40 executes various processes, or data generated by the processes of the processor 40. Further, the data memory 43 may store various programs executed by the processor 40. For example, the data memory 43 may store a program for registration processing and a program for accounting processing.

[0028] Also, the RAM 42 or the data memory 43 stores information regarding products (registered products) registered in the registration process. The RAM 42 or the data memory 43 may record information regarding registered products as data in a list format (registered product list). Further, the RAM 42 or the data memory 43 stores captured images captured by the camera 28. The data memory 43 may store video data obtained by adding time stamp data indicating the capture time to the captured images captured by the camera 28.

[0029] The processor 40 is connected to the scanner 25 via the scanner IF 44. The processor 40 acquires an image (scanned image) read by the scanner 25 via the scanner IF 44. Further, the processor 40 is connected to the hand scanner 26 via the scanner IF 45. The processor 40 acquires an image (scanned image) read by the hand scanner 26 via the scanner IF 45. The processor 40 identifies the product indicated by the barcode by decoding the barcode included in the scanned image of the scanner 25 or the hand scanner 26.

[0030] The processor 40 is connected to the display 21 via the display interface 46. The processor 40 controls the display content displayed by the display device 211 of the display 21 via the display interface 46. Further, the processor 40 acquires information indicating the touch position detected by the touch sensor 212 of the display 21 via the display interface 46. For example, the processor 40 displays on the display 21 a message as an alert or an indicator for notifying the existence of a product before the registration process (unregistered). Further, the processor 40 detects an input to each icon (touch key) displayed on the display 21 based on the information indicating the touch position.

[0031] The processor 40 is connected to the card reader / writer 22 via the card interface 47. The processor 40 controls the processing of the card by the card reader / writer 22 via the card interface 47. For example, the processor 40 executes a settlement process using a credit card processed by the card reader / writer 22.

[0032] The processor 40 is connected to the printer 23 via the printer interface 48. The processor 40 controls the printer 23 via the printer interface 48. For example, the processor 40 issues a receipt indicating the result of the accounting process by the printer 23.

[0033] The processor 40 is connected to the speaker 24 via the audio interface 49. The processor 40 controls the sound output from the speaker 24 via the audio interface 49. For example, the processor 40 outputs, via the speaker 24, a sound (buzzer) as an alert for notifying the existence of a product before the registration process.

[0034] The processor 40 is connected to the cash processor 27 via the deposit / withdrawal IF 50. The processor 40 controls the cash processor 27 via the deposit / withdrawal IF 50. For example, the processor 40 causes the cash processor 27 to count the cash deposited by the customer and obtains information indicating the amount of the deposit counted by the cash processor 27. Further, the processor 40 controls the cash processor 27 to dispense the change calculated from the deposited amount and the total amount of the registered items (item price).

[0035] The processor 40 is connected to the camera 4 via the camera IF 51. The processor 40 obtains the captured image captured by the camera 4 via the camera IF 51. For example, during the registration process, the processor 40 obtains a captured image including an image of the opening of the basket set on the fixture 3 in the first product area Ra captured by the camera 4. Further, during the registration process, the processor 40 continuously (for example, at a predetermined cycle) obtains a captured image including an image of the opening of the basket set on the fixture 3 in the first product area Ra captured by the camera 4.

[0036] The processor 40 is connected to an external device via the communication IF 52. For example, the processor 40 is connected to an attendant terminal that monitors the operating status of the checkout device via the communication IF 53. The processor 40 transmits information indicating the operating status to the attendant terminal via the communication IF 52. For example, the processor 40 may transmit an alert to the attendant terminal when an error in the registration process is detected in the process described later.

[0037] Further, the processor 40 may be connected to a computer such as a management server via the communication IF 52. For example, the processor 40 transmits information regarding the items registered in the registration process in the checkout device 1 to the management server via the communication IF 52. Further, the processor 40 may appropriately obtain information regarding the items from the store computer that communicates via the communication IF 52.

[0038] Next, the operation as the first embodiment in the checkout device 1 configured as described above will be described. FIG. 3 is a flowchart for explaining an operation example of a checkout process including a registration process and an accounting process by the checkout device 1 according to the first embodiment. First, it is assumed that a customer visits in front of the checkout device 1 with the merchandise to be purchased. The customer or the staff member places the merchandise to be purchased (merchandise before registration) at a predetermined position. In the first embodiment, the customer or the staff member sets the basket containing the merchandise to be purchased on the fixture 3 installed in the first merchandise area Ra. As a result, the basket (shopping basket) containing the merchandise before registration is set at the predetermined position defined by the fixture 3 in the first merchandise area Ra.

[0039] The processor 40 of the checkout device 1 receives an instruction to start a registration process for registering merchandise information in a standby state (ACT11). The processor 40 detects the start of the registration process in response to an instruction input by an operator (the customer himself or the staff member). After the operator sets the basket containing the merchandise before registration on the fixture 3, the operator inputs an instruction to start the registration process for registering the merchandise information. For example, the processor 40 detects the start of the registration process by detecting that the operator has touched a touch key instructing the start of the registration process displayed on the display 21.

[0040] Alternatively, the processor 40 may detect the start of the registration process by detecting that the basket has been set on the fixture 3 installed in the first merchandise area Ra. Further, the processor 40 may recognize the movement of the operator from the image captured by the camera 4 and detect the start of the registration process according to the recognized movement of the operator.

[0041] When the processor 40 of the checkout device 1 detects the start of the registration process (ACT11, YES), it executes a registration process (ACT12) to register (input) the product information of the product to be purchased (for example, information including identification information such as a product code). In the registration process (registration mode), the operator places the product taken out of the basket at the reading position of the scanner 25. The processor 40 reads the product information attached to the product by the scanner 25 according to the operator's operation. For example, the scanner 25 reads a barcode indicating the product information attached to the product. The processor 40 acquires the product information read by the scanner 25. When the processor 40 acquires the information read using the scanner 25, it registers the product information such as the product identification information included in the acquired information.

[0042] Also, in the registration process, the processor 40 may register the product information of the product specified by the operator on the display 21. For example, the processor 40 registers the product information of the product specified by the operator operating the touch keys displayed on the display 21. In such a case, it is assumed that the operator performs an operation (operation) of taking out the product from the basket after specifying the product with the touch keys or immediately before specifying the product with the touch keys.

[0043] Note that the checkout device 1 may be allowed such that the product registered (input) using the hand scanner 26 is outside the basket set on the fixture 3. When assuming such an operation, for example, the processor 40 stores the information indicating the product registered using the hand scanner 26 in a memory such as the RAM 42. Thereby, the processor 40 may be able to count the number of products registered using the hand scanner 26.

[0044] During the execution of the registration process, the processor 40 receives an instruction to end the registration process (or transition to the accounting process) (ACT13). For example, when the operator (the customer himself or a staff member) wants to end the registration process, they touch the touch key on the display 21 that instructs the end of the registration process (transition to the accounting process). In this case, the processor 40 detects the end of the registration process by detecting that the operator has touched the touch key indicating the end of the registration process displayed on the display 21.

[0045] Also, the processor 40 may detect the end of the registration process by detecting that there are no products in the basket from the image captured by the camera 4. Further, the processor 40 may recognize the movement of the operator from the image captured by the camera 4 and detect the end of the registration process according to the recognized movement of the operator.

[0046] When the end of the registration process is detected (ACT13, YES), the processor 40 determines whether the number of products registered in the registration process is appropriate (ACT14). The processor 40 performs a product number check process for determining the validity of the number of registered products in parallel with the above-described registration process. The product number check process is executed as a process shown in FIG. 4 described later. The processor 40 determines whether the number of products is appropriate based on the result of the product number check process.

[0047] When it is determined that the number of registered products is not appropriate (ACT14, NO), the processor 40 aborts the checkout process without transitioning to the accounting process. In this case, the processor 40 outputs an alert. For example, the processor 40 displays an alert on the display 21 indicating that there are unregistered products. When an alert is output, the processor 40 may resume the checkout process by confirmation input of the registration content by the operator, confirmation input by the supervisor, or additional registration of products (redoing the registration process).

[0048] When it is determined that the number of registered items is appropriate (ACT14, YES), the processor 40 proceeds to the accounting process. When proceeding to the accounting process, the processor 40 executes an accounting process for settling the total amount of all registered items (ACT15). For example, the processor 40 identifies the price of each registered item and calculates the total amount of the prices of each item. The processor 40 displays the calculated total amount on the display 21 and performs a process of receiving the total amount from the customer. When the reception of the total amount of the items is completed, the processor 40 ends the accounting process for the registered items.

[0049] Next, the item number checking process in the checkout device 1 according to the first embodiment will be described. FIG. 4 is a flowchart for explaining an operation example of the item number checking process in the checkout device 1 according to the first embodiment. The processor 40 performs the item number checking process as shown in FIG. 4 in parallel with the process shown in FIG. 3 including the registration process as described above. The processor 40 sets a basket in the fixture 3 in the first item area Ra (ACT21) and waits for the start of the registration process to be detected as the item number checking process (ACT22).

[0050] When the processor 40 detects the start of the registration process (ACT22, YES), the processor 40 acquires an image captured by the camera 4 via the camera IF51 (ACT23). In the first embodiment, the camera 4 captures a shooting range including the opening of the basket set in the fixture 3. The position of the fixture 3 (the position where the fixture 3 holds the basket) in the first item area Ra is fixed. If the basket has a predetermined shape, the position (image area) of the opening of the basket in the image captured by the camera 4 is specified based on the position of the fixture 3 and the shape of the basket.

[0051] If the camera 4 is set to photograph a shooting range including the opening of the basket under certain shooting conditions, the image of the opening of the basket appears in a predetermined area in the image photographed by the camera 4. That is, the processor 40 can extract the image of the predetermined area in the image photographed by the camera 4 as the image of the opening of the basket.

[0052] When the processor 40 acquires the image photographed by the camera 4, the processor 40 executes a hand detection process based on the image photographed by the camera 4. The hand detection process is a process of detecting the entry and exit of the operator's hand with respect to the basket. Here, the hand is a part such as the palm of the hand when the operator performs an operation (work) of grasping and taking out the product in the basket. In the hand detection process, the processor 40 detects that the operator has taken out the product from the basket based on the change in the state of the entry and exit of the hand with respect to the basket.

[0053] Also, in the hand detection process, the processor 40 performs a process of counting the number of products taken out by the operator from the basket. Thereby, the processor 40 acquires information indicating the number of products taken out from the basket as a result of the hand detection process. Note that the hand detection process will be described in detail later with reference to FIG. 6.

[0054] During the registration process, the processor 40 detects an instruction to end the above-described registration process (or transition to the accounting process) (ACT25). The processor 40 repeatedly executes the processes of ACT23 and ACT24 until the product registration process ends (ACT25, NO).

[0055] When the end of the registration process is detected (ACT25, YES), the processor 40 counts the number of products registered in the registration process (ACT26). For example, the processor 40 provides a registration count counter in a memory such as the RAM 42 to count the number of registered products (registration count). The processor 40 increments the registration count counter each time product information is registered (input) in response to the operator's operation. In this case, the processor 40 can count the number of registered products based on the value of the registration count counter when the registration process ends. After counting the number of registered products, the processor 40 determines whether the number of registered products is appropriate (ACT27).

[0056] For example, the processor 40 determines whether the number of products is appropriate based on whether the number of registered products matches the number of products taken out of the basket obtained as a result of the hand detection process. Thereby, the processor 40 can confirm whether all the products taken out of the basket have been registered by the number of products.

[0057] However, it is assumed that products registered (input) using the hand scanner 26 may be outside the basket. Therefore, the processor 40 may determine whether the number of products is appropriate in consideration of the number of products registered using the hand scanner 26. For example, the processor 40 counts the number of products registered using the hand scanner 26. Thereby, the processor 40 can determine the appropriateness of the number of products in consideration of the number of products registered using the hand scanner 26.

[0058] For example, the processor 40 may be configured to determine whether the number of items is appropriate when the number obtained by subtracting the number of items registered using the hand scanner 26 from the total number of registered items matches the number of items taken out. This determination is valid when all items registered using the hand scanner 26 are outside the basket. Also, the processor 40 may be configured to determine that the registered number is appropriate when the number obtained by subtracting the number of items registered using the hand scanner 26 from the total number of registered items is greater than or equal to the number of items taken out. This determination is valid when both cases where items are registered using the hand scanner 26 outside the basket and cases where items are taken out from inside the basket are mixed.

[0059] When it is determined that the number of registered items is appropriate (ACT27, YES), the processor 40 ends the item number check process. According to the operation example shown in FIG. 3, after determining that the number of items is appropriate, the processor 40 proceeds to the accounting process for settling the price of the registered items.

[0060] Also, when it is determined that the number of registered items is not appropriate (ACT27, NO), the processor 40 outputs an alert (ACT28). For example, as the output of the alert, the processor 40 displays an alert on the display 21 indicating that the number of registered items is not appropriate (or that there are unregistered items).

[0061] Further, the processor 40 may notify an alert indicating that the number of items is not appropriate to an attendant terminal as an external device that communicates via the communication IF 52. Thereby, the attendant terminal for the supervisor to monitor the state of the checkout device can also notify an alert indicating that the number of registered items is not appropriate (or that there are unregistered items).

[0062] FIG. 5 is a diagram showing an example display of an alert to be displayed on the display 21 of the checkout device 1. In the example shown in FIG. 5, the display 21 displays a message indicating that there are unregistered products in response to an alert output instruction from the processor 40. After displaying an alert as shown in FIG. 5, the processor 40 may prompt the clerk (monitor) to take action, or may prompt the operator to confirm the registered products. In the former case, the processor 40 may proceed to the accounting process after the clerk takes action (clerk operation). In the latter case, the processor 40 may proceed to the accounting process after the operator's confirmation instruction or the additional registration of unregistered products has been carried out.

[0063] Note that the processor 40 may notify the alert to the mobile terminal held by the monitor (administrator) as an output of the alert. For example, the processor 40 may send an email including the alert notification to a predetermined email address (for example, the email address of the monitor). Further, the processor 40 may turn on the display (warning light) provided on the checkout device 1 to notify the alert.

[0064] Next, the hand detection process in the checkout device 1 according to the first embodiment will be described. FIG. 6 is a flowchart for explaining an operation example of the hand detection process in the checkout device 1 according to the first embodiment. First, the processor 40 determines whether the position of the operator's hand is inside the basket based on the image captured by the camera 4 (ACT31). The processor 40 stores the information indicating the hand position in the RAM 42. The processor 40 detects whether the hand position is outside or inside the basket by a hand exit detection process or a hand entry detection process described later. Also, when starting the registration process, the processor 40 detects whether the position of the operator's hand is inside or outside the basket from the image captured by the camera 4. Also, when starting the registration process, the processor 40 may assume that the position of the operator's hand is outside the basket as an initial state.

[0065] When it is determined that the operator's hand is inside the basket (ACT31, YES), the processor 40 executes a hand withdrawal detection process to detect that the hand has withdrawn from inside the basket (ACT32). The processor 40 detects that the hand has withdrawn from inside the basket based on an image of the opening of the basket extracted from the image captured by the camera 4. For example, the processor 40 detects that the operator's hand has withdrawn from inside the basket based on the amount of change in the image of the opening of the basket.

[0066] Here, it is assumed that the processor 40 holds an image of the opening of the basket extracted from the image captured by the camera 4 in a memory such as the RAM 42. In the first embodiment, the basket is assumed to be set on the fixture 3. For this reason, the processor 40 can specify the position where the opening of the basket is located as a fixed value based on the position where the fixture 3 holds the basket. Further, since the processor 40 can specify the position where the opening of the basket is located as a fixed value, the processor 40 can easily extract an image of the opening of the basket from the image captured by the camera 4.

[0067] Specific examples of the hand withdrawal detection process include the following methods. For example, the processor 40 calculates a displacement vector indicating the displacement of the feature point position in the current image with respect to the previous image for the image of the opening of the basket captured by the camera 4. The processor 40 determines whether the norm value of the calculated maximum displacement vector is less than the threshold value for withdrawal detection. When the norm value of the calculated maximum displacement vector is less than the threshold value for withdrawal detection, the processor 40 detects that the hand has withdrawn from inside the basket.

[0068] However, the method for detecting the withdrawal of the hand is not limited to the method described above. For example, the processor 40 may determine whether the maximum amount of change in the pixel values in the images of the opening of the basket captured by the camera 4 in time series is less than the threshold value. In this case, when the maximum amount of change in the pixel values in the image of the opening of the basket is less than the threshold value, the processor 40 may detect that the hand has withdrawn from inside the basket.

[0069] When it is detected that the hand has exited the basket (ACT33, YES), the processor 40 determines that the operator has taken out the merchandise from the basket. The processor 40 counts the number of items taken out as the number of items taken out from the basket (ACT34). For example, the processor 40 provides a count-up counter for counting the number of items taken out (the number of items taken out) in a memory such as the RAM 42. The processor 40 increments (adds 1) the number of items taken out by counting up the count-up counter every time it detects that the hand has exited the basket.

[0070] When it is detected that the hand has exited the basket (ACT33, YES), the processor 40 records information indicating that the operator's hand is outside the basket in a memory such as the RAM 42. When it cannot be detected that the hand has exited the basket (ACT33, NO), the processor 40 ends the hand detection process assuming that the position of the operator's hand with respect to the basket has not changed.

[0071] Also, when it is determined that the operator's hand is outside the basket (ACT31, NO), the processor 40 executes a hand entry detection process for detecting that the hand has entered the basket (ACT36). The processor 40 detects the entry of the hand based on the image of the opening of the basket extracted from the image captured by the camera 4. For example, the processor 40 detects the entry of the hand based on the amount of change in the image of the opening of the basket extracted from the image captured by the camera 4 and held in the RAM 42 or the like.

[0072] For example, the processor 40 calculates a displacement vector indicating the displacement of the feature point positions in the previous image and the current image as the amount of change in the image of the opening of the basket. The processor 40 determines whether the norm value of the calculated maximum displacement vector exceeds the threshold for entry detection. The processor 40 detects that the hand has entered the basket when the norm value of the calculated maximum displacement vector exceeds the threshold for entry detection.

[0073] However, the method for detecting the entry of a hand is not limited to the method described above. For example, the processor 40 may determine whether the maximum change amount of the pixel values in the image of the opening of the basket captured by the camera 4 over time exceeds a threshold value. In this case, when the maximum change amount of the pixel values in the image of the opening of the basket exceeds the threshold value, the processor 40 may detect that a hand or something has entered the basket.

[0074] When it is detected that a hand has entered the basket (ACT37, YES), the processor 40 records information indicating that the operator's hand is in the basket in a memory such as the RAM 42. When it cannot be detected that a hand has entered the basket (ACT37, NO), the processor 40 ends the hand detection process assuming that the position of the operator's hand with respect to the basket has not changed.

[0075] The processor 40 executes the hand detection process as shown in FIG. 6 every time it acquires an image captured by the camera 4 until the registration process ends. As a result, the processor 40 can count the number of products (number of removals) taken out by the operator from the basket during the period from the start to the end of the registration process as a result of the hand detection process.

[0076] As described above, the checkout device according to the first embodiment detects that a product has been taken out from a basket set at a predetermined position based on the image captured by the camera. If the number of registered products is not appropriate for the number of products taken out from the basket, the checkout device outputs an alert indicating that there is an unregistered product. Thereby, the checkout device can detect that the product taken out from the basket is not correctly registered, and can notify that there is an unregistered product or the like.

[0077] The checkout device according to the first embodiment acquires an image captured by a camera of a shooting range including the opening of a basket set at a fixed position by a fixture. The checkout device acquires an image of the opening of the basket based on the fixed position defined by the fixture from the image captured by the camera. Thereby, the checkout device can surely acquire an image of the opening of the basket from the image captured by the camera, and can perform the hand detection process with high accuracy. As a result, the checkout device can accurately count the number of products taken out of the basket and can accurately detect unregistered products.

[0078] (Second Embodiment) Next, the second embodiment will be described. The checkout device 1 according to the second embodiment is assumed to have the same hardware configuration as the checkout device described in the first embodiment. That is, the checkout device 1 related to the second embodiment is assumed to be a device having the configuration as shown in FIG. 2 described in the first embodiment. For this reason, detailed description of the hardware configuration of the checkout device 1 according to the second embodiment will be omitted.

[0079] The checkout device 1 according to the second embodiment is different from the first embodiment in that it detects that a product before registration has been taken out from an image of a basket placed at an arbitrary position or a shopping cart (hereinafter referred to as a cart). That is, the checkout device 1 according to the second embodiment is used in a system in which the position where the basket containing the product before registration is placed or the position where the cart on which the product before registration is placed stops is an arbitrary position. For example, the checkout device 1 according to the second embodiment is applied to a system having a configuration in which the fixture 3 is omitted in the configuration example shown in FIG. 1.

[0080] When the fixture 3 is omitted in the configuration shown in FIG. 1, it is assumed that the basket containing the product before registration is placed at an arbitrary position in the first product area Ra. When the basket is placed at an arbitrary position in the first product area Ra, the camera 4 related to the second embodiment is installed so as to capture a shooting range including the entire first product area Ra.

[0081] Also, the checkout device 1 according to the second embodiment may be configured to detect that a product before registration placed on a cart stopped at a predetermined stop position (cart stop position) has been taken out. Here, the cart is assumed to include a basket part (basket) for putting products before registration. The basket part of the cart may directly hold products, or may be a shopping basket placed on the cart.

[0082] Also, the cart stop position for arranging (stopping) a cart with products before registration is provided, for example, in a region included in the first product area Ra or a region adjacent to the first product area Ra. In the checkout device 1 according to the second embodiment, the registration process is started with products before registration placed in the basket part of a cart stopped at a predetermined stop position (cart stop position).

[0083] The checkout device according to the second embodiment can also target for checking products taken out by an operator from the basket part of a cart stopped at the cart stop position. When products taken out from the basket part of the cart are also targeted for checking, the camera 4 according to the second embodiment is installed to photograph a shooting range including the opening of the basket part of the cart stopped at the cart stop position. In the following description, the opening of the basket is also considered to correspond to the opening of the basket part of the cart stopped at the cart stop position.

[0084] Next, the product quantity checking process in the checkout device 1 according to the second embodiment will be described. FIG. 7 is a flowchart for explaining an operation example of the product quantity checking process in the checkout device 1 according to the second embodiment. Also in the second embodiment, the processor 40 executes the processes as shown in FIG. 3 described in the first embodiment as the registration process and the accounting process. The processor 40 performs the product quantity checking process as shown in FIG. 7 in parallel with the processes as shown in FIG. 3.

[0085] After the basket is placed in the first product area Ra (ACT121), the processor 40 enters a waiting state for the start of the registration process (ACT122). Also, when detecting the removal of a product from the cart, the processor 40 accepts the start of the registration process after the cart has stopped at a predetermined cart stop position. When detecting the removal of a product from the cart, the camera 4 is assumed to capture an imaging range including the opening of the basket part (basket or shopping basket placed on the cart) of the cart stopped at the cart stop position.

[0086] When the processor 40 detects the start of the registration process (ACT122, YES), it acquires the image captured by the camera 4 via the camera IF51 (ACT123). Here, assuming the case of detecting the removal of a product from the basket placed in the first product area Ra, the camera 4 will be described as capturing an imaging range including the first product area Ra. The basket is placed at an arbitrary position in the first product area Ra. Therefore, the processor 40 detects the basket (the image of the basket) based on the image captured by the camera 4 (ACT131).

[0087] For example, the processor 40 detects the image of the basket from the pixel values of each pixel constituting the image captured by the camera 4 based on a threshold value set based on the color information of the basket to be used. In this case, the processor 40 specifies the image area of the opening of the basket from the image detected as the image of the basket. However, the processor 40 is not limited to the above-described image processing and may apply known image processing techniques to detect the basket. For example, the processor 40 may detect the image of the basket from the image captured by the camera 4 using an image processing technique such as template matching.

[0088] The processor 40 extracts the image of the opening of the basket from the image captured by the camera 4 according to the detected position of the basket. The image of the opening of the basket (the image of the opening of the basket captured by the camera 4) is assumed to be stored in a memory such as the RAM 42 that stores images. The processor 40 executes a hand detection process based on the image of the opening of the basket captured by the camera 4 stored in the RAM 42.

[0089] Note that the hand detection process is the same as the process described in the first embodiment, and is a process of detecting the insertion and removal of the operator's hand into and out of the basket. For example, the processor 40 executes the hand detection process as shown in FIG. 6 described in the first embodiment. As described above, through the hand detection process, the processor 40 acquires information indicating the number of products (removal quantity) detected as having been removed by the operator from the basket.

[0090] During the registration process, the processor 40 receives an instruction to end the above-described registration process (or transition to the accounting process) (ACT125). The processor 40 repeatedly executes the processes of ACT123 and 124 until the product registration process ends (ACT125, NO).

[0091] When detecting the end of the registration process (ACT125, YES), the processor 40 counts the number of products registered in the registration process (ACT126). After counting the number of registered products, the processor 40 determines whether the number of registered products is appropriate (ACT127).

[0092] The processor 40 determines whether the number of products is appropriate based on whether the number of products registered is the same as the number of products removed from the basket acquired as a result of the hand detection process. Thereby, the processor 40 can confirm whether all the products removed from the basket have been registered based on the number of products.

[0093] However, as described in the first embodiment, products registered (input) using the hand scanner 26 may also be assumed to be outside the basket. In this case, the processor 40 may determine whether the number of products is appropriate in consideration of the number of products registered using the hand scanner 26 as described in the first embodiment.

[0094] When it is determined that the number of registered products is appropriate (ACT127, YES), the processor 40 ends the check process for the number of products. According to the operation example shown in FIG. 3, after determining that the number of products is appropriate, the processor 40 proceeds to the accounting process for settling the price of the registered products.

[0095] When it is determined that the number of registered products is not appropriate (ACT127, NO), the processor 40 outputs an alert (ACT128). For example, the processor 40 displays an alert on the display 21 indicating that the number of registered products is not appropriate. Also, the processor 40 may output an alert indicating that the number of products is not appropriate to an attendant terminal as an external device that communicates via the communication IF.

[0096] Also, similar to the description in the first embodiment, the processor 40 may notify an alert to a mobile terminal held by a supervisor (administrator) as an output of the alert. For example, the processor 40 may send an email including an alert notification to a predetermined email address (for example, the supervisor's email address). Further, the processor 40 may turn on a display (warning light) provided in the checkout device 1 to notify the alert.

[0097] As described above, the checkout device according to the second embodiment detects a basket (or cart) placed at an arbitrary position based on the image captured by the camera. The checkout device extracts an image of the opening of the basket from the image captured by the camera 4 based on the detected basket. The checkout device detects that a product has been taken out based on the image of the opening of the basket captured by the camera 4. If the number of products taken out from the basket is not appropriate for the number of registered products, the checkout device outputs an alert indicating that there is an unregistered product. Thereby, the checkout device can detect that the products taken out from the basket are not correctly registered and can notify that there is an unregistered product.

[0098] According to the checkout device according to the second embodiment, a basket can be detected from an image captured by a camera, and an image of the opening of the basket can be extracted from the detected position of the basket. As a result, the checkout device according to the second embodiment can acquire an image of the opening of the basket even in an operation where the basket is placed at an arbitrary position, and can perform hand detection processing. As a result, the checkout device can accurately count the number of products taken out of the basket and can accurately detect unregistered products.

[0099] Note that the above-described first and second embodiments may be applied to a checkout device dedicated to registration processing that does not execute accounting processing. The checkout device dedicated to registration processing may transmit information regarding the products registered by the registration processing to an accounting device or an accounting system that performs accounting processing. Even in the checkout device dedicated to registration processing, it is possible to detect that there are unregistered products by executing the product number check processing including the above-described hand detection processing.

[0100] Furthermore, in the first and second embodiments, the description has been made on the assumption that the functions for implementing the present invention are pre-recorded inside the device. However, the present invention is not limited to this, and similar functions may be downloaded from a network to the device, or a device in which similar functions are stored in a recording medium may be installed in the device. As the recording medium, any form may be used as long as it can store a program such as a CD-ROM and can be read by the device. Also, the functions obtained in advance by installation or download in this way may cooperate with an OS (Operating System) or the like inside the device to realize the functions.

[0101] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and its equivalent scope.

Explanation of Symbols

[0102] 1…Checkout device, 2…Workbench, Ra…First product area, 3…Fixture, 4…Camera, 21…Display, 40…Processor, 51…Camera interface.

Claims

1. A checkout device comprising: an interface for acquiring an image captured by a camera including an area in which a basket containing products to be purchased is placed; a scanner for acquiring product information from an image having product-related information attached to the product to be purchased; a processor that counts the number of times a person's hand is detected entering and leaving the basket from the image captured by the camera and acquired through the interface as the number of products removed from the basket, and detects whether the number of products counted from the product information acquired by the scanner is appropriate for the number of products removed from the basket; A checkout device having a

2. The processor outputs an alert when the number of products counted from the product information acquired by the scanner is not valid with respect to the number of products removed from the basket. The checkout device according to claim 1 .

3. The interface acquires an image captured by a camera including, in its imaging range, the cage held in a predetermined position by a fixture installed in a predetermined position, The processor extracts an image of an opening of the basket from the image captured by the camera based on an installation position of the fixing tool, and detects the person's hand entering and leaving the basket in the image of the opening of the basket. The checkout device according to claim 1 .

4. The processor detects the basket from the image captured by the camera, extracts an image of an opening of the basket, and detects the person's hand entering and leaving the basket in the image of the opening of the basket. The checkout device according to claim 1 .

5. On the computer, Acquire an image captured by a camera including an area in which a basket containing the product to be purchased is placed; acquiring product information from an image containing information about the product attached to the product to be purchased using a scanner; Counting the number of products to be registered in the checkout device based on the acquired product information; The number of times that the camera detects a person's hand entering and leaving the basket is set as the number of products taken out of the basket, detecting whether the number of products registered in the checkout device is appropriate to the number of products removed from the basket; A program to check the number of items to execute this.

Images