Item registration device and cart POS system equipped with the item registration device

Abstract

To provide an article registration device and a cart POS system that perform registration or registration cancellation of a purchased article by package recognition or identification code recognition.SOLUTION: An article registration device includes an imaging device, an illumination unit, and an information terminal. The imaging device photographs an article that passes through a photographing area. The illumination unit illuminates the photographing area from the imaging device side. The information terminal executes in a switchable manner a package recognition step of recognizing the article based on a package of the article included in a photographed image and an identification code recognition step of recognizing the article based on an identification code of the article, and performs registration or registration cancellation of the article based on a direction passing through the photographing area by using either a result of recognition of the article by means of the package or a result of recognition of the article acquired from the identification code photographed at a reading position suggested by illumination light.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an item registration device for registering or canceling registration of items to be purchased, and a cart POS system equipped with the item registration device.

Background Art

[0002] Normally, when purchasing goods, goods are registered and accounting is performed using a register installed in a store. A cart POS (Point Of Sales) system has been proposed for retail stores equipped with shopping carts. Such a cart POS system has an information terminal equipped with a touch panel, a barcode scanner, etc. on the shopping cart. A customer who is a user using the shopping cart, when placing the item to be purchased on the shopping cart, has the identification code (barcode or two-dimensional barcode (QR code (registered trademark))) of the purchased product read by a barcode scanner, and registers the product using barcode recognition technology.

[0003] Furthermore, within the scope of this system, there is a cart POS system that utilizes AI (artificial intelligence), which is called a frictionless cart, an AI cart, a smart cart, or an intelligent cart, etc. This shopping cart has an imaging device arranged around the basket, and when the goods to be purchased are put in and taken out of the basket, the goods are photographed, the product package is recognized from the image, the type of the product is automatically identified, and package recognition technology for automatically registering the product is used. This package recognition technology can implement registration or cancellation of the product as long as the user puts the product in and takes it out of the basket, without reading the barcode attached or printed on the product.

[0004] However, current package recognition technology is not yet reliable enough to recognize packages, so it is necessary to use barcode recognition technology in conjunction with it, and barcode scanners, as mentioned above, are often installed alongside the package recognition system. In such a configuration, it is necessary to switch between the imaging device, which serves as the input means, and the barcode scanner. Furthermore, the information terminal equipped with the barcode scanner is installed on the outside of the shopping cart, making it difficult to see where to hold the barcode for scanning. In addition, because products must be removed from the cart and their barcodes scanned, the path taken to register or cancel product registration becomes long, resulting in poor usability and requiring extra effort. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2016-57813 [Overview of the project] [Problems that the invention aims to solve]

[0006] The problem that the embodiments of the present invention aim to solve is to provide an item registration device and a cart POS system equipped with the item registration device that, when product recognition fails by AI-based package recognition, does not require the use of a separate barcode scanner, but instead uses the imaging device and lighting unit used for package recognition to indicate the area where the identification code should be scanned by illumination, and performs identification code recognition. [Means for solving the problem]

[0007] An item registration device according to one embodiment includes: a plurality of imaging devices for photographing items passing through a shooting area; an illumination unit provided corresponding to each of the imaging devices for illuminating the shooting area from the corresponding imaging device side; a package recognition step capable of recognizing the item based on the package of the item contained in the captured image and obtaining a first recognition result; and an identification code recognition step capable of recognizing the item by analyzing the identification code of the item contained in the image and obtaining a second recognition result; and an information terminal that, when the item cannot be recognized by the package recognition step, switches from the package recognition step to the identification code recognition step and uses the first or second recognition result to register or cancel the registration of the item to be purchased based on the direction in which it passes through the shooting area. When the information terminal switches to the identification code recognition step, Excluding one or more of the aforementioned imaging devices A specific imaging device is selected, and the selected imaging device is switched to a close-up mode that shortens the exposure time and focal length compared to the shooting mode of the package recognition process. At the same time, only the illumination unit corresponding to the imaging device switched to the close-up mode is illuminated to indicate the reading position of the identification code affixed to the article. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is an overall configuration diagram of a cart POS system equipped with an item registration device according to one embodiment. [Figure 2] Figure 2 is a schematic diagram showing the main data structure of the information terminal table. [Figure 3] Figure 3 is a block diagram showing the main circuit configuration of an information terminal. [Figure 4] Figure 4 shows the overall configuration of a shopping cart equipped with an information terminal and an item registration device. [Figure 5] Figure 5 is a flowchart illustrating the procedure for registering items using an information terminal. [Figure 6]Figure 6 is a flowchart illustrating the recovery procedure when an item cannot be recognized by the information terminal. [Modes for carrying out the invention]

[0009] The following describes an embodiment of a cart POS system equipped with an item registration device, with reference to the drawings. The item registration device 100 of this embodiment includes an information terminal 7 (described later), a camera 8 which is an imaging device, and a lighting unit 9 which is a lighting device, and is incorporated into the cart POS system 1. In the following description, barcodes and two-dimensional barcodes (QR codes®) are known as typical identification codes affixed to or printed on products for managing and identifying products.

[0010] [First Embodiment] Figure 1 is an overall configuration diagram of the cart POS system 1 according to the first embodiment. The cart POS system 1 includes a store server 2, a cart controller 3, an access point (shown as AP in Figure 1) 4, and a cashier 5. The store server 2, cart controller 3, access point 4, and cashier 5 are connected to a network 6 such as a LAN (Local Area Network) or wireless LAN. The cart POS system 1 includes a plurality of information terminals 7 connected to the network 6 via the access point 4.

[0011] The item registration device 100 is a device that allows customers to register the items they purchase themselves using an information terminal 7 and a camera 8. The information terminal 7 and camera 8 are installed on a shopping cart 10 used by customers when shopping in the store. Hereinafter, the shopping cart 10 will be simply referred to as cart 10. Multiple carts 10 are each equipped with an information terminal 7, a camera 8, and a lighting unit 9. The cart 10 is an example of a transporter that carries the items purchased by the customer using that cart 10. It is also equipped with a rechargeable battery 18 to power the information terminal 7, camera 8, and lighting unit 9.

[0012] Information terminal 7 is an information processing device that functions as a user interface for operations related to product registration or cancellation of registration (hereinafter simply referred to as "cancellation"). Information terminal 7 is equipped with a wireless unit as a means of communication with access point 4. Access point 4 relays communication between each device connected to network 6, namely the store server 2, cart controller 3, and checkout machine 5, and information terminal 7. Although only one access point 4 is shown in Figure 1, there may be two or more depending on the size of the store, etc.

[0013] The payment terminal 5 is a device that allows either a store employee or a customer to pay for registered items. For example, when a store employee is paying, a well-known counter-style POS terminal is used, and when a customer is paying, a well-known self-service POS terminal is used. There is no particular limit on the number of payment terminals 5.

[0014] Store Server 2 supports store operations such as accounting, purchasing, inventory management, and customer service (customer support, complaint handling, etc.). To support these operations, Store Server 2 manages the product database (DB) 11, the member database (DB) 12, and other databases.

[0015] Product Database 11 stores product records that describe product data for each product sold in the store. Product records include product code, price, product name, etc. The product code is the identification code for the product. Sales Database stores sales records that describe sales data for each item such as product, department, time of day, date, week, month, etc. Sales data includes the number of items sold, sales amount, etc.

[0016] Also, in this embodiment, the product database 11 stores pre - learned training data 13 for AI to recognize the products stored in the product database 11. This training data 13 is a collection of per - product training data regarding information on each product sold in the store (including the contents, external shape, package wrapping pattern, product name (character form) described, etc.). Each per - product training data is incorporated into the training data 13 in association with, for example, a product code. The training data 13 can be updated for each per - product training data regarding the product in accordance with the update of the product database 11 in response to the addition or replacement of products sold in the store. The per - product training data for each product may be created by the provider of the product and provided to the store, or may be created independently by the store.

[0017] The member database 12 stores member records that describe member data for each member. The member data includes a member ID, name, gender, email address, etc. The member ID is a code such as a unique member number set for each member to individually identify each member. A member owns a recording medium on which the member ID is recorded. The recording medium is, for example, a magnetic card, a contact - type IC (Integrated Circuit) card, a non - contact - type IC card, a smartphone, etc.

[0018] Note that although not particularly shown in the figure, the store server 2 may further manage a member settlement database. The member settlement database stores member settlement records that describe data necessary for payment for each member in the member settlement record. The member settlement record includes, in addition to the member ID, an electronic money identification code and an electronic money user ID as data necessary for payment of the amount.

[0019] The cart POS system 1 according to this embodiment restricts the payment method for non-cash settlement where a member does not use the accounting machine 5, i.e., non-cash payment, to one type of electronic money. The electronic money identification code is the identification code of the electronic money that a member identified by the member ID has registered in advance for use in paying for goods or services. The electronic money user ID is identification information for identifying a user of electronic money by the electronic money center EMC, and is assigned by the electronic money center EMC.

[0020] In order to enable payment by electronic money, a member registers these electronic money identification codes and the electronic money ID in advance, for example, at the time of member registration. If no prior registration for this electronic money is performed, NULL or 0000 shall be stored in the electronic money identification code and the electronic money ID in the accounting settlement record. Also, when electronic money can be used in this way, the store server 2 has the function of a communication server, or the cart POS system 1 includes a server. The store server 2 or the communication server supports data communication processing performed via an external communication network such as the Internet or an electronic money dedicated network between the accounting machine 5 or the cart controller 3 and the electronic money center that supports settlement by electronic money.

[0021] The cart controller 3 cooperates with the information terminal 7 to support the information terminal 7 in functioning as a POS terminal. The cart controller 3 has an information terminal table 14 corresponding to each information terminal 7. In this embodiment, an example of providing the information terminal table 14 in the cart controller 3 will be described, but a storage area may be provided in the auxiliary storage device 23 of the information terminal 7 and the information terminal table 14 may be set for each of them. In such a case, the store server 2 side may manage the information terminal table 14 together with the information terminal 7. Note that the information terminal table 14 is a part of the constituent parts of the article registration device 100.

[0022] Figure 2 is a schematic diagram showing an example of an information terminal table 14. As shown in Figure 2, the information terminal table 14 has a terminal ID area 14a, a member ID area 14b, one or more purchased product data areas 14c, and a subtotal area 14d. The terminal ID area 14a stores the terminal ID, which is a code such as a terminal number that uniquely identifies the information terminal 7 to which the information terminal table 14 corresponds. The member ID area 14b stores the member ID. The purchased product data area 14c creates a purchased product list each time a customer registers a product they intend to purchase on the information terminal 7, that is, it stores purchased product data related to the product to be purchased. Alternatively, each time a customer decides not to purchase a product that has been registered on the information terminal 7 and deletes the registration, the purchased product list is updated. The purchased product list consists of the product code, unit price, number of items sold (quantity), product name, sales amount, tax amount, etc. of the purchased product purchased by the customer. The subtotal area 14d is monetary data that shows the total sales price and tax amount for all purchased items by the customer, calculated by aggregating the list of purchased items stored in the purchased item data area 14c. This monetary data is updated each time the customer registers or cancels an item they intend to purchase, just like the purchased item list.

[0023] Figure 3 is a block diagram showing the main circuit configuration of the information terminal 7. Figure 4 is a schematic diagram showing the mounting positions of the information terminal 7, camera 8, and lighting unit 9 on the cart 10. A reader 16 is positioned adjacent to the information terminal 7. The information terminal 7 may be a dedicated device or a general-purpose device such as a tablet computer.

[0024] The information terminal 7 comprises a processor 21, main memory 22, auxiliary storage device 23, wireless unit 27, touch panel 28, speaker 29, and system transmission line 30. The system transmission line 30 includes an address bus, data bus, control signal lines, etc. The information terminal 7 connects the processor 21, main memory 22, auxiliary storage device 23, wireless unit 27, touch panel 28, and speaker 29 to the system transmission line 30. The reader 16 and camera 8 are connected to the system transmission line 30 via an interface (not shown). In the information terminal 7, the computer is composed of the processor 21, main memory 22, auxiliary storage device 23, and the system transmission line 30 connecting them. The system transmission line 30 also includes a power line for supplying power to each component from the battery 18.

[0025] The processor 21 corresponds to the central part of the computer described above. The processor 21 controls various parts to realize various functions as an information terminal 7, including the functions of registering and canceling product processing, package recognition process, and identification code recognition process, according to the operating system or application program. The processor 21 controls the camera's shooting timing (so-called shutter timing), in this case the timing of reading pixel signals from the image sensor, using an arbitrarily set trigger signal. If product recognition fails during package recognition, the processor 21 switches from the application program that executes the package recognition process to the application program that executes the identification code recognition process, and further switches the camera's shooting mode from the continuous image mode for package recognition to the identification code close-up mode, which will be described later, for identification code recognition. Conversely, if the application program is switched back from the identification code recognition process to the package recognition process, the camera's shooting mode is also switched back from the identification code close-up mode to the continuous image mode.

[0026] This processor 21 uses, for example, a CPU (Central Processing Unit) as its device, but is not limited to this. The processor 21 may also be multi-core / multi-threaded, capable of executing multiple processes in parallel. Furthermore, the processor 21 may incorporate an AI chip specialized for AI. In addition, the processor 21 may be implemented in a variety of other forms, including integrated circuits such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (field-programmable gate array), and DSP (Digital Signal processor).

[0027] Main memory 22 corresponds to the main memory portion of the computer described above. Main memory 22 includes a non-volatile memory area and a volatile memory area. In the non-volatile memory area of ​​main memory 22, at least the operating system or application programs are stored. Main memory 22 may also store data necessary for the processor 21 to perform processing to control each part in the non-volatile or volatile memory area. Main memory 22 uses the volatile memory area as a work area where data is rewritten as appropriate by the processor 21. The non-volatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0028] The auxiliary storage device 23 corresponds to the auxiliary storage portion of the computer described above. For example, EEPROM (Electric Erasable Programmable Read-Only Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive) can be the auxiliary storage device 23. The auxiliary storage device 23 stores data used by the processor 21 in performing various processes, or data created by the processing performed by the processor 21. The auxiliary storage device 23 may also store application programs that perform the above functions and processes. In this embodiment, the information terminal 7, camera 8, and lighting unit 9 store a terminal program 24 as an application program for performing the functions of the item registration device according to this embodiment (item registration and cancellation function, package recognition process, and identification code recognition process).

[0029] Furthermore, the terminal program 24 includes input limit data 25 that defines the upper limit of the number of products that can be simultaneously input. The terminal program 24 may also be stored in the main memory 22. The auxiliary storage device 23 stores training data 26, which has been learned for AI product recognition, for each product sold at the store and stored in the product database 11 of the store server 2. This training data 26 may be a copy of the training data 13 contained in the product database 11 of the store server 2. The timing of copying this training data 13 as training data 26 may be a specific timing, such as each time the information terminal 7 is started up, or it may be at any timing, such as when the training data 26 is updated and distributed from the store server 2 to each information terminal 7.

[0030] The wireless unit 27 transmits data wirelessly to the access point 4 in accordance with the wireless communication protocol. The touch panel 28 is a device that combines the input device and display device of the information terminal 7. The touch panel 28 displays an image based on image data output from the processor 21, and also detects the touch position on the displayed image and outputs the touch position information to the processor 21. As shown in Figure 4, the information terminal 7 is mounted on the cart 10 in an orientation and position that allows the customer to visually see the display on the touch panel 28 and easily perform touch operations.

[0031] The speaker 29 is housed within the casing of the information terminal 7 and plays sounds based on sound data output from the processor 21. The sounds played include voice guidance, confirmation sounds, alert sounds, etc. The voice guidance, for example, instructs the system to read the product's identification code and perform identification code recognition if product package recognition fails. It may also emit voice information related to calling a store employee (such as a notification that an employee is being called, or the voice of the employee who responded).

[0032] The reader 16 reads the data recorded on the recording medium and outputs the read data to the processor 21. If the recording medium is a magnetic card, the reader 16 is a magnetic card reader; if it is a contact-type IC card, the reader 16 is an IC card reader. In the case of a contactless IC card or a recording medium using RFID (Radio Frequency Identification), such as a smartphone, an RFID reader is used as the reader 16. The mounting position of the reader 16 on the cart 10 is not specifically defined. For example, as shown in Figure 4, it can be mounted adjacent to the information terminal 7.

[0033] As shown in Figure 4, the cart 10 is equipped with four cameras 8, which are imaging devices, so that the inside of the basket 15 of the cart 10 can be photographed from all four sides (front, back, left, and right) above. In this example, one camera 8 and two lighting units 9 form one imaging unit, and one imaging unit is arranged on each side so as to surround the square edge of the top of the basket 15. Here, an example is shown in which the lighting units 9 are arranged on both sides of the camera 8, but a configuration in which one lighting unit 9 is arranged on either side of the camera 8 is also possible. In this embodiment, an example is shown in which the lighting units 9 are arranged adjacent to the camera 8 on the side, but the position in which the lighting units 9 are arranged is not limited as long as the inside of the shooting area can be illuminated from the camera 8 side, for example, they may be arranged above the camera 8 at the opening of the basket 15. Furthermore, to prevent collisions between products and the camera 8 and lighting units 9 when products are put in or taken out of the basket 15, the thickness of the edge portion is increased and the equipment is configured to be housed within that edge portion. This embodiment describes an example configuration in which four cameras are arranged in a four-way configuration, but of course, it is possible to implement this embodiment using at least one camera.

[0034] Camera 8 is used to monitor the items in the cart 15 by taking continuous images or burst shots of items being placed in or removed from the cart 15. LED (Light Emitting Diode) lamps are provided on both sides of camera 8 as illumination units 9. These illumination units 9 are used in environments where the store is dark and the exposure is insufficient for continuous image capture inside the basket 15, or when performing identification code recognition using camera 8. Identification code recognition is more significantly affected by blur in the captured image than package recognition. Therefore, it is necessary to keep the exposure time as short as possible to prevent blurring of the captured bar. However, to compensate for the shortened exposure time and achieve a brighter image, lighting is needed to increase the exposure of the subject. Also, due to the effect of light and dark contrast, the reading depth is shortened or the depth of field becomes shallower. Therefore, in continuous image mode (or normal mode) for capturing moving or sequential images for package recognition, the shooting mode that shortens the exposure time and the focal length (corresponding to shortening the reading depth of field (DOF) in a barcode reader) in order to perform identification code recognition is called the identification code close-up mode.

[0035] In this embodiment, the illumination light of the lighting unit 9 is set to light up at the same time that the camera 8 captures the identification code, for example, using an LED lamp. The processor 21 controls the operation of the camera 8 and the lighting unit 9. Generally, in terms of exposure when the camera 8 captures the entire contents of the basket, package recognition using continuous image mode is superior to identification code recognition, i.e., it is easier to capture. Therefore, normally, product recognition is performed using only package recognition without illumination light, and only if recognition is not possible is the illumination light shone on the product to capture the identification code and register the product. In this way, the processor 21 controls the light emission of the lighting unit 9, thereby suppressing the power consumption of the battery 18 mounted on the cart 10.

[0036] Furthermore, the information terminal 7 may be equipped with a warning light 17, as shown by the dashed line in Figure 3. The warning light 17 is connected to the processor 21 via a system transmission path 30 through an interface (not shown). The warning light 17 is controlled by the processor 21 and is a lamp that signals the location of a customer requiring attention and warns customers using the cart 10 and store staff.

[0037] Figure 5 is a flowchart showing the main information processing steps executed by the processor 21 of the information terminal 7 according to the terminal program 24. Figure 6 is a flowchart showing the recovery process using identification code recognition when package recognition fails.

[0038] The following describes the main operations of the cart POS system 1 when a customer self-services the process from registering purchased items to making a payment using the information terminal 7 on the cart 10, referring to the flowchart shown in Figure 5. Note that the operations described below are just one example. The procedure is not particularly limited as long as similar results can be obtained.

[0039] First, a customer entering the store pulls out a shopping cart 10, operates the information terminal 7, and the reader 16 reads data (ID) from their ID card or smartphone, allowing them to log in. During the registration of this member ID, the information terminal table 14 of the cart controller 3, which is associated with the pulled-out shopping cart 10, is initialized (ACT1). That is, previously registered member IDs, products, and the number of registered products are reset, and then the new customer's member ID is registered (stored) in the member ID area 14b of the information terminal table 14.

[0040] The processor 21 of the information terminal 7 starts the package recognition process (ACT2) by launching an application program for package recognition and simultaneously starting to take images with the camera 8. At this time, an application program for recognition of the identification code may also be launched and left in a standby state. Image data of the product captured by the camera 8 is stored in the main memory 22. This main memory 22 has an area reserved to store image data for multiple frames of the captured product. When the processor 21 stores new image data in this area, it erases the oldest image data and stores the new image data in the reserved area.

[0041] Next, it is determined whether the customer has placed or removed an item from the basket 15. First, it is determined whether there has been a change in the captured image (ACT3). This change is determined by the processor 21 comparing the latest image data stored in the main memory 22 with the previously stored older image data to determine whether there has been a change in the image. If there is no change in the captured image (ACT3:NO), the shooting continues. On the other hand, if there is a change in the captured image (ACT3:YES), it is determined whether the item has been placed in or removed from the basket 15 (ACT4). Whether an item has been placed in the basket 15 of the cart 10 can be determined by the direction in which the item moves within the shooting area. For example, if in ACT3 it is determined that the item crossed the shooting area in a downward direction towards the bottom surface of the basket 15, in ACT4 it is determined that the item has been "placed", and the process moves to the next ACT5. Conversely, in ACT4, if it is determined that the product has crossed the shooting area in a direction away from the bottom and upward, it is determined that the product has been "removed," and the system proceeds to ACT10, which will be described later. In addition, in ACT4, if it is determined from the image that the product has not crossed the shooting area vertically, but has moved a short distance diagonally downward or horizontally, it is considered that the product in the basket 15 has simply shifted position, and there has been no change in the detected image. Furthermore, in addition to determining the direction of product movement, a weighing scale (not shown) may be placed on the bottom of the basket 15, and the addition / removal of the product may be detected by changes in weight.

[0042] In the judgment of ACT4 above, if it is determined that a product has been placed in the basket 15 (ACT4: Placement), the processor 21 uses an AI-based package recognition process to recognize the placed product (ACT5). Specifically, the processor 21 recognizes the package (shape, color, design, symbol, etc.) of the moving product in the image. The processor 21 sets a rectangular area called a bounding box for the product in the latest image, and based on the learning data 26 stored in the auxiliary storage device 23, performs image recognition of the product within the bounding box to recognize which product it is.

[0043] Next, the processor 21 determines whether or not the product has been recognized by the package recognition process (ACT6). If the recognition result is that the product has been recognized by its package [first recognition result] (ACT6:YES), then the processor registers the product to be purchased in the purchase product list of the information terminal table 14 as described above (ACT7) and updates the purchase product list (ACT8). Specifically, the information terminal 7 sends a product registration notification to the cart controller 3. Upon receiving the product registration notification, the cart controller 3 updates the purchase product list by adding the purchase product data for the product to be purchased to the purchase product list in the purchase product data area 14c of the information terminal table 14, which is associated with the member ID of the information terminal 7. The purchase product list is updated each time a customer registers a product they intend to purchase using the information terminal 7.

[0044] On the other hand, if the processor 21 fails to recognize the product in ACT6 (ACT6:NO), the system switches to the identification code recognition process in the subroutine shown in Figure 6 (described later) to perform a recovery process that reads the identification code of the product to be purchased (ACT9), thereby recognizing the product.

[0045] Next, if the product is recognized by the identification code recognition process using the subroutine, the process proceeds to ACT7, where the product purchased by the customer at the information terminal 7 is registered in the purchase product list in the information terminal table 14, and the purchase product list is updated in ACT8. Also, if the processor 21 determines in ACT4 that a product that has already been registered has been removed from the basket 15 (ACT4: removal), the processor 21 uses the AI-based package recognition process described above to recognize the removed product (ACT10).

[0046] Next, the processor 21 determines whether or not the product has been recognized by package recognition (ACT11). If the product is recognized (ACT11:YES), the processor 21 cancels the registration of the product in the purchased product list in the information terminal table 14 (ACT12), proceeds to ACT8, and updates the purchased product list. Specifically, the information terminal 7 sends a product cancellation notification to the cart controller 3. Upon receiving the product cancellation notification, the cart controller 3 updates the purchased product list by deleting the purchased product data for the product to be canceled from its memory in the purchased product list in the purchased product data area 14c of the information terminal table 14, which is associated with the member ID of the information terminal 7, thereby deleting the registration. The purchased product list is updated each time a customer cancels an item that has already been registered in the shopping cart 15.

[0047] On the other hand, in ACT11, if the product is not recognized by the package recognition process (ACT11:NO), the system switches to the identification code recognition process in a subroutine described later, performs a recovery process to read the identification code of the product whose registration is to be canceled (ACT13), and recognizes the product.

[0048] Next, if the product is recognized by the identification code recognition process, the process proceeds to ACT12, where any products already registered in the purchase product list in the information terminal table 14 are removed, and the purchase product list is updated in ACT8.

[0049] Next, it is determined whether or not to proceed with payment (ACT14). If the customer decides to proceed with payment by touching the payment button on the product registration screen of the information terminal 7 (ACT14:YES), the process moves to the payment processing step in ACT15. If the customer still wishes to purchase or return the product (ACT14:NO), the process returns to ACT3, and the product registration or retrieval process by package recognition continues.

[0050] The payment process for ACT15 is described below. By touching the payment button on the product registration screen, the processor 21 can proceed to the payment process. Although not specifically illustrated, in the payment process, for example, the processor 21 generates a barcode based on the terminal ID assigned to the information terminal 7 and displays that barcode on the touch panel 28. The payment machine 5 reads the barcode and sends a transaction inquiry command including the terminal ID to the cart controller 3. The cart controller 3 generates payment data based on the list of purchased products registered in the purchased product data area 14c of the information terminal table 14 corresponding to the terminal ID included in the transaction inquiry command, and sends that payment data back to the payment machine 5, the source of the transaction inquiry command. Based on the received payment data, the payment machine 5 prompts the customer to make a payment, processes the payment for the purchased products, and ends the main routine related to the purchase.

[0051] Furthermore, if the cart POS system 1 is configured as a system that can be paid for with electronic money, the processor 21 transmits an electronic payment command including the terminal ID to the cart controller 3 via the access point 4 using the wireless unit 27. The cart controller 3 generates payment data based on the list of purchased items registered in the purchased item data area 14c of the information terminal table 14, which corresponds to the terminal ID included in the electronic payment command, and transmits this payment data to the store server 2 along with the member ID stored in the member ID area 14b of the same information terminal table 14. The store server 2 uses the member ID sent with the payment data to obtain the electronic money user ID from the member payment record in the member payment database (not shown), and uses this electronic money user ID to perform payment with the electronic money center EMC based on the payment data.

[0052] Next, referring to the flowchart shown in Figure 6, we will explain the subroutine for the recovery process using the aforementioned identification code recognition.

[0053] First, the application being executed is switched from the package recognition process to the identification code recognition process (ACT21). When switching to this identification code recognition process, camera 8 switches from the continuous image mode used for package recognition to a shooting mode that enables close-up reading. This shooting mode can be switched to, for example, an identification code close-up mode that performs continuous shooting with the exposure time and focal length as short as possible. If a configuration is provided with multiple cameras 8, one or more of these cameras 8 can be selected and switched to the identification code close-up mode. As mentioned above, identification code recognition is more affected by blur in the captured image than package recognition. Therefore, it is necessary to prevent blurring in the captured bar, but since the exposure time is shortened, illumination light is needed to increase the exposure amount of the subject.

[0054] Next, the lighting unit 9 positioned adjacent to the camera 8, which has been switched to identification code close-up mode, is turned on, and illumination light is emitted (ACT22). This illumination light not only increases the exposure amount of the subject as described above, but also functions as an indicator to suggest the reading position for the customer to read the product's identification code. At this time, if the store is dark and the lighting unit 9 is already on as auxiliary light while package recognition is being performed, the lighting unit 9 on the side of the camera 8 that is switched to identification code close-up mode will remain lit, and the lighting unit 9 on the side of the camera 8 that was not switched to identification code close-up mode will turn off. For example, if only the two cameras 8 positioned on the left and right sides of the basket 15 are switched to identification code close-up mode, only the lighting units 9 provided adjacent to those cameras 8 will light up, and the lighting units 9 positioned in front of and behind the basket 15 will turn off or not light up. When the camera 8 reads the identification code, the illumination light may be blinking as well as on, as long as the exposure amount necessary for analyzing the identification code is obtained. Furthermore, when reading the identification code while the illumination light is flashing, it is necessary to synchronize the timing of the flashing with the timing of the shutter opening of camera 8.

[0055] Next, the information terminal 7's speaker 29 informs the customer via voice guidance that the product packaging recognition failed and instructs them to read the product's identification code to perform identification code recognition (ACT23). Specifically, the voice guidance instructs the customer to hold the product's identification code in front of the camera 8 (shooting area) and in front of the illuminated light. In addition to this voice guidance, the information terminal 7's touch panel 28 may also display a message indicating that the product packaging recognition failed and prompting the customer to read the identification code.

[0056] The customer receives an audio guide, picks up the product that failed to be recognized by the package, and holds it in front of camera 8 so that the product's identification code is illuminated by the light. If the identification code cannot be read on the first attempt, the audio guide may be played again to prompt the customer to try reading it again. The processor 21 of the information terminal 7 recognizes the identification code captured by camera 8 (ACT24). If the processor 21 can detect the product code, which is product information, from the read identification code, it considers the identification code to have been recognized.

[0057] Next, the processor 21 of the information terminal 7 determines whether or not the identification code has been recognized (ACT25). If the recognition result is that the product has been recognized by the identification code [second recognition result] (ACT25: YES), the illumination light of the illumination unit 9 is turned off (ACT26). After that, the application program to be executed is switched in order to change from the identification code recognition process to the package recognition process (ACT27). In addition, along with the switch to the package recognition process, the shooting mode of the camera 8 is adjusted from the identification code close-up mode to the normal mode, which is suitable for package recognition, by increasing the exposure time and increasing the reading depth. After switching from the identification code recognition process to the package recognition process, the recovery processing subroutine is terminated and the process moves to ACT7 or ACT12 of the main routine shown in Figure 5.

[0058] Next, in the judgment of ACT25 described above, if the information terminal 7 is unable to recognize the identification code (ACT25: NO), it is judged as an identification code recognition error, and the store server 2 notifies the store terminal (not shown) or the intercom carried by the store clerk that a customer has called due to an error. At this time, the touch panel 28 of the information terminal 7 displays a comment to the customer suggesting that they wait in place (ACT26). This comment includes the instruction to stay in their current location and that they have called a store clerk. The warning light 17 may also be turned on to inform the store clerk of the customer's location. Note that the touch panel 28 of the information terminal 7 displays an error clearing button (not shown), and by touching the error clearing button, the waiting state due to the error notification is cleared, and the system can return to ACT7 or ACT12 of the main routine shown in Figure 5.

[0059] As described above, according to the cart POS system equipped with the item registration device of the embodiment, when the cameras positioned around the basket fail to recognize the product through package recognition, the cameras are switched to an identification code close-up mode with a shorter exposure time and reading depth, and used to read the identification code. In other words, instead of using a dedicated identification code reading device that scans a light beam to read the identification code, the camera's shooting settings are changed to enable proper capture of the identification code. Furthermore, by turning on the lighting unit, the amount of exposure light necessary to analyze the identification code is supplemented, and the reading position where the customer should hold the identification code is intuitively easy to understand. This eliminates the need for a dedicated device to read the identification code. Also, since the identification code can be read simply by facing the side of the product with the identification code attached to the camera under the illumination light within the basket 15, the distance traveled for reading the product can be shortened. In addition, if package recognition is not possible, the system automatically switches to identification code recognition, and the product's identification code can be read simply by holding it over the illuminated area (shooting area), thus reducing the effort required for shopping cart users.

[0060] Furthermore, this embodiment of the present invention is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. The original claims of this application are included below. [C1] At least one imaging device for photographing objects passing through a shooting area, An illumination unit that illuminates the shooting area from the imaging device side, A package recognition step is performed in a switchable manner to recognize the article based on the package of the article contained in the captured image, and an identification code recognition step is performed in a switchable manner to recognize the article by analyzing the identification code of the article contained in the captured image, and a first recognition result of the article based on the package, Alternatively, if the item cannot be recognized by the package recognition step, the information terminal switches from the package recognition step to the identification code recognition step, illuminates the illumination unit to indicate the reading position of the identification code, and uses either of the second recognition results of the item obtained by analyzing the identification code of the item photographed by the imaging device to register or cancel the registration of the item to be purchased, based on the direction in which it passes through the shooting area. An item registration device equipped with the following features. [C2] When the information terminal switches to the identification code recognition process, The imaging device is changed to a close-up mode, which shortens the exposure time and focal length compared to the shooting mode of the package recognition process. The article registration device according to C1, wherein the illumination unit illuminates only the shooting area of ​​the imaging device used in the identification code recognition process from the imaging device side, and indicates the reading position of the identification code affixed to the article. [C3] The information terminal comprises a processor and a storage unit, and the package recognition step and the identification code recognition step are executed by the processor in accordance with an application program stored in the storage unit, as described in C1. [C4] Multiple carts equipped with the item registration device described in C1 and functioning as POS (Point of Sales) terminals, A cart controller having an information terminal table which is rewritten each time an item is registered or canceled, and which has a purchase item list set for each information terminal that stores the registration or cancellation of the item. A product database that stores rewritable product information used for certification in the package recognition process and the identification code recognition process, and a store server that stores training data learned for product recognition using artificial intelligence, A cart POS system equipped with an item registration device, which includes an accounting machine that performs payment for the items registered in the information terminal table by the aforementioned information terminal. [C5] The cart is equipped with a basket for storing the items, The cart POS system according to C4, wherein the imaging device and the illumination unit of the item registration device are arranged at the top of the opening of the basket. [C6] The aforementioned information terminal communicates individually with the cart controller, store server, and accounting machine, which are connected to the network via an access point, as described in C4, for the cart POS system. [Explanation of symbols]

[0061] 1... Cart POS system, 2... Store server, 3... Cart controller, 4... Access point, 5... Cash register, 6... Network, 7... Information terminal, 8... Camera, 9... Lighting unit, 10... Shopping cart (cart), 11... Product database, 12... Member database, 13... Learning data, 14... Information terminal table, 14a... Terminal ID area, 14b... Member ID area, 14c... Purchased product data area, 14d... Subtotal area, 15... Basket, 16... Reader, 17... Warning light, 21... Processor, 22... Main memory, 23... Auxiliary storage device, 24... Terminal program, 25... Input quantity limit data, 26... Learning data, 27... Wireless unit, 28... Touch panel, 29... Speaker, 30... System transmission line, 100... Item registration device.

Claims

1. Multiple imaging devices that photograph objects passing through the shooting area, Each of the aforementioned imaging devices is provided with a corresponding illumination unit that illuminates the shooting area from the corresponding imaging device side, The system is capable of performing a package recognition step, which recognizes the article based on the package of the article contained in the captured image and obtains a first recognition result, and an identification code recognition step, which recognizes the article by analyzing the identification code of the article contained in the image and obtains a second recognition result. If the package recognition step fails to recognize the item, the process switches from the package recognition step to the identification code recognition step. The system includes an information terminal that uses the first recognition result or the second recognition result to register or cancel the registration of items to be purchased based on the direction in which the items pass through the shooting area, When the information terminal switches to the identification code recognition process, From among the multiple imaging devices, one or more imaging devices are excluded, and the selected imaging device is switched to a close-up mode that shortens the exposure time and focal length compared to the shooting mode of the package recognition process. An item registration device that illuminates only the illumination unit corresponding to the imaging device switched to the close-up mode to indicate the reading position of the identification code affixed to the item.

2. The article registration device according to claim 1, wherein the information terminal comprises a processor and a storage unit, and the package recognition step and the identification code recognition step are executed by the processor in accordance with an application program stored in the storage unit.

3. A plurality of carts equipped with the item registration device described in claim 1 and functioning as POS (Point of Sales) terminals, A cart controller having an information terminal table which is rewritten each time an item is registered or canceled, and which has a purchase item list set for each information terminal that stores the registration or cancellation of the item. A product database that stores rewritable product information used for certification in the package recognition process and the identification code recognition process, and a store server that stores training data learned for product recognition using artificial intelligence, A cart POS system equipped with an item registration device, comprising an accounting machine that performs payment for the items registered in the information terminal table by the information terminal.

4. The cart is equipped with a basket for storing the items, The cart POS system according to claim 3, wherein the imaging device and the illumination unit of the item registration device are arranged in the upper part of the opening of the basket.

5. The cart POS system according to claim 3, wherein the information terminal communicates individually with the cart controller, store server, and accounting machine, which are connected to the network via an access point.

Images