Product Identification System

Abstract

To provide an article identification system that enables reliable article registration using AI-based image recognition.SOLUTION: An article identification system that identifies, by an identification unit, articles placed in a cart based on images acquired by an image capturing unit installed in the cart, allows a notification unit to inform that the accuracy of identification of the articles by the identification unit is low when the number of articles in the images acquired by the image capturing unit exceeds a specified maximum number.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments of the present invention relate to a product identification system.

Background Art

[0002] A cart POS (Point Of Sales) system has been proposed for retail stores equipped with shopping carts. Such a cart POS has an information terminal provided with a touch panel, a scanner, etc. on the shopping cart. A user who is a customer using the shopping cart registers the product by having the identification code of the product to be purchased, so-called purchased product, read by the scanner when placing it on the shopping cart. Thus, since the cart POS system allows the user to register products while shopping himself / herself, it can relieve the congestion at the checkout, so-called cash register. <​​​​​​​​​​​​​​​​

[0006] [Patent Document 1] Japanese Patent Publication No. 2021-87578 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] The problem that the embodiments of the present invention aim to solve is to provide a product identification system that enables highly reliable product registration using AI-based image recognition. [Means for solving the problem]

[0008] In one embodiment, the product identification system comprises an imaging unit, a identification unit, and an alert issuing unit. The imaging unit is installed in the cart. The identification unit identifies the products placed in the cart based on the image acquired by the imaging unit. The alert issuing unit determines the number of products included in the image identified by the identification unit. The number of products to be introduced is the number of products included in the image acquired by the imaging unit. An alert will be issued when the specified upper limit is exceeded. [Brief explanation of the drawing]

[0009] [Figure 1] Overall configuration diagram of a cart POS system applying the product identification system according to the first embodiment. [Figure 2] A schematic diagram showing the main data structure of the information terminal table of a mobile controller. [Figure 3] A block diagram showing the main circuit configuration of an information terminal. [Figure 4] A schematic diagram showing the mounting position of an information terminal on a shopping cart. [Figure 5] This diagram shows the first part of a flowchart illustrating the main information processing steps performed by the processor of an information processing device in an information terminal, according to the terminal program. [Figure 6] This diagram shows the second part of a flowchart illustrating the main information processing steps performed by the processor of an information processing device in an information terminal, according to the terminal program. [Figure 7] A block diagram showing the main circuit configuration of the mobile controller according to the second embodiment. [Figure 8] This diagram shows the first part of a flowchart illustrating the main information processing steps performed by the mobile controller's processor according to the cart control program. [Figure 9] This diagram shows the second part of a flowchart illustrating the main information processing steps performed by the mobile controller's processor according to the cart control program. [Modes for carrying out the invention]

[0010] The following describes an embodiment of the product identification system with reference to drawings. Note that the following embodiment is applied to a shopping cart POS system.

[0011] [First Embodiment] Figure 1 is an overall configuration diagram of a cart POS system 1 to which the product identification system according to the first embodiment is applied. The cart POS system 1 includes a store server 2, a cart controller 3, an access point (shown as AP in Figure 1) 4, and a cashier 5. The store server 2, cart controller 3, access point 4, and cashier 5 are connected to a network 6 such as a LAN (Local Area Network) or wireless LAN. The cart POS system 1 includes a plurality of information terminals 7 connected to the network 6 via the access point 4.

[0012] Information terminal 7 is a device that allows customers to self-enter data related to the registration of purchased items. Information terminal 7 is installed in shopping carts 8 used by customers when shopping in the store. Hereinafter, shopping carts 8 will be simply referred to as cart 8. Multiple carts 8 are each equipped with an information terminal 7. A cart 8 is an example of a transport vehicle that carries the purchased items of a customer using that cart 8.

[0013] The information terminal 7 is provided on the cart 8. The information terminal 7 is an information processing terminal that functions as a user interface for operations related to registering purchased goods. The information terminal 7 includes a wireless unit. The information terminal 7 performs wireless communication with the access point 4. The access point 4 relays communication between each device connected to the network 6, namely the store server 2, the cart controller 3, and the accounting machine 5, and the information terminal 7. Although only one access point 4 is shown in FIG. 1, there may be two or more depending on the scale of the store, etc.

[0014] The accounting machine 5 is a device that enables a store clerk or a customer to perform accounting for purchased goods. The accounting machine 5 is a well-known self-service type POS terminal. The accounting machine 5 may also be a well-known face-to-face type POS terminal. The number of accounting machines 5 is not particularly limited.

[0015] The store server 2 supports store operations. For this support, the store server 2 manages a product database (DB) 21, a membership database (DB) 22, etc.

[0016] The product database 21 stores product records that describe product data for each product sold in the store. The product records include a product code, a price, a product name, etc. The product code is an identification code for the product. The sales database stores sales records that describe sales data for each item such as product, department, time zone, date, week, month, etc. The sales data includes the number of sales, the sales amount, etc.

[0017] Furthermore, in this embodiment, the product database 21 stores pre-trained learning data 211 for AI to recognize the products stored in the product database 21. This learning data 211 is a collection of learning data for each product sold in the store, and the product-specific learning data is incorporated into the learning data 211, for example, by associating it with a product code. The learning data 211 can be updated in accordance with updates to the product database 21 in response to additions or replacements of products sold in the store, and the product-specific learning data for each product can be created and provided to the store by the product provider, or it can be created independently by the store. In addition to the above-mentioned learning data, the learning data 211 may also be trained with data other than products, such as data of a person's hand.

[0018] The member database 22 stores member records that describe member data for each member. Member data includes member ID, name, gender, email address, etc. The member ID is a unique code, such as a member number, assigned to each member to identify them individually. Members possess a recording medium on which their member ID is recorded. Recording mediums include, for example, magnetic cards, contact IC (Integrated Circuit) cards, contactless IC cards, smartphones, etc.

[0019] Note that although not particularly shown in the figure, the store server 2 may further manage a membership settlement database. The membership settlement database stores membership settlement records that describe data necessary for payment of the price for each member. The membership settlement record includes, in addition to the member ID, an electronic money identification code and an electronic money user ID as data necessary for payment of the price. The cart POS system 1 according to the present embodiment limits the payment method when the member does not use the cash register 5 to one type of electronic money. The electronic money identification code is an identification code of electronic money pre-registered as being used by the member identified by the member ID for payment of the price. The electronic money user ID is an identification code for the electronic money center EMC to identify the user of the electronic money, and is assigned by the electronic money center EMC. In order to enable payment by electronic money, the member registers these electronic money identification codes and electronic money IDs in advance, for example, at the time of membership registration. If the pre-registration for this electronic money has not been performed, NULL or 0000 shall be stored in the electronic money identification code and the electronic money ID of the membership settlement record. Further, when electronic money can be used in this way, the store server 2 has a function of a communication server, or the cart POS system 1 includes a communication server. The store server 2 or the communication server supports data communication processing performed via an external communication network such as the Internet or an electronic money dedicated network between the cash register 5 or the cart controller 3 and an electronic money center that supports settlement by electronic money.

[0020] The cart controller 3 works in cooperation with the information terminal 7 to assist in making it appear as if the information terminal 7 is functioning as a POS terminal. The cart controller 3 has an information terminal table 31 corresponding to each information terminal 7. Figure 2 is a schematic diagram showing an example of the information terminal table 31. As shown in Figure 2, the information terminal table 31 has a terminal ID area 311, a member ID area 312, one or more purchased product data areas 313, and a subtotal area 314. The terminal ID area 311 stores the terminal ID, which is a code such as a terminal number that uniquely identifies the information terminal 7 to which the information terminal table 31 corresponds. The member ID area 312 stores the member ID. The purchased product data area 313 stores purchased product data for each product that a customer intends to purchase using the information terminal 7. The purchased product data consists of the product code, unit price, number of items sold, product name, sales amount, tax amount, etc. of the purchased product purchased by the customer. The subtotal area 314 is monetary data that shows the total sales price and tax amount for all purchased items by the customer, calculated based on the purchased item data stored in the purchased item data area 313. This can be updated each time the customer registers an item they intend to purchase.

[0021] Figure 3 is a block diagram showing the main circuit configuration of the information terminal 7. Figure 4 is a schematic diagram showing the mounting position of the information terminal 7 on the cart 8. The information terminal 7 includes an information processing device 71, a reader 72, and a camera 73. The information processing device 71 may be a dedicated device or a general-purpose device such as a tablet computer. The information processing device 71 includes a processor 711, a main memory 712, an auxiliary storage device 713, a wireless unit 714, a touch panel 715, a speaker 716, and a system transmission line 717. The system transmission line 717 includes an address bus, a data bus, control signal lines, etc. The information processing device 71 connects the processor 711, the main memory 712, the auxiliary storage device 713, the wireless unit 714, the touch panel 715, and the speaker 716 to the system transmission line 717. The reader 72 and the camera 73 are connected to the system transmission line 717 via an interface (not shown). In the information processing device 71, the computer is composed of a processor 711, a main memory 712, an auxiliary storage device 713, and a system transmission line 717 connecting them.

[0022] The processor 711 corresponds to the central part of the computer described above. The processor 711 controls each part to realize various functions as an information processing device 71 according to the operating system or application program. The processor 711 is, for example, a CPU (Central Processing Unit), but is not limited to that. The processor 711 may be multi-core / multi-threaded and capable of executing multiple processes in parallel. Furthermore, the processor 711 may include an AI chip specialized for AI. In addition, the processor 711 may be implemented in various other forms, including integrated circuits such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (field-programmable gate array), and DSP (Digital Signal processor).

[0023] Main memory 712 corresponds to the main memory portion of the computer described above. Main memory 712 includes a non-volatile memory area and a volatile memory area. In the non-volatile memory area, main memory 712 stores the operating system or application programs. Main memory 712 may also store data necessary for the processor 711 to perform processing to control each part in the non-volatile or volatile memory area. Main memory 712 uses the volatile memory area as a work area where data is rewritten as needed by the processor 711. The non-volatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0024] The auxiliary storage device 713 corresponds to the auxiliary storage portion of the computer described above. For example, an EEPROM (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 713. The auxiliary storage device 713 stores data used by the processor 711 in performing various processes, or data created by processing by the processor 711. The auxiliary storage device 713 may also store the application program described above. In this embodiment, the terminal program 7131 is stored as the application program to cause the information processing device 71, the reader 72, and the camera 73 to function as the information terminal 7 according to the embodiment. The terminal program 7131 includes input limit value data 71311 that defines the upper limit of the number of items that can be simultaneously inserted. The terminal program 7131 may be stored in the main memory 712. Furthermore, the auxiliary storage device 713 stores training data 7132 for each product sold at the store, which is stored in the product database 21 of the store server 2, for AI-based product recognition. This training data 7132 may be a copy of the training data 211 contained in the product database 21 of the store server 2. The timing of copying this training data 211 as training data 7132 may be a specific timing, such as each time the information processing device 71 is started up, or it may be at any timing, such as when the training data 211 is updated and distributed from the store server 2 to each information processing device 71.

[0025] The wireless unit 714 transmits data wirelessly to the access point 4 in accordance with the wireless communication protocol.

[0026] The touch panel 715 is a device that combines the input device and display device of the information terminal 7. The touch panel 715 displays an image based on image data output from the processor 711, and also detects the touch position on the displayed image and outputs the touch position information to the processor 711. As shown in Figure 4, the information processing device 71 is mounted on the cart 8 in a position and orientation that allows customers who are moving while pushing the cart 8 to view the display of the touch panel 715 and to perform touch operations.

[0027] Speaker 716 plays sounds based on sound data output from processor 711. The sounds played include confirmation sounds, alert sounds, etc.

[0028] The reader 72 reads the data recorded on the recording medium and outputs the read data to the processor 711. If the recording medium is a magnetic card, the reader 72 is a magnetic card reader; if it is a contact IC card, the reader 72 is an IC card reader. In the case of a contactless IC card or a recording medium using RFID (Radio Frequency Identification), such as a smartphone, an RFID reader is used as the reader 72. The mounting position of the reader 72 on the cart 8 is not specifically defined. For example, it can be mounted next to the information processing device 71.

[0029] As shown in Figure 4, camera 73 is mounted on the cart 8 so as to be able to photograph the basket receiving section 81 of the cart 8 from above. Camera 73 is for monitoring the basket receiving section 81 of the cart 8 or the loading or return of goods to or from the baskets placed in the basket receiving section 81. An LED (Light Emitting Diode) lamp 731 for illumination may be installed on camera 73. The operating status of camera 73 and LED lamp 731 can be controlled by processor 711 via system transmission line 717. Although only one camera 73 is shown in Figures 3 and 4, there may be two or more cameras. For example, four cameras 73 may be placed on all four sides or corners of the basket receiving section 81 of the cart 8. Depending on the type of camera 73, sufficient brightness may be ensured, so it may be possible to operate without installing an LED lamp 731.

[0030] Furthermore, the information terminal 7 may also include a warning light 74, which is connected to the system transmission path 717 via an interface (not shown) as indicated by the dashed line in Figure 3, and whose operating state is controlled by the processor 711. The warning light 74 is a lamp that alerts customers using the cart 8 and nearby store employees.

[0031] Figures 5 and 6 are flowcharts showing the main information processing steps executed by the processor 711 of the information processing device 71 in the information terminal 7 according to the terminal program 7131.

[0032] The following flowchart illustrates the main operations of the cart POS system 1 when a customer uses the information terminal 7 on the cart 8 to register and pay for their purchases themselves. Note that the operations described below are just one example; the procedure is not limited to any other method as long as similar results can be achieved. Furthermore, it is assumed that the data recorded on the recording medium has been read by the reader 72 and that the user is already logged in. That is, the member ID area 312 of the information terminal table 31 on the cart controller 3, which corresponds to the information terminal 7, already stores the member ID read from the customer's recording medium.

[0033] The processor 711 of the information processing device 71, as ACT11, turns on the LED lamp 731 of the camera 73 and causes the camera 73 to capture an image. The image data of the captured image is stored in the main memory 712. The main memory 712 has an area reserved to store multiple frames of this image data, and when the processor 711 stores new image data in this area, it erases the oldest image data and stores the new image data in the reserved area. After that, the processor 711 proceeds to ACT12.

[0034] As ACT12, the processor 711 detects moving products in the image from the latest image data stored in the main memory 712. Specifically, the processor 711 performs object detection on objects in the latest image, sets a rectangular area called a bounding box, and detects objects in the image based on the training data 7132 stored in the auxiliary storage device 713. The detected object and its detection position in the image data are associated with the image data stored in the main memory 712 and stored in the main memory 712. After that, the processor 711 proceeds to ACT13.

[0035] In ACT13, the processor 711 compares the objects and detection positions in the latest image data stored in the main memory 712 with the objects and detection positions in the older image data to determine if there has been a change. When goods are placed into the basket receiving section 81 of the cart 8 or into the basket placed in the basket receiving section 81, or when goods are removed from the basket receiving section 81 or into the basket, i.e., when goods are returned, the goods in motion for placement or return are photographed by the camera 73. Therefore, when goods are placed in or returned, a difference occurs between the objects and detection positions in the latest image data and the objects and detection positions in the older image data, so the processor 711 can determine that there has been a change. If it is determined that there has been no change, the processor 711 determines NO in ACT13 and proceeds to ACT11. If it is determined that there has been a change, the processor 711 determines YES in ACT13 and proceeds to ACT14.

[0036] The image capture process is repeated through the loop consisting of ACT11, ACT12, and ACT13 described above. When there is a change in the captured image, processor 711 proceeds to ACT14.

[0037] Processor 711, as ACT14, recognizes the moving product in the image. Specifically, based on the training data 7132 stored in the auxiliary memory device 713, processor 711 performs image recognition on the object within the bounding box detected in ACT12 and stored in the main memory 712, and identifies which product the object is. After that, processor 711 proceeds to ACT15.

[0038] The processor 711 determines in ACT15 whether or not the product has been identified. If only a small portion of the product is visible in the image, it is not possible to identify what the product is. If it is determined that the product could not be identified, the processor 711 determines NO in ACT15 and proceeds to ACT11. In this embodiment, the field of view of the camera 73 and the image capture cycle repeated in the loop of ACT11, ACT12, ACT13 and the image capture cycle repeated in the loop of ACT11, ACT12, ACT13, ACT14, ACT15 are set so that at least one image showing the entire product can be captured when a product of a predetermined maximum size is in free fall. Of course, it is also possible to configure the system to generate an image showing the entire product by combining multiple images and then perform product recognition. If it is determined that the product has been identified, the processor 711 determines YES in ACT15 and proceeds to ACT16.

[0039] In ACT16, the processor 711 determines whether or not an item has been placed in the basket receiving section 81 of the shopping cart 8 or in the basket placed in the basket receiving section 81. For example, the processor 711 can determine whether an item has been placed in or removed by looking at the direction of movement of the same item shown in the most recent image and past images. Alternatively, a weighing scale (not shown) can be placed on the bottom of the basket receiving section 81 of the shopping cart 8, and the presence or absence of an item can be detected by whether the weight has increased or decreased. If it is determined that an item has not been placed in, i.e., that the item has been removed, the processor 711 determines NO in ACT16 and proceeds to ACT21. If it is determined that an item has been placed in, the processor 711 determines YES in ACT16 and proceeds to ACT17.

[0040] The processor 711, as ACT17, determines whether the number of items to be loaded, which is the number of items recognized and identified in ACT14, is less than the maximum number of items to be loaded simultaneously, which is stored in the auxiliary storage device 713 as loading limit data 71311. In this embodiment, the maximum number of items to be loaded simultaneously is "2". In other words, ACT17 determines whether multiple items have been loaded simultaneously. If the number of items to be loaded is less than the maximum number of items to be loaded simultaneously, that is, if only one item has been loaded, the item will be recognized and identified with high accuracy. On the other hand, if multiple items are loaded simultaneously, the accuracy of identifying the items will drop drastically. Therefore, in ACT17, the processor 711 determines whether the number of items to be loaded exceeds the maximum number of items to be loaded simultaneously, that is, whether multiple items have been loaded. Note that the number of items to be loaded can be the number of bounding boxes set in the image used for image recognition, that is, the number of objects detected in ACT12. If the number of items being loaded exceeds the maximum number of items that can be loaded simultaneously, meaning that multiple items have been loaded, the processor 711 determines NO in ACT17 and proceeds to ACT24. Conversely, if the number of items being loaded is less than the maximum number of items that can be loaded simultaneously, meaning that only one item has been loaded, the processor 711 determines YES in ACT17 and proceeds to ACT18.

[0041] If the processor 711 determines in ACT 17 that the number of items to be loaded is less than the maximum number of items to be loaded simultaneously, that is, that only one item has been loaded, it determines that it has identified the item with accuracy. In this case, the processor 711 sends an item loading notification to the cart controller 3 via the access point 4 using the wireless unit 714 as ACT 18. The item loading notification may include the terminal ID assigned to the information terminal 7, which includes the information processing device 71, and the item code indicating the item recognized and identified in ACT 14. After that, the processor 711 proceeds to ACT 19.

[0042] When the cart controller 3 receives a product input notification, it registers the product being input. Specifically, the cart controller 3 queries the store server 2 for product information using the product code included in the product input notification and retrieves the product information stored in the product database 21 managed by the store server 2. The cart controller 3 then adds this retrieved product information as new purchased product data to the purchased product data area 313 of the information terminal table 31 which has the terminal ID included in the product input notification. The cart controller 3 then creates a registered product screen to notify the registered product based on the purchased product data stored in the purchased product data area 313 and sends this registered product screen to the sender of the product input notification.

[0043] As ACT19, processor 711 receives the registered product screen transmitted from cart controller 3 via access point 4 by wireless unit 714. Then, processor 711 proceeds to ACT20.

[0044] Processor 711, as ACT20, displays the received registered product screen to the customer using cart 8 via touch panel 715. After that, processor 711 proceeds to ACT11.

[0045] In this way, when an item is placed in the basket receiving section 81 of the cart 8 or in a basket placed in the basket receiving section 81, the information terminal 7 identifies the item using image recognition and sends an item entry notification to the cart controller 3 that includes an item code indicating the identified item, thereby allowing the cart controller 3 to register the item as a purchased item.

[0046] Furthermore, if the processor 711 determines in ACT16 that an item has been removed, it sends an item return notification to the cart controller 3 via the access point 4 using the wireless unit 714 as ACT21. The item return notification may include the terminal ID assigned to the information terminal 7, which includes the information processing device 71, and the item code indicating the item recognized and identified in ACT14. After that, the processor 711 proceeds to ACT22.

[0047] When the cart controller 3 receives a product return notification, it removes the returned product from the registered purchased products. Specifically, the cart controller 3 removes the purchased product data for the product code included in the product return notification, which is stored in the purchased product data area 313 of the information terminal table 31 that has the terminal ID included in the product return notification, from the purchased product data area 313. Then, the cart controller 3 creates a registered product screen to notify the registered product based on the purchased product data stored in the purchased product data area 313, and sends this registered product screen to the sender of the product input notification.

[0048] Processor 711, as ACT22, receives the registered product screen transmitted from the cart controller 3 via the access point 4 by the wireless unit 714. Then, processor 711 proceeds to ACT23.

[0049] Processor 711, as ACT23, displays the received registered product screen to the customer using cart 8 via touch panel 715. After that, processor 711 proceeds to ACT11.

[0050] In this way, when an item is removed from the basket receiving section 81 of the cart 8 or from a basket placed in the basket receiving section 81, the information terminal 7 identifies the item using image recognition and sends a product return notification to the cart controller 3 that includes the product code indicating the identified item, thereby causing the cart controller 3 to remove the item from the purchased items. Generally, returned items are removed one by one and returned to the shelves, so the quantity is not determined as it was when items were put into stock. Of course, it is also possible to perform quantity determination when items are returned.

[0051] Furthermore, if the processor 711 determines in ACT 17 that the number of items being loaded exceeds the maximum number of items that can be loaded simultaneously, that is, that multiple items have been loaded, it issues an alert as ACT 24. The method of issuing this alert is not particularly limited, but for example, it may be by flashing the LED lamp 731 for illumination attached to the camera 73, or by playing a specific notification sound through the speaker 716, or both may be used. Also, if the cart 8 is equipped with a warning light 74, the warning light 74 may be illuminated in place of or together with the other devices. After that, the processor 711 proceeds to ACT 25.

[0052] As ACT25, the processor 711 displays a message to the customer using cart 8 via the touch panel 715 prompting them to add more items. This message is superimposed on the registered items screen displayed on the touch panel 715, for example. The message prompting the customer to add more items should preferably include specific instructions, such as, "[Please return the items you have added. After removing them, press the OK button and add the items one by one.]" The processor 711 also displays an image of the OK button on the touch panel 715 along with this message prompting the customer to add more items. After that, the processor 711 proceeds to ACT26.

[0053] Processor 711, in ACT26, determines whether the OK button displayed on the touch panel 715 has been pressed. That is, processor 711 determines whether the customer has touched the location of the OK button image displayed on the touch panel 715. If the customer has touched the location of the OK button image, processor 711 determines that the OK button has been pressed. If it is determined that the OK button has been pressed, processor 711 determines YES in ACT26 and proceeds to ACT33. If it is determined that the OK button has not been pressed, processor 711 determines NO in ACT26 and proceeds to ACT27.

[0054] Processor 711, as ACT27, captures an image using camera 73, similar to ACT11, and stores the image data of the captured image in main memory 712. After that, processor 711 proceeds to ACT28.

[0055] Processor 711, as ACT28, detects moving products in the image from the latest image data stored in main memory 712, similar to ACT12. The detected object and its detection position in the image data are associated with the image data stored in main memory 712 and stored in main memory 712. After that, processor 711 proceeds to ACT29.

[0056] In ACT29, the processor 711, similar to ACT13, compares the object and detection location in the latest image data stored in main memory 712 with the object and detection location in the older image data to determine if there is a change. If it determines that there is no change, the processor 711 determines NO in ACT29 and proceeds to ACT26. If it determines that there is a change, the processor 711 determines YES in ACT29 and proceeds to ACT30.

[0057] Processor 711, as ACT30, recognizes and identifies the moving product in the image, similar to ACT14 described above. After that, processor 711 proceeds to ACT31.

[0058] Processor 711, as ACT31, determines whether the product has been identified, similar to ACT15 above. If it determines that the product has not been identified, processor 711 determines NO in ACT31 and proceeds to ACT26 above. If it determines that the product has been identified, processor 711 determines YES in ACT31 and proceeds to ACT32.

[0059] In ACT32, the processor 711 determines, similar to ACT16, whether or not an item has been placed in the basket receiving section 81 of the cart 8 or in the basket placed in the basket receiving section 81. If it determines that an item has been placed, the processor 711 determines YES in ACT32 and proceeds to ACT34. If it determines that an item has not been placed, i.e., that the item has been removed, the processor 711 determines NO in ACT32 and proceeds to ACT26.

[0060] It is reasonable to conclude that the determination in ACT32 that an item has been removed was made by the customer removing multiple items they had placed in the cart, in accordance with a predetermined message displayed on the touch panel 715. The possibility of another item being removed is small. Therefore, the processor 711 proceeds to ACT26 without sending an item return notification to the cart controller 3 as in ACT21. After returning the item, the customer presses the OK button in accordance with the predetermined message displayed on the touch panel 715. As a result, the processor 711 determines that the OK button was pressed in ACT26 and proceeds to ACT33.

[0061] As ACT33, the processor 711 terminates the activation of alerts such as the flashing of the LED lamp 731 and clears the message displayed on the touch panel 715 prompting the user to insert another item. Subsequently, the processor 711 proceeds to ACT11 and repeats the aforementioned operation to determine whether to insert or remove the next item.

[0062] Furthermore, the determination in ACT32 that an item has been inserted is presumed to be that, following a predetermined message displayed on the touch panel 715, the customer removed an item that had been inserted multiple times, but did not press the OK button, and instead inserted the removed item. In this case, the processor 711, as ACT34, terminates the issuance of alerts such as the flashing of the LED lamp 731 and clears the message displayed on the touch panel 715 prompting the customer to insert another item. After that, the processor 711 proceeds to ACT17 and makes the determination of the number of items that have been inserted, as described above.

[0063] As described above, purchased items are registered and deleted. When it is time to pay, pressing the payment button on the product registration screen allows the processor 711 to proceed to the payment process. This payment process is not related to the product identification system of this embodiment, so it will be briefly explained below. Although not specifically illustrated, in the payment process, for example, the processor 711 generates a barcode based on the terminal ID assigned to the information terminal 7, which includes the information processing device 71, and displays the barcode on the touch panel 715. The payment machine 5 reads the barcode and sends a transaction inquiry command including the terminal ID to the cart controller 3. The cart controller 3 generates settlement data based on the purchased product data registered in the purchased product data area 313 of the information terminal table 31 corresponding to the terminal ID included in the transaction inquiry command, and sends the settlement data back to the payment machine 5, the source of the transaction inquiry command. The payment machine 5 then settles the payment for the purchased items based on the settlement data.

[0064] Furthermore, if the cart POS system 1 is configured as a system that can be paid for with electronic money, the processor 711 transmits an electronic payment command including the terminal ID to the cart controller 3 via the access point 4 using the wireless unit 714. The cart controller 3 generates payment data based on the purchased product data registered in the purchased product data area 313 of the information terminal table 31 corresponding to the terminal ID included in the electronic payment command, and transmits this payment data to the store server 2 along with the member ID stored in the member ID area 312 of the same information terminal table 31. The store server 2 uses the member ID sent with the payment data to obtain the electronic money user ID from the member payment record in the member payment database (not shown), and uses this electronic money user ID to perform payment with the electronic money center EMC based on the payment data.

[0065] As described above, in the product identification system according to this embodiment, the camera 73 of the information terminal 7 functions as an imaging unit provided on the cart 8. Furthermore, the processor 711 of the information processing device 71 of the information terminal 7 functions as an identification unit that identifies products placed in the cart 8 based on the image acquired by the camera 73, specifically using AI-based image recognition, and as a notification unit that, when the number of products included in the image acquired by the camera 73 exceeds a predetermined upper limit stored as input quantity upper limit data 71311, an alert is issued by an LED lamp 731 installed on the camera 73 and / or a speaker 716 of the information processing device 71, or by a warning light 74 if the information terminal 7 is equipped with one, by that warning light 74, indicating that the accuracy of identifying the identified products is low. Thus, in the product identification system according to this embodiment, if the accuracy of product identification by image, that is, the accuracy of product identification, drops drastically, the user who added the product to cart 8 can be notified of this. Therefore, the user who receives this notification can remove the product and add it back to cart 8 to have it correctly identified. Thus, a product identification system that enables highly reliable product registration using AI-based image recognition can be provided.

[0066] Here, the notification provided by the processor 711, which functions as a notification unit, may include a message output via the touch panel 715 of the information processing device 71, prompting the user to remove items that have been placed in more than the upper limit from the cart 8 and to re-insert items in a quantity that does not exceed the upper limit. This allows users of Cart 8 to understand what they need to do and perform the actions indicated by the message.

[0067] The camera 73, which functions as an imaging unit, repeatedly acquires images, and the processor 711, which functions as an identification unit, starts identifying the product when there is a change between the images repeatedly acquired by the camera 73. Therefore, even if a product that has already been identified by being placed in the basket receiving section 81 of the cart 8 or in a basket placed in the basket receiving section 81 is photographed by the camera 73, there is no change in the images acquired in chronological order, so the identified product will not be re-identified.

[0068] Furthermore, the touch panel 715 functions as an operation unit that receives confirmation operations such as pressing the OK button by the user using the cart 8, and the processor 711, which functions as a notification unit, terminates the notification, including the above-mentioned alert and message, when the touch panel 715 receives the OK button press operation. Therefore, after receiving confirmation from the user following the removal of a product with low identification accuracy, the notification can be terminated, and the identification of the removed product to be reinserted can be started.

[0069] Furthermore, the processor 711, which functions as a notification unit, terminates the notification, including the above alerts and messages, when the product is identified after the notification has started. Therefore, if a user forgets to perform a confirmation operation after removing an item with low identification accuracy and reinserts the removed item, the notification can be terminated once the reinserted item is identified, preventing the notification from continuing.

[0070] The specified upper limit for the number used by the processor 711, which functions as a notification unit, to determine whether or not to send a notification is "1". Therefore, when multiple products are loaded simultaneously and the image contains images of all those products, that is, when multiple products are loaded simultaneously and the accuracy of identifying the products drops drastically, the processor 711 can provide notification.

[0071] Furthermore, the processor 711 functions as a specific product notification unit that transmits a product code indicating the identified product to an external source. Therefore, when the product identification system according to this embodiment is applied to the cart POS system 1, the cart controller 3 that receives this product code can register the identified product as a product purchased by the user.

[0072] Here, the processor 711, which functions as a product notification unit, does not transmit the product code of the specified product when the processor 711, which functions as a notification unit, issues a notification, that is, when the number of products exceeds the upper limit. Therefore, product codes for items with low identification accuracy will not be transmitted, preventing incorrect items from being registered as purchased items by the user.

[0073] Furthermore, in the product identification system according to this embodiment, the processor 711, which functions as an identification unit, can not only identify products placed in the cart 8, but also identify products removed from the cart 8 based on images acquired by the camera 73. The processor 711, which functions as an identified product notification unit, then transmits a product code indicating the identified removed product to an external source. Therefore, when the product identification system according to this embodiment is applied to the cart POS system 1, the cart controller 3 that receives this product code can remove the identified product from the user's purchased items.

[0074] [Second Embodiment] In the first embodiment described above, product recognition is performed by the information terminal 7, but in this embodiment, product recognition is performed by the cart controller 3.

[0075] The cart POS system to which the product identification system in this embodiment is applied includes, like the first embodiment, a store server 2, a cart controller 3, an access point 4, multiple cashiers 5, a network 6, and multiple information terminals 7, each including an information processing device 71. In this embodiment, the configuration and operation of the information processing devices 71 of the cart controller 3 and the information terminals 7 differ from the first embodiment. Hereinafter, parts that are the same as in the first embodiment will be denoted by the same reference numerals as in the first embodiment and their descriptions will be omitted, and the parts that differ from the first embodiment will be described.

[0076] Figure 7 is a block diagram showing the main circuit configuration of the cart controller 3 according to the second embodiment. The cart controller 3 includes a processor 32, main memory 33, auxiliary storage device 34, network interface 35, and system transmission path 36. The system transmission path 36 includes an address bus, data bus, control signal lines, etc. The cart controller 3 connects the processor 32, main memory 33, auxiliary storage device 34, and network interface 35 to the system transmission path 36. In the cart controller 3, the computer is composed of the processor 32, main memory 33, auxiliary storage device 34, and the system transmission path 36 connecting them.

[0077] The processor 32 corresponds to the central part of the computer described above. The processor 32 controls each part to realize various functions as a cart controller 3 according to the operating system or application program. The processor 32 is, for example, a CPU, but may be implemented in a variety of other forms, including integrated circuits such as AI chips, GPUs, ASICs, FPGAs, and DSPs.

[0078] The main memory 33 corresponds to the main memory portion of the computer described above. The main memory 33 includes a non-volatile memory area and a volatile memory area. The main memory 33 includes a non-volatile memory area such as ROM and a volatile memory area such as RAM. The volatile memory area stores an information terminal table 31 corresponding to each information terminal 7 as described in the first embodiment.

[0079] The auxiliary storage device 34 corresponds to the auxiliary storage portion of the computer described above. For example, an EEPROM, HDD, or SSD can be the auxiliary storage device 34. In this embodiment, the auxiliary storage device 34 stores a cart control program 341 as an application program that causes the cart controller 3 to execute a product registration function using the information terminal 7. The cart control program 341 includes input limit value data 3411 that defines the upper limit of the number of products that can be simultaneously entered, similar to the input limit value data 71311 described in the first embodiment. The cart control program 341 may also be stored in the main memory 33. Furthermore, the auxiliary storage device 34 stores learning data 342 for each product sold at the store, stored in the product database 21 of the store server 2, similar to the learning data 7132 in the first embodiment. This learning data 342 may be a copy of the learning data 211 included in the product database 21 of the store server 2, similar to the first embodiment. Since the cart controller 3 stores the input limit data 3411 and the learning data 342, the information processing device 71 of the information terminal 7 does not need to store the input limit data 71311 and the learning data 7132.

[0080] The network interface 35 is an interface for network 6. The processor 32 can send and receive data between the store server 2, access point 4, and accounting machine 5 via network 6 using this network interface 35.

[0081] Figures 8 and 9 are flowcharts illustrating the main information processing steps performed by the processor 32 of the cart controller 3 according to the cart control program 341. These flowcharts show the processing steps for one information terminal 7, and the processor 32 performs similar processing in parallel for each of the multiple information terminals 7.

[0082] The following flowchart illustrates the main operations of the cart POS system 1 when a customer uses the information terminal 7 on the cart 8 to register and pay for their purchases themselves. Note that the operations described below are just one example. The procedure is not particularly limited as long as similar results can be obtained. Also, here we assume that the data recorded on the recording medium has been read by the reader 72 on the information terminal 7 and that the customer has already logged in. That is, the member ID area 312 of the information terminal table 31 of the cart controller 3, which corresponds to the information terminal 7, already stores the member ID that was read from the customer's recording medium.

[0083] The processor 32 of the cart controller 3, as ACT41, receives image data of the captured image transmitted from the information terminal 7 via the access point 4 through the network interface 35. As described in the first embodiment, the information terminal 7 lights up the LED lamp 731 of the camera 73 and takes an image with the camera 73. The information terminal 7 then transmits the image data of the captured image to the cart controller 3. The processor 32 stores the received image data in the main memory 33. The main memory 33 has an area reserved to store multiple frames of this image data, and when the processor 32 stores new image data in this area, it erases the oldest image data and stores the new image data in the reserved area. After that, the processor 32 proceeds to ACT42.

[0084] Processor 32, as ACT42, detects moving products in the image from the latest image data stored in main memory 33. The detected object and its detection position in the image data are associated with the image data stored in main memory 33 and stored in main memory 33. After that, processor 711 proceeds to ACT43.

[0085] In ACT43, the processor 32 compares the object and detection location in the latest image data stored in the main memory 33 with the object and detection location in the older image data to determine if there is a change. If it determines that there is no change, the processor 32 determines NO in ACT43 and proceeds to ACT41. If it determines that there is a change, the processor 32 determines YES in ACT43 and proceeds to ACT44.

[0086] The loop consisting of ACT41, ACT42, and ACT43 described above receives image data transmitted from the information terminal 7 and repeatedly checks for any changes. If there is a change in the image, the processor 32 proceeds to ACT44.

[0087] Processor 32, as ACT44, recognizes the moving product in the image. Specifically, based on the training data 342 stored in the auxiliary memory device 34, processor 32 performs image recognition on the object within the bounding box detected in ACT42 and stored in the main memory 33, and identifies which product the object is. After that, processor 32 proceeds to ACT45.

[0088] Processor 32 determines in ACT45 whether or not the product has been identified. If it determines that the product has not been identified, processor 32 determines NO in ACT45 and proceeds to ACT41. If it determines that the product has been identified, processor 32 determines YES in ACT45 and proceeds to ACT46.

[0089] In ACT46, the processor 32 determines whether or not an item has been placed in the basket receiving section 81 of the cart 8 or in the basket placed in the basket receiving section 81. For example, the processor 32 can determine whether an item has been placed in or removed by looking at the direction of movement of the same item shown in the most recent image and past images. If it determines that the item has not been placed in, i.e., that the item has been removed, the processor 32 determines NO in ACT46 and proceeds to ACT51. If it determines that an item has been placed in, the processor 32 determines YES in ACT46 and proceeds to ACT47.

[0090] In ACT47, the processor 32 determines whether the number of items to be inserted, which is the number of items recognized and identified in ACT44, is less than the maximum number of items to be inserted simultaneously, which is stored in the auxiliary storage device 34 as the upper limit data 3411. In this embodiment as well, the upper limit for the number of items to be inserted simultaneously is "2". The number of items to be inserted can be the number of bounding boxes set in the image used for image recognition, that is, the number of objects detected in ACT42. If the processor determines that the number of items to be inserted is greater than or equal to the upper limit for the number of items to be inserted simultaneously, that is, that multiple items have been inserted, the processor 32 determines NO in ACT47 and proceeds to ACT54. If the processor determines that the number of items to be inserted is less than the upper limit for the number of items to be inserted simultaneously, that is, that only one item has been inserted, the processor 32 determines YES in ACT47 and proceeds to ACT48.

[0091] If the processor 32 determines in ACT 47 that the number of items to be entered is less than the upper limit for simultaneous entry, that is, that only one item has been entered, it determines that it has identified the item accurately. In this case, the processor 32 registers the item recognized and identified in ACT 44 as ACT 48. That is, the processor 32 queries the store server 2 for product information using the product code of the identified item and retrieves the product information stored in the product database 21 managed by the store server 2. Then, it adds the retrieved product information as new purchased product data to the purchased product data area 313 of the corresponding information terminal table 31. After that, the processor 32 proceeds to ACT 49.

[0092] As ACT49, processor 32 creates a registered product screen to notify registered products based on the purchased product data stored in the purchased product data area 313 of the corresponding information terminal table 31. After that, processor 32 proceeds to ACT50.

[0093] The processor 32, as ACT 50, transmits the generated registered product screen to the corresponding information terminal 7 via the network interface 35 and access point 4, causing the information terminal 7's touch panel 715 to display the registered product screen. After that, the processor 32 proceeds to ACT 41.

[0094] In this way, when an item is placed in the basket receiving section 81 of the cart 8 to which the corresponding information terminal 7 is attached, or in a basket placed in the basket receiving section 81, the cart controller 3 can identify what the item is by image recognition of the image captured by the camera 73 of the information terminal 7, and register the item as a purchased item.

[0095] Furthermore, if the processor 32 determines in ACT 46 that an item has been retrieved, it performs ACT 51 by deleting the returned item from the registered purchased items. That is, the processor 32 deletes the purchased item data of the item recognized and identified in ACT 44, which is stored in the purchased item data area 313 of the corresponding information terminal table 31, from that purchased item data area 313. After that, the processor 32 proceeds to ACT 52.

[0096] Processor 32, as ACT52, creates a registered product screen to notify registered products based on the purchased product data stored in the purchased product data area 313 of the corresponding information terminal table 31. After that, processor 32 proceeds to ACT53.

[0097] As ACT53, the processor 32 transmits the generated registered product screen to the corresponding information terminal 7 via the network interface 35 and access point 4, causing the information terminal 7's touch panel 715 to display the registered product screen. After that, the processor 32 proceeds to ACT41.

[0098] In this way, when an item is removed from the basket receiving section 81 of the cart 8 to which the corresponding information terminal 7 is attached, or from a basket placed in the basket receiving section 81, the cart controller 3 can identify what the item is by image recognition of the image captured by the camera 73 of the information terminal 7, and remove the item from the purchased items.

[0099] Furthermore, if the processor 32 determines in ACT 47 that the number of items being loaded exceeds the maximum number of items that can be loaded simultaneously, that is, if multiple items have been loaded, it instructs the corresponding information terminal 7 to issue an alert via the network interface 35 and access point 4 as ACT 54. After that, the processor 32 proceeds to ACT 55. Upon receiving this alert instruction, the information terminal 7 may, for example, flash the LED lamp 731 attached to the camera 73, or play a specific notification sound via the speaker 716. Alternatively, if the cart 8 is equipped with a warning light 74, it may illuminate the warning light 74 in place of or in conjunction with the other warning lights.

[0100] Furthermore, as ACT 55, the processor 32 instructs the information terminal 7 via the network interface 35 through the access point 4 to display a message prompting the customer to add items again. After that, the processor 32 proceeds to ACT 56. Upon receiving this instruction to display a message prompting the customer to add items again, the information terminal 7 displays the message prompting the customer to add items again and an image of an OK button on the touch panel 715. The message and the image of the OK button are superimposed on, for example, the registered product screen displayed on the touch panel 715. Alternatively, instead of instructing the processor 32 to display a message prompting the customer to add items again, the processor 32 may generate a registered product screen with the message and the image of the OK button superimposed and send it to the information terminal 7 for display on the touch panel 715.

[0101] The processor 32, as ACT 56, determines whether or not it has received an OK notification transmitted from the information terminal 7 via the network interface 35 through the access point 4. The OK notification is transmitted from the information terminal 7 when the OK button displayed on the touch panel 715 is pressed. If the processor 32 determines that it has received an OK notification, it determines YES in ACT 56 and proceeds to ACT 63. If the processor 32 determines that it has not received an OK notification, it determines NO in ACT 56 and proceeds to ACT 57.

[0102] Processor 32, as ACT57, receives image data of the captured image transmitted from the information terminal 7 via the access point 4 through the network interface 35, similar to ACT41. After that, processor 32 proceeds to ACT58.

[0103] Processor 32, as ACT58, detects moving products in the image from the latest image data stored in main memory 33, similar to ACT42. The detected object and its detection position in the image data are associated with the image data stored in main memory 33 and stored in main memory 33. After that, processor 32 proceeds to ACT59.

[0104] In ACT59, the processor 32, similar to ACT43, compares the object and detection location in the latest image data stored in the main memory 33 with the object and detection location in the older image data to determine whether there has been a change. If it determines that there has been no change, the processor 32 determines NO in ACT59 and proceeds to ACT56. If it determines that there has been a change, the processor 32 determines YES in ACT59 and proceeds to ACT60.

[0105] Processor 32, as ACT60, recognizes the moving product in the image, similar to ACT44 above. After that, processor 32 proceeds to ACT61.

[0106] Processor 32, as ACT61, determines whether the product has been identified, similar to ACT45 above. If it determines that the product has not been identified, processor 32 determines NO in ACT61 and proceeds to ACT56 above. If it determines that the product has been identified, processor 32 determines YES in ACT61 and proceeds to ACT62.

[0107] In ACT62, the processor 32 determines, similar to ACT46, whether or not an item has been placed in the basket receiving section 81 of the cart 8 or in the basket placed in the basket receiving section 81. If it is determined that an item has been placed, the processor 32 determines YES in ACT62 and proceeds to ACT64. If it is determined that an item has not been placed, i.e., that the item has been removed, the processor 32 determines NO in ACT62 and proceeds to ACT56.

[0108] As ACT63, the processor 32 instructs the corresponding information terminal 7 via the network interface 35 through the access point 4 to terminate the alert and clear the displayed message. Subsequently, the processor 32 proceeds to ACT41 and repeats the aforementioned operation to determine whether to insert or remove the next product. Upon receiving this instruction to clear the message, the information terminal 7 clears the message prompting the user to insert another product, which is displayed on the touch panel 715. Alternatively, instead of instructing the user to clear the message, the processor 32 may generate a registered product screen with the message and OK button image cleared, and send it to the information terminal 7 for display on the touch panel 715.

[0109] As ACT64, the processor 32 instructs the corresponding information terminal 7 via the network interface 35 and access point 4 to terminate the alert and delete the displayed message. Subsequently, the processor 32 proceeds to ACT47 and makes the determination of the number of products that have been inserted, as described above. Upon receiving this instruction to delete the message, the information terminal 7 deletes the message prompting the insertion of products again, which is displayed on the touch panel 715. Alternatively, instead of instructing the deletion of the message, the processor 32 may generate a registered product screen with the message and OK button image deleted, and send it to the information terminal 7 for display on the touch panel 715.

[0110] In this way, purchased items are registered and deleted.

[0111] As described above, in the product identification system according to this embodiment, the processor 32 of the cart controller 3 in the cart POS system 1 communicates with the information terminal 7 via the network interface 35 to acquire images obtained by the camera 73 of the information terminal 7 and, based on those images, specifically using AI-based image recognition, identifies the products placed in the cart 8. The processor 32 also functions as an alert unit that, when the number of products included in the image acquired by the camera 73 exceeds a predetermined upper limit stored as input quantity upper limit data 3411, communicates with the information terminal 7 via the network interface 35 to notify an alert indicating that the accuracy of identifying the identified products is low, using the LED lamp 731 installed on the camera 73 and / or the speaker 716 provided in the information processing device 71 of the information terminal 7, or, if the information terminal 7 is equipped with a warning light 74, the warning light 74. Thus, in the product identification system according to this embodiment, similar to the product identification system according to the first embodiment, if the accuracy of product identification by image, i.e., the accuracy of product identification, drops drastically, the user who added the product to cart 8 can be notified of this. Therefore, the user who receives this notification can remove the product and add it back to cart 8 to have it correctly identified. Thus, a product identification system that enables highly reliable product registration using AI-based image recognition can be provided.

[0112] Here, the notification provided by the processor 32, which functions as a notification unit, may include a message output via the touch panel 715 of the information processing device 71, prompting the user to remove items that have been placed in the cart 8 in excess of the upper limit and to re-insert items in quantities that do not exceed the upper limit. This allows users of Cart 8 to understand what they need to do and perform the actions indicated by the message.

[0113] The camera 73, which functions as an imaging unit, repeatedly acquires images, and the processor 32, which functions as an identification unit, starts identifying the product when there is a change in the images repeatedly acquired by the camera 73. Therefore, even if a product that has already been identified by being placed in the basket receiving section 81 of the cart 8 or in a basket placed in the basket receiving section 81 is photographed by the camera 73, there is no change in the images acquired in chronological order, so the identified product will not be re-identified.

[0114] Furthermore, the touch panel 715 functions as an operation unit that receives an OK button press operation, which is a confirmation operation by the user using the cart 8. The processor 32, which functions as a notification unit, terminates the notification by the information terminal 7, including the alert and message mentioned above, when it receives an OK notification via the network interface 35 indicating that the touch panel 715 has accepted the OK button press operation. Therefore, after receiving confirmation from the user following the removal of a product with low identification accuracy, the notification can be terminated, and the identification of the removed product to be reinserted can be started.

[0115] Furthermore, the processor 32, which functions as a notification unit, terminates the notification, including the alerts and messages from the information terminal 7, when the product is identified after the notification has started. Therefore, if a user forgets to perform a confirmation operation after removing an item with low identification accuracy and reinserts the removed item, the notification can be terminated once the reinserted item is identified, preventing the notification from continuing.

[0116] Although the first and second embodiments have been described above, the embodiments are not limited thereto. For example, in the first and second embodiments described above, the trigger for starting product recognition in ACT14 and ACT44 was set to when there was a change in the image, but it is not limited to this. For example, at least one sensor that optically detects objects present within the field of view of camera 73 may be placed at any position on the cart 8, and product recognition may be started in response to the sensor detecting an object being placed in or removed from the cart 8.

[0117] Furthermore, the decision to add / remove items in ACT16 and ACT46 is not limited to being determined from images. For example, a weighing scale can be placed on the bottom of the basket support section 81 of the shopping cart 8, and the decision to add / remove items can be made based on whether the weight has increased or decreased.

[0118] Furthermore, if it is possible to determine whether to insert / remove items based on weight, it will also be possible to determine whether or not items inserted simultaneously have been removed in response to alerts and messages. This will allow the store staff to be notified at that time or at the time of payment at checkout machine 5 if additional items are inserted without the simultaneously inserted items being removed.

[0119] Furthermore, in the first embodiment described above, it was assumed that the terminal program 7131 was pre-stored in the auxiliary storage device 713 or main memory 712 of the information processing device 71 of the information terminal 7. In this regard, a terminal program that has been transferred separately from the information processing device 71 may be written to a writable storage device provided by the information processing device 71 in response to user operations. Similarly, in the second embodiment described above, it was assumed that the cart control program 341 was pre-stored in the auxiliary storage device 34 or main memory 33 of the cart controller 3, but a cart control program that has been transferred separately from the cart controller 3 may be written to a writable storage device provided by the cart controller 3 in response to user operations. The transfer of these terminal programs and cart control programs, etc., can be done by recording them on a removable recording medium or by communication over a network. The recording medium can be of any form as long as it can store a program and is readable by the device, such as a CD-ROM or memory card.

[0120] In addition, several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope of the invention, as well as within the scope of the invention and its equivalents as described in the claims. The invention described in the original claims of this application is listed below. [1] An imaging unit installed in the cart, Based on the image acquired by the aforementioned imaging unit, an identification unit identifies the product placed in the cart, When the number of products included in the image acquired by the imaging unit exceeds a predetermined upper limit, a notification unit notifies that the accuracy of product identification by the identification unit is low. A product identification system equipped with the following features. [2] The notification unit further outputs a message prompting the user to remove the items that have been placed in the cart in excess of the upper limit and to place the items again in quantities that do not exceed the upper limit. [1] The product identification system described above. [3] The imaging unit repeatedly acquires the image, The identification unit starts identifying the product when there is a change between the images repeatedly acquired by the imaging unit. [1] The product identification system described above. [4] The cart is further equipped with an operation unit that receives confirmation operations from the user using the cart, The notification unit terminates the notification when the operation unit receives the confirmation operation. A product identification system as described in any of [1] through [3]. [5] The notification unit further terminates the notification when the product is identified by the identification unit after the notification has started. [4] The product identification system described. [Explanation of symbols]

[0121] 1... Cart POS system, 2... Store server, 3... Cart controller, 4... Access point (AP), 5... Cash register, 6... Network, 7... Information terminal, 8... Shopping cart (cart), 21... Product database (DB), 22... Member database (DB), 31... Information terminal table, 32,711... Processor, 33,712... Main memory, 34,713... Auxiliary storage device, 35... Network interface, 36,717... System transmission path, 71... Information processing device, 72... Reader, 73... Camera, 74... Warning light, 81... Cart receiving unit, 211,342,7132... Learning data, 311... Terminal ID area, 312... Member ID area, 313... Purchased product data area, 314... Subtotal area, 341... Cart control program, 714... Wireless unit, 715...Touch panel, 716...Speaker, 731...LED lamp, 3411,71311...Input limit data, 7131...Terminal program.

Claims

1. The imaging unit installed in the cart, Based on the image acquired by the aforementioned imaging unit, an identification unit identifies the product placed in the cart, An alert issuing unit issues an alert when the number of products identified by the identification unit exceeds a predetermined upper limit for the number of products included in the image acquired by the imaging unit. A product identification system equipped with the following features.

2. The system further includes a message output unit that, when the number of items added exceeds the upper limit, removes the items added in excess of the upper limit from the cart and outputs a message prompting the user to add the items again in quantities that do not exceed the upper limit. The product identification system according to claim 1.

3. The imaging unit repeatedly acquires the image, The identification unit starts identifying the product when there is a change between the images repeatedly acquired by the imaging unit. The product identification system according to claim 1.

4. The cart is further equipped with an operation unit that receives confirmation operations from the user using the cart. The alert issuing unit terminates the issuance of the alert when the operation unit receives the confirmation operation. A product identification system according to any one of claims 1 to 3.

5. The alert issuing unit further terminates the alert when the product is identified by the identification unit after the alert has been issued. The product identification system according to claim 4.

6. The cart is further equipped with an operation unit that receives confirmation operations from the user using the cart. The message output unit terminates outputting the message when the operation unit accepts the confirmation operation. The product identification system according to claim 2.

7. The message output unit further terminates the message output when the product is identified by the identification unit after the start of message output. The product identification system according to claim 6.

Images