Two-way product verification by reverse lookup

Abstract

Disclosed herein is a novel system and method to reduce theft and errors in the self-checkout process. The system and methods disclosed herein reconcile purchased products with receipts for those products by corresponding, analyzing, and / or comparing information on self-checkout transaction receipts with captured images of purchase products from camera feeds to ensure that all products purchased via a self-checkout system have been properly accounts for and paid for.

Description

TWO-WAY PRODUCT VERIFICATION BY REVERSE LOOKUPRelated Applications

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 554,687, filed February 16, 2024, the contents of which are incorporated herein in its entirety.Background

[0002] Retailers are increasingly adopting self-checkout stations to cut staff costs and improve operations. Self-checkout stations are machines that allow customers to complete their own purchases with a retailer without using a staffed checkout. In the typical operation of a self-checkout station, customers scan item barcodes appearing on the products using a laser scanner, or are able to lookup product SKUs at the self-checkout station (some products, for example produce at a grocery store, may also be weighed at the self-checkout station, and the self-checkout station may calculate a cost based on the weight of the product). As the barcode for each scanned product is scanned, a corresponding product UPC is added to the receipt. These receipt barcodes are used to compute the total bill of the transaction. This process is illustrated in FIG. 1.

[0003] The self-checkout station also provides a means for accepting payment for the products from the customer and providing a receipt summarizing the customer's purchases. Customers also typically bag their own products. As such, the customer exits the store without the need for one-to-one staff assistance.

[0004] For shoppers, self-checkout may provide a more convenient and speedier experience. For retailers, cost savings may be realized as the need for multiple staff members may be reduced to a single employee who oversees multiple self-checkout stations.

[0005] However, the steady adoption of self-checkouts by retail businesses has been accompanied by certain challenges. For example, due to human error or malicious intent, incorrect barcodes may be scanned, or products may be left unscanned, but still placed in the bag (i.e., theft). This can result in incorrect information on the generated receipt and, consequently, a significant loss to the retailers. Thus, there exists a need for a solution to improve selfcheckout accuracy and verification of receipts and purchases.Summary of the Invention

[0006] To address the issues identified above, disclosed herein is a novel system and method relating to product verification. In some embodiments, the disclosed invention includes processes and methods which correspond, analyze, and / orcompare information on checkout transaction receipts with captured images from visual camera feeds. In some embodiments, visual cameras may be positioned at self-checkout counters. In some embodiments, a multi-step verification process may be implemented to ensure checkout transactions are valid (i.e., that all products have been correctly scanned and appear on the receipt). The disclosed invention adds robustness to the self-checkout process and is capable of flagging fraudulent self-checkout transactions to discourage or prevent theft.Brief Description of the Drawings

[0007] By way of example, specific exemplary embodiments of the disclosed system and method will now be described, with reference to the accompanying drawings, in which:

[0008] FIG. 1 is an illustration of the self-checkout process

[0009] FIG. 2 is a flow chart showing the label lookup process during checkout.

[0010] FIG. 3. Is a flow chart showing the label lookup process using product matching during checkout.

[0011] FIG. 4 is an illustration of individual product segmentation from verification counter captured image.

[0012] FIG. 5 is a flow chart showing the reverse lookup process after checkout.Detailed Description

[0013] The processes and methods of the invention are generally broken down into two aspects which, in some embodiments, may overlap. In a first aspect, processes are specified for building and verifying a list of purchased products, which eventually becomes a physical or virtual receipt provided to the customer at the conclusion of the transaction. During the checkout process, each scanned product adds a corresponding product UPC to the receipt. These receipt UPCs are used to compute the total bill of the transaction. In the second aspect of the invention, the purchased products are scanned and identified to verify that each of the purchased products appears on the receipt.

[0014] In the first aspect of the invention, the customer performs a standard selfcheckout process at the end of the shopping experience. The environment of the self-checkout area may be provided with one or more cameras to image the product from one or more viewpoints as the barcode on the product is scanned by the customer using the laser of the scanner. Cameras may be located, for example, above or to the side of the scanner, or may be collocated internally with the scanner.

[0015] In some embodiments, customers are permitted to scan their products with a hand scanner as products are placed into a cart or buggy. A barcode on a checkout station is then scanned using the hand scanner and the customer ispresented with a list of scanned products and provided with an opportunity to tender payment. In such situations, cameras may be located, for example, on the cart, at various locations physically attached to the store or fixtures within the store or may be collocated with the hand scanner. The hand scanner may be, for example, a hand laser scanner provided to the customer upon entry to the store, or the product barcodes may be read using an app executing on a user smart device that utilizes a built-in camera. In such cases, the built in camera may also capture an image of the product as the barcode is read from the product.

[0016] In all embodiments of the invention, a product databased is used. The product database contains, for each individual product, a plurality of feature vectors that have been extracted using a trained feature extractor from various images of the products collected from a plurality of different viewpoints. Additionally, each feature vector is associated with a UPC code assigned to the product, such that a match between a feature vector collected from an image of the product during the self-checkout process and a feature vector in the product database can be matched and the UPC of the product is added to the receipt.

[0017] In the first aspect of the invention, as the products are scanned, each scanned barcode, encoding the UPC of the product, is verified using a visual inspection. One embodiment of this process is shown in FIG. 2. After the customer scans a product at 202 and a receipt barcode is added at 204, astandard image of the product 206 corresponding to the UPC of the scanned product is looked up in the product database. From the visual feed or camera frame 210, the checked-out product image is isolated at 212, for example, by cropping. If multiple cameras are present, all of the product crops can be used to do ensemble matching. The product image is then matched with a standard image of the product to verify if the product scanned is the same as the corresponding product UPC added to the receipt. Any known method of image matching may be used. For example, feature vectors may be extracted from the cropped image 212 and compared to feature vectors from a standard image of the product. If a match between the product images exists, then the product checkout is verified. Otherwise, the product is flagged.

[0018] FIG. 3 shows a second embodiment of the first aspect of the invention in which a slightly different process is used to verify the checkout using ensemble matching with UPC prediction. After the customer scans a product at 302 and a receipt UPC is added at 304, a standard image of the product 306 corresponding to the UPC of the scanned product is retrieved from the product database. From the visual feed or camera frame 310, the checked- out product image is isolated 312, for example, by cropping. However, instead of directly comparing product crop(s) from camera feed(s), a product matcher 314 is used to predict the product UPC of each crop 316. Using the matcher predicted UPC, the corresponding image 318 is pulled up. As withthe process of FIG. 2, the product matcher predicted image 318 can be matched 320 with the UPC appearing of the receipt 306 to verify the checkout process. The product checkout is then either verified or flagged, depending on whether or not match is detected.

[0019] In the second aspect of the invention, a verification procedure is performed after the checkout process to verify that the items on the transaction receipt and the product images captured at the verification station match.

[0020] The verification station allows capturing of a post-checkout view of the checked-out products. The post-checkout view may be, for example, a photo of the products in the shopping cart, as shown in FIG. 4, or it may be an area where products are placed after they have been scanned. Preferably, the post-checkout verification step occurs prior to the products being bagged, such that at least a portion of each product from any angle is available for imaging.

[0021] The post-checkout image may be captured, in one embodiment, by one or more stationary cameras mounted at the verification station at or near the self-checkout station, or somewhere between the self-checkout stations and the exit of the store. In one embodiment, the customer may capture the verification image using a personal smart device. The smart device may run an app that instructs the customer on how to capture the image and captures the image with a built-inn camera of the smart device. The customer may be provided with an identifying indicia, for example, a bar code or a QR code,that is used to identify the customer as the customer makes their way to the exit. The identifying indicia must be scanned, and the customer will be cleared to exit the store if the verification procedure (described below) for that customer has passed. The customer may bag their products after the verification image is captured and prior to scanning the identifying indicia.

[0022] A flow chart of the verification procedure is shown in FIG. 5 and comprises a reverse lookup process. At 502 the verification image is captured showing multiple products at the verification station. The verification image is segmented, and the individual product images are extracted at 504. This process may be performed, for example, by a trained machine learning product detector. The individual product images may be placed in a bounding box by the trained detector. The segmented product crops are run through a product matcher at 506. The product matcher matches the individual product crops against a product database of standard product images and predicts the UPC for each product crop image.

[0023] At 508, it is determined if the matching confidence score is below a predetermined threshold. For product crops that have a UPC prediction with a matching confidence score less than the predetermined threshold, the system may abstain from predicting the UPC to avoid false positives at 510. Some embodiments of the invention may avoid further verification of these crops as they may contain occluded product images, off-angle images and / ornewly introduced store products that are not yet present in the product database.

[0024] A list of the successfully predicted UPCs 510 is produced and compared with the UPCs appearing on the customer receipt 516. If the UPCs predicted by the product matcher 506 for each product crop is found on the receipt, then the verification process is complete at 516. No further verification is required as all of the checked-out products are accounted for.

[0025] If there is no exact match between the predicted UPC list and the receipt, the items in the matcher-predicted UPC list are separated into two sets 532. A first set 534 contains products whose matcher-predicted UPCs are found on customer receipt 516. For these products, the checkout is verified at 536. A second set contains matcher-predicted UPCs which are not found on the customer receipt 516. If the matcher-predicted UPC for a particular product crop is not found on the receipt at 518, then a soft matching is performed between all of the collected images corresponding to the UPC 530 and the standard image of the matcher-predicted UPC.

[0026] Preferably, the product database contains feature vectors extracted from multiple images of each of the products available at the store captured at different angles and or viewpoints. To complete the soft match, at 520 a feature vector is extracted from the product crop from the verification image and is compared to all of the feature vectors stored in the product database and associated with the matcher-predicted UPC. A confidence score isproduced for each comparison. If it is determined at 522 that the top matching confidence score is greater than or equal to a soft-match threshold, then the predicted UPC is reassigned to that of the top matching receipt UPC 524 and the checkout for that product if verified 526. If the top matching confidence score is less than the soft-match threshold, then none of the receipt UPCs are close to the product image prediction and hence the product is flagged as a theft item 528. The confidence score may be based on a distance between the feature vectors, for example, a cosine distance.

[0027] Note that there is no direct correspondence between the order of UPCs listed on the transaction receipt 516 and the captured product image placement. Also, the number of listed receipt UPCs should be equal to or higher than those of the products captured from the camera, to allow for occlusion resulting from product placement in the verification image.

[0028] The product database containing feature vectors of each product captured from multiple viewpoints may be automatically built. Each time a particular product is scanned, one or more feature vectors can be extracted from one or more captured images of the product. The feature vectors may be stored in the product database and associated with the UPC of the item. Each feature vector may be compared with feature vectors for that item already stored in the product database and, if the feature vectors are within a certain threshold distance of a feature vector already present in the product database for that item (indicating that a similar viewpoint of the item isalready in the product database), the feature vector may not be added.However, if the feature vectors captured from the one or more images are sufficiently different from any other feature vector already stored in the product database for that item (indicating a new viewpoint of the item) the new feature vector may be stored in the product database and associated with the UPC for that item. Images for each product may be captured from all checkouts within the store, including both self-checkouts and employee- assisted checkouts. Additionally, images of the same item captured at different stores may be added to the product database such as to have as many possible viewpoints of the item as possible stored in the product database.

[0029] In various embodiments of the invention, the system upon which the processes and methods are executed may consist of software implementing the processes and executing on a general purpose computing device. The software may include a trained product detector capable of detecting individual products in an image of multiple products. The software may further include a trained feature extractor for extracting feature vectors from images of individual products. The software may further include a product database containing multiple feature vectors extracted from multiple images of individual products. Each feature vector in the product database may be associated with a UPC of the product depicted in the image from which the feature vector was extracted. The system may further comprise one or morecameras coupled to the computing device and associated with various selfcheckout stations and one or more cameras coupled to the computing device and associated with a verification station. The system may further include the laser scanners used by customers during the self-checkout process to scan barcodes present on the products. The system may further include an application executing on user smart devices to assist in the scanning of products and / or in the verification of the order, as described herein. All components of the system may communicate via wireless connections, for example, Wi-Fi.

[0030] The present invention illustrates various techniques and configurations that enable verification of self-checkout orders. As would be realized, many configurations of the system and / or the software are possible and are intended to be within the scope of the invention. As used herein, the terms "barcode" and "UPC" are used interchangeably, however, as would be realized, a UPC is typically used to refer to a number that is encoded in a barcode that appears on the product. Further, although the invention is presented in terms of using the UPC to identify each object, the use of the term "UPC" is meant to include any number or other indicia identifying a product.

Claims

Claims:

1. A method for verifying self-checkout purchases comprising: capturing a verification image of multiple products; detecting individual objects within the verification image and cropping images of each individual object; predicting a UPC for each object based on the cropped image of that object; comparing the predicted UPCs for each object with UPCs appearing on a receipt from the self-checkout; and verifying the self-checkout when there is a one-to-one correspondence between the predicted UPCs and the UPCs appearing on the receipt.

2. The method of claim 1 further comprising, when one or more of the predicted UPCs do not appear on the receipt: verifying checkout of a predicted UPC when the predicted UPC matches a UPC on the receipt.

3. The method of claim 2 further comprising: retrieving one or more unverified UPCs from the receipt;matching an individual object image from the verification image with multiple views of products in a product database corresponding to the unverified UPCs and determining a confidence score for each matching; and verifying checkout of the object if at least one confidence score is above a predetermined threshold.

4. The method of claim 3 further comprising: marking a corresponding UPC on the receipt as verified if at least one confidence score for the object is above a pre-determined threshold.

5. The method of claim 3 wherein the product database contains a plurality of feature vectors extracted from multiple views of the object, the step of matching comprising: extracting a feature vector from the individual object image from the verification image; and determining a distance between the feature vector from the individual object image and the plurality of feature vectors; wherein the confidence score for a match between the feature vector from the individual object image and the plurality of feature vectors is based on the determined distance.

6. The method of claim 1 wherein individual objects are detected within the verification image by a trained object detector.

7. A method of verifying objects scanned in a self-checkout comprising: obtaining one or more images of the object being scanned at a selfcheckout station; cropping the one or more images to isolate the object; receiving a UPC from the self-checkout station; retrieving one or more views of an object associated with the receivedUPC from a product database; and verifying the self-checkout of the object if at least one of the one or more cropped images matches at least one of the one or more views retrieved from the product database.

8. A method of verifying objects scanned in a self-checkout comprising: obtaining one or more images of the object being scanned at a selfcheckout station; cropping the one or more images to isolate the object; receiving a UPC from the self-checkout station;predicting a UPC of the object based on the one or more cropped images; and verifying the self-checkout of the object if the predicted UPC matches the UPC from received from the self-checkout station.

9. The method of claim 8 where the step of predicting the UPC compromises: matching the one or more cropped images with multiple views of objects in a product database; and receiving a predicted UPC associated an object in the product database when one of the cropped images matches an image of the object in the product database with a confidence score exceeding a pre-determined threshold.

10. The method of claim 9 wherein the product database contains a plurality of feature vectors extracted from multiple views of multiple objects, the step of matching comprising: extracting feature vectors from the one or more cropped images; determining a distance between the feature vectors from the one or more cropped images and the plurality of feature vectors; wherein the confidence score for a match between the feature vector from the one of the cropped images and a feature vector in the product database is based on the determined distance.

11. A system for verifying self-checkout purchases comprising: a processor; one or more cameras, coupled to the processor; a trained object detector; and software that, when executed by the processor, causes the system to : capture a verification image of multiple products using the one or more cameras; detect individual objects within the verification image and crop images of each individual object; predict a UPC for each object based on the cropped image of that object; compare the predicted UPCs for each object with UPCs appearing on a receipt from the self-checkout; and verify the self-checkout when there is a one-to-one correspondence between the predicted UPCs and the UPCs appearing on the receipt.

12. The system of claim 11 wherein the software, when one or more of the predicted UPCs do not appear on the receipt, further causes the system to: retrieve one or more unverified UPCs from the receipt;match an individual object image from the verification image with multiple views of products in a product database corresponding to the unverified UPCs and determining a confidence score for each match; and verify checkout of the object if at least one confidence score is above a pre-determined threshold.

13. The system of claim 12, further comprising: a trained feature extractor; wherein the product database contains a plurality of feature vectors extracted from multiple views of the object, the match comprising: extracting a feature vector from the individual object image from the verification image using the trained feature extractor; and determining a distance between the feature vector from the individual object image and the plurality of feature vectors; wherein the confidence score for a match between the feature vector from the individual object image and the plurality of feature vectors in the product database is based on the determined distance.

14. A system for verifying objects scanned in a self-checkout comprising: a processor;one or more cameras located at or near a self-checkout station and coupled to the processor; and software that, where executed by the processor, causes the system to: obtain one or more images of the object being scanned at the self-checkout station using the one or more cameras; crop the one or more images to isolate the object; receive a UPC from the self-checkout station; retrieve one or more views of an object associated with the received UPC from a product database; and verify the self-checkout of the object if at least one of the one or more cropped images matches at least one of the one or more views retrieved from the product database.

15. A system for verifying objects scanned in a self-checkout comprising: a processor; one or more cameras located at or near a self-checkout station and coupled to the processor; and software that, where executed by the processor, causes the system to: obtain one or more images of the object being scanned at a self-checkout station using the one or more cameras; crop the one or more images to isolate the object;receive a UPC from the self-checkout station; predict a UPC of the object based on the one or more cropped images; and verify the self-checkout of the object if the predicted UPC matches the UPC from received from the self-checkout station.

16. The system of claim 15, the predicting compromising: matching the one or more cropped images with multiple views of objects in a product database; and receiving a predicted UPC associated an object in the product database when one of the cropped images matches an image of the object in the product database with a confidence score exceeding a pre-determined threshold.

17. A method of building a product database comprising: receiving, from a checkout station, a scanned UPC of an object and one or more images of the scanning of the object; cropping images of the object from the one or more images of the scanning of the object; extracting, from each of the cropped images, a feature vector using a trained feature extractor;determining a distance between each extracted feature vector and one or more feature vectors stored in a product database and associated with the scanned UPC; adding a feature vector to the product database if the distance falls below a pre-determined threshold, indicating that the feature vector represents a new view of the object; and associating the scanned UPC with the added feature vector.

18. The method of claim 17 wherein the scanned UPC of an object and one or more images of the scanning of the object are received from one or more checkout stations located in one or more stores.

19. The method of claim 18 wherein the scanned UPC does not appear in the product database, further comprising: adding the feature vectors extracted from the one or more cropped images to the product database; and associating the scanned UPC with each of the added feature vectors.

Images