Two-way product verification by reverse lookup using multi-view enrollments

Abstract

Disclosed herein is a novel system and method to reduce theft and errors in the self-checkout process. The system and methods disclosed herein reconcile purchased products with receipts for those products by corresponding, analyzing, and / or comparing information on self-checkout transaction receipts with captured images of purchase products from camera feeds to ensure that all products purchased via a self-checkout system have been properly scanned and accounted for.

Description

TWO-WAY PRODUCT VERIFICATION BY REVERSE LOOKUP USING MULTI-VIEW ENROLLMENTSRelated Applications

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 565,729, filed March 15, 2024, the contents of which are incorporated herein in its entirety.Background

[0002] Retailers are increasingly adopting self-checkout stations to cut staff costs and improve operations. Self-checkout stations are machines that allow customers to complete their own purchases with a retailer without using a staffed checkout. In the typical operation of a self-checkout station, customers scan item barcodes appearing on the products using a laser scanner, or are able to lookup product SKUs at the self-checkout station (some products, for example produce at a grocery store, may also be weighed at the self-checkout station, and the self-checkout station may calculate a cost based on the weight of the product). As the barcode for each scanned product is scanned, a corresponding product UPC is added to the receipt. These receipt barcodes are used to compute the total bill of the transaction.

[0003] The self-checkout station provides a means for accepting payment for the products from the customer and providing a receipt summarizing the customer's purchases. Customers also typically bag their own products. As such, the customer exits the store without the need for one-to-one staff assistance.

[0004] For shoppers, self-checkout may provide a more convenient and speedier experience. For retailers, cost savings may be realized as the need for multiple staff members may be reduced to a single employee who oversees multiple self-checkout stations.

[0005] However, the steady adoption of self-checkouts by retail businesses has been accompanied by certain challenges. For example, due to human error or malicious intent, incorrect barcodes may be scanned, or products may be left unscanned, but still placed in the bag (i.e., theft). This can result in incorrect information on the generated receipt and, consequently, a significant loss to the retailers. Thus, there exists a need for a solution to improve selfcheckout accuracy and verification of receipts and purchases.Summary of the Invention

[0006] To address the issues identified above, disclosed herein is a novel system and method relating to product verification. The disclosed invention includes one or more processes and methods to validate information on checkouttransaction receipts using captured images from one or more cameras at a self-checkout station. In some embodiments, checkout validation may be performed during the checkout process (single verification mode). In some embodiments, checkout validation may be performed after the checkout process (batch verification mode). In some embodiments, checkout validation may be performed as a store-wide audit (for example, at regular intervals, at discretion, and / or at random).

[0007] In a first embodiment of the invention multiple, different views of products in a store catalog may be captured and enrolled into a product gallery. The product gallery also contains a standard UPC image of the product. The different views may include views of different faces of the product and views of the product faces in different orientations (e.g., rotated, askew, upside down, etc.).

[0008] In a second embodiment of the invention, a reverse lookup process is disclosed for verifying a match between a scanned UPC and one or more images of the product captured as the product is scanned during the selfcheckout process.

[0009] In third embodiment of the invention, a batch reverse lookup process is disclosed in which an image containing multiple products is captured and verified against all products appearing on the receipt from a single transaction, typically after the checkout process is complete.Brief Description of the Drawings

[0010] By way of example, specific exemplary embodiments of the disclosed system and method will now be described, with reference to the accompanying drawings, in which:

[0011] FIG. 1 is a flowchart showing the multi-view enrollment process.

[0012] FIG. 2 shows the various machine learning models used to process the images.

[0013] FIG. 3 is a flowchart showing a reverse lookup process for a single scanned product.

[0014] FIG. 4 is a flowchart showing a batch reverse lookup process.

[0015] FIG. 5 is a flowchart showing the matching process.Definitions

[0016] As used herein, the terms "barcode" and "UPC" are used interchangeably, however, as would be realized, a UPC is typically used to refer to a number that is encoded in a barcode that appears on the product. Further, although the invention is presented in terms of using the UPC to identify each object, the use of the term "UPC" is meant to include any number or other indicia capable of uniquely identifying a product.

[0017] As used herein, the term "isolated product image” refers to an image extracted from a larger image which shows only the product (i.e., segmented"), shows the product with a solid or blurred background ("i.e.,cropped") or shows the product with the background from the larger image(i.e., product is isolated within a "bounding box"). A trained machine learning model may produce any of these versions of an isolated product image.Detailed Description

[0018] Some embodiments may include one or more processes and / or methods to validate the information on a checkout transaction receipt and captured images from a visual camera feed on a checkout counter. In some embodiments, robust verification steps may ensure that products scanned by a user through the self-checkout process correspond to the items added to the receipt. In some embodiments, fraudulent transactions, including, but not limited to, unscanned items and incorrectly scanned items, may be flagged and / or reported to authorities, which may reduce theft and revenue loss. In some embodiments, during the product checkout process, as the user scans each item, the corresponding Unique Product Code (UPC) are added to the receipt. In some embodiments, at the end of the transaction, these UPCs may be used to calculate the overall bill for the transaction.

[0019] In some embodiments, if an item is being scanned by computer vision, barcode, and / or some other manner, it may also be validated by the methods described herein to check if the product resembles the returned scanned ID. In some embodiments, if the product does not pass the validation check, itmay be flagged. In some embodiments, this may be done in real-time or after-the-fact.

[0020] The store verification gallery 112 is a database preferably containing feature vectors extracted from images of each product captured from multiple viewpoints, orientation information for each feature vector and, optionally, the actual images from which the feature vectors were extracted, all associated with the unique identifier of the product depicted in the images (e.g., a UPC). Store verification gallery 112 may be automatically built. Each time a particular product is scanned, one or more feature vectors can be extracted from one or more captured images of the product. The feature vectors may be stored in the product database and associated with the UPC of the item. Each feature vector may be compared with feature vectors for that item already stored in the product database and, if the feature vectors are within a certain threshold distance of a feature vector already present in the product database for that item (indicating that a similar viewpoint of the item is already in the product database), the feature vector may not be added. However, if the feature vectors captured from the one or more images are sufficiently different from any other feature vector already stored in the product database for that item (indicating a new viewpoint of the item) the new feature vector may be stored in the product database and associated with the UPC for that item. Images for each product may be captured from all checkouts within the store, including both self-checkoutsand employee-assisted checkouts. Additionally, images of the same item captured at different stores may be added to the product database such as to have as many possible viewpoints of the item as possible stored in the product database.

[0021] In a first aspect of the invention, products are enrolled in the store verification gallery 112 by the process 100 shown in FIG. 1, which shows the enrollment of product 102. In one embodiment, a standard UPC image 108 of the product 102 is retrieved. The standard image 108 may be retrieved using a lookup process 106, wherein the image is provided, for example, by the manufacturer of product. Alternatively, standard UPC image 108 is captured as a standard view of the product using any camera. The standard UPC image 108 is typically a frontal image showing an unobstructed and unobscured view of the primary face of product 102.

[0022] In addition to standard UPC image 108, multiple additional images 104 of products in various orientations may be added to gallery 112. The multiple additional images 104 may be captured, in one embodiment, by one or more cameras located adjacent to or within the checkout station as product 102 is being scanned by customers in the store or in multiple other stores. In other embodiments, multiple images 104 may be captured using any camera. Multiple images 104 may include images of product 102 captured at different orientations, angles and or viewpoints and / or showing different faces of product 102 and with varying degrees of obscuration.

[0023] The captured images of products may be processed by a machine learning product detection model 202, shown in FIG. 2, trained to place a bounding box around product 102. The images within bounding boxes 203 may then be cropped by a machine learning cropping model 204 trained to extract the bounding boxes from a captured image, modify the background of the bounding box or segment the image such as to remove all background, leaving only an isolated product image 120 of product 102. In other embodiments, any means of detecting and isolating the products within the captured images may be used.

[0024] Isolated product images 120 of product 102 (including both the multiple views 104 and the standard UPC image 108) may be further processed by a trained feature extractor 206, which extracts a feature vector 122 from the isolated product image. In some embodiments, both the feature vectors 122 and the isolated product images 120 are enrolled in gallery 112 at 110, while in other embodiments, only the feature vectors 120 are enrolled in gallery 112.

[0025] In addition, an indication of the orientation 124 of the product depicted in isolated product image 120 may also be stored in gallery 112. The orientation may be determined, in some embodiments, by a machine learning orientation model 208 trained to determine the orientation, or by any other means.

[0026] Lastly, each isolated product image and / or feature vector of product 102 is associated, in gallery 112, with the UPC code of the product 126, or is associated with other identifying information (e.g., a SKU number) uniquely identifying product 102.

[0027] The store verification gallery 112 is a database preferably containing feature vectors extracted from images of each product captured from multiple viewpoints, orientation information for each feature vector and, optionally, the actual images from which the feature vectors were extracted, all associated with the unique identifier of the product depicted in the images (e.g., a UPC). Store verification gallery 112 may be automatically built. Each time a particular product is scanned, one or more feature vectors can be extracted from one or more captured images of the product. The feature vectors may be stored in the product database and associated with the UPC of the item. Each feature vector may be compared with feature vectors for that item already stored in the product database and, if the feature vectors are within a certain threshold distance of a feature vector already present in the product database for that item (indicating that a similar viewpoint of the item is already in the product database), the feature vector may not be added. However, if the feature vectors captured from the one or more images are sufficiently different from any other feature vector already stored in the product database for that item (indicating a new viewpoint of the item) the new feature vector may be stored in the product database andassociated with the UPC for that item. Images for each product may be captured from all checkouts within the store, including both self-checkouts and employee-assisted checkouts. Additionally, images of the same item captured at different stores may be added to the product database such as to have as many possible viewpoints of the item as possible stored in the product database.

[0028] In a second aspect of the invention, products may be verified as a user scans the product at a self-checkout station (single verification mode). The selfcheckout station typically consists of one or more lasers oriented to capture and read a bar code present on the product. The barcode typically encodes the UPC or SKU of the product, or other information that can uniquely identify the product. To verify the products as they are scanned, one or more cameras may be located at different positions with respect to the scanner, or within the body of the scanner, to capture one or more images of the product as the user scans the barcode.

[0029] The process 300 for verifying the product as the products are being scanned as shown in FIG. 3. Process 300 starts when the product is scanned at 310 and the UPC of the product is obtained at 312. As would be realized, steps 310 and 312 are part of the standard scanning process of the self-checkout system. As the product is being scanned at 310 one or more images are captured containing the scanned product at 302. It should be realized that as the product is being scanned, the user may hold the product at anyconvenient orientation with respect to the scanner and with respect to the one or more cameras and that portions of the product may be obscured by the hand of the user, as the user is holding the product to move it over the scanner. In other embodiments, the one or more cameras may be oriented to capture the image(s) immediately after the user puts the product down after scanning. This would alleviate difficulties arising from portions of the product being obscured by the user's hand, but may not work in some situations, for example, wherein the user places the product directly into a bag after scanning.

[0030] Once the one or more images are captured, they are processed in a manner similar to the process used for enrolling products in gallery 112, shown in FIG.2, to obtains the isolated product image. In one embodiment, the product may be detected in the image by product detection model 112, which may place the product into abounding box. In other embodiments, any method of detecting the product in the image may be used. The image may then be cropped to isolate the bounding box. The bounding box may then be processed to replace the background with a solid color, or other effect, or the product may be isolated by a segmentation process, which removes the background, leaving only the isolated product image. The cropping, background processing and / or segmentation may be performed by cropping model 204 or by any other means.

[0031] Once the image of the product is isolated, feature extractor 206 extracts a feature vector from the isolated product image. The feature vector is used in a matching process at 316 to match the product image against one or more enrolled images of the product (i.e., with feature vectors extracted from the enrolled images), which were retrieved at 314. The enrolled images of the product may be indexed by the UPC of the product stored in gallery 112, such that only images of the product corresponding to the scanned UPC are retrieved. The details of the matching process will be discussed later herein, but, at 318, if there is a confident match with at least one of the enrolled images corresponding to the scanned UPC, the product scan is validated at 320. If no confident match occurs, then an alert is raised at 322.

[0032] In yet another aspect of the invention, images containing multiple products may be scanned as a batch (batch verification mode). This may occur after the scanning process is complete and all products have been scanned. One or more images containing all or a portion of the products may by captured by one or more cameras situated where the products may be viewed as a group, for example, in the user's shopping cart or on a post-checkout platform where the products are waiting to be bagged. This process 400 is shown in flowchart form in FIG. 4.

[0033] At 402, one or more images of the batch of products is captured. The products are detected in the image at 404 and cropped to isolated product images at 406, using the methods previously discussed. At 407, one isolatedproduct image is selected for validation. A feature vector is extracted from the isolated product image using feature extractor 206.

[0034] At 408, the list of scanned UPC is retrieved and images corresponding to all scanned products are retrieved from gallery 112 at 410. At 414, it is determined of there is a confident match between the isolated image of the product and an image enrolled in the gallery 112 of all products on the receipt, using feature vectors extracted from the respective images. If there is a match with confidence at 416, the product scan is verified at 418. If not, an alert is raised at 420. At 422, it is determined if all products in the batch image have been processed and, if not, the next isolated product image is selected at 407 and processed as described, until all products in the batch image have been processed at 424.

[0035] The matching process 500 is shown in FIG. 5. Process 500 can be used for the matching of both individual products at checkout time (single verification mode) and products isolated from a batch image of the products (batch verification mode). At 502, the enrolled standard UPC image 108 corresponding to the scanned UPC (single verification mode) is retrieved. If the isolated product image cropped from the captured image matches the standard image with a confidence above a pre-determined threshold at 504, then the match is verified at 514. If the matching is being performed in batch verification mode, the enrolled standard UPC images 108 from all products listed on the receipt are retrieved, and if a match with at least one of themexceeds the pre-defined confidence threshold at 504, then the match is verified at 514.

[0036] If no match can be verified using the enrolled standard UPC image(s) 108, then, at 505, the orientation of the product in the isolated product image is determined. The orientation may be determined using orientation model 208 or by any other means. In single verification mode, all enrolled images associated with the scanned UPC of the product and indicating the same or a similar orientation as the isolated product image are retrieved at 506. If at least one of the retrieved images matches the isolated product image with a confidence exceeding the threshold at 508, then the match is verified at 514. In batch verification mode, the enrolled images from all products listed on the receipt having the same or similar orientation are retrieved and compared with the isolated product image. If at least one of these images matches with a confidence exceeding the threshold at 508, then the match is verified at 514.

[0037] If there is still no verified match, then all remaining enrolled images associated with the scanned UPC are retrieved at 510. In batch verification mode, all remaining images associated with all UPCs listed on the receipt are retrieved. If the isolated product image matches any of the retrieved enrolled images with a confidence exceeding the threshold at 512, then the match is verified at 514.

[0038] If no match can be verified at any of the 3 stages of the matching process 500, then no match is indicated at 516.

[0039] As would be realized by one of skill in the art, references herein to "retrieving images" from gallery 112 may be interpreted to mean retrieving the feature vectors extracted from the images and stored in gallery 112. Likewise, the process of "matching images" may be interpreted to mean that feature vectors extracted from each image are compared and the match is assigned a probability based on the similarity of the feature vectors. A match is indicated when the confidence level exceeds a pre-determined threshold.The similarity of feature vectors may be determined by any known means, for example, by cosine distance. The confidence level may be assigned by any known means, based on the similarity between the feature vectors, including by a trained machine learning model.

[0040] The disclosed invention may include one or more algorithms which flag fraudulent transactions to authorities when an alert is raise at step 322 of process 300 shown in FIG. 3 or at step 420 of process 400 shown in FIG. 4, which may help prevent shrinkage. In some embodiments, a general principle of verification logic against false alarms may be used, where the customer receives the benefit of the doubt when the lookup result is within a certain confidence score range.

[0041] In various embodiments of the invention, the system upon which the processes and methods are implemented may consist of softwareimplementing the processes and executing on a general purpose computing device. The software may include a trained product detector capable of detecting individual products in an image of multiple products. The software may further include a trained cropping model. The software may further include a trained feature extractor for extracting feature vectors from images of individual products. The software may further include an orientation model capable of determining the orientation of a product in an image. The software may further include the store verification gallery containing multiple feature vectors extracted from multiple images of individual products. Each feature vector in the product database may be associated with a UPC of the product depicted in the image from which the feature vector was extracted. The system may further comprise one or more cameras coupled to the computing device and associated with various self-checkout stations and one or more cameras coupled to the computing device and associated with a verification station (i.e., for capturing images in batch verification mode). The system may further include the laser scanners used by customers during the self-checkout process to scan barcodes present on the products. The system may further include an application executing on user smart devices to assist in the scanning of products and / or in the verification of the order, as described herein. All components of the system may communicate via wireless connections, for example, Wi-Fi.

[0042] The present invention illustrates various techniques and configurations that enable verification of self-checkout orders. As would be realized, many configurations of the system and / or the software are possible and are intended to be within the scope of the invention.

Claims

Claims:

1. A method for building a product gallery comprising: obtaining a standard image of a product; obtaining one or more isolated images of the product from different viewpoints; for each of the one or more isolated images, determining an orientation of the product in the image; extracting a feature vector from the standard image and each of the one of more isolated images; storing, in a database, each of the extracted feature vectors, the associated orientation information for each feature vector and a unique identifier of the product for each feature vector.

2. The method of claim 1 further comprising: storing the standard image and the one or more isolated images in the database, associated with the feature vector extracted therefrom.

3. The method of claim 1 wherein obtaining an isolated image of the product comprises: capturing an image containing the product; detecting the product in the image; and cropping the detected product to obtain the isolated image.

4. The method of claim 3 wherein detecting the product comprises: inputting the image to a machine learning model trained to detect products in an image and place a bounding box around the product.

5. The method of claim 4 wherein cropping the detected product comprises: inputting the image to a machine learning model trained to crop products in an image.

6. The method of claim 5 wherein the cropping comprises one of: extracting the bounding box from the image; extracting the bounding box from the image and modifying the background within the bounding box; or segmenting the bounding box to remove the background.

7. The method of claim 3 wherein the images containing the product are captured by one or more cameras located at a self-checkout station.

8. The method of claim 1 wherein the standard image is an image of the product showing a primary face of the product.

9. A method of verifying scanning of a product at a self-checkout station comprising: obtaining a product identifier of the product as it is being scanned capturing one or more images of the product as it is being scanned or immediately thereafter; extracting isolated images of the product from the one or more captured images; extracting a product feature vector from each of the one or more isolated images; matching the product feature vectors to one or more enrolled feature vectors retrieved from a database and associated with the product identifier; verifying the scan if one of the product feature vectors matches at least one of the enrolled feature vectors with a confidence exceeding a predetermined threshold.

10. The method of claim 9 wherein matching the product feature vector to one or more enrolled feature vectors comprises:retrieving, from the database, an enrolled feature vector extracted from a standard image of the product and associated with the product identifier; verifying the scan if a similarity between the product feature vector and the enrolled feature vector from the standard image match with a predetermined degree of confidence; if the scan is not verified, then: determining an orientation of the product in the isolated image; retrieving, from the database, one or more enrolled feature vectors extracted from isolated images of the product having a similar orientation to the orientation of the product in the isolated image and associated with the product identifier; verifying the scan if a similarity between the product feature vector and the one or more enrolled feature vectors from images having similar orientation of the product match with a pre-determined degree of confidence; if the scan is not verified, then: retrieving, from the database, all remaining enrolled feature vectors associated with the product identifier; verifying the scan if a similarity between the product feature vector and the remaining enrolled feature vectors match with a pre-determined degree of confidence; andif the scan is not verified, then: declaring a mismatch between the product identifier and the scanned product.

11. The method of claim 9 further comprising: raising an alert if the scan of the product is not verified.

12. The method of claim 9 wherein obtaining isolated images of the product comprises: detecting the product in the captured images; and cropping the detected product to obtain the isolated image.

13. The method of claim 12 wherein detecting the product comprises: inputting the image to a machine learning model trained to detect products in an image and place a bounding box around the product.

14. The method of claim 13 wherein cropping the detected product comprises: inputing the image to a machine learning model trained to crop products in an image.

15. The method of claim 14 wherein the cropping comprises one of:T2.extracting the bounding box from the image; extracting the bounding box from the image and modifying the background within the bounding box; or segmenting the bounding box to remove the background.

16. A method of verifying scanning of one or more products after scanning at a self-checkout station comprising: obtaining product identifiers of all products that have been scanned as part of a single transaction; capturing one or more images containing all or some of the products; extracting isolated images of the products from the one or more captured images; for each product: extracting a product feature vector from each of the one or more isolated images; matching the product feature vectors to one or more enrolled feature vectors retrieved from a database and associated with the product identifiers; and verifying the scan if at least one of the product feature vectors matches at least one of the enrolled feature vectors with a confidenceexceeding a pre-determined threshold.

17. The method of claim 16 wherein matching the product feature vectors to one or more enrolled feature vectors comprises: retrieving, from the database, enrolled feature vectors extracted from standard images associated with all obtained product identifiers; verifying the scan if a similarity between the product feature vector and at least one of the enrolled feature vector from the standard images match with a pre-determined degree of confidence; if the scan is not verified, then: determining an orientation of the product in the isolated image; retrieving, from the database, one or more enrolled feature vectors extracted from images of the product having a similar orientation to the orientation of the product in the isolated image and associated with all of the obtained product identifiers; verifying the scan if a similarity between the product feature vector and the one or more enrolled feature vectors from images having similar orientation of the product match with a pre-determined degree of confidence; if the scan is not verified, then:retrieving, from the database, all remaining enrolled feature vectors associated with the obtained product identifiers; verifying the scan if a similarity between the product feature vector and the remaining enrolled feature vectors match with a pre-determined degree of confidence; and if the scan is not verified, then: declaring a mismatch between the product identifier and the scanned product.

18. The method of claim 16 further comprising: raising an alert if the scan of one or more products are not verified.

19. The method of claim 16 wherein obtaining isolated images of the products comprises: detecting one or more products in the captured images; and cropping the detected products to obtain the isolated images.

20. The method of claim 19 wherein detecting the product comprises: inputting the image to a machine learning model trained to detect products in an image and place bounding boxes around the products.

21. The method of claim 20 wherein cropping the detected product comprises:inputting the image to a machine learning model trained to crop products in an image.

22. The method of claim 21 wherein the cropping comprises one of: extracting bounding boxes from the image; extracting bounding boxes from the image and modifying the background within the bounding boxes; or segmenting the bounding boxes to remove the background.

Images