Artificial intelligent (AI) based self-checkout scan verification system

Abstract

Methods and apparatus for performing AI (AI) - based scan validation for a self-checkout process using a self-checkout system are described.SOLUTION: In an exemplary method, a first image of a plurality of items in a shopping cart is displayed on a display system of a self-checkout system. A scan of a first item of the plurality of items using a handheld scanner associated with the self-checkout system is detected. A second image of at least a second item of the plurality of items in the shopping cart is generated. The second image lacks a representation of the first item. The second image is displayed on the display system.SELECTED DRAWING: Figure 3A

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION Embodiments of the present invention relate to an artificial intelligence (AI) based self-checkout scan verification system. [Background technology]

[0002]

[0001] Self-checkout systems include a variety of features that allow a user (e.g., a customer) to perform some or all of the steps of a self-checkout process, such as scanning items, weighing the items, bagging the items, submitting a payment amount, and providing a paper or electronic receipt. For example, a self-checkout system may include shelves for holding shopping baskets, one or more scanners for scanning items that the customer wishes to purchase, a bagging area for bagging the items after they are scanned, and a point-of-sale (POS) device for processing payment for the items scanned by the customer. Summary of the Invention [Means for solving the problem]

[0003] An embodiment computer-implemented method includes the steps of: displaying a first image of a plurality of items in a shopping cart on a display system of a self-checkout system; detecting that a first item of the plurality of items has been scanned by a handheld scanner of the self-checkout system; generating a second image in response to the detection, the second image including at least a second item of the plurality of items in the shopping cart, the second image not including a representation of the first item; and displaying the second image on the display system. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a diagram illustrating an exemplary self-checkout system, according to one embodiment. [Figure 2]FIG. 2 is a flowchart of a method for scan verification of a self-checkout process, according to one embodiment. [Figure 3A] FIG. 3A illustrates an exemplary scenario for scan confirmation of a self-checkout process, according to one embodiment. [Figure 3B] FIG. 3B illustrates an exemplary scenario for scan confirmation of a self-checkout transaction, according to one embodiment. [Figure 3C] FIG. 3C illustrates an exemplary scenario for scan confirmation of a self-checkout transaction, according to one embodiment. [Figure 3D] FIG. 3D illustrates an exemplary scenario for scan confirmation of a self-checkout transaction, according to one embodiment. [Figure 3E] FIG. 3E illustrates an exemplary scenario for scan confirmation of a self-checkout process, according to one embodiment. [Figure 4] FIG. 4 is a flowchart of another method for scan verification of a self-checkout process, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0005]

[0006] Self-checkout systems may include a variety of features to facilitate scanning of items by customers. For example, self-checkout systems may include various types of scanners, including, by way of illustrative and non-limiting examples, handheld scanners, integrated scanners (e.g., scanners integrated into the self-checkout system, such as flatbed scanners), and scanners with integrated load cells for measuring the weight of items.

[0006]

[0007] In some cases, a customer may use a handheld scanner to scan items in a customer's shopping cart (or basket) without physically removing the items from the customer's shopping cart. For example, a customer's shopping cart (or basket) may contain multiple items, one or more heavy items, one or more rounded (or irregular) items, or a combination thereof, making it infeasible and / or impractical for the customer to remove the items from the shopping cart, scan the items using the self-checkout system's integrated scanner (e.g., a flatbed scanner), and place the items in the bagging area of ​​the self-checkout system. For example, a customer may be unable to remove a heavy item from the shopping cart without the assistance of another user (e.g., a customer or a store clerk).

[0007]

[0008] One potential problem with allowing customers to scan items in their shopping carts with a handheld scanner (e.g., without removing the items) is that the customer may not be able to remember which items have been scanned and whether all items in the shopping cart have been scanned. This scenario can result in a poor customer experience, a less efficient self-checkout process, and retail shrinkage.

[0008]

[0009] Embodiments herein describe techniques for using image processing and artificial intelligence (AI) / machine learning (ML) to verify whether a customer has scanned all items in their shopping cart using a handheld scanner. For example, in some embodiments, a scan verification system for a self-checkout system may use one or more camera devices to capture images (and / or video) of items in a customer's shopping cart. The one or more camera devices may include camera devices from the handheld scanner, camera devices at the self-checkout system, camera devices in the environment in which the self-checkout system is located, or a combination thereof. The scan verification system may cause images of the items to be displayed on a display system associated with the self-checkout system.

[0009]

[0010] As each item from the shopping cart is scanned by the customer, the scan confirmation system may edit the image displayed on the self-checkout system's display system (e.g., using one or more image processing and / or AI / ML technologies) to remove the scanned item from the image. In this way, the scan confirmation system can provide the customer with an indication of whether they have scanned each item in the shopping cart with the handheld scanner. Once all items have been scanned, the scan confirmation system may cause an indication on the display system that scanning is complete. If the customer attempts to complete the self-checkout process before scanning all items, the scan confirmation system may cause an indication on the display system that one or more items have not been scanned and prompt the customer to scan the unscanned items.

[0010]

[0011] Advantageously, the embodiments described herein can reduce retail shrink caused by customers failing to scan all items in a shopping cart with a handheld scanner. Thus, the embodiments can significantly increase the efficiency of self-checkout processes in retail stores and improve the customer experience.

[0011]

[0012] It should be noted that the techniques described herein for AI / ML-based scan confirmation of self-checkout systems may be incorporated into (e.g., embodied within or performed by) various wired or wireless devices. In some embodiments, the device may provide connectivity to a network (e.g., a wide area network (WAN) such as the Internet or a cellular network) via a wired or wireless communication link. In some embodiments, the device may include a self-checkout system or a computing system disposed in a cloud computing environment.

[0012]

[0013] Although the features of the self-checkout system are generally described within the context of a shopping environment, such as within a retail store, it is contemplated that the techniques disclosed herein may be applied to other environments, such as, by way of illustrative and non-limiting examples, libraries, museums, classrooms, and hospitals.

[0013]

[0014] The Benefits of AI-Based Scan Verification for Self-Checkout Processing Self-checkout systems may include multiple different scanners to provide customers with flexibility in scanning items as part of the self-checkout process. For example, many self-checkout systems may include handheld scanners that allow customers to scan items in a customer's shopping cart without removing the items from the cart. Embodiments herein provide an automated method, e.g., without human intervention, for using image processing to automatically detect whether a customer has scanned all items in the shopping cart with the handheld scanner. Embodiments herein may also provide a visual indication to the user of whether each item has been scanned, e.g., by automatically editing an image displayed on the self-checkout system's display system to remove from the image each item scanned by the customer. Doing so may reduce the amount of retail shrink that occurs due to using handheld scanners to complete the self-checkout process.

[0014]

[0015] 1 is a diagram illustrating an exemplary implementation of a self-checkout system 100, according to one embodiment. The self-checkout system 100 generally includes functionality that allows a customer and / or a companion to perform some or all of the steps of a self-checkout process, such as scanning items, weighing items, bagging items, submitting payment, and providing a paper or electronic receipt. The self-checkout system 100 may be located within an environment 190, such as a retail environment (e.g., a grocery store, a clothing store, an electronics store, etc.).

[0015]

[0016] The self-checkout system 100 includes a point-of-sale (POS) terminal 102 having a display system 105 that presents information viewable by a user 170 (e.g., a customer or companion) during various stages of the self-checkout process. The display system 105 may use any suitable display technology, such as, by way of illustrative and non-limiting examples, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display. The display system 105 is communicatively coupled to one or more computer processors, which may be integrated into the POS terminal 102 or external to the POS terminal 102. For example, the one or more computer processors may be included in a computing device integrated with the POS terminal 102, which may further be networked with other computing devices. In some embodiments, the display system 105 includes an input device 115 that accepts input from the user 170 during the self-checkout process. For example, the input device 115 may include a touch-sensitive screen that uses any suitable sensing technology, such as, by way of illustrative and non-limiting examples, capacitive or resistive sensing.

[0016]

[0017] The POS terminal 102 further includes one or more product scanners 110-1, 110-2 (collectively or generically referred to as product scanners 110). Each product scanner 110 is communicatively coupled to one or more computer processors and operates in conjunction with the one or more computer processors to visually identify the product during scanning. For example, each product scanner 110 may detect a coded portion on the product (e.g., a Universal Product Code (UPC) or a Quick Response (QR) code) and / or compare an image of the product to a reference image to identify the type of product.

[0017]

[0018] In some embodiments, item scanner 110-1 includes a handheld scanner that can be pointed at an item by a user to scan the item, and item scanner 110-2 is integrated into the surface of POS terminal 102. In some embodiments, item scanner 110-2 further includes one or more load cells positioned near the surface to measure the weight of the item.

[0018]

[0019] The self-checkout system 100 further includes a printer 125 that prints or generates a tangible receipt for the self-checkout transaction. As shown, the printer 125 is located on a front panel 150 of the self-checkout system 100 below the item scanner 110-2, although the printer 125 may be located in any other suitable location. The printer 125 is communicatively coupled to one or more computer processors. In some embodiments, the printer 125 generates a paper receipt for the self-checkout transaction and / or coupons.

[0019]

[0020] Self-checkout system 100 further includes a payment terminal 120 capable of accepting electronic payments from user 170. In some embodiments, payment terminal 120 includes one or more of a credit card reader, a display device, a PIN pad, a near field communication (NFC) reader, or the like. In some embodiments, payment terminal 120 includes one or more computer processors, memory, or the like that are distinct from the computing hardware of POS terminal 102. In this manner, payment terminal 120 may be effectively isolated from POS terminal 102 to ensure compliance with the Payment Card Industry Data Security Standard (PCI DSS) or other standards.

[0020]

[0021] The self-checkout system 100 may further include a cash acceptor on the front panel 150 configured to accept bills and / or coins from the user 170 as payment for the self-checkout transaction. In some cases, the cash acceptor may be configured to dispense bills and / or coins to the user 170 as change.

[0021]

[0022] Self-checkout system 100 further includes a bagging area 130 including a plurality of bagging stations 140-1, 140-2 (collectively or generically referred to as bagging stations 140). Each bagging station 140 includes structure suitable for dispensing, supporting, suspending, and / or holding disposable shopping bags and / or reusable bags or carrier bags. As shown, bagging stations 140-1, 140-2 are positioned on a fixed platform 135, and bagging station 140-1 engages a handle of a reusable carrier bag 145 to hold the reusable carrier bag 145 on the fixed platform 135. Other configurations of bagging area 130, such as a different number and / or arrangement of bagging stations 140, as well as a rotatable carousel, are also contemplated.

[0022]

[0023] In some cases, user 170 may bring shopping cart 160 (containing one or more items 165) to POS terminal 102 to begin the self-checkout process. User 170 may scan items 165 using item scanner 110-2 (e.g., an integrated flatbed scanner) and place the scanned items in bagging area 130, for example, in a reusable tote bag 145 or on stationary platform 135. In some cases, user 170 may scan items 165 using item scanner 110-1 (e.g., a handheld scanner) without removing items 165 from shopping cart 160. For example, shopping cart 160 may contain one or more heavy and / or rounded (or irregular) items that make it impractical or infeasible for user 170 to remove the items from the shopping cart during the self-checkout process. In such cases, item scanner 110-1 provides the user with a convenient option to scan items without removing them from the shopping cart. After scanning an item 165 with item scanner 110-1, user 170 may keep the scanned item in shopping cart 160 as opposed to placing the scanned item in bagging area 130.

[0023]

[0024] During the self-checkout process, the POS terminal 102 may update the list of items included in the self-checkout process as the user 170 adds items to the bagging area 130 or removes items from the shopping cart 160. The POS terminal 102 may also provide guidance (e.g., instructions) to the user 170 via the display system 105. The user 170 may request assistance by input to the POS terminal 102 (e.g., touch input to the input device 115).

[0024]

[0025] The self-checkout process may be monitored by one or more cameras disposed in environment 190. For example, in some embodiments, overhead camera 172 is mounted on top of the housing of self-checkout system 100, and its field-of-view (FOV) includes self-checkout system 100 (including shopping cart 160). In that manner, camera 172 can capture images (and video) of user 170 moving items onto item scanner 110-2 and / or user 170 using item scanner 110-1 to scan items 165 in shopping cart 160. In some embodiments, item scanner 110 may include an integrated camera. For example, item scanner 110-1 (e.g., a handheld scanner) and / or item scanner 110-2 (e.g., a flatbed scanner) may include an integrated camera that captures images (and / or video) of the items being scanned.

[0025]

[0026] In some embodiments, environment 190 includes one or more cameras 195 positioned external to self-checkout system 100. Cameras 195 may be part of a monitoring system for environment 190. Cameras 195 may be communicatively coupled to self-checkout system 100 and / or a computing system communicatively coupled to the self-checkout system. In some embodiments, shopping cart 160 may be equipped with one or more cameras 155. Cameras 155 may be configured to capture images of items (or other contents) in shopping cart 160. Cameras 155 may be installed or positioned at various locations on shopping cart 160, such as on the sides of shopping cart 160, the bottom of shopping cart 160, and the front of shopping cart 160. Cameras 155 may be communicatively coupled to self-checkout system 100 and / or a computing system communicatively coupled to the self-checkout system.

[0026]

[0027] In some embodiments, images (and / or video) captured from camera 195, camera 155, camera 172, and / or the integrated camera of item scanner 110 may be transmitted to self-checkout system 100 and / or another computing system for processing and analysis, such as identifying the type of item in the shopping cart and generating images of the items for display on display system 105. In some cases, visual data (e.g., images and / or video) captured from multiple cameras may be aggregated and registered to generate images of the items displayed on the display system. In some embodiments, the collected visual data, the processed visual data, or a combination thereof may be stored in a storage system (e.g., storage 188) communicatively coupled to self-checkout system 100.

[0027]

[0028] Self-checkout system 100 also includes computing system 180. Computing system 180 may be integrated into the housing of self-checkout system 100 (e.g., as part of POS terminal 102) or may be located external to self-checkout system 100 and communicatively coupled to self-checkout system 100 using, for example, an Ethernet cable or a wireless connection. Computing system 180 may represent any number of computing devices. For example, computing system 180 may be embodied by a computer device located at POS terminal 102, may be a server located elsewhere in environment 190, may be one or more computer devices located in a cloud computing environment, or a combination thereof.

[0028]

[0029] Computing system 180 includes processor 182, memory 184, and storage 188. Processor 182 represents one or more processing elements, each of which may include one or more processing cores. Memory 184 may be volatile memory, non-volatile memory, or a combination thereof. Memory 184 includes various instructions executable by processor 182 to perform one or more techniques described herein, for example, to verify that a customer has scanned all items in a shopping cart using a handheld scanner. In this regard, memory 184 includes a POS application 186 (e.g., a software application) that controls the operation of self-checkout system 100. For example, POS application 186 may include any number of software modules (or suites of software applications) that communicate with POS terminal 102 (including item scanner 110, display system 105, and input device 115), payment terminal 120, camera 172, camera 195, camera 155, other components within self-checkout system 100, other computing systems within environment 190, or a combination thereof. The POS application 186 can receive inputs from and send instructions to these components. For example, the POS application 186 can cause certain visual components (e.g., graphic features, structures, and / or elements comprising virtual reality (VR), mixed reality (MR), and / or artificial intelligence (AI)-generated imagery) generated using the techniques described herein to be displayed on the display system 105.

[0029]

[0030] In some embodiments, POS application 186 performs AI-based scan verification as part of the self-checkout process. For example, POS application 186 may process visual data of items 165 in shopping cart 160 captured from one or more cameras in environment 190 (e.g., camera 155, camera 195, camera 172, and / or the integrated camera of item scanner 110), generate images of the items (e.g., using image processing techniques), and display the images on display system 105 for view by user 170 during the self-checkout process. As user 170 scans each item 165 in shopping cart 160 with item scanner 110-1 (e.g., a handheld scanner), POS application 186 may edit the image to remove the scanned item from the image. In this manner, POS application 186 may provide user 170 with an indication of which items have been scanned and which have not.

[0030]

[0031] The instructions may be one or more graphic patterns, one or more graphic structures, and / or one or more graphic elements that form (or comprise) a virtual reality (VR), mixed reality (MR), and / or artificial intelligence (AI)-generated image, among other visual components. In an illustrative, non-limiting example, the instructions may include a graphical user interface (GUI) component that can be displayed on display system 105. The GUI component may include, for example, code that generates a hypertext markup language (HTML) component or HTML component that can be passed to display system 105 and rendered as the graphic pattern / structure / element thereon. The GUI component may further include instructions executable by processor 182 to display the graphic pattern / structure / element using language-specific or operating system-specific GUI components (e.g., Abstract Window Toolkit, Swing API components on the Java platform, etc.). In general, instructions capable of rendering a GUI on display system 105 may include computer-executable code generated by compiling and / or interpreting C (or variants thereof), Java, PHP, Ruby, HTML, Javascript, Python, AJAX, VBscript, and other programming or scripting languages ​​used to construct and present the GUI.

[0031]

[0032] Once all items have been scanned, the POS application 186 may cause an indication on the display system 105 that scanning is complete. In some embodiments, such an indication may be in the form of an image of an empty shopping cart. However, as noted, in general, such an indication may include any graphic patterns, structures, and / or elements that make up VR, MR, and / or AI-generated imagery, among other visual components. In some embodiments, if the user 170 attempts to complete the self-checkout process before scanning all items in the shopping cart 160, the POS application 186 may cause an indication on the display system 105 that one or more items 165 have not been scanned and a prompt on the display system 105 requesting the user 170 to scan the unscanned items 165.

[0032]

[0033] It should be noted that FIG. 1 illustrates an exemplary reference configuration of a self-checkout system 100 in which the techniques presented herein may be implemented, and that other configurations of a self-checkout system consistent with the functionality described herein are contemplated.

[0033]

[0034] 2 is a flowchart of a method 200 for performing scan confirmation of a self-checkout transaction, according to one embodiment. Method 200 may be performed by a computing system (e.g., computing system 180). In some cases, method 200 may be performed in the course of a customer using a self-checkout system (e.g., self-checkout system 100) for a self-checkout transaction.

[0034]

[0035] Method 200 may begin at block 202, where a computing system detects a shopping cart in proximity to a self-checkout system. For example, the computing system may detect the shopping cart using one or more proximity sensors, a camera device (e.g., camera device 172), or a combination thereof.

[0035]

[0036] In block 204, the computing system receives an indication that the customer wishes to perform a self-checkout transaction using a handheld scanner. For example, the customer may use the POS terminal 102 to indicate that the customer wishes to perform a self-checkout transaction using a handheld scanner.

[0036]

[0037] In block 206, the computing system captures images of the items in the shopping cart with one or more camera devices, such as, for example, camera device 172, the integrated camera of item scanner 110-1, camera device 195, camera device 155, or a combination thereof.

[0037]

[0038] In block 208, the computing system causes the image to be displayed on a display system (e.g., display system 105) of the self-checkout system. Using the reference example shown in FIG. 3A , the computing system may capture images of items 165-1 through 165-3 in shopping cart 160 and generate image 310-1 based on the captured images. In some cases, generated image 310-1 may be the same as the captured image (e.g., the computing system may capture a single image of items 165-1 through 165-3). In other cases, generated image 310-1 may be a registered image from multiple images captured from one or more camera devices. As shown in FIG. 3A , the computing system may cause generated image 310-1 to be displayed on display system 105. Generated image 310-1 includes an indication 365-1 of item 165-1 in shopping cart 160, an indication 365-2 of item 165-2 in shopping cart 160, and an indication 365-3 of item 165-3 in shopping cart 160. The instructions 365-1 through 365-3 may be in the form of a graphic pattern, a graphic structure, and / or one or more graphic elements that make up the VR, MR, and / or AI-generated image, among other visual components.

[0038]

[0039] 2 , one or more of blocks 210, 212, 214, and 216 may be executed while items are being scanned with a handheld scanner. In block 210, the computing system determines whether the position of at least one of the items in the shopping cart has changed. In particular embodiments, the computing system may determine whether the position of at least one of the items in the shopping cart has changed based on performing a difference operation (e.g., an image diff) between one or more images captured by one or more camera devices. In such embodiments, the computing system may continuously capture images of the self-checkout area, including the self-checkout system and the shopping cart.

[0039]

[0040] If the computing system determines that the position of at least one item in the shopping cart has changed, then method 200 proceeds to block 212. If the position of the items has not changed, then method 200 proceeds to block 214.

[0040]

[0041] In block 212, the computing system captures new images of the items in the shopping cart using one or more camera devices, such as, for example, camera device 172, an integrated camera in item scanner 110-1, camera device 195, camera device 155, or a combination thereof.

[0041]

[0042] In block 214, the computing system modifies the image to remove the previously scanned item from the image. In block 216, the computing system causes the modified image to be displayed on the display system. For example, with reference to FIG. 3B , the computing system may modify image 310-1 to remove designation 365-3 of third item 165-3 from image 310-1 and cause modified image 310-2 to be displayed on display system 105.

[0042]

[0043] 2, the computing system may continue executing one or more of blocks 210, 212, 214, and 216 while an item is being scanned with the handheld scanner. For example, with reference to FIG. 3C, after a user scans item 165-2 with a handheld scanner (e.g., item scanner 110-1), the computing system may modify image 310-2 to remove designation 365-2 for item 165-2 from image 310-2 and cause modified image 310-3 to be displayed on display system 105. Further, with reference to FIG. 3D, after a user scans item 165-1 with the handheld scanner, the computing system may modify image 310-3 to remove designation 365-1 for item 165-1 from image 310-3 and cause modified image 310-4 to be displayed on display system 105. While the example shown in FIG. 3D illustrates a computing system causing an image of an empty shopping cart to be displayed on a display system, it should be noted that the computing system may cause any indication to be displayed on the display system indicating that all items 165 in the shopping cart 160 have been scanned.

[0043]

[0044] 2, at block 218, the computing system determines whether the user selected the "Finish and Pay" option on the display system 105 (or whether the user has otherwise indicated that they are finished scanning items with the handheld scanner). If so, method 200 proceeds to block 222. At block 222, the computing system determines whether there are any unscanned items remaining in the shopping cart. If so, method 200 proceeds to block 220. If not, method 200 proceeds to block 224.

[0044]

[0045] In block 220, the computing system causes an indication on the display system that there is at least one unscanned item in the shopping cart and a prompt on the display system requesting the user to scan the unscanned item. For example, referring to FIG. 3E , if the user attempts to complete the self-checkout transaction without scanning item 165-1, the computing system may cause image 310-5 including prompt 350 along with an indication 365-1 of unscanned item 165-1 to be displayed on display system 105.

[0045]

[0046] 2, at block 224, the computing system may complete the self-checkout process, such as by prompting the user to pay. Method 200 may then end.

[0046]

[0047] It should be noted that the computing system may use various AI / ML and / or image processing techniques to modify images displayed on the display system. For example, in some cases, the computing system may use an AI image generator to generate / modify images for scan verification. AI image generators are generally based on deep learning neural networks that are trained on large datasets of images. AI image generators can be used to generate new images by predicting pixel values ​​based on patterns learned from the training data. Examples of AI image generators may include, but are not limited to, style transfer, generative adversarial networks (GANs), variational autoencoders, DeepDream, neural style transfer, and stable diffusion, among others.

[0047]

[0048] 4 is a flowchart of another method 400 for performing scan confirmation of a self-checkout transaction, according to one embodiment. Method 400 may be performed by a computing system (e.g., computing system 180). In some cases, method 400 may be performed in the course of a customer using a self-checkout system (e.g., self-checkout system 100) for a self-checkout transaction.

[0048]

[0049] Method 400 may begin at block 402, where a computing system causes a first image (e.g., image 310-1) of a plurality of items (e.g., items 165-1 through 165-3) in a shopping cart to be displayed on a display system (e.g., display system 105) of a self-checkout system. The first image may include respective instructions (e.g., instructions 365) for each item in the shopping cart.

[0049]

[0050] In block 404, the computing system detects that a first item (e.g., item 165-3) of the plurality of items has been scanned using a handheld scanner (e.g., item scanner 110-1) associated with the self-checkout system.

[0050]

[0051] In block 406, the computing system generates a second image (e.g., image 310-2) of at least a second item of the plurality of items in the shopping cart in response to the detection, the second image lacking any representation (e.g., instructions 365-3) of the first item.

[0051]

[0052] In block 408, the computing system causes the second image to be displayed on the display system.

[0052]

[0053] In certain embodiments, generating the second image (at block 406) may include (i) modifying the first image to remove the first item from the first image, and (ii) using the modified first image as the second image.

[0053]

[0054] In certain embodiments, method 400 may further include, upon detecting a change in at least one of the position or orientation of at least one item of the plurality of items, acquiring a third image of the plurality of items in the shopping cart. In such embodiments, generating the second image (at block 406) may include (i) modifying the third image to remove the first item from the third image, and (ii) using the modified third image as the second image. Method 400 may further include, upon detecting a change in at least one of the position or orientation of at least one item of the plurality of items, the second image further lacking a representation of the third item.

[0054]

[0055] In certain embodiments, method 400 may further include the computing system (i) receiving a request to complete a self-checkout process for a plurality of items, (ii) after receiving the request, determining that at least one of the plurality of items has not been scanned using the handheld scanner, and (iii) in response to the determination, causing a display system to display instructions requesting a scan of at least one of the plurality of items that has not been scanned. In such embodiments, method 400 may further include the computing system (in response to the determination) displaying a prompt on the display system requesting a user of the self-checkout system to scan at least one of the plurality of items that has not been scanned. In some cases, at least one of the plurality of items that has not been scanned may include at least a second item of the plurality of items, and the instructions requesting a scan of at least one of the plurality of items that has not been scanned may include a second image.

[0055]

[0056] In certain embodiments, method 400 may further include the computing system (i) receiving a request to complete a self-checkout process for a plurality of items, (ii) after receiving the request, determining that each of the plurality of items has been scanned using the handheld scanner, and (iii) in response to the determining, causing a third image to be displayed on the display system indicating that each of the plurality of items has been scanned. In such embodiments, the third image may lack an indication of any of the plurality of items.

[0056]

[0057] In certain embodiments, method 400 may further include the computing system (i) acquiring one or more third images of the plurality of items in the shopping cart, and (ii) generating the first image from the one or more third images. The one or more third images may be acquired from at least one of (i) a first camera device integrated into the handheld scanner, (ii) a second camera device attached to the self-checkout system, (iii) a third camera device disposed in an environment in which the self-checkout system is located, or (iv) a fourth camera device disposed on the shopping cart, each of the first camera device, second camera device, third camera device, and fourth camera device having a field of view capable of capturing at least one item of the plurality of items and communicatively coupled to the self-checkout system.

[0057]

[0058] As used herein, "processor," "at least one processor," or "one or more processors" generally refers to a single processor configured to perform one or more operations, or to multiple processors configured to jointly perform one or more operations. In the case of multiple processors, performance of one or more operations may be divided among different processors, although one processor may perform multiple operations and multiple processors may jointly perform a single operation. Similarly, "memory," "at least one memory," or "one or more memories" generally refers to a single memory configured to store data and / or instructions, or to multiple memories configured to jointly store data and / or instructions.

[0058]

[0059] The descriptions of various embodiments are presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0059]

[0060] The foregoing references have been made to embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the described embodiments. Instead, any combination of the following features and elements, whether associated with different embodiments, is contemplated for implementation and practice of contemplated embodiments. Furthermore, the embodiments disclosed herein may achieve advantages over other possible solutions or the prior art, and whether an advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Accordingly, the following aspects, features, embodiments, and advantages are merely exemplary and are not considered elements or limitations of the appended claims unless expressly recited in the claims. Similarly, references to "the present disclosure" should not be construed as a generalization of the inventive subject matter disclosed herein, and should not be considered elements or limitations of the appended claims unless expressly recited in the claims. Furthermore, when elements of an embodiment are described in the form of "at least one of A and B" or "at least one of A or B," it is understood that embodiments including only element A, only element B, and both element A and B are contemplated, respectively.

[0060]

[0061] Aspects of the described embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be generally referred to herein as a "circuit," "module," or "system."

[0061]

[0062] One or more of the described embodiments may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the embodiment.

[0062]

[0063] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction-execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures with instructions recorded thereon, and any suitable combination of the above. Computer-readable storage medium, as used herein, should not be construed as a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0063]

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0064]

[0065] The computer-readable program instructions for carrying out the operations of the described embodiments may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the described embodiments.

[0065]

[0066] Aspects of the described embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0066]

[0067] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, form means for performing the functions / acts specified in the flowchart and / or block diagram blocks. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in the manner described, such that the computer-readable storage medium having stored thereon instructions comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0067]

[0068] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions, executing on the computer, other programmable apparatus, or other device, perform the functions / acts specified in the flowchart and / or block diagram blocks.

[0068]

[0069] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may possibly be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0069]

[0070] Embodiments may be provided to end users via a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides an abstraction between computing resources and their underlying technical architecture (e.g., servers, storage, network), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in the "cloud" regardless of the underlying physical systems (or the location of those systems) used to provide the computing resources.

[0070]

[0071] Typically, cloud computing resources are provided to users on a pay-per-use basis, with users being charged for the computing resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualization systems instantiated by the user). Users can access all of the resources present in the cloud at any time from anywhere via the Internet. In the context of the described embodiment, users may access applications (e.g., POS applications 186, AI / ML models, visual data 192) and / or related data (e.g., captured images and / or videos) available in the cloud. Doing so allows users to access this information from any computing system connected to a network (e.g., the Internet) connected to the cloud.

[0071]

[0072] While the forgoing is directed to one or more embodiments, other and further embodiments may be devised without departing from the basic scope thereof, the scope of which is determined by the claims that follow. [Explanation of symbols]

[0072] 100 Self-checkout system 102 POS terminals 105 Display System 110 Product Scanner 110-1 Product scanner 110-2 Product scanner 115 Input Devices 120 payment terminals 125 printer 130 Bagging Area 135 Fixed Platform 140 Bagging Station 140-1 Bagging Station 140-2 Bagging Station 145 Carrying Bag 150 Front Panel 155 Camera 160 Shopping Cart 165 items 165-1 item 165-2 items 165-3 items 170 users 172 overhead camera 180 Computing Systems 182 processors 184 memory 186 POS Applications 188 Storage 190 Environment 192 Visual Data 195 cameras 310-1 Generated image 310-2 Corrected image 310-3 Corrected image 310-4 Corrected image 310-5 Image 350 prompts 365 Instructions 365-1 Instructions 365-2 Instructions 365-3 Instructions

Claims

1. 1. A computer-implemented method comprising: displaying a first image of the plurality of items in the shopping cart on a display system of the self-checkout system; detecting that a first item of the plurality of items has been scanned by a handheld scanner of the self-checkout system; generating a second image in response to said detecting, said second image including at least a second item of said plurality of items in said shopping cart, said second image not including a representation of said first item; displaying the second image on the display system; 20. A computer-implemented method comprising:

2. The step of generating the second image comprises: modifying the first image to remove the first item from the first image; using the modified first image as the second image; The computer-implemented method of claim 1 , comprising:

3. capturing a third image of the plurality of items in the shopping cart upon detecting a change in at least one of a position or an orientation of at least one item of the plurality of items; further comprising The step of generating the second image comprises: modifying the third image to remove the first item from the third image; using the modified third image as the second image; The computer-implemented method of claim 1 , comprising:

4. 4. The computer-implemented method of claim 3, wherein modifying the third image comprises modifying the third image to remove from the third image a third product of the plurality of products scanned using the handheld scanner, and the second image further lacks representation of the third product.

5. receiving a request to complete a self-checkout transaction for the plurality of items; determining, after receiving the request, that at least one of the plurality of items has not been scanned using the handheld scanner; In response to the determination, displaying instructions on the display system requesting scanning of at least one item of the plurality of items that has not been scanned; The computer-implemented method of claim 1 further comprising:

6. 6. The computer-implemented method of claim 5, further comprising the step of: in response to the determination, displaying a prompt on the display system, the prompt requesting a user of the self-checkout system to scan at least one item of the plurality of items that has not been scanned.

7. at least one item of the plurality of items that is not scanned includes at least the second item of the plurality of items; the instructions for at least one item of the plurality of items that has not been scanned include the second image; 6. The computer-implemented method of claim 5.

8. receiving a request to complete a self-checkout transaction for the plurality of items; determining, after receiving the request, that each of the plurality of items has been scanned using the handheld scanner; displaying a third image on the display system in response to the determination, the third image indicating that each of the plurality of items has been scanned; The computer-implemented method of claim 1 further comprising:

9. The computer-implemented method of claim 8 , wherein the third image lacks a representation of any of the plurality of products.

10. capturing one or more third images of the plurality of items in the shopping cart; generating the first image from the one or more third images; The computer-implemented method of claim 1 further comprising:

11. 11. The computer-implemented method of claim 10, wherein the one or more third images are acquired from at least one of (i) a first camera device integrated within the handheld scanner, (ii) a second camera device attached to the self-checkout system, (iii) a third camera device disposed in an environment in which the self-checkout system is located, or (iv) a fourth camera device disposed on the shopping cart, wherein each of the first camera device, the second camera device, the third camera device, and the fourth camera device has a field of view capable of capturing at least one item of the plurality of items and is communicatively coupled to the self-checkout system.

12. A self-checkout system, A handheld scanner and A display system; a computing system for controlling the self-checkout system, the computing system comprising: displaying a first image of a plurality of items in the shopping cart on the display system; detecting that a first item of the plurality of items has been scanned by the handheld scanner; generating a second image in response to said detecting, said second image including at least a second item of said plurality of items in said shopping cart, said second image not including a representation of said first item; Displaying the second image on the display system. a computing system configured to: A self-checkout system comprising:

13. To generate the second image, the computing system: modifying the first image to remove the first item from the first image; Use the modified first image as the second image The self-checkout system of claim 12, configured as follows:

14. the computing system is further configured to, upon detecting a change in at least one of a position or an orientation of at least one item of the plurality of items, capture a third image of the plurality of items in the shopping cart; To generate the second image, the computing system: modifying the third image to remove the first item from the third image; Use the modified third image as the second image It is configured as follows: The self-checkout system of claim 12.

15. the computing system is further configured to modify the third image to remove from the third image a third product of the plurality of products scanned with the handheld scanner, such that the second image further lacks representation of the third product. The self-checkout system of claim 14.

16. The computing system includes: receiving a request to complete a self-checkout transaction for the plurality of items; determining, after receiving the request, that at least one of the plurality of items has not been scanned using the handheld scanner; In response to the determining, displaying instructions on the display system requesting scanning of at least one item of the plurality of items that has not been scanned. The self-checkout system of claim 12 further configured to:

17. 17. The self-checkout system of claim 16, wherein the computing system is further configured to, in response to the determining, cause a prompt to be displayed on the display system, the prompt requesting a user of the self-checkout system to scan at least one item of the plurality of items that has not been scanned.

18. at least one item of the plurality of items that is not scanned includes at least the second item of the plurality of items; the instructions for at least one item of the plurality of items that has not been scanned include the second image; 17. The self-checkout system of claim 16.

19. The computing system includes: receiving a request to complete a self-checkout transaction for the plurality of items; determining, after receiving the request, that each of the plurality of items has been scanned using the handheld scanner; In response to the determining, displaying a third image on the display system indicating that each of the plurality of items has been scanned. The self-checkout system of claim 12 further configured to:

20. A non-transitory computer-readable medium containing computer-executable instructions that, when jointly executed by one or more processors of a computing system, cause the computing system to: Displaying a first image of the plurality of items in the shopping cart on a display system of the self-checkout system; Detecting that a first item of the plurality of items has been scanned by a handheld scanner associated with the self-checkout system; generating a second image in response to the detection, the second image including at least a second item of the plurality of items in the shopping cart, the second image not including a representation of the first item; displaying the second image on the display system; and A non-transitory computer-readable medium for causing a computer to perform operations including:

Images