Point of sale information management item prediction and validation

Abstract

To provide a technique for predicting items for purchase in a point of sale information management (POS) system.SOLUTION: One or more images of one or more items for purchase, captured in a POS system, are specified, and measured properties of the one or more items for purchase are specified. The measured properties include at least one of (i) a first weight measured using a weight sensor associated with the POS system and (ii) a footprint measured using one or more pressure or weight sensors associated with the POS system. These techniques include predicting the items for purchase based on an analysis of the one or more images using a machine learning (ML) model, and validating the predicted items for purchase using the measured properties.SELECTED DRAWING: Figure 1B

Description

[Background technology]

[0001] Point of sale (POS) terminals can be used by purchasers to purchase items for sale. For example, a retail store can include one or more self-checkout kiosks to allow purchasers to purchase items. These self-checkout kiosks can function as POS terminals and can be used to specify items for purchasers and allow the purchasers to purchase the specified items. [Brief explanation of the drawings]

[0002] [Figure 1A] FIG. 1A illustrates an example checkout area with POS item prediction, according to one embodiment. [Figure 1B]

[0003] FIG. 1B illustrates an example POS kiosk for item prediction, according to one embodiment. [Figure 2]

[0004] FIG. 2 is a block diagram illustrating a controller for POS item prediction according to one embodiment. [Figure 3]

[0005] FIG. 3 is a flow chart illustrating POS item prediction using weight detection according to one embodiment. [Figure 4]

[0006] FIG. 4 illustrates the use of a kiosk for POS item prediction using weight detection, according to one embodiment. [Figure 5]

[0007] FIG. 5 is a flow chart illustrating weight validation for POS item prediction according to one embodiment. [Figure 6]

[0008] FIG. 6 is a flowchart illustrating training a product weight prediction machine learning (ML) model according to one embodiment. [Figure 7]

[0009] FIG. 7 is a flowchart illustrating the use of an ML model to estimate product weight, according to one embodiment. [Figure 8]

[0010] FIG. 8 is a flow chart illustrating POS item prediction using weight and item footprint according to one embodiment. [Figure 9]

[0011] FIG. 9 illustrates the use of a kiosk for POS item prediction using weight and item footprint, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0003]

[0012] Identifying items at a POS terminal is a challenging problem. For example, a POS system may include one or more image capture devices (e.g., cameras) for capturing images of items being purchased. Computer vision techniques (e.g., suitable ML models) may be used to detect items that are visible to the camera. However, computer vision has significant challenges with identifying items that are not visible to the camera (e.g., items that overlap each other) or items that appear visually similar but are actually different (e.g., items available in different sizes).

[0004]

[0013] For example, a purchaser may place a smaller item below a larger item at a POS terminal, such that the larger item is visible to a camera associated with the POS terminal, but the smaller item is not. Computer vision that relies solely on cameras will not identify items that are not visible to the camera, leading to an incorrect prediction of the item being purchased. Furthermore, computer vision may misidentify identical items of different sizes. For example, a purchaser may attempt to purchase two cans of soda of the same brand but different sizes. A computer vision system may incorrectly identify the two cans as the same item or may be unable to distinguish between the cans and therefore present the purchaser with a list of available sizes of cans (e.g., instead of correctly predicting the can actually being purchased).

[0005]

[0014] One or more techniques described below may alleviate one or more of these problems. For example, weight verification may be used for item recognition (e.g., in addition to or instead of visual item recognition). In one embodiment, a POS system may include a scale, and a weight deviation may be identified between the actual recorded weight of an item for purchase and a predicted weight (e.g., predicted using a suitable ML model). This weight deviation may be used to identify errors in item prediction, to improve item prediction (e.g., in conjunction with visual item recognition), or for any other suitable technique. This is further described below with respect to FIGS. 3-7.

[0006]

[0015] As another example, an item footprint may be used for item recognition. In one embodiment, a POS terminal may include a weight or pressure sensor that identifies pressure points on an item for purchase. This may be used to generate a footprint for the item (e.g., the footprint of the item as it rests on the POS terminal). This footprint may be compared to a shape identified from a captured image (e.g., captured from above the item for purchase) and used to predict the item for purchase, to verify the purchase transaction, or both. This is described further below with respect to FIGS. 8-9. For example, a purchaser may be purchasing an item (e.g., a can of soda) that is available in different sizes. Computer vision techniques may be used to identify the category of the item (e.g., the brand and type of soda), but may not be able to predict the size. The item footprint, the item weight, or both may be used to refine the prediction and select from options (e.g., from among available sizes).

[0007] [Advantages of predictive validation]

[0016] As described above, in one embodiment, weight, footprint, or both may be used to verify or improve predictions about items at a POS terminal. This has numerous technical advantages. For example, computer vision techniques may be inadequate to accurately predict items for purchase. This may be because some items are not visible in the captured image (e.g., a smaller item may be placed below a larger item), because items may be available in visually similar but different variations (e.g., different sizes for a desired product), or due to various other options. Using weight, footprint, or both to verify item predictions may result in significantly more accurate predictions. This solves technical problems inherent in computer vision techniques by using additional data (e.g., weight, footprint, or both) to supplement the computer vision analysis.

[0008]

[0017] Furthermore, one or more techniques described below may reduce computational resources used for prediction by reducing resources used for computer vision prediction. For example, fewer training resources may be used to train a computer vision ML model because the computer vision ML model does not distinguish between items that may instead be distinguished based on footprint, weight, or both. As another example, inference using a computer vision ML model may be less computationally intensive due to the improved predictions provided by using footprint, weight, or both.

[0009]

[0018] 1A illustrates an example checkout area 100 with POS item prediction, according to one embodiment. In one embodiment, checkout area 100 is associated with a retail environment (e.g., a grocery store). This is merely an example, and checkout area 100 may be associated with any suitable environment.

[0010]

[0019] One or more purchasers 102 use checkout area 110 (e.g., to pay for purchases). In one embodiment, checkout area 110 includes multiple point-of-sale (POS) systems 120A-N. For example, one of purchasers 102 may use one of self-checkout POS systems 120A-N to purchase items. Checkout area 110 also includes employee station 126. For example, an employee (e.g., a retail store employee) may use employee station 126 to monitor purchasers 102 and POS systems 120A-N. Self-checkout is merely one example, and POS systems 120A-N may be any suitable systems. For example, POS system 120A may be an assisted checkout kiosk where an employee assists purchasers with checkout, or checkout area 110 may be fully purchaser-focused and not include employee assistance.

[0011]

[0020] In one embodiment, each of POS systems 120A-N includes components used by a purchaser for self-checkout. For example, POS system 120A includes a scanner 122 and one or more sensors 124A-N (e.g., an image capture device, a weight sensor, a pressure sensor, or any other suitable sensor). In one embodiment, purchaser 102 may use scanner 122 to scan a UPC on an item. Furthermore, in one embodiment, scanner 122 may be integrated with or replaced by one or more of sensors 124A-N. For example, sensors 124A-N may include one or more image capture devices, which may be used to identify the UPC of an item. An example POS system (e.g., POS system 120A) is further described below with respect to FIG. 1B .

[0012]

[0021] In one embodiment, POS system 120A may communicate with management system 140 using network 130. Network 130 may be any suitable communications network, including a local area network (LAN), a wide area network (WAN), a cellular communications network, the Internet, or any other suitable communications network. POS system 122A may communicate with network 130 using any suitable network connection, including a wired connection (e.g., an Ethernet connection), a WiFi connection (e.g., an 802.11 connection), or a cellular connection.

[0013]

[0022] In one embodiment, POS system 120A may communicate with management system 140 to identify items scanned by purchaser 102 and to perform other functions related to self-checkout. Management system 140 is further described below with respect to FIGURE 2. For example, POS system 120A may use management system 140 to identify items (e.g., using scanner 122, sensors 124A-N, or any combination thereof).

[0014]

[0023] 1A illustrates a management system 140 connected to checkout area 110 using communications network 130. Management system 140 may transmit identifying information about the item (e.g., an alphanumeric UPC, PLU code, SKU code, price, text description, or any other suitable information) back to POS system 120A. This is merely an example, and management system 140 may be maintained completely or partially on a local computer accessible to POS system 120A (e.g., maintained on POS system 120A itself or in a local storage repository) without using a network connection.

[0015]

[0024] Additionally, in one embodiment, sensors 124A-N may also be components of POS system 120A and may be used to identify items that a purchaser is attempting to purchase. For example, sensors 124A-N may include image capture devices used to capture one or more images of items that purchaser 102 is attempting to purchase. POS system 120A may transmit the images to management system 140 to identify the items depicted in the images. Management system 140 may then use a suitable trained ML model to identify the items depicted in the images and return identification information about the identified items to POS system 120A.

[0016]

[0025] For example, management system 140 may send a code (e.g., PLU) identifying an item to POS system 120A. POS system 120A may use this code to look up the item and present the item to the user (e.g., display an image of the item and a text description of the item). In one embodiment, information about the item (e.g., stock image and text description) presented to the user is maintained at POS system 120A. Alternatively, this information may be maintained in another suitable location. For example, POS system 120A may communicate with any suitable storage location (e.g., a local storage location or a cloud storage location) to retrieve information (e.g., using an identification code for the item). Alternatively or additionally, management system 140 may provide the information (e.g., image and text description) to the user.

[0017]

[0026] FIG. 1B illustrates an example POS kiosk 150 for item prediction, according to one embodiment. In one embodiment, POS kiosk 150 provides an example of POS systems 120A-N illustrated in FIG. 1A. POS kiosk 150 includes image capture devices 162A-N (e.g., cameras), one or more displays 164A-N, one or more pads 170, and a payment sensor 180. As illustrated, POS kiosk 150 includes three image capture devices 162A-N, two displays 164A-N, and one pad 170. This is merely an example, and POS kiosk 150 may include any suitable number of image capture devices, displays, pads, and other components.

[0018]

[0027] In one embodiment, image capture devices 162A-N capture images of items for purchase (e.g., items placed on pad 170). POS kiosk 150 may use image recognition techniques (e.g., suitable ML models), available UPC codes, and any other suitable information to identify items for purchase. Additionally, in one embodiment, pad 170 may include or be associated with one or more sensors. These may include weight sensors, pressure sensors, or any other suitable sensors. For example, pad 170 may be used for weight and footprint detection, as described below in connection with FIGS. 3-11.

[0019]

[0028] 2 is a block diagram illustrating a controller 200 for POS item prediction, according to one embodiment. In one embodiment, the controller 200 is used for the management system 140 illustrated in FIG. 1A. The controller 200 includes a processor 202, a memory 210, and a network component 220. The processor 202 generally retrieves and executes programming instructions stored in the memory 210. The processor 202 may represent a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a graphics processing unit (GPU) with multiple execution paths, and the like.

[0020]

[0029] Network component 220 includes components for interfacing controller 200 with a suitable communications network (e.g., communications network 130 illustrated in FIG. 1A). For example, network component 220 may include wired, WiFi, or cellular network interface components and associated software. Although memory 210 is shown as a single entity, memory 210 may include one or more memory devices having blocks of memory associated with physical addresses, such as random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory.

[0021]

[0030] Memory 210 generally includes program code for performing various functions associated with use of controller 200. The program code is generally described as various functional "applications" or "modules" within memory 210, although alternative implementations may have different functionality and / or combinations of functionality. Within memory 210, a prediction validation service 212 facilitates validation of predictions for item predictions (e.g., using weight, item footprint, or both), as further described below with respect to FIGS. 3-9.

[0022]

[0031] While FIG. 2 illustrates prediction validation service 212 as located in memory 210, that representation is provided merely as an example for clarity. More generally, controller 200 may include one or more computing platforms, such as, for example, computer servers, which may be co-located, separate, or may form an interactively linked but distributed system, such as a cloud-based system (e.g., a public cloud, a private cloud, a hybrid cloud, or any other suitable cloud-based system). Consequently, processor 202 and memory 210 may correspond to distributed processor and memory resources within a computing environment. Furthermore, in one embodiment, prediction validation service 212 may be divided across any suitable number of computing systems or compute nodes (e.g., in a cloud computing system), including being fully or partially integrated within a POS device (e.g., POS kiosk 150 illustrated in FIG. 1B ).

[0023]

[0032] 3 is a flowchart 300 illustrating POS item prediction using weight detection, according to one embodiment. In block 302, a prediction verification service (e.g., prediction verification service 212 illustrated in FIG. 2 or any other suitable service) predicts items for purchase using images. For example, a POS kiosk (e.g., POS kiosk 150 illustrated in FIG. 1B) may include one or more image capture devices (e.g., cameras).

[0024]

[0033] The image capture device may capture images of the items for purchase, and the predictive verification service may use the images to predict the items for purchase. For example, the predictive verification service may use a suitable computer vision ML model (e.g., a deep neural network (DNN), a support vector machine (SVM), or any other suitable ML model) to infer the items for purchase from one or more images captured by the image capture device. In one embodiment, if the POS kiosk includes multiple image capture devices, multiple images (e.g., captured from different angles) may be used together to predict the items for purchase.

[0025]

[0034] In block 304, the predictive verification service measures weights for the items for purchase. For example, a POS kiosk may include one or more weight sensors. The predictive verification service may use the weight sensors to measure the weights of the items for purchase, as further illustrated below with respect to FIG. 4. In one embodiment, changes in the measured weights are captured over time. For example, an average weight (e.g., a weight mean or a weight median) may be captured over a period of time. As another example, a maximum weight or a minimum weight may be captured. These are merely examples, and any suitable weight measurement may be used.

[0026]

[0035] In one embodiment, weight is merely one example of a characteristic that may be used to validate an item prediction. For example, as described below in connection with FIGS. 8-9 , an item footprint may be determined (e.g., using pressure or weight sensors) and used instead of or in addition to weight measurements. As another example, a model (e.g., a three-dimensional model) of the item may be generated (e.g., using image recognition, infrared imaging, lidar, radar, sonar, or any other suitable technique). The item model may be used to validate the prediction (e.g., instead of or in addition to weight or footprint).

[0027]

[0036] At block 306, the prediction validation service validates the measured weight. In one embodiment, the prediction validation service predicts a weight for the identified item (e.g., using a suitable ML model) and determines whether the predicted weight is within a deviation threshold of the measured weight, as further described below with respect to FIG. 5.

[0028]

[0037] If the predictive verification service confirms the validity of the measured weight, flow proceeds to block 308. In block 308, the predictive verification service continues with the transaction. For example, the predictive verification service may allow the purchaser to complete the transaction (e.g., make payment and receive the items for purchase). As another example, the predictive verification service may allow the purchaser to remove items from the POS kiosk (e.g., if the purchaser placed some of their items at the POS kiosk), add additional items to the POS kiosk, and continue the transaction. In one embodiment, the POS kiosk may specify multiple items at a time, as shown in FIG. 4, but the purchaser may still have more items for purchase than can be specified at the POS kiosk at one time.

[0029]

[0038] Returning to block 306, if the predictive verification service was unable to confirm the validity of the measured weight (e.g., the deviation of the measured weight from the predicted weight exceeded a threshold), flow proceeds to block 310. In block 310, the predictive verification service intervenes in the transaction. In one embodiment, the predictive verification service triggers an intervention action. For example, the predictive verification service may notify the purchaser that there is an apparent error and may ask the purchaser to rearrange or replace items on the POS kiosk (e.g., if some items are obscured from view by the image capture device, as illustrated in FIG. 4). As another example, the predictive verification service may notify an employee or other personnel to assist with the transaction. As another example, the predictive verification service may attempt to correct the problem (e.g., by using image recognition or an alternative technique for item verification). These are merely examples, and the predictive verification service may take any suitable verification action.

[0030]

[0039] 4 illustrates the use of a kiosk for POS item prediction using weight detection, according to one embodiment. In one embodiment, POS kiosk 400 is used to purchase two items: snack bag 424 and candy bar 422. POS kiosk 400 includes an image capture device and a pad (e.g., as illustrated in FIG. 1B ). The items are placed on the POS kiosk pad such that candy bar 422 (shown in view 412B) is placed below snack bag 424 and is not visible in images captured by the image capture device at the POS kiosk. This is illustrated in view 412C.

[0031]

[0040] In one embodiment, a prediction verification service (e.g., prediction verification service 212 illustrated in FIG. 2 or any other suitable service) compares predicted weight 434 based on the captured image (e.g., based on expecting one item, namely, snack bag 424, to be present) with actual measured weight 432 (e.g., including both snack bag 424 and candy bar 422, as illustrated in view 412B) according to the techniques illustrated above in connection with FIG. 3. The prediction verification service performs weight verification 436, as described below in connection with FIG. 5, and determines that the deviation between the predicted weight and the actual weight exceeds a threshold. In that case, the prediction verification service takes intervening action (e.g., asking the purchaser to move the items so that they are flat on the pad and visible to the image capture device).

[0032]

[0041] Figure 5 is a flow chart illustrating weight validation for a POS item prediction, according to one embodiment. In one embodiment, Figure 5 corresponds to block 306 illustrated in Figure 3. In block 502, a prediction validation service (e.g., prediction validation service 212 illustrated in Figure 2 or any other suitable service) predicts a weight for an item for purchase.

[0033]

[0042] In one embodiment, as described above in connection with block 302 illustrated in FIG. 3 , the predictive verification service uses a suitable ML model to identify items for purchase (e.g., using images captured using one or more image capture devices at a POS kiosk). The predictive verification service may further use an ML model (e.g., a different ML model) to predict weights for the identified items. In one embodiment, the predicted weights for the items are inferred using an ML model (e.g., a trained ML model), and the predictive verification service may use the captured images, the identified items (e.g., identified using computer vision ML techniques as described above), or any other suitable data to predict weights for the items. This is further described below in connection with FIGS. 6-7 .

[0034]

[0043] In block 504, the prediction verification service calculates the weight deviation between the measured weight and the predicted weight. In one embodiment, both the predicted weight and the measured weight are likely to be somewhat inaccurate. Directly comparing the predicted weight to the measured weight is likely to result in a larger number of false positives, wasting computational resources (e.g., from inference using an ML model), and compromising the shopping experience. To reduce (or avoid) false positives, a weight deviation may be calculated. For example, the weight deviation may be calculated using the following formula: |(Measured Weight - Predicted Weight)| ÷ (Expected Weight) ÷ Predicted Number of Items. This is merely an example, and any suitable formula or other technique (e.g., suitable ML model or algorithmic technique) may be used to determine the weight deviation.

[0035]

[0044] At block 506, the prediction validation service identifies a deviation threshold. In one embodiment, the prediction validation service validates a purchase as long as the weight deviation (e.g., calculated above at block 506) is within the threshold. For example, a threshold of 2% may be used. Any weight deviation above 2% will not be validated, while anything within the 2% threshold will be validated. In one embodiment, the threshold may be provided by an administrator (e.g., using a suitable user interface), coded (e.g., hard-coded) into the prediction validation service, or dynamically determined (e.g., using a suitable ML model or algorithmic technique).

[0036]

[0045] At block 508, the prediction validation service determines whether the weight deviation is within the deviation threshold. If so, flow continues to block 510, where the prediction validation service validates the transaction. If not, flow continues to block 512, where the prediction validation service does not validate the transaction.

[0037]

[0046] 6 is a flowchart 600 illustrating training an item weight prediction ML model according to one embodiment. This is merely an example, and in one embodiment, a suitable unsupervised technique may be used (e.g., without requiring training). At block 602, a training service (e.g., a human administrator or a software or hardware service) collects item weight data. For example, a prediction validation service (e.g., prediction validation service 212 illustrated in FIG. 2) may be configured to act as the training service and collect item data reflecting predicted weights for various items.

[0038]

[0047] At block 604, a training service (or other suitable service) preprocesses the collected item weight data. For example, the training service may create feature vectors that reflect the values ​​of various features for items with associated weight data. At block 606, the training service receives the feature vectors and uses them to train a trained item weight prediction ML model 610.

[0039]

[0048] In one embodiment, at block 604, the training service also collects additional product data. For example, the training service may use UPC data or any other suitable data to further identify the products. At block 606, the training service may also preprocess this additional product data. For example, feature vectors corresponding to the product weight data may be further annotated using the additional product data. Alternatively or additionally, additional feature vectors corresponding to the additional product data may be created. At block 608, the training service generates a trained product weight prediction ML model 610 using the preprocessed additional product data during training.

[0040]

[0049] In one embodiment, preprocessing and training may be performed as batch training. In this embodiment, data is preprocessed simultaneously (e.g., item weight data and additional item data) and provided to the training service in block 608. Alternatively, preprocessing and training may be performed in a streaming manner. In this embodiment, data is streaming and continuously preprocessed and provided to the training service. For example, a streaming approach may be desirable for scalability. Training data sets may be very large, and therefore it may be desirable to preprocess data and provide it to the training service in a streaming manner (e.g., to avoid computational and storage limitations). Furthermore, in one embodiment, a federated learning approach may be used in which multiple entities contribute to training a shared model.

[0041]

[0050] 7 is a flowchart 700 illustrating predicting item weight using an ML model, according to one embodiment. In one embodiment, a processing service 720 (e.g., the prediction validation service 212 illustrated in FIG. 2 or any other suitable software service) is associated with an item weight prediction ML model 610. In one embodiment, the item weight prediction ML model 610 is trained to predict predicted weights 730 for one or more identified items 702. For example, the item weight prediction ML model 610 may predict a total weight for items for purchase, as described above in connection with FIGS. 3-5.

[0042]

[0051] In one embodiment, predicted weight 730 reflects a predicted weight for the identified item 702. Alternatively or additionally, predicted weight 730 identifies multiple proposed matches (e.g., a range of predicted weights). In one embodiment, item weight prediction ML model 610 provides a confidence score for the possible predicted weight 730, which may be factored into the weight deviation calculated in block 504 of FIG. 5 above.

[0043]

[0052] 8 is a flowchart 800 illustrating POS item prediction using weight and item footprint, according to one embodiment. At block 802, a prediction validation service (e.g., the prediction validation service 212 illustrated in FIG. 2 or any other suitable software service) captures images of the items for purchase. For example, a POS kiosk (e.g., the POS kiosk 150 illustrated in FIG. 1B) may include one or more image capture devices (e.g., cameras) for capturing images of the items for purchase.

[0044]

[0053] At block 804, the predictive validation service predicts an item using the captured image. As described above in connection with block 302 illustrated in FIG. 3 , the predictive validation service may use computer vision techniques (e.g., a suitable ML model) to predict the item depicted in the image. In one embodiment, the predictive validation service identifies items that are visible in the captured image. Alternatively, or additionally, the predictive validation service may not be able to distinguish between visually similar-looking items. For example, a purchaser may be purchasing soda cans that are available in different sizes. The predictive validation service may be able to identify the brand and type of soda, but may not be able to distinguish between the available sizes of the soda cans. The predictive validation service may predict multiple options for the item (e.g., the available sizes of soda cans for the item).

[0045]

[0054] In block 806, the prediction verification service determines the item footprint. In one embodiment, the POS kiosk includes pressure or weight sensors that can be used to identify pressure points of an item resting on a pad at the POS kiosk. This can be used to determine a footprint for the item when it is resting on the pad. In one embodiment, a suitable ML model can be used to infer an item footprint, an item prediction, or both, from the pressure or weight measurements of the POS kiosk. For example, an ML model can be trained using measured pressure or weight data for various items and item footprints. The trained ML model can then be used to infer an item footprint, a prediction, or both, from the measured pressure or weight data.

[0046]

[0055] In block 808, the prediction verification service predicts the item using the footprint. In one embodiment, the prediction verification service compares the item shape from a top-down view (e.g., as captured in an image by a POS kiosk) with the item shape from below (e.g., in an item footprint captured using a pressure or weight sensor). For example, the prediction verification service may use Intersection Over Union (IoU) to verify or modify the item prediction (e.g., determined in block 804 using the captured image) by comparing two perspective views of the item (e.g., a top perspective view determined from the captured image and a footprint captured from a pressure or weight sensor). In one embodiment, IoU is an object detection technique that may be used to verify the prediction. The use of IoU is merely one example, and any suitable technique may be used. Additionally, the prediction verification service may use the footprint to select between predicted item options from the image (e.g., soda can sizes, as described above in connection with block 804). That is, computer vision techniques may be used to identify a set of predicted items from a captured image, and the item footprints may be used to select from among the set of predicted items.

[0047]

[0056] At block 810, the prediction validation service validates the prediction using the weight. In one embodiment, the weight of the items for purchase may be used to further validate the prediction. For example, as described above in connection with FIGS. 3-7, the weight of the items for purchase may be used to validate an image-based prediction of items for purchase (e.g., when one or more items are not visible in the captured image). These techniques described above in connection with FIGS. 3-7 may be further used to validate the footprint-based predicted items determined in block 808. While FIG. 8 illustrates using weight and footprint for the prediction, this is merely an example. In one embodiment, the prediction validation service may use footprint data (e.g., as described in connection with block 808), weight data (e.g., as described above in connection with FIGS. 3-7), or any combination thereof.

[0048]

[0057] Additionally, in one embodiment, the predictive verification service may use changes in footprint, weight, or both over time. For example, a POS kiosk may record changes in item footprint and weight over time. The predictive verification service may record historical values ​​for item footprint, weight, or both, and use those historical values ​​and changes in item prediction and verification. For example, historical values ​​for weight or footprint may be used to identify the state of the POS kiosk (e.g., purchasing an item) and to predict appropriate actions or interventions based on this state.

[0049]

[0058] In block 812, the prediction verification service uses the item prediction in the POS system. For example, the prediction verification service may present the predicted item to the purchaser (e.g., using a display on a POS kiosk). The purchaser may then decide whether the prediction is correct and whether to proceed with the transaction. As another example, the prediction verification service may use the prediction to verify a previous selection by the purchaser or another prediction (e.g., an image-based prediction only) and to determine whether to continue or intervene in the transaction. This is described above in connection with blocks 310 and 312 illustrated in FIG. 3.

[0050]

[0059] FIG. 9 illustrates the use of a kiosk for POS item prediction using weight and item footprint, according to one embodiment. Similar to FIG. 4 above, a POS kiosk 900 is used to purchase two items: a snack bag 924 and a candy bar 922. The POS kiosk 900 includes an image capture device (e.g., as illustrated in FIG. 1B ) and a pad. The items are placed on the POS kiosk pad such that the candy bar 922 (shown in view 912B) is placed below the snack bag 924 and is not visible in the image captured by the image capture device at the POS kiosk. This is illustrated in view 912C.

[0051]

[0060] In one embodiment, a prediction verification service (e.g., the prediction verification service 212 illustrated in FIG. 2 or any other suitable service) uses the footprint data, the weight data, or both to predict and verify items for purchase according to the techniques illustrated above in connection with FIG. 8 . For example, the POS kiosk 900 may measure the footprint and weight of the items for purchase (e.g., using pressure or weight sensors). The prediction verification service may determine footprint verification 938 by comparing a top-down shape of the item (e.g., based on a captured image) with a bottom-up footprint (e.g., captured using pressure or weight sensors) to predict the item (e.g., using IoU). Further, the prediction verification service may determine weight verification 936 using the measured weight of the items for purchase. The prediction verification service may use footprint verification 938 and weight verification 936 to predict items for purchase or to verify a previous prediction. For example, the prediction verification service may present the predicted items on a display of the POS kiosk 900 or may take suitable intervention action.

[0052]

[0061] The descriptions of various embodiments are presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements to technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0053]

[0062] In the foregoing description, reference has been made to embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the described embodiments. Instead, any combination of the following features and elements, whether associated with different embodiments, is contemplated to implement and practice the contemplated embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether or not an advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Accordingly, the following aspects, features, embodiments, and advantages are merely exemplary and should not be considered elements or limitations of the appended claims unless expressly recited in the claim(s). Similarly, references to "the present disclosure" should not be construed as a generalization of any inventive subject matter disclosed herein, nor should they be considered elements or limitations of the appended claims unless expressly recited in the claim(s).

[0054]

[0063] Aspects of the disclosed embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects which may be generally referred to herein as a "circuit," "module," or "system."

[0055]

[0064] The disclosed embodiments may be systems, methods, and / or computer program products that may include computer-readable storage medium(s) having computer-readable program instructions thereon for causing a processor to perform aspects of the disclosed embodiments.

[0056]

[0065] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transitory signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted through wires.

[0057]

[0066] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device, for example, via the Internet, a local area network, a wide area network, and / or a wireless network. The network may comprise copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0058]

[0067] The computer-readable program instructions for carrying out the operations of the disclosed embodiments may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, and traditional procedural programming languages ​​such as the "C" programming language or similar. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the disclosed embodiments.

[0059]

[0068] Aspects of the disclosed embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0060]

[0069] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions executing on the processor of the computer or other programmable data processing apparatus generate means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, whereby the computer-readable storage medium having instructions stored therein comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0061]

[0070] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0062]

[0071] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, depending on the functionality involved, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0063]

[0072] While the foregoing is directed to exemplary embodiments, other and further embodiments may be devised without departing from the basic scope thereof, which scope is determined by the following claims.

Claims

1. Identifying an image of an item for purchase captured at a point of sale (POS) system; Identifying characteristics of the item for purchase, the characteristics including at least one of: (i) a first weight measured using a weight sensor associated with the POS system; or (ii) a footprint measured using one or more pressure or weight sensors associated with the POS system; predicting the items for purchase based on the images; and validating the predicted item for purchase using the characteristics; and A method for providing the above.

2. the characteristics include the first weight; The method further comprises predicting a second weight of the predicted item for purchase. The method of claim 1.

3. validating the predicted item for purchase using the characteristics includes: calculating a weight deviation between a first weight for the item for purchase and the predicted second weight; The method of claim 2.

4. validating the predicted item for purchase using the characteristics includes: determining that the weight deviation is within a deviation threshold; The method of claim 3.

5. The weight deviation is calculated using the following formula: | (measured weight - predicted weight) | ÷ (predicted weight) ÷ predicted number of items The method of claim 4, wherein the calculated value is calculated according to:

6. the characteristics include the footprint measured using one or more pressure or weight sensors associated with the POS system; The method includes identifying a top-down shape of the item for purchase based on the identified one or more images; comparing the top-down shape to the footprint for the item; The method of claim 1 further comprising:

7. 7. The method of claim 6, wherein comparing the top-down shape to the footprint for the item comprises determining an Intersection Over Union (IoU) for the top-down shape compared to the footprint for the item.

8. the characteristics include both the first weight measured using a weight sensor associated with the POS system and the footprint measured using one or more pressure or weight sensors associated with the POS system; validating the predicted item for purchase using the characteristic includes validating based on both the first weight and the footprint. The method of claim 1.

9. Verifying the prediction of the item for purchase includes: determining that a transaction to purchase the item should not be validated; intervening in said transaction; The method of claim 1 , comprising:

10. Intervening in said transaction is modifying a user interface associated with the POS system to indicate that an item should be moved at the POS system; 10. The method of claim 9.

11. A non-transitory computer program product, comprising: one or more non-transitory computer-readable media containing computer program code in any combination, the computer program code, when executed by operation of one or more processors in any combination, Identifying an image of an item for purchase captured at a point of sale (POS) system; Identifying characteristics of the item for purchase, the characteristics including at least one of: (i) a first weight measured using a weight sensor associated with the POS system; or (ii) a footprint measured using one or more pressure or weight sensors associated with the POS system; predicting the items for purchase based on the images; and validating the predicted item for purchase using the characteristics; and 1. A non-transitory computer program product that performs operations comprising:

12. the characteristics include the first weight; The operation predicts a second weight of the predicted item for purchase.

12. The non-transitory computer program product of claim 11.

13. validating the predicted item for purchase using the characteristics includes: calculating a weight deviation between a first weight for the item for purchase and the predicted second weight; determining that the weight deviation is within a deviation threshold; 13. The non-transitory computer program product of claim 12, comprising:

14. the characteristics include the footprint measured using one or more pressure or weight sensors associated with the POS system; The operations include: determining a top-down shape of the item for purchase based on the determined one or more images; comparing the top-down shape to the footprint for the item; The non-transitory computer program product of claim 11 , further comprising:

15. the characteristics include both the first weight measured using a weight sensor associated with the POS system and the footprint measured using one or more pressure or weight sensors associated with the POS system; validating the predicted item for purchase using the characteristic includes validating based on both the first weight and the footprint.

12. The non-transitory computer program product of claim 11.

16. 1. A system comprising: one or more processors; one or more memories having stored thereon a program, the program, when executed on any combination of the one or more processors, performing operations, the operations including: Identifying an image of an item for purchase captured at a point of sale (POS) system; Identifying characteristics of the item for purchase, the characteristics including at least one of: (i) a first weight measured using a weight sensor associated with the POS system; or (ii) a footprint measured using one or more pressure or weight sensors associated with the POS system; predicting the items for purchase based on the images; and validating the predicted item for purchase using the characteristics; and A system comprising:

17. the characteristics include the first weight; The operation predicts a second weight of the predicted item for purchase.

17. The system of claim 16.

18. validating the predicted item for purchase using the characteristics includes: calculating a weight deviation between a first weight for the item for purchase and the predicted second weight; determining that the weight deviation is within a deviation threshold; 20. The system of claim 17, comprising:

19. the characteristics include the footprint measured using one or more pressure or weight sensors associated with the POS system; The operations include: determining a top-down shape of the item for purchase based on the determined one or more images; comparing the top-down shape to the footprint for the item; The system of claim 16 further comprising:

20. the characteristics include both the first weight measured using a weight sensor associated with the POS system and the footprint measured using one or more pressure or weight sensors associated with the POS system; validating the predicted item for purchase using the characteristic includes validating based on both the first weight and the footprint.

17. The system of claim 16.

Images