Product identification system and method

Abstract

To provide a method for identifying a product taken out from a cabinet.SOLUTION: A method for identifying a product taken out from a cabinet includes detecting a visual feature of the product taken out from the cabinet by using a camera in the cabinet. The method may include detecting an identifier on the product taken out from the cabinet by using an identifier sensor, and comparing the visual feature and the identifier with a product information database. The product taken out from the cabinet may be identified on the basis of the comparison between the visual feature and the identifier, and the product information database.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The embodiments described herein generally relate to systems and methods for identifying products. In particular, the embodiments described herein relate to systems and methods for identifying products retrieved by a consumer from an unattended vending machine. [Background technology]

[0002] Vending machines generally require consumers to enter a payment, make a product selection, and wait for the product to be dispensed by the vending machine. However, consumers may encounter several problems when using vending machines. First, the vending machine may not accept the consumer's form of payment. For example, the vending machine may not accept folded or wrinkled bills. The vending machine may not properly register the receipt of bills or coins, and thus the consumer may not receive credit for the payment made. The vending machine may not be configured to accept mobile payments, which can be inconvenient for consumers. Furthermore, the vending machine may not be able to read payment cards, such as credit or debit cards. As a result, the consumer may be unable to make a purchase, or the consumer may become frustrated and decide not to use the vending machine.

[0003] Second, once payment is made, a consumer may incorrectly enter the code corresponding to the desired product. As a result, a different product may be dispensed than the consumer expected. The consumer may not be able to return the incorrect product, and the consumer may not be able to request a larger or smaller product. Furthermore, vending machines generally allow consumers to purchase only one product at a time; to purchase multiple products, the consumer must repeat the process of making payment and selecting the product. Repeating the same steps is time-consuming and frustrating, which may discourage consumers from making multiple purchases.

[0004] Third, the vending machine may not be able to properly deliver the selected product to the user. For example, the screw drive may not be able to move the product into the dispensing opening of the vending machine, or the gate holding the product in place may not fully open. Additionally, the product may be jammed or stuck within the vending machine and out of the consumer's reach. As a result, the consumer does not receive the product and is unable to obtain a declined payment.

[0005] Vending machines have various additional drawbacks, such as not allowing consumers to individually select a specific product. Instead, consumers simply select a type of product but cannot choose the exact product that is dispensed. Furthermore, consumers cannot handle or inspect the product before purchase. As a result, consumers may not be able to learn about the product, such as by reading the label, ingredients, or nutritional information. This may deter consumers from purchasing products with which they are unfamiliar. Dispensed products may be damaged, expired, or otherwise defective. These various factors can contribute to a poor consumer experience.

[0006] Therefore, what is desired is an improved vending machine that provides a simple and easy purchasing experience. Further, what is desired is a vending machine that allows a consumer to individually select one or more products and ensures the dispensing of the desired product. Summary of the Invention

[0007] Some embodiments described herein relate to a method for identifying a product removed from a cabinet, the method including detecting a visual feature of the product removed from the cabinet by a camera within the cabinet; detecting an identifier of the product removed from the cabinet by an identifier sensor; comparing the visual feature and the identifier with a product information database; and identifying the product removed from the cabinet based on the comparison of the visual feature and the identifier with the product information database.

[0008] Some embodiments described herein are methods for identifying products removed from a cabinet, the method including capturing a first image of a plurality of products in the cabinet with an internal camera in the cabinet; removing a product from the plurality of products from the cabinet; capturing a second image of the plurality of products in the cabinet with the internal camera after removing the product; determining an identity of the product removed from the cabinet by analyzing the first image and the second image; and confirming the identity of the product removed from the cabinet by detecting an identifier of the product removed from the cabinet.

[0009] Some embodiments described herein relate to a method for identifying a product removed from a cabinet, the method including: detecting a location where the product was removed from the cabinet via at least one of a camera and a sensor; determining visual characteristics of the product based on data from the at least one of the camera and the sensor; determining a predicted identity of the product based on the visual characteristics; determining the identity of the product at the location based on a digital map of the product in the cabinet; and verifying that the predicted identity of the product corresponds to the identity of the product based on the digital map.

[0010] In any of the various embodiments discussed herein, the visual characteristics may include the shape of the product.

[0011] In any of the various embodiments discussed herein, the visual characteristics may include the color of the product.

[0012] In any of the various embodiments discussed herein, the identifier sensor may be a camera.

[0013] In any of the various embodiments discussed herein, the identifier may be a barcode.

[0014] In any of the various embodiments discussed herein, the method may further include verifying the identity of the product by determining a weight of the product removed from the cabinet with a weight sensor disposed within the cabinet. In some embodiments, determining the weight of the product removed from the cabinet may include determining a first weight of the product in the cabinet via the weight sensor, determining a second weight of the product in the cabinet via the weight sensor after removing the product from the cabinet, and calculating a difference between the first weight and the second weight.

[0015] In any of the various embodiments discussed herein, the method may further include detecting visual characteristics of the product removed from the cabinet at a location outside the cabinet with an external camera, and verifying the identity of the removed product based on the visual characteristics detected by the external camera.

[0016] In any of the various embodiments discussed herein, the method may further include detecting a user with an external camera, detecting tampering by the user with the external camera, and locking the cabinet when tampering is detected.

[0017] In any of the various embodiments discussed herein, the method may further include detecting data related to the removed product via an optical sensor and verifying the identity of the removed product using the data from the optical sensor.

[0018] In any of the various embodiments discussed herein, the optical sensor may include a LIDAR sensor.

[0019] In any of the various embodiments discussed herein, the method may further include updating the digital map of the products in the cabinet after the products are removed from the cabinet.

[0020] In any of the various embodiments discussed herein, the method may further include generating a digital map of the products in the cabinet using data received from at least one of the camera and the sensor.

[0021] In any of the various embodiments discussed herein, the digital map may include the location and model of each product within the cabinet.

[0022] In any of the various embodiments discussed herein, the method may further include detecting an identifier of the product removed from the cabinet via the identifier sensor. [Brief explanation of the drawings]

[0023] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate the present disclosure and, together with the description, serve to further explain the principles of the present disclosure and to enable those skilled in the art to make and use the present disclosure. [Figure 1] FIG. 1 is a front view of a vending machine configured to identify products removed from a cabinet, according to one embodiment. [Figure 2] 1 is a schematic diagram of components of a system for identifying products removed from a vending machine cabinet, according to one embodiment. [Figure 3] FIG. 1 is a top view of a shelf in a vending machine showing camera placement, according to one embodiment. [Figure 4] FIG. 1 illustrates a top view of a shelf in a vending machine showing camera locations, according to one embodiment. [Figure 5] FIG. 1 is a perspective view of a vending machine with a camera, according to one embodiment. [Figure 6] 1 illustrates an exemplary method for determining the identity of a product, according to one embodiment. [Figure 7] 1 illustrates an exemplary method for determining the identity of a product according to one embodiment. [Figure 8]FIG. 1 illustrates a side view of a vending machine having a weight sensor, according to one embodiment. [Figure 9] 1 illustrates an exemplary method for determining the identity of a product, according to one embodiment. [Figure 10] 1 illustrates an exemplary method for determining whether a product has been returned to a cabinet, according to one embodiment. [Figure 11] 1 is a diagram of products in a shelf as determined by an optical sensor, according to one embodiment. [Figure 12] FIG. 1 is a perspective view of a digital map of vending machine cabinets and products, according to one embodiment. [Figure 13] FIG. 1 is a top view of a system for identifying products by tracking the position of a consumer's hand, according to one embodiment. [Figure 14] 1 illustrates an exemplary method for determining the identity of a product, according to one embodiment. [Figure 15] FIG. 1 is a schematic block diagram of an exemplary computer system on which embodiments may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0024] Reference will now be made in detail to representative embodiments, as illustrated in the accompanying drawings. It should be understood that the following description is not intended to limit the embodiments to a single preferred embodiment. On the contrary, the invention is intended to cover alternatives, modifications, and equivalents, which may be included within the spirit and scope of the embodiments as defined by the appended claims.

[0025] Some vending machines may provide consumers with access to the compartments where products are stored. In this manner, consumers may inspect the products to consider labels, nutritional information, etc. when deciding whether to purchase the product. Furthermore, consumers may select exactly the products they wish to purchase. Consumers may easily purchase multiple products in a single transaction.

[0026] Such vending machines can accept a payment source from a consumer or identify the consumer, provide the consumer with access to a cabinet where products are stored, detect products removed from the cabinet by the consumer, and charge the consumer for the selected products. While such vending machines can provide additional convenience to consumers, accurately detecting the products selected by the consumer presents many technical challenges. If a product removed from the cabinet is not identified and charged to the consumer, the owner of the vending machine may lose revenue. Furthermore, if the removed product is not correctly identified, the consumer may be charged an incorrect price and the vending machine's inventory may be incorrectly maintained. Care must also be taken to ensure that consumers do not tamper with the products or otherwise engage in fraudulent activity.

[0027] To ensure accurate identification of products dispensed from a vending machine, the vending machine must be able to distinguish between various products, many of which may be similar in appearance. For example, many beverage bottles, particularly those from the same manufacturer, may be identical or similar in size and shape. Thus, some products may differ only in packaging details, such as product name or color.

[0028] Vending machines must also be able to detect products in various orientations. Products may be placed in the cabinet in various orientations, and consumers may remove products from the vending machine in different ways. Consumers may select products in a manner that obscures their visibility and hampers identification. For example, consumers may grasp multiple products in one hand, thereby making it difficult to detect the individual selected products. Consumers may also remove products and return them in a different position and orientation than when they were originally placed.

[0029] To ensure proper use of vending machines, care must be taken to prevent fraudulent activities, such as theft, tampering with products, or damaging the vending machine. A consumer may attempt to remove a product without being detected so that the consumer is not charged for the product. Alternatively, a consumer may attempt to deceive the vending machine by inserting an external object into the vending machine in place of the product to make it appear that the product has been returned. If a consumer is able to steal or tamper with the product, the owner of the vending machine may suffer loss of revenue. If the product available for purchase is damaged, other consumers may choose not to use the vending machine.

[0030] Some embodiments described herein relate to systems and methods for identifying products removed from a cabinet using a camera and an identifier sensor, where data collected by the camera and identifier sensor is compared to a product information database. In this way, products can be accurately identified without the consumer having to manually scan or enter information about the product, simplifying the purchase of the product from a vending machine. Some embodiments described herein relate to systems and methods for identifying products removed from a cabinet, including generating a digital map of the products within the cabinet. The digital map provides a baseline of the product's location and identity within the cabinet and can be used to confirm the identity of products removed from the cabinet by a camera or sensor within the cabinet.

[0031] In some embodiments, vending machine 100 may include cabinet 110 with multiple products 200 stored therein, as shown in FIG. 1 . Cabinet 110 may further include door 118 that can be opened to provide access to the multiple products 200 within cabinet 110. Cabinet 110 may include one or more cameras 120, 130 or sensors 140, 150, 160 for identifying products. Specifically, the cameras and sensors are configured to identify products removed from cabinet 110, such that consumers are charged only for products removed from cabinet 110 and not returned to cabinet 110. Data collected by the sensors and cameras is analyzed, such as by control unit 180 (see FIG. 2 ), to determine the identity of the removed products. The analysis may include comparing data from the cameras and sensors to a database of product information and / or product inventory.

[0032] The detection systems and methods described herein can be used in vending machines that allow a user to manually select and retrieve products from cabinets where the products are stored. Vending machines that allow consumers to manually select and retrieve products are described, for example, in U.S. Patent Application No. 16 / 559,300, filed September 3, 2019, which is incorporated herein by reference in its entirety. Exemplary vending machines incorporating product identification systems and methods are described herein for illustrative purposes only. Those skilled in the art will understand that the product identification systems and methods described herein can also be used with other types of vending machines or retailers and utilized in other environments for product identification.

[0033] Vending machine 100 may have, for example, the components shown in Figure 2. However, vending machine 100 need not have all the components shown in Figure 2 and may include additional components.

[0034] The vending machine 100 may be configured to authenticate the identity of a consumer. The vending machine 100 may include an external camera 130 for identifying the consumer through facial recognition, or a biometric sensor 172 for obtaining biometric information from the consumer, such as a thumbprint or iris. In some embodiments, the vending machine 100 may alternatively or additionally include a communication device 174, such as a wireless transceiver for communicating with a mobile device, such as a mobile phone, so that the consumer can authenticate or make a payment via the mobile device. In such embodiments, the mobile device may have a software application that facilitates interaction with the vending machine 100. The consumer's identity may be linked to a consumer profile that contains information about the consumer, such as a payment source, so that the consumer does not need to manually make a payment when using the vending machine 100, and the consumer's purchases are automatically postpaid to the consumer's profile.

[0035] Vending machine 100 may not require consumer authentication and may simply accept a form of payment from the consumer. Vending machine 100 may include a payment processing unit 170, which may include one or more slots for receiving bills, coins, or tokens. Payment processing unit 170 may include a card reader including a near-field communication (NFC) antenna for reading magnetic stripes or electronic chips on credit cards, debit cards, gift cards, etc., or for receiving contactless payments from contactless payment cards. Payment processing unit 170 may include a communication device for accepting mobile payments, such as a cell phone, watch, laptop, tablet, etc., or payment processing unit 170 may include a scanner for scanning payment codes, such as Quick Response (QR) codes.

[0036] Upon authenticating the consumer's identity or receiving payment from the consumer, the door 118 of the vending machine 100 may automatically unlock to allow the consumer access to the plurality of products 200. The products removed by the consumer may be identified by methods described herein. A virtual shopping cart displayed on the user interface 176 or on the user's mobile device may list the products removed from the cabinet 110 along with the product prices and the total price of the products.

[0037] When the consumer closes the door 118 of the vending machine 100, they may complete the purchase of the dispensed product. To complete the purchase, the consumer may provide input, for example, make a selection to complete the transaction on the user interface 176 of the vending machine 100, or perform a gesture on the user interface 176 that has a touch screen, such as swiping along a path. Alternatively, the purchase may be automatically completed once the door 118 has been closed for a predetermined period of time.

[0038] In some embodiments, the vending machine 100 may include one or more internal cameras 120 within the cabinet 110 for identifying products, as shown in FIG. 3 . The cameras 120 may be configured to capture static images, the cameras 120 may capture video, or both. In some embodiments, multiple cameras 120 may be positioned to capture images or video of products on each shelf 112 of the cabinet 110. The cameras 120 may be positioned at one or more corners of the cabinet 110 above each shelf 112. Further, in some embodiments, the cameras 120 may be positioned centrally above each shelf 112 to capture images of the center of the shelf 112. For example, a camera 120 may be positioned at each of the four corners of the cabinet 110, and a fifth camera may be positioned above the center of the shelf 112. In this manner, the cameras 120 may capture products from different angles and detect products that are obscured in the field of view of a particular camera 120. Additionally, the cameras 120 may view any product within the cabinet 110 and may have overlapping fields of view. For example, a camera 120 located at the front 116 of the cabinet 110 may not fully capture a product located at the back 114 of the cabinet 110. The best image from the camera 120 may be selected for analysis to determine the identity of the product, or a composite image incorporating the various images may be generated and analyzed.

[0039] In some embodiments, one or more cameras 120 may be configured to capture images or video of products exiting (or entering) the cabinet 110, as shown in FIG. 4 . The cameras 120 may be located at the front 116 of the cabinet 110. In some embodiments, the cameras 120 may be positioned at corners of the front 116 of the cabinet 110 or may be positioned around the periphery of the front 116 of the cabinet 110. The cameras 120 may define a plane P parallel to the front 116 of the cabinet 110. In this manner, product 200B exiting or entering the cabinet 110 must pass through plane P and thus be detected by the cameras 120. In such embodiments, product 200A within the cabinet 110 may not be detected by the cameras 120. The cameras 120 may capture images or video to identify the product 200B removed from the cabinet 110 or to return it to the cabinet 110. The camera 120 on the front 116 of the cabinet 110 may have a clear view of the product 200 as it is being removed by the consumer so that the product 200 is not obscured by other products in the cabinet.

[0040] The camera 120 may be used to detect visual characteristics of the product. The visual characteristics may include product shape, product dimensions, product color, or a combination thereof. The camera may also be used to determine the location of the product within the cabinet 110.

[0041] The product shape may be a silhouette or 2D view of the product, such as a front profile, side profile, back profile, top view, or bottom view. For example, if the product is a can, the shape may be circular when viewed from the top or generally rectangular when viewed in side profile. In some embodiments, the shape may be a 3D view, such as a perspective view of the product. The 3D view may be generated by combining 2D views from various cameras. In some embodiments, the cameras may be used to generate a model of each product. The model may be a 2D model including the shape and color or color palette. In some embodiments, the model may be a 3D model including the product's shape, dimensions, and color or color palette. The camera 120 may have a depth sensor to aid in the generation of the 3D model. The camera may determine the product's dimensions so that products with similar shapes can be distinguished. For example, a 12-ounce can and a 16-ounce can may be distinguishable despite both being cylindrical. In some embodiments, to ensure accuracy, the camera may be configured to determine the product's dimensions within ±5 mm, ±3 mm, or ±1 mm.

[0042] Visual characteristics may include the color of the product. The color may be the color of any part of the product, or a pattern or combination of colors, such as a color palette. For example, the visual characteristic may be the color of the packaging, the color of the text, the color of a logo or marking on the packaging, among other coloring items. For example, if the product is a bottled beverage, the color may be the color of the bottle (e.g., clear, green), the color of the liquid in the bottle, the color of the bottle cap, the color of the label, or the lettering or marking on the label, as well as combinations thereof.

[0043] In some embodiments, identifier sensor 150 (see, e.g., FIG. 1 ) may detect identifier 210 of product 200. Product identifier 210 may include a label, barcode, QR code, text (such as a brand, product, or flavor name), logo, or other markings on the product. In some embodiments, identifier sensor 150 may be camera 120. In some embodiments, identifier sensor 150 may be a separate component, such as a scanner for scanning barcodes or QR codes. Identifier sensor 150 may have a sufficiently high resolution so that it can read text on product 200. In some embodiments, control unit 180 of vending machine 100 may perform optical character recognition (OCR) to identify text in a captured image of the product. The captured image or video may have a pixel density sufficient to allow accurate identification of text. In some embodiments, the minimum pixel density for identifying text may be approximately 2 pixels / mm. Additionally, identifier sensor 150 may have a high frame rate to provide a clear image that facilitates OCR.

[0044] In some embodiments, a convolutional neural network (CNN) may be used to detect the identifier 210 on the product 200 and analyze the identifier 210 for product recognition, as will be understood by those skilled in the art. The CNN may be trained based on the products available in the vending machine 100 to increase accuracy. Furthermore, the identifier sensor 150 may have sufficient resolution to resolve differences in the identifiers 210 of related products (e.g., Pepsi, Diet Pepsi, Cherry Pepsi). In some embodiments, for example, accurate product identification may require a minimum pixel density of 1.5 pixels / mm.

[0045] Identifier sensor 150 may assist in determining a particular stock keeping unit (SKU). For example, one or more cameras 120 may detect the size and shape of a product, but multiple products in cabinet 110 may be the same size and shape. Thus, identifier sensor 150 may help determine a particular type of product by detecting the product's identifier 210. Alternatively, if only camera 120 can determine the identity of a removed product, the information provided by identifier sensor 150 may be used to increase confidence that the product has been correctly identified or to confirm that the product's identification based on camera 120 is correct.

[0046] Control unit 180 may be configured to receive and analyze data from camera 120 and identifier sensor 150 to determine the identity of a product. Control unit 180 may also store a product information database. The database may include information about products stored in cabinet 110. The database may include, for example, a list of products. For each product, the database may include corresponding visual characteristics such as the product's shape or silhouette, dimensions, and packaging color, product weight, and additional information about the product's label and identifier. To identify a product removed from the cabinet, the analysis may determine a product in the database that has visual characteristics that correspond to or best match the visual characteristics determined based on the data from camera 120 and sensor 150. In some embodiments, control unit 180 may execute a sensor fusion algorithm to determine the identity of a product based on the data from camera 120 and sensor 150. Artificial intelligence and machine learning may be used to analyze the data from camera 120 and sensor 150 in combination with the product information database to determine the identity of a product. In some embodiments, artificial intelligence may assign a confidence level to a product identification. As will be appreciated by those skilled in the art, computer vision techniques may be used to analyze data, such as images or video, from cameras and sensors. In some embodiments, artificial intelligence or computer vision techniques may be used remotely from vending machine 100. For example, cloud computing, edge computing, or a combination thereof may be used to analyze data from cameras 120 and sensors.

[0047] In some embodiments, the control unit 180 may also maintain a product inventory for the vending machine 100 such that it is known which products are within the cabinet 110. Thus, identification of a retrieved product is limited to products known to be within the cabinet 110 from which the product 200 was retrieved, or on the particular shelf 112. In some embodiments, the control unit 180 may generate and store a digital map of the products within the cabinet 110, which can further assist with product identification, as discussed in more detail below with respect to FIG.

[0048] In some embodiments, as shown in FIG. 5, the vending machine 100 may include an interior camera 120. The vending machine 100 includes a cabinet 110 having shelves 112 on which products may be stored. A first plurality of cameras 120B may be positioned above each shelf 112 inside the cabinet 110 to detect products on each shelf 112. The first plurality of cameras 120B may include cameras 120B at the corners of each shelf 112 and in the center of each shelf 112. A second plurality of cameras 120A may be positioned at the front 116 of the cabinet 110. The second plurality of cameras 120A may be configured to detect products removed from or returned to the cabinet 110, as described above with respect to FIG. 4. The second plurality of cameras 120A may be positioned around the periphery of the front 116. However, in some embodiments, fewer or additional cameras 120 may be included.

[0049] In some embodiments, a method 600 for determining the identity of a product removed from a cabinet may include using a camera to capture video of a product entering or exiting a cabinet, as shown in FIG. 6 , for example. The camera may be activated when the cabinet door is open and deactivated when the door is closed, so that video is captured only when the cabinet is being accessed by a consumer. The camera may detect visual characteristics of the product removed from the cabinet 610. An identifier of the product removed from the cabinet may be detected by an identifier sensor 620. The visual characteristics and identifier may be analyzed to determine the identity of the product 630, which may be based on a product information database. In some embodiments, the visual characteristics and identifier may be analyzed for correspondence with a product inventory or a digital map of products in the cabinet. The identity of the product may be determined based on an analysis of the visual characteristics and identifier and the product information database 640. In some embodiments, the visual characteristics or identifier may be used to identify the product, and the other of the visual characteristics or identifier may be used to confirm the identity of the product.

[0050] In some embodiments, a method 700 for determining product identity may include the use of a camera, as shown in FIG. 7 . The camera may capture images at set intervals, or the camera may capture images when the cabinet door is closed. If multiple cameras are used, the images from the multiple cameras may be combined into a composite image, or the best image may be used. The method may include capturing a first image 710 of multiple products in a cabinet. The first image may be a baseline image showing the products before a consumer accesses the cabinet. The consumer may remove one or more products from the cabinet 720. The camera may then capture a second image 730 of the multiple products in the cabinet. The identity of the products may be determined, in part, by analyzing the first and second images to determine which products have been removed 740. Analysis of the images may be used to determine visual characteristics of the removed products and / or the location from which the products were removed. The analysis may include comparing the visual characteristics to a product information database, and the product's location may be used to determine products known to be stored at that location. Additionally, an identifier for the dispensed product may be detected by an identifier sensor 750 to verify the identity of the dispensed product. By detecting the product's identifier, the accuracy of product identification may be increased relative to using data from a camera alone.

[0051] In some embodiments, a combination of images and video captured by a camera can be used to identify a product. In such embodiments, a camera can capture an image of a product in a cabinet. Another camera can capture video of the product being removed from the cabinet. After the product is removed, a second image of the product in the cabinet can be captured. The video can be analyzed to determine visual characteristics of the product, and artificial intelligence can use the visual characteristics and a product information database to determine the identity of the product. The identity of the product determined based on the captured video can be confirmed by analysis of the first and second images to determine the location or visual characteristics of the removed product. Alternatively, product identification can be performed by analyzing the first and second images, and data from the video can be used to confirm the identification.

[0052] In some embodiments, the identity of the product may be determined, in part, using weight sensor 140 (see, for example, FIG. 1). Weight sensor 140 may be configured to determine the weight of the product in cabinet 110. In some embodiments, weight sensor 140 may be one or more load cells. Weight sensor 140 may be positioned on or inside shelf 112, such that weight sensor 140 may determine the weight of the product placed on shelf 112.

[0053] In other embodiments, the weight sensors 140 may be positioned as shown in Figure 8. Each shelf 112 in the cabinet 110 of the vending machine 100 may be supported on one or more weight sensors 140. In some embodiments, a weight sensor 140 is located at each corner of the rectangular shelf 112. Additionally, the weight sensors 140 may be located on brackets 111 within the cabinet 110. Thus, the weight sensors 140 may detect the weight of the shelf 112 and the products on the shelf to determine whether the products have been removed from or returned to the shelf 112.

[0054] An exemplary method 900 for determining the identity of a product using a weight sensor is shown in FIG. 9. A first weight of a product on a shelf may be determined by a weight sensor 910. The product may be removed from the shelf by a consumer 920. A second weight of the product on the shelf may be determined by a weight sensor 930. A difference between the first weight and the second weight may be calculated to determine the weight of the product or the removed product. The calculated weight may be analyzed to determine one or more products in a product information database that correspond to or best match the calculated weight of the removed product. The weight of the removed product may be used to confirm the identity of the removed product determined based on the camera or identifier sensor.

[0055] In some embodiments, weight sensors 140 can also be used to provide information regarding the location of products within cabinet 110. A shelf 112 within cabinet 110 may include multiple weight sensors 140. Thus, weight sensors 140 can help indicate the location where a product is removed, depending on which weight sensor 140 the product is placed on. The more weight sensors 140 included in cabinet 110, the greater the ability of weight sensors 140 to determine the product's exact location. Additionally, when a consumer returns a product, weight sensors 140 can help determine the location where the returned product will be placed. This information can be used to update a digital map of products within cabinet 110. Weight sensors 140 can also detect products that have been dropped or are obscured by other products and therefore cannot be easily seen by camera 120.

[0056] In some embodiments, as shown in FIG. 10 , a weight sensor may be used to determine consumer fraud. In such embodiments, the weight of a removed product may be determined by a weight sensor 1010, as described above with respect to FIGS. 8 and 9. The weight sensor may also determine the weight of a product returned to the cabinet 1020. For example, a consumer may remove a product to read the label or nutritional information, decide to purchase the product, and return the product to the cabinet. The weight of the removed product is compared 1030 with the return weight of the returned product. If the weight of the returned product is the same as the weight of the removed product, the return is accepted 1040. If the weight of the returned product is different from the weight of the returned product, the return is not accepted 1050. Thus, the consumer may be charged for the removed but not returned product. If a consumer attempts to deceive the system by returning an external item instead of the product removed from the cabinet, the system may detect that the weight of the external product is different from the weight of the removed product. Furthermore, if the consumer samples the product, such as by consuming a portion of the product, the weight of the product will be reduced and the return will not be accepted.

[0057] In some embodiments, determining whether a product is properly returned may be aided by camera 120. Camera 120 may detect visual characteristics of the removed and returned product to determine whether the visual characteristics are the same. If the visual characteristics of the returned product are different from the visual characteristics of the removed product, the consumer may have tampered with the product or attempted to return a different item.

[0058] In some embodiments, a light sensor 160 may be used to determine the visual characteristics and / or location of a product (see, e.g., FIG. 1). The light sensor 160 may be located within the cabinet 110 and be able to view substantially the entire interior of the cabinet. The light sensor 160 may use various wavelengths of light. In some embodiments, the light sensor 160 may be used to assist in determining the location, size, and shape of each object within the cabinet 110, as shown, for example, in FIG. 11. Data from the light sensor may determine the size, shape, and location of products that may be obscured from the camera's view. Additionally, the light sensor 160 may be used to determine the size, shape, and location of products that may be obscured from the camera's view. If the product is removed and replaced in a different location, the optical sensor can determine the position of the product.

[0059] In some embodiments, the optical sensor may be an RFID sensor. In such embodiments, cabinet 110 may include an RFID sensor configured to detect the presence of RFID-tagged products. Thus, when a product is removed from the cabinet, the RFID sensor can determine the identity of the removed product. In some embodiments, the optical sensor may be a light detection and ranging (LIDAR) sensor or a magnetic resonance imaging (MRI) sensor, among others. Data from optical sensor 160 may be used to confirm the identity of the product removed from the cabinet, as determined by other sensors or cameras. This may help to increase the accuracy of product identification.

[0060] In some embodiments, the vending machine may generate a digital map of the products in its cabinets, as shown in FIG. 12. Digital map 300 may include a digital representation of the cabinet and the products therein. Thus, digital map 300 may be a real-time digital twin of cabinet 110 and the products within cabinet 110. For example, as shown in FIG. 12, shelf 112 of cabinet 110 may include products 201-204 arranged on shelf 112 at specific locations. Digital map 300 may include a representation of the cabinet and shelf 312, and may include models, such as 3D models, of products 201'-204' on shelf 312. Thus, digital map 300 may include the locations and identities of products within cabinet 110 of vending machine 100. Digital map 300 may also function as a product inventory, such that the number of each product in the cabinet is known.

[0061] In some embodiments, cameras 120, sensors, or a combination thereof can be used to generate digital map 300. For example, as an operator places each product in cabinet 110, cameras 120 and sensors can detect the product's location and ID and further generate a 3D model of the product. In this manner, a digital map can be generated as cabinet 110 is filled. In some embodiments, an operator can manually enter or confirm the ID and location of each product in the cabinet.

[0062] The digital map 300 may provide a baseline of information about the products in the cabinet before a consumer accesses the cabinet. When a consumer removes a product from a particular location in the cabinet, the identity of the product at that location is known from the digital map. Thus, the digital map can be used to confirm product identification made based on data from the camera 120 or other sensors of the vending machine 100. Furthermore, as a consumer removes products from the cabinet, the digital map may be updated using data from the camera 120 and sensors. For example, the digital map may be updated to reflect that one or more products have been removed, that one or more products have been refilled, or that one or more products have been moved or rearranged within the cabinet. The optical sensor 160 may help determine the location of products within the cabinet, as the view of some products in the cabinet may be obscured from the camera's field of view.

[0063] In some embodiments, the identity of a product may be determined, in part, by tracking the location of a consumer's hand, as shown in FIG. 13 . The location of the consumer's hand may be tracked using computer vision techniques based on data from the camera 120. The coordinates of the consumer's hand 1200 within the cabinet 110 may be detected by one or more cameras 120 to determine which shelf the consumer is accessing. In some embodiments, the consumer's hand 1200 may be tracked to identify the product 200 at the location of the consumer's hand. The coordinates of the consumer's hand may be determined in two dimensions, such as a top view of a shelf. For example, the coordinates of the consumer's hand 1200 may include a position along an X-axis extending from the front to the back of the cabinet 110 and a position along a horizontal or Y-axis. The identity and location of each product 200 may be known, such as via a digital map. Thus, if the location of the consumer's hand is known, the product 200 at the coordinates of the consumer's hand 1200 can be easily identified by identifying the product at its location in the digital map. Computer vision may further detect specific movements or gestures by the consumer's hand 1200 to determine whether the consumer is picking up or putting back a product. In some embodiments, the trajectory of the consumer's hand 1200 may be detected to determine the product selected by the consumer.

[0064] FIG. 14 shows an exemplary method 1400 for determining the identity of a product using a digital map. The location where a product is removed from a cabinet may be detected 1410 by at least one of a camera and a sensor. The visual characteristics of the product removed from the cabinet may be determined 1420 based on data from the camera or sensor. A predicted identity of the product at the location may be determined 1430 based on the visual characteristics. For example, artificial intelligence and machine learning techniques may be used to predict the identity of the product based on a pre-trained machine learning model. For example, the visual characteristics may be the shape of the product, and the artificial intelligence may analyze products in product inventory that have a known shape that most closely matches the shape of the product. To confirm that the predicted identity is correct, the identity of the product at the location may be determined 1440 based on a digital map of the products in the cabinet. If the predicted identity corresponds to the identity of the product based on the digital map, the predicted identity may be confirmed 1450.

[0065] In some embodiments, the vending machine 100 may include an external camera 130, as shown in FIG. 1. The external camera 130 may be configured to view an area external to the vending machine 100. The external camera 130 may be positioned outside the cabinet 110, or may be disposed within the cabinet 110 to view an area external to the vending machine 100. For example, the external camera 130 may be disposed on a door of the vending machine 100 or on the exterior of the cabinet 110, or the camera 130 may be separate from the vending machine 100. In some embodiments, the external camera 130 may be activated when the presence of a consumer is detected near the vending machine 100. The presence of a consumer may be detected by a proximity sensor 135 (see, for example, FIG. 2).

[0066] The external camera 130 may be configured to capture images or video of one or more consumers and / or products removed from the cabinet 110. In some embodiments, the external camera 130 may be configured to identify one or more consumers. The captured images or videos may be used for facial recognition of the consumers. In some embodiments, the external camera 130 may be configured to identify the consumers' hands and their movements.

[0067] Similar to the operation of internal camera 120, external camera 130 may capture images or video that may be analyzed to determine the visual characteristics of a product removed from cabinet 110. Thus, a product removed from cabinet 110 may be detected by external camera 130 to determine the product's shape, size, or color. Data from external camera 130 may be used to confirm the identity of the product removed from cabinet 110, as determined by other cameras or sensors described herein.

[0068] In some embodiments, the external camera 130 may be used to determine whether a consumer is engaging in fraudulent activity. In some embodiments, the external camera 130 may be remotely monitored by an operator, such as a security officer. In some embodiments, data from the external camera 130 may be analyzed by artificial intelligence that is pre-trained to detect fraudulent activity. The artificial intelligence may detect the consumer or the consumer's hand gestures or movements. For example, the artificial intelligence may be programmed to specifically detect consumers who strike the vending machine 100 or block the camera 120 to prevent product identification. If fraudulent activity is detected, a responsible authority, such as a security officer or local authority, may be alerted. Additionally, the cabinet door may be locked so that the consumer cannot open the door and can no longer access the products within the cabinet. In some embodiments, if fraudulent activity is detected, an alarm 178 may be activated (see, for example, FIG. 2).

[0069] In some embodiments, the identity of a product removed from cabinet 110 may be determined using one or more of internal camera 120, weight sensor 140, light sensor 160, identifier sensor 150, or external camera 130, as described herein. The identity of a product removed from cabinet 110 may be determined based on data from internal camera 120 and identifier sensor 150, but one or more of light sensor 160, weight sensor 140, and external camera 130 may be used to verify that the identity is correct. Additionally, additional sensors or cameras may serve as backups in case the camera or identifier is unable to operate correctly.

[0070] In one example of product identification, a camera may capture an image of a product and identify the product's shape (e.g., bottle shape). However, the product's shape may correspond to multiple possible product identities (e.g., Pepsi, Diet Pepsi, or Cherry Pepsi). An identifier sensor may detect an identifier on the product, such as text (e.g., Diet Pepsi), which may correspond to multiple product identities (e.g., can or bottle). Thus, data from the camera and identifier sensor may be combined and analyzed to determine a predicted product identity (e.g., a bottle of Diet Pepsi). The analysis may limit the possible product identifications to products in the product inventory. Additional data may be collected to verify that the product identification is correct. For example, a weight sensor may determine a calculated weight of the product, which may correspond to a bottle of Diet Pepsi to confirm the product identification. An optical sensor may indicate that the product has been removed from a location in a cabinet where bottles of Diet Pepsi are stored. Furthermore, an external camera may detect visual features, such as bottle shape, once the product is removed from the cabinet. Thus, additional cameras and sensors may help verify that the product has been correctly identified.

[0071] In another example, an internal camera may capture images before and after a consumer removes a product from the cabinet. The images may be analyzed to determine the location within the cabinet from which the product was removed. Data from the camera indicating the product's location may be analyzed using a digital map of the products within the cabinet to determine the identity of the product at that location. To confirm the identity of the product, a camera that detects products entering or leaving the cabinet may determine the visual characteristics of the product removed from the cabinet to confirm the identity of the product identification based on the digital map. Alternatively, an identifier sensor may detect the identifier of the product removed from the cabinet.

[0072] In some embodiments, artificial intelligence may determine the confidence of a product identification based on the camera or sensor. A sensor fusion algorithm may determine the product's identity based on the confidence of the identification made by each camera or sensor. If the data matches, the product's identity is confirmed. For example, if a first camera determines that a removed product is product A with 80% confidence, and a second camera determines that the removed product is product B with 30% confidence, the algorithm may determine that product A is the correct product identification with the higher confidence. In some embodiments, data from a particular camera or sensor may have greater weight in determining identity. In some embodiments, if the confidence is below a predetermined threshold, for example, 30%, the data may be ignored. In some embodiments, if the confidence is below a predetermined threshold, an alert may be sent for audit or review.

[0073] 15 illustrates an exemplary computer system 1500 in which embodiments or portions thereof may be implemented as computer readable code. The control unit 180 discussed herein may be a computer system having all or some of the components of computer system 1500 for implementing the processes discussed herein.

[0074] Where programmable logic is used, such logic may be executed on a commercially available processing platform or a special purpose device. Those skilled in the art will appreciate that embodiments of the disclosed subject matter may be practiced with a variety of computer system configurations, including multi-core multiprocessor systems, minicomputers and mainframe computers, computers linked or clustered with distributed functionality, and pervasive or small computers that may be embedded in virtually any device.

[0075] For example, at least one processor device and memory may be used to implement the above-described embodiments. The processor device may be a single processor, multiple processors, or a combination thereof. The processor device may have one or more processor "cores."

[0076] Various embodiments of the present invention may be implemented by this exemplary computer system 1500. After reading this description, it will become apparent to one skilled in the art how one or more aspects of the present invention may be implemented using other computer systems and / or computer architectures. While operations may be described as sequential processes, some of the operations may actually be performed in parallel, concurrently, and / or in a distributed environment, and may be performed by program code stored locally or remotely for access by single or multi-processor machines. In some embodiments, edge computing, cloud computing, or a combination thereof may be used. Additionally, in some embodiments, the order of operations may be rearranged without departing from the spirit of the disclosed subject matter.

[0077] The processor device 1504 may be a dedicated or general-purpose processor device. As will be appreciated by those skilled in the art, the processor device 1504 may also be a single processor in a multi-core / multi-processor system, operating singly or in a cluster of computing devices operating in a cluster or server farm. The processor device 1504 is connected to a communications infrastructure 1506, such as a bus, message queue, network, or multi-core message passing scheme.

[0078] The computer system 1500 also includes a main memory 1508, e.g., random access memory (RAM), and may also include a secondary memory 1510. The secondary memory 1510 may include, for example, a hard disk drive 1512 or a removable storage drive 1514. The removable storage drive 1514 may include a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like. The removable storage drive 1514 reads from and / or writes to a removable storage unit 1518 in a well-known manner. The removable storage unit 1518 may include a floppy disk, magnetic tape, optical disk, universal serial bus (USB) drive, or the like, which is read from and written to by the removable storage drive 1514. As will be appreciated by those skilled in the art, the removable storage unit 1518 includes a computer-usable storage medium having stored thereon computer software and / or data.

[0079] Computer system 1500 (optionally) includes a display interface 1502 (which may include input and output devices such as a keyboard, mouse, etc.), which transfers graphical, textual, and other data from a communications infrastructure 1506 (or from a frame buffer, not shown) for display on a display unit 1530.

[0080] In alternative implementations, secondary memory 1510 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 1500. Such means may include, for example, a removable storage unit 1522 and interface 1520. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units 1522 and interfaces 1520 that can transfer software and data from the removable storage unit 1522 to computer system 1500.

[0081] Computer system 1500 may also include a communications interface 1524. Communications interface 1524 allows software and data to be transferred between computer system 1500 and external devices. Communications interface 1524 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 1524 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1524. These signals may be provided to communications interface 1524 via communications path 1526. Communications path 1526 carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0082] As used herein, the terms "computer program medium" and "computer usable medium" are used generally to refer to media such as removable storage unit 1518, removable storage unit 1522, and a hard disk installed in hard disk drive 1512. Computer program medium and computer usable medium may also refer to memory, such as main memory 1508 and secondary memory 1510, which may be memory semiconductors (e.g., DRAM, etc.).

[0083] Computer programs (also called computer control logic) are stored in main memory 1508 and / or secondary memory 1510. Computer programs may also be received via communications interface 1524. When executed, such computer programs enable computer system 1500 to implement the embodiments discussed herein. Specifically, when executed, the computer programs enable processor device 1504 to perform the processes of the embodiments discussed herein. Such computer programs thus represent controllers of computer system 1500. When an embodiment is implemented using software, the software may be stored in a computer program product and loaded into computer system 1500 using removable storage drive 1514, interface 1520, and hard disk drive 1512, or communications interface 1524.

[0084] Embodiments of the present invention may also be directed to computer program products including software stored on any computer-usable medium. Such software, when executed on one or more data processing devices, causes the data processing devices to operate as described herein. Embodiments of the present invention may employ computer-usable or readable media. Examples of computer-usable media include, but are not limited to, primary storage devices (e.g., any type of random access memory), secondary storage devices (e.g., hard drives, floppy disks, CD ROMs, ZIP disks, tapes, magnetic and optical storage devices, MEMS, nanotechnology storage devices, etc.).

[0085] It is understood that the "Detailed Description" section, and not the "Summary" and "Abstract" sections, is intended to be used to interpret the claims. The Summary and Abstract sections may set forth one or more, but not all, exemplary embodiments of the invention as contemplated by the inventor(s), but are not intended to limit the scope of the invention and the appended claims in any way.

[0086] The present invention has been described above with the aid of functional building blocks that illustrate the performance of certain functions and their relationships. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and their relationships are appropriately performed.

[0087] The foregoing description of specific embodiments makes the general nature of the present invention fully apparent, and others, by applying the knowledge of those skilled in the art, may readily modify and / or adapt such specific embodiments to various uses without undue experimentation and without departing from the general concept of the present invention. Such adaptations and modifications are therefore intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology used herein is for the purpose of description and not of limitation, and therefore should be interpreted by those skilled in the art in light of the teaching and guidance provided herein.

[0088] The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. 1. A method for identifying a product removed from a cabinet, comprising: detecting a visual characteristic of the product removed from the cabinet with a camera within the cabinet; detecting, with an identifier sensor, an identifier on the product removed from the cabinet; comparing the visual characteristics and the identifier with a product information database; and identifying the product removed from the cabinet based on the visual characteristics and the comparison of the identifier to the product information database.

2. The method of claim 1 , wherein the visual characteristics include a shape of the product.

3. The method of claim 1 , wherein the visual characteristics include the color of the product.

4. The method of claim 1 , wherein the identifier sensor is the camera.

5. The method of claim 1 , wherein the identifier is a barcode.

6. 10. The method of claim 1, further comprising verifying the identity of the product by determining the weight of the product removed from the cabinet with a weight sensor located within the cabinet.

7. determining the weight of the product removed from the cabinet; determining a first weight of the product in the cabinet via the weight sensor; and determining a second weight of the product in the cabinet via the weight sensor. removing the product from the cabinet; and calculating the difference between the first weight and the second weight.

8. detecting, by an external camera, visual characteristics of the product removed from the cabinet at a location outside the cabinet; The method of claim 1 , further comprising: verifying the identity of the product based on the visual characteristics detected by the external camera.

9. Detecting a user by an external camera; The method of claim 1 , further comprising detecting tampering by the user with the external camera, and locking the cabinet when the tampering is detected.

10. 10. The method of claim 1, further comprising: detecting data regarding the removed product via an optical sensor; and using the data from the optical sensor to verify the identity of the removed product.

11. 1. A method for identifying a product removed from a cabinet, comprising: capturing a first image of a plurality of products within the cabinet with an internal camera within the cabinet; removing a product from the plurality of products from the cabinet; capturing a second image of the plurality of products in the cabinet with the internal camera after removing the products; determining an identity of the product removed from the cabinet by analyzing the first image and the second image; and verifying the identity of the product removed from the cabinet by detecting an identifier of the product removed from the cabinet.

12. 12. The method of claim 11, further comprising verifying the identity of the product by detecting a weight of the product removed from the cabinet via a weight sensor.

13. detecting, by an external camera, visual characteristics of the product removed from the cabinet at a location outside the cabinet; The method of claim 11 , further comprising: verifying the identity of the product based on the visual characteristics detected by the external camera.

14. 12. The method of claim 11, further comprising detecting data regarding the removed product via an optical sensor and using the data from the optical sensor to verify the identity of the removed product.

15. The method of claim 14 , wherein the optical sensor comprises a LIDAR system.

16. 1. A method for identifying a product removed from a cabinet, comprising: detecting via at least one of a camera and a sensor the location at which the product is removed from the cabinet; determining a visual characteristic of the product based on data from at least one of a camera and a sensor; determining a predicted identity of the product based on the visual characteristics; and determining an identity of the product at the location based on a digital map of products within the cabinet. and verifying that the predicted ID of the product corresponds to the ID of the product based on the digital map.

17. 17. The method of claim 16, further comprising updating the digital map of products in the cabinet after the products are removed from the cabinet.

18. 17. The method of claim 16, further comprising generating the digital map of the products in the cabinet using data received from at least one of a camera or a sensor.

19. The method of claim 16 , wherein the digital map includes the location and model of each product in the cabinet.

20. 17. The method of claim 16, further comprising detecting an identifier of the product removed from the cabinet via an identifier sensor.

Images