Article detection and estimation system

Abstract

To provide a POS system capable of detecting an error when an article is measured, scanned, and purchased.SOLUTION: There are provided a method and a system that include: receiving image information including an article being measured and corresponding to a POS transaction; receiving an indication of an article weight or an indication of an article identifier as to an article related to the POS transaction; determining an anticipated article weight or an anticipated identifier as to the article based upon the image information; determining that there is a state between the article weight and anticipated article weight or between the article identifier and anticipated article identifier; and managing the POS transaction based upon the state.SELECTED DRAWING: Figure 3

Description

[Background technology]

[0001]

[0001] Self-checkout stations are commonly used by consumers in retail locations such as grocery stores. In operation, a customer can scan an item into the self-checkout system and place the scanned item on a conveyor of the self-checkout system. Some items, such as produce, are sold by weight. When purchasing these items, the customer must place the item on a scale and enter information identifying the item into the self-checkout station, for example, using a produce code or by scanning the item barcode. Errors in the self-checkout process can occur when a customer misidentifies an item, forgets to scan each item, or otherwise weighs or scans an item improperly. These errors can result in lost sales, also known as "shrinkage."

[0002]

[0002] The present disclosure is described with reference to the following figures: [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a diagram illustrating an example retail environment in accordance with some embodiments of the present disclosure. [Diagram 2]

[0004] FIG. 2 is a diagram illustrating an example checkout terminal according to some embodiments of the disclosure. [Diagram 3]

[0005] FIG. 3 illustrates an example weighing platter in accordance with some embodiments of the present disclosure. [Figure 4] FIG. 4 illustrates an example weighing platter in accordance with some embodiments of the present disclosure. [Diagram 5] FIG. 5 illustrates an example weighing platter in accordance with some embodiments of the present disclosure. [Figure 6]

[0006] FIG. 6 is a flow diagram illustrating a routine according to some embodiments of the present disclosure. [Figure 7]

[0007] FIG. 7 is a basic block diagram of a data processor that can be used to process data in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004]

[0008] In retail and grocery stores, some items, such as produce, are sold by weight. In both traditional and self-checkout lanes, the produce is placed on a weighing surface or platter of a scale. The scale weighs the item. The consumer or store employee then enters the item identification code, which may include a stock-keeping unit (SKU), into a point-of-sale (POS) system via a user interface. The weight of the item, the unit price, and the resulting price of the weighed item are shown to the consumer or store employee on a display.

[0005]

[0009] Errors in the checkout process may occur when a customer or store employee misidentifies an item and / or fails to properly weigh an item. A consumer or store employee may not place the entire item on the scale or weighing platter, resulting in an inaccurate weight measurement. A portion of an item may accidentally or intentionally extend beyond the weighing platter. These errors may result in lost sales, or shrinkage. According to some embodiments of the present disclosure, a POS system can detect errors in weighing, scanning, and purchasing items to reduce shrinkage and address other imperfections during POS transactions.

[0006]

[0010] In some examples, when using a self-checkout station at a grocery store, a consumer may place a bag of apples on a weighing platter of a scale. A bag may contain eight apples, but be positioned such that only six apples are on the weighing platter. Two apples may be resting, for example, on a counter adjacent to the weighing platter. As a result, the scale cannot measure the full weight of the entire bag of apples, and more precisely, the scale only measures the weight of the six apples that are on the weighing platter. Thus, the customer is charged for six apples when eight apples were intended for purchase. In accordance with the present disclosure, images and / or computer vision may be used to recognize that a partial or incomplete amount of an item has been weighed. In this example, a camera at the checkout station may detect that six apples were on the weighing surface, but two apples were in close proximity to the weighing platter and intended for purchase. The POS system may then determine that eight apples are part of the POS transaction and update the transaction details accordingly, editing the quantity from six to eight.

[0007]

[0011] FIG 1 illustrates one embodiment of a retail environment 100 including a network 150, a point of sale (hereinafter "POS") system 110, an item analysis system 120, an imaging system 170, and an item database 160. For ease of explanation and to avoid limiting the present disclosure, FIG 1 illustrates only one POS system 110, although multiple POS systems located at the same or different locations may be used. Although the present disclosure generally refers to environments in retail locations, it should be understood that the environment 100 is not limited to retail locations. Rather, the environment 100 may include other types of environments, such as, for example, office buildings, hospitals, and transportation hubs.

[0008]

[0012] Any of the aforementioned components or systems of environment 100 may communicate via network 150. Although only one network 150 is illustrated, multiple distinct and / or distributed networks may exist. Network 150 may include any type of communication network. For example, network 150 may include one or more of a wide area network (WAN), a local area network (LAN), a cellular network (e.g., LTE, HSPA, 3G, and other cellular technologies), an ad-hoc network, a satellite network, a wired network, a wireless network, and the like. In some embodiments, network 150 may include the Internet.

[0009]

[0013] The various components and systems of environment 100 may be implemented in or may include various computing devices, such as processors, distributed processing systems, servers, or isolated execution environments (IEEs), such as virtual machines or virtual containers, or shared computing resources, etc. Any of the foregoing components or systems of environment 100 may be combined and / or include software, firmware, hardware, or any combination or combinations of software, firmware, or hardware suitable for the purposes described.

[0010]

[0014] The POS system 110 facilitates POS transactions through the user terminal 20, the scale 30, and the processing system 114. A POS transaction broadly refers to any part of a process for conducting a transaction at a retail location, such as a POS system or a self-checkout station. The specific steps may vary depending on the retailer or POS system, but may include a series of actions performed during the exchange of goods or services, including, but not limited to, scanning or weighing an item, entering an item code, and finalizing the checkout procedure. The POS system 110 may include or communicate with hardware and software components, such as, for example, a display, a scanner, a printer, a database, a scale, a conveyor belt, a dimensional analyzer, etc., to support the POS transaction. The POS system 110 may include a user terminal 20, such as a touch screen 22 or a keyboard, to allow a person to interact with the POS system 110 during a POS transaction, and a scale 30 to weigh items to be purchased. As described herein, a user may interact with the POS system to perform a checkout procedure, including weighing items to be purchased with the scale 30. A checkout procedure can refer to the sequence of steps a customer follows to complete a purchase, while a POS transaction can be considered a subset of a checkout procedure that focuses on the formal actions performed within the POS system.

[0011]

[0015] According to the present disclosure, the item analysis system 120 may be responsible for ensuring the accuracy and completeness with which an item is weighed and entered into the POS system 110 during a POS transaction. This responsibility may include tasks such as identifying when incorrect or incomplete data is provided about an item during a POS transaction, for example, identifying when an item has been improperly weighed or improperly identified by a user, estimating correct data, and / or updating or adjusting the provided data so that the final data accurately reflects the item being purchased. For example, the item analysis system 120 may detect when an item has been improperly weighed, estimate the proper weight of the item, and adjust or correct the improper weight if there is a discrepancy between the estimated proper weight and the improper weight. The item analysis system 120 may include an image recognition system 122 and a verification system 126.

[0012]

[0016] The image recognition system 122 can identify the items and produce being purchased and additional objects around the checkout station during a POS transaction. The item analysis system 120 receives and analyzes the images and image information, and the images can be received from the image system 170. The images can include a user interacting with the POS system 110, such as a user weighing the items or produce at a self-checkout station, or an employee scanning the items or produce at a traditional checkout station. The image recognition system 122 can identify features in the image and / or identify various objects, items, or produce in the image. The features can include, for example, the type of item, the number of items, the number of items in a set or group, the number of items in a container, the container, the type of container, the location of the items relative to the weighing area or surface, the weighing area or surface, the perimeter or border of the weighing area or surface, the name of the item, the size of the item, different or dissimilar items and / or produce being weighed together, unintended items placed on the weighing surface, obstacles on the weighing surface, etc.

[0013]

[0017] The image recognition system 122 may use computer vision techniques to identify items, produce, and features in the image. To process the image information, the image recognition system 122 may use various image segmentation and analysis techniques. The system may perform object recognition algorithms to distinguish between different types of items, produce, objects, components, equipment, etc. Edge detection algorithms may be utilized to delineate boundaries between different objects and the background. Color segmentation may be used to distinguish areas based on hue, saturation, and brightness values. Additionally, the system may use machine learning models trained on the items and produce to optimize the identification process. In some examples, the image recognition system 122 may determine whether a user is purchasing a bag of eight apples, a box of six apples, two heads of lettuce, a rotisserie chicken, or a pound of burrata.

[0014]

[0018] The verification system 126 can compare information and data from the POS system 110 with information and data identified by the image recognition system 122. By doing so, the verification system 126 determines whether the POS transaction is properly performed without problems, mistakes, errors, or fraud. The POS transaction can be deemed to be properly performed if no error condition or discrepancy is identified by the verification system 126. The POS transaction can be deemed to be properly performed when the information collected by the image matches the information entered by the user. The error condition or discrepancy can indicate an interruption or error in the POS transaction and can be based on a variety of criteria. The verification system 126 can detect that there is an error condition based on detecting a discrepancy between an item associated with the POS transaction shown in the image and information entered by the user, including item weight, item code, or other item identifier. For example, the image recognition system 122 can detect that an item is being weighed for purchase in the image, and therefore the verification system 126 expects an item weight, code, or identifier that corresponds to the item being weighed. However, the user may enter a code that identifies a different item. The verification system 126 detects a mismatch between the expected code and the code entered by the user and recognizes an error condition. For example, the image recognition system 122 detects that an apple is being weighed on the scale, and therefore the verification system 126 expects an apple code to be entered by the user. When the verification system 126 receives a banana code, an error condition is detected because there is a mismatch between the expected apple code and the entered banana code. As another example, the verification system 126 can recognize an error condition when an item is imperfectly loaded onto a designated weighing surface, which can result in an inaccurate weight measurement. This detection can be done through a combination of weight sensor feedback and visual analytics, such as using an integrated camera to capture the actual placement of the item and compare it to the expected dimensions of the weighing surface.

[0015]

[0019] The verification system 126 can analyze multiple information and data sets to verify that the user properly weighs the item or produce being purchased, enters the correct SKU or barcode, properly identifies the item and produce being purchased, and otherwise completes the POS transaction. The information and data sets can be received from the POS system 110, the imaging system 170, the network 150, or the item database 160. In some examples, the verification system 126 recognizes when an item is not positioned within the weighing area, resulting in an improper weight reading. In another example, the verification system 126 can recognize when two dissimilar items are weighed together. In some examples, the verification system 126 recognizes when a user enters the produce identity code of a banana while weighing an apple, which may result in a lower price. In this manner, the verification system 126 can help detect problems, mistakes, errors, or fraud during a POS transaction.

[0016]

[0020] The verification system 126 can determine an expected weight or expected price of produce or items that may not have been properly weighed or properly identified during a POS transaction to reduce the impact of problems, mistakes, errors, or fraud during a POS transaction. The verification system 126 can detect or estimate portions of items that are within or outside the perimeter of the weighing area and estimate the actual weight of the item. In some examples, the verification system 126 can detect that a butternut squash has been weighed but 1 / 3 of the butternut squash is on an adjacent counter and is not on the weighing surface, resulting in an incorrect item weight. The verification system 126 can estimate the weight of the 1 / 3 of the butternut squash that is not on the weighing surface. The verification system 126 may estimate the weight by proportionally increasing or decreasing the weight measured by the scale, or the verification system may rely on an item database 160 that includes average weights, prices, etc. for butternut squash. In some examples, an item may be properly weighed but incorrectly identified by the user, resulting in a discrepancy between the entered price and the expected price. Thus, the verification system 126 can detect a discrepancy between the image of the item being weighed and the identification information entered into the POS system and update or modify the sales information to reflect the appropriate item. For example, while a user is weighing apples, the user may identify bananas as the produce to be purchased, resulting in a lower price. The verification system can determine that a discrepancy exists between the data entered by the user into the POS system 110 and the data received from the image recognition system 122. The verification system 126 can look up the price of the apples from the item database 160 to determine the expected price of the weighed apples and update the sales information accordingly.

[0017]

[0021] Imaging system 170 may capture image data of produce, items, or people present within environment 100. For example, imaging system 170 may capture image data associated with a weighing procedure, weighing system, scanning system, or scanning procedure performed by a user or employee during a POS transaction. Image data captured during a POS transaction may include various information and features that may be used to determine whether the weighing or scanning procedure was performed correctly and whether the information and data from POS system 110 can be considered reliable or contains problems, errors, mistakes, or fraud.

[0018]

[0022] The image data may include images of a user or employee placing an item or produce on the scale, weighing the item or produce, or entering data into the user interface. The image data may be used to ensure that the user or employee is following the proper protocol of the weighing and scanning procedure and that the item is being weighed or scanned correctly. The image data may include images of the item itself, including any packaging or labeling. This may be used to verify that the item being weighed matches the data of the scanned item and matches the data of the item identified by the image recognition system 122. Additionally, the image data may include images of tools and equipment used during the weighing and scanning procedure, such as the scale, user interface, or display. This may be used to ensure that the equipment is functioning properly and that the measurements obtained are accurate. Additionally, the image data may include images of the checkout station and any objects around the checkout station.

[0019]

[0023] Imaging system 170 may be positioned at various locations throughout environment 100 to capture image data of items, produce, users, employees, or POS system 110. For example, imaging system 170 may be placed proximate POS system 110 to capture images of users weighing, scanning, and purchasing items and produce.

[0020]

[0024] The item database 160 can store references, characteristics, or other information about produce and items, such as item descriptions, weights, measurements, dimensions, prices, or other details. The database 160 can include associations between various types of data, such as image data and reference data, to enable more accurate and efficient system operation. For example, captured image data can be analyzed in conjunction with stored data and the image recognition system 122. This integrated approach leverages multiple data sources to accurately identify items, produce, errors, and problems in the images.

[0021]

[0025] The article database 160 can be implemented as a cloud storage, such as, for example, Amazon's Simple Storage Service (S3), Elastic Block Storage (EBS), or CloudWatch, Google Cloud Storage, Microsoft Azure Storage, InfluxDB, etc. The article database 160 can include one or more databases that store data referenced by the article analysis system 120, the POS system 110, or the imaging system 170. The article database 160 can be configured to provide highly available and highly resilient low-loss data storage. The article database 160 can include Amazon CloudWatch metrics and can store multiple copies of data in the same and different geographic locations, across different types of data stores (e.g., solid state, hard drives, tape, etc.).

[0022]

[0026] Referring now to FIG. 2, a POS system 10 for a self-checkout station 12 is illustrated in accordance with some embodiments of the present disclosure. The POS system 10 includes a self-checkout station 12 at a first location 14. The checkout station 12 includes a terminal 20 and a bagging station 50. The terminal 20 may include a customer interface, such as a touch screen 22, a payment console 24, a counter 26, and a base or housing 28. The checkout station 12 may also include an indicator, such as a pole 16 having a light 18 attached thereto, for example, to indicate when the checkout station is open or closed and / or when a consumer requires assistance from a store employee. Checkout stations in retail and grocery stores include both self-checkout stations and staffed checkout stations, and are typically installed in groups or clusters, with multiple checkout stations located in the same area or vicinity. The POS system 10 may be an embodiment of the POS system 110 of FIG. 1, or may include one or more components thereof.

[0023]

[0027] 2 is provided by way of example only, and it will be understood that embodiments of the present disclosure are not limited thereto. Additionally, checkout stations may include traditional or manned checkout stations, as well as mobile, pop-up, or transportable checkout stations.

[0024]

[0028] The terminal 20 includes a scanner 40 adjacent to a scale 30. The scale 30 includes a weighing platter 32 having a perimeter 34. The scale 30 is received within an opening in the counter 26 such that the counter 26 surrounds a portion of the perimeter 34 of the scale 30. In other embodiments, the counter may be adjacent to one or more perimeters of the scale. As shown in FIG. 3, the perimeter 34 includes perimeters 34a, 34b, 34c, and 34d. The shape of the perimeter of the scale may vary. Objects, including goods and produce, are weighed by placing the object on the weighing platter 32 such that the total weight of the object is transferred to the weighing platter within the perimeter 34 that defines the weighing area of ​​the scale 30. When a first object contacts a second object or another surface other than the weighing platter 32, a portion of the weight of the first object may be transferred to the second object or surface and the first object may not be weighed properly. As a result, the weigh platter does not receive the full weight of the first object, resulting in an inaccurate weight measurement.

[0025]

[0029] The checkout station 12 includes at least one edge camera 70, which may be connected to the network 150, the imaging system 170, or the item analysis system 120. Each edge camera 70 is associated with a corresponding field of view (FOV) 72, which may include a desired target area. The edge cameras 70 may be positioned such that the FOVs 72 overlap one another and / or form a complete view of the checkout station 12. The edge cameras 70 may also be positioned such that the FOV 72 is in focus on the desired target area, e.g., a view of the terminal 20 overhead, the bagging area 50, the payment console 24, etc.

[0026]

[0030] Each checkout station typically includes multiple fixed cameras mounted thereon. The cameras may be positioned to provide a viewing angle of a desired target area. The target area may include, for example, the scanner and weighing platter, the cart or basket, the bottom of the basket, the bagging area, and the payment area. The target area may encompass the entire store and may include overlapping areas and fields of view. Cameras mounted at checkout stations next to or near the main station may provide alternative viewing angles that may be of value as input to the main station, especially in the case of self-checkout stations. The cameras may be smart cameras and / or edge cameras and may be connected to each other and / or to additional nodes forming a vision mesh network.

[0027]

[0031] The edge cameras 70 may implement the article analysis system 120 and the imaging system 170. Each edge camera 70 is connected to at least one other node or edge camera forming a vision mesh network 74. In some examples, each edge camera 70 at a checkout station 12 is directly connected to the other cameras 70 and to the network 150. Each edge camera 70 may be connected to any number of other edge cameras 70 as desired. The edge cameras 70 are connected to each other wired or wirelessly, or any combination thereof. Using a wired setup may reduce latency and interference.

[0028]

[0032] Each edge camera 70 can include article analysis system 120 and imaging system 170, thus collecting and processing information close to the source of the data, i.e., at the edge camera 70, and performing calculations, including analysis of images and other data, on the edge camera 70. Edge computing reduces the need to send images and other data to a central server or cloud service for processing, which can increase processing speeds and reduce stress on the overall network.

[0029]

[0033] The edge camera 70 captures and provides data input, e.g., image or video capture, and processes the data input through a computer vision module, which may include a produce recognition module, a known average weight module, and a face and body recognition module, that outputs a behavior decision based on the input data. Accuracy is directly related to the quality and quantity of input data. The computer vision module includes methods for acquiring, processing, analyzing, and understanding digital images, and for extracting high dimensional data from the real world to form a decision. Various embodiments of the computer vision module described herein acquire, process, analyze, and understand images and data to provide feedback and behavior decisions regarding POS transactions.

[0030]

[0034] According to the present disclosure, a customer may initiate a transaction by pressing a "start" button or by entering a loyalty number on touch screen 22. A scanner 40 may be used to scan the item, which is then placed in a bagging area 50. The touch screen 22 may be used to locate items that need to be weighed, e.g., fruits and vegetables such as bananas or apples, which are weighed on scale 30 and then placed in bagging area 50. Alternatively, the customer may enter an item identification code via touch screen 22. The item identification code may be located on a label, bar code, or sticker on the item.

[0031]

[0035] The price of items purchased by weight is dependent upon each item being weighed accurately. In general, an item 80 is properly weighed by the scale 30 when a single item 80 is placed on the weighing platter 32 and the item 80 is within the perimeter 34 of the scale 30, as illustrated in FIG. 5, so that the entire weight of the item is transferred to the weighing platter 32. An item 80 cannot be properly weighed when the item 80 spans the perimeter 34, as illustrated in FIGS. 3 and 4, because a portion of the item's weight is transferred to another surface and cannot be measured by the scale 30. As illustrated, the item 80 is partially resting on the counter 26. Additionally, an item cannot be properly weighed when an additional item is inadvertently placed on the scale. For example, a carrier bag, a child's toy, or a bottle of water 82 (FIG. 5) may be accidentally placed on the scale by a customer, inadvertently affecting the weight of the desired item.

[0032]

[0036] According to the present disclosure, the article analysis system 120 and imaging system 170 can determine whether an article 80 placed on the scale 30 has been properly weighed. The article analysis system 120, imaging system 170, and camera 70 can capture and process images and data to determine whether the article 80 has been placed on the scale 30 such that the entire weight of the article 80 is on the weighing platter 32, or whether the article 80 is resting on another surface or object, for example, by exceeding the perimeter 34 or any of the perimeters 34a, 34b, 34c, 34d. The article analysis system 120 and imaging system 170 can also detect whether an additional article 82 has inadvertently been placed on the scale, thereby affecting the desired article weight, as shown in FIG.

[0033]

[0037] 3 and 4, when using the self-checkout station 12 at a grocery store, a customer has placed an item 80 on the weighing platter 32 of the scale 30. The item 80 is a bag containing eight apples, but six of the apples are positioned on the weighing platter 32 and the remaining two apples are on the counter 26. As a result, the scale 30 is unable to weigh the entire item 80 (eight apples), but weighs only a portion of the item 80 (six apples) on the weighing platter 32. Thus, without the present invention, the customer would be charged for six apples when eight apples were intended for purchase.

[0034]

[0038] According to the present disclosure, an edge camera 70 having an FOV 72 of the scale 30, embodying and implementing the article analysis system 120 and the imaging system 170, recognizes that a partial or incomplete amount of an article 80 has been weighed. As shown in FIGS. 1, 2, and 3, the camera 70 has a view of the weighing platter 32 and the perimeter 34. The camera 70 recognizes that the article 80 spans the perimeter 34 of the weighing platter 32. The camera 70 processes the data input through a computer vision module that obtains data input, e.g., image or video capture, and outputs an action decision based on the input data. For example, the computer vision module may send a command to the user interface 22 to inform the customer that a weighing error has occurred and provide the customer with an opportunity to remedy the error. As illustrated in FIG. 5, the article 80 is placed on the weighing platter 32 within the perimeter 34 of the scale 30 for weighing. The edge camera 70 can detect and recognize when the article 80 has been properly weighed and provide this information to the user terminal 20.

[0035]

[0039] In some examples, multiple cameras 70 at the checkout station 12 include a view of the scale 30. When this occurs, one camera may be designated a primary camera and receive input from the remaining secondary cameras. The primary camera processes the input from all the cameras and communicates with the user terminal 20 regarding the scale 30.

[0036]

[0040] According to some embodiments of the present disclosure, a combination of image information and known average weight and price may be used to account for the weight or price of all items, even if some items or portions of items are off the scale. The item analysis system 120 recognizes when an item 80 has been improperly weighed because it is not on the scale or is partially off the scale. For example, the item analysis system 120 recognizes that six of eight apples have been weighed, resulting in two apples being missing from the price calculation. The item analysis system 120 may send an instruction to the terminal 20 to charge the customer for the additional two apples. The charge may be based, for example, on the known average weight or by estimating or extrapolating the weight from the weighed portions of the item. If apples are known to weigh an average of 0.25 lbs each, the customer may be charged an additional 0.5 lbs for the two apples that are outside the perimeter 34 of the scale 30. Alternatively, if the six apples weighed by the customer weighed a total of 1.2 lbs, the customer may be charged an additional 0.4 lbs for the two apples that were not on the scale. In this manner, the item analysis system 120, POS system 110, and imaging system 170 identify and recognize errors during checkout, thereby reducing shrinkage and fraud.

[0037]

[0041] In some examples, the item analysis system 120 can use information from the mesh network 74, imaging system 170 to determine when there is a discrepancy between an item that has not been properly identified by a customer and a known average weight stored in the item database 160. A customer may place an item 80 containing eight apples on the scale, but enter an item identification code for bananas into the touch screen 22, which would traditionally result in a lower price since bananas are less expensive than apples. The item analysis system 120 determines that the item 80 on the scale 30 contains apples and verifies that the weight of the item 80 matches the known average weight of apples. A message may be displayed on the touch screen 22 informing the customer of the error, or the item analysis system 120 may modify or adjust the sales information accordingly. The message may request customer input, and the customer may be asked to verify that apples are on the scale 30 and not, for example, bananas. In this manner, the item analysis system 120 can recognize discrepancies regarding item identification and errors during weighing. In at least one embodiment, user interface 22 may generate item identification suggestions based on images recognized by camera 70. In at least one embodiment, user interface 22 may generate item identification suggestions based on images recognized by camera 70. Thus, article analysis system 120 identifies the item being purchased via images, computer vision, and image recognition to predict the item being purchased. Article analysis system 120 may then compare the predicted item to information entered by the customer or received by terminal 20.

[0038]

[0042] According to some embodiments of the present disclosure, the article analysis system 120 can detect if two dissimilar articles are placed on the scale 30 for weighing. As illustrated in FIG. 5, a customer places an article 80, i.e., a bag of apples, on the weighing platter 32 within the scale perimeter 34, but a bottle of water 82 is also mistakenly placed on the scale 30. The article analysis system 120 detects and recognizes that two dissimilar articles are being weighed simultaneously and communicates this to the POS system 110, which may cause the user interface 22 to alert the customer that two different articles 80, 82 are both on the scale 30, display an error, and / or request customer input.

[0039]

[0043] 6 is a flow diagram illustrating one embodiment of a routine 300 performed by a computing device of environment 100. Although described as being performed by item analysis system 120, it will be understood that the elements outlined for routine 300 may be performed by one or more computing devices / components associated with environment 100, such as, but not limited to, POS system 110, checkout terminal 20, edge camera 70, user interface 22, processing system 114, or scale 30. Thus, the following exemplary embodiment should not be considered limiting. Additionally, fewer, more, or different blocks may be used as part of routine 300, and blocks may occur in a different order than as illustrated.

[0040]

[0044] In block 310, the item analysis system 120 receives image information from the imaging system 170 corresponding to the POS transaction. The image information may include images of a user and / or objects during a POS transaction, including images of a user scanning or weighing an object, and the item or produce being purchased. With reference to FIGS. 2-5 , the image information may include images of the checkout terminal 20, the scale 30, the scanner 40, the user, and the POS system 10. The image information may include information indicating that the item was not positioned on the scale 30 for accurate weighing. For example, the image information may include an image showing the location of the item relative to the circumference 34 of the weighing platter 32.

[0041]

[0045] In block 320, the item analysis system 120 receives an item weight or item price for an item associated with the POS transaction. The item analysis system 120 may receive the item weight or item price from the POS system 110 or the item database 160. With reference to FIGS. 2-5, the item weight or item price may be based on information entered into the scale 30 or the scanner 40. In some examples, a user places an item on the scale 30 and enters an item code into the user interface 22. The scale 30 records the weight of the item, and the POS system 110 sends the item weight and item code to the item analysis system 120.

[0042]

[0046] In block 330, the item analysis system 120 determines an expected item weight or expected item price for the item based on the image information. The received image information may be forwarded to a processing unit in the item analysis system 120 for further analysis to determine an expected item weight or expected item price. The image information may be processed using various algorithms and techniques, including but not limited to computer vision algorithms, object detection, and image segmentation, to isolate characteristics relevant to determining an expected item weight or expected item price for an item associated with a POS transaction. An image is captured and processed to determine whether a checkout procedure is performed properly. Specifically, the image may be processed to determine whether an item was properly weighed on a scale, whether an item was properly scanned, or whether an item was properly entered into a user interface. The number of items on the scale, the type or identity of the items on the scale, and the location of the items relative to the scale are some of the details considered. The expected weight or expected price of the item may be determined by retrieving weight or price information corresponding to the identified item from the item database 160 or the network 150. In some examples, eight apples are identified in the image of the weighing area, and thus an expected item weight and expected item price for the eight apples are determined. The determining may include accessing item database 160 to determine how much the apples weigh and what the price of the apples is. The expected weight and expected price for the eight apples may then be calculated using information from item database 160.

[0043]

[0047] In block 350, the item analysis system 120 can manage the POS transaction in response to the error condition or discrepancy. The item analysis system 120 can modify the total item weight or the total item price in response to the discrepancy to correct the error condition. Thus, the user's sales list can automatically reflect the appropriate charge for the item or produce being purchased. In the previous example, the user would be charged for the entire bag 80 containing eight apples, rather than being charged for the six apples weighed by the scale 30. In this regard, the amount of loss or shrinkage can be reduced. In another example, if the user intentionally or mistakenly enters a banana code when weighing the apples, the item analysis system 120 can change the banana code entered by the user to an apple code to provide an accurate total item price when the item analysis system 120 determines from the image information that apples are being purchased, not bananas.

[0044]

[0048] For example, consider the scenario shown in Figures 3 and 4 where a user places a bag of apples 80 on the scale 30. The bag 80 may be positioned such that the bag 80 extends beyond the weighing platter 32 of the scale 30. The item analysis system 120 may detect through the image information that the bag 80 has not been weighed accurately because a portion of the bag 80 extends beyond the perimeter 34 onto the counter 26. In response, the item analysis system 120 may determine an expected weight or price of the apples in the entire bag 80 based on the received image. Thus, in this example, because only a portion of the apples in the bag 80 were weighed, the expected weight or price of the items will not be the same as the received item weight or the received item price, and the determined expected item weight or price will be higher because the estimate is for the apples in the entire bag 80. Thus, in block 340, the item analysis system 120 may determine that there is a discrepancy between the item weight and the expected item weight, or between the item price and the expected item price. A discrepancy exists when the expected item weight or expected item price differs from the item weight or item price. When the item analysis system 120 identifies that a discrepancy exists, the item analysis system 120 can manage the POS transaction accordingly. The POS transaction may update the item weight or price, or may change the item code or item identifier, or may require user input or employee intervention. The item analysis system 120 can be programmed to perform other actions, such as generating and / or sending a message in response to the discrepancy, which may be indicated to the POS system 110 and / or a user via the user interface 22, generating and / or sending an alert to store employees and / or the POS system 110, and / or communicating with the item database 160.

[0045]

[0049] 7, a data processor 600 is described that communicates with an item module 690 that receives image input from the vision mesh 74 (FIGS. 2-5). It will be understood that the data processor may be included in any component of the system without departing from the scope of the present disclosure. For example, the data processor may reside in the POS system 10 (FIG. 2) or may be centrally located. The item module 690 may increase the likelihood that the POS data is accurate, e.g., that all items have been properly weighed and scanned, that no items have been lost or stolen, or that each entered item includes the appropriate corresponding stock keeping unit (SKU).

[0046]

[0050] As illustrated, FIG. 7 is a block diagram of an example of a data processing system 600 suitable for use in a system according to an embodiment of the present disclosure. Data processing may occur in any of the devices in the system (or all of the devices, such as each edge camera 70 (FIGS. 2-4)), the checkout terminal 20, or the POS system, without departing from the scope of the present disclosure. As illustrated in FIG. 7, the data processing system 600 includes a user interface 644, such as a keyboard, keypad, touchpad, or voice activation circuit, an I / O data port 646, and a memory 636 in communication with a processor 638. The I / O data port 646 may be used to transfer information between the data processing system 600 and another computer system or network. These components may be conventional components such as those used in many conventional data processing systems and may be configured to operate as described herein. An item module 690, which may include a produce recognition module 692 and an item recognition module 694, processes image input from the mesh network 74 and communicates with the mesh network 74 and the data processing system 600. The item module may also include a known average weight module 696 to aid in processing input from the mesh network 74 .

[0047]

[0051] The foregoing flow logic and / or methods illustrate the functionality and operation of the various services and applications described herein. If embodied in software, each block may represent a module, segment, or portion of code that includes program instructions for implementing the specified logical function(s). The program instructions may be embodied in the form of source code, including human-readable statements written in a programming language, or machine code, including numerical instructions recognizable by a suitable execution system, such as a processor in a computer system or other system. Machine code may be translated from source code, etc. Other suitable types of code include compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc. These examples are not intended to be limiting in this context.

[0048]

[0052] When embodied in hardware, each block may represent one circuit or multiple interconnected circuits for implementing the specified logical function(s). The circuitry may include any of a variety of commercially available processors, including, but not limited to, AMD® Athlon®, Duron®, and Opteron® processors, ARM® application, embedded, and secure processors, IBM® and Motorola® DragonBall® and PowerPC® processors, IBM and Sony® Cell processors, Qualcomm® Snapdragon®, Intel® Celeron®, Core(2)Duo®, Core i3, Core i5, Core i7, Itanium®, Pentium®, Xeon®, Atom®, and XScale® processors, and similar processors. Other types of multi-core processors and other multi-processor architectures may also be used as part of the circuitry. According to some examples, the circuitry may also include an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), and the modules may be implemented as hardware elements of an ASIC or FPGA. Furthermore, embodiments may be provided in the form of a chip, chipset, or package.

[0049]

[0053] Each of the flow logics and / or methods described above shows a specific order of execution, but it is understood that the order of execution may differ from that shown. Also, operations shown sequentially in the flow charts may be performed simultaneously or partially simultaneously. Furthermore, in some embodiments, one or more of the operations may be skipped or omitted. Furthermore, any number of counters, state variables, alert semaphores, or messages may be added to the logic flows or methods described herein, such as for purposes of improving usability, accounting, measuring performance, or providing troubleshooting assistance. It is understood that all such variations are within the scope of the present disclosure. Furthermore, not all operations illustrated in the flow logic or methods may be required for a novel implementation.

[0050]

[0054] If any operation or component described herein is implemented in the form of software, any one of a number of programming languages ​​may be used, such as, for example, C, C++, C#, Objective C, Java, Javascript, Perl, PHP, Visual Basic, Python, Ruby, Delphi, Flash, or other programming languages. A software component is stored in a memory and is executable by a processor. In this respect, the term "executable" refers to a program file in a form that can ultimately be executed by a processor. An example of an executable program may be, for example, a compiled program that can be converted into machine code in a format that can be loaded into a random access portion of memory and executed by a processor, source code expressed in a suitable format such as object code that can be loaded into a random access portion of memory and executed by a processor, or source code that can be interpreted by another executable program to generate instructions in a random access portion of memory to be executed by a processor, etc. An executable program may be stored in any portion or component of memory. In the context of this disclosure, a "computer-readable medium" may be any medium (e.g., memory) capable of containing, storing, or maintaining logic or applications described herein for use by or in connection with an instruction execution system.

[0051]

[0055] Memory is defined herein as an article of manufacture and includes volatile and / or non-volatile memory, removable and / or non-removable memory, erasable and / or non-erasable memory, and writeable and / or re-writable memory, etc. A volatile component is a component that does not retain a data value upon loss of power. A non-volatile component is a component that retains data upon loss of power. Thus, memory may include, for example, random access memory (RAM), read-only memory (ROM), hard disk drives, solid state drives, USB flash drives, memory cards accessed via memory card readers, floppy disks accessed via associated floppy disk drives, optical disks accessed via optical disk drives, magnetic tapes accessed via suitable tape drives, and / or other memory components, or combinations of any two or more of these memory components. Additionally, RAM may include, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM), and other such devices. The ROM may include, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other similar memory device.

[0052]

[0056] The devices described herein may include multiple processors and multiple memories, each operating in parallel processing circuits. In such cases, a local interface, such as a communication bus, may facilitate communication between any two of the multiple processors, between any processor and any memory, or between any two of the memories, etc. The local interface may include additional systems designed to coordinate this communication, including, for example, performing load balancing. The processors may be electrical or of any other available structure.

[0053]

[0057] A method is provided, the method comprising: receiving image information corresponding to the POS transaction, the image information including an item being weighed and associated with the POS transaction; receiving an indication of an item weight or an indication of an item identifier for an item associated with a POS transaction; determining an expected item weight or an expected item identifier for the item based on the image information; determining that a condition exists between the item weight and the expected item weight or between the item identifier and the expected item identifier; and managing the POS transaction based on the state.

[0054]

[0058] The method also includes, considered alone or in combination, detecting a position of the article relative to the weighing area based on the image information; estimating a portion of the article that is outside the weighing area based on the image information; identifying one or more item identifiers for the items based on the image information; communicating the status to the user during the POS transaction; the one or more item identifiers include a location of the item being weighed relative to the weighing area; Determining an expected item weight or an expected item identifier comprises detecting whether the location of the item is outside a weighing area; Determining an expected item weight or an expected item identifier comprises estimating a portion of the item that is outside the weighing area; Determining an expected item weight or expected item identifier comprises estimating an expected item weight or expected item identifier based on the item weight or item identifier and a portion of the item that is inside the weighing area; Managing the POS transaction includes using the estimated expected weight or the estimated expected item identifier; Managing the POS transaction includes using the predicted weight or the estimated predicted item identifier; Determining an expected item weight or expected item identifier comprises referencing a table of weights or prices; Determining the expected item weight or expected item identifier comprises a table of item weights or item identifiers and weights or prices; Determining the expected item weight or expected item identifier comprises a table of weights or prices and a location of the item relative to the weighing area; Determining an expected item weight or expected item price comprises estimating a portion of the item that is outside the weighing area and a table of weights or prices; Managing the POS transaction comprises referencing a table of average weights or prices; The one or more article identifiers can be used to detect whether two dissimilar articles are weighed together; The multiple item identifiers include an item type, a number of items in a set, a number of items in a container, an item position relative to the weighing area, an item name, or an item size.

[0055]

[0059] A POS system is provided. The POS system comprises: A POS system comprising one or more processors, the one or more processors comprising: receiving image information corresponding to the POS transaction, the image information including an item being weighed and associated with the POS transaction; receiving an indication of an item weight or an indication of an item identifier for an item associated with a POS transaction; determining an expected item weight or an expected item identifier for the item based on the image information; determining that a condition exists between the item weight and the expected item weight or between the item identifier and the expected item identifier; and managing the POS transaction based on the state.

[0056]

[0060] A non-transitory computer readable medium (CRM) is provided that, when executed by one or more processors, causes the one or more processors to: receiving image information corresponding to the POS transaction, the image information including an item being weighed and associated with the POS transaction; receiving an indication of an item weight or an indication of an item identifier for an item associated with a POS transaction; determining an expected item weight or an expected item identifier for the item based on the image information; determining that a condition exists between the item weight and the expected item weight or between the item identifier and the expected item identifier; and managing the POS transaction based on the state.

[0057]

[0061] A method is provided, the method comprising: receiving image information corresponding to the POS transaction, the image information including an item associated with the POS transaction; receiving at least one identifying characteristic of an item associated with a POS transaction; identifying at least one image feature corresponding to a feature of an item associated with the POS transaction based on the image information; and verifying that the at least one identifying feature corresponds to the at least one image feature.

[0058]

[0062] A method is provided, the method comprising: receiving first information corresponding to an item weighing procedure as part of a POS transaction, the first information comprising at least one of image data, a weight metric, or an item identifier; indicating at least one of a discrepancy between the measured item weight and an expected item weight or a discrepancy between the item identifier and the expected item identifier based at least in part on the first information, and identifying an error condition associated with the item weighing procedure; and managing an outcome of the POS transaction based at least in part on at least one of the expected item identifier or the expected item weight.

[0059]

[0063] Exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. However, the present disclosure may be embodied in many different forms and should not be construed as being limited to the specific exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0060]

[0064] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations described for a clear understanding of the principles of the present disclosure. Of course, it is not possible to describe every conceivable combination of components and / or methods, but one skilled in the art may recognize that many further combinations and permutations are possible. That is, many variations and modifications can be made to the above-described embodiment(s) without substantially departing from the spirit and principles of the present disclosure. All such modifications and variations are intended to be included within the scope of the present disclosure and protected by the following claims.

[0061]

[0065] In this disclosure, reference is made to a "Point of Sale (POS) system." The term "POS system" refers to any system used to process transactions in a retail store or other business, including self-checkout (SCO) systems where a person can scan, pay, or bag their items. POS systems are used for a variety of purposes, such as completing sales transactions, processing returns, or responding to inquiries. POS systems can be found in a variety of contexts, including traditional brick-and-mortar retail stores, online storefronts, and mobile sales environments. As used herein, it will be understood that a POS system includes more than one checkout system adjacent to or near other similar systems with associated cameras.

[0062]

[0066] As used herein, "edge camera" generally refers to a camera that features on-board storage. Edge cameras can be used as security cameras that store recorded video footage data on internal storage hardware. Recording at the edge thus refers to recording video and storing it either on the camera or on a directly attached storage device, rather than transporting it over a network to a centralized recording mechanism such as a digital video recorder.

[0063]

[0067] As used herein, a "vision mesh network" refers to a network that acts as a single network of multiple cameras, e.g., at least one camera connected to another camera or node via a network that includes a Wi-Fi router. Rather than a single camera, there may be multiple data sources.

[0064]

[0068] As used herein, a computer vision module includes methods for acquiring, processing, analyzing, and understanding digital images, and for extracting high dimensional data from the real world to form decisions. Various embodiments of computer vision modules described herein acquire, process, analyze, and understand images and data from a vision mesh network to provide feedback and operational decisions regarding a customer, a POS system, a POS transaction checkout process, and / or a checkout station.

[0065]

[0069] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting to other embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "including," and / or "including," "having," and / or "having," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. An element described as "for" performing a function, action, and / or operation may be configured or otherwise structured to do so. As used herein, the term "and / or" includes any and all combinations of one or more of the associated list items and may be abbreviated as " / ."

[0066]

[0070] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the various embodiments described herein belong. Furthermore, it will be understood that the terms used herein should be interpreted as having a meaning consistent with the meaning in the context of the present specification and the related art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0067]

[0071] While the foregoing is directed to aspects of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, which scope is determined by the following claims.

Claims

1. receiving image information corresponding to a POS transaction including an item being weighed; receiving an indication of an item weight or an indication of an item identifier for the item associated with the POS transaction; determining an expected item weight or an expected item identifier for the item based on the image information; determining that a predetermined condition exists between an item weight and the expected item weight or between an item identifier and the expected item identifier; managing the POS transaction based on the state; and A method comprising:

2. detecting a position of the item relative to a weighing area based on the image information. The method of claim 1.

3. and estimating a portion of the item that is outside a weighing area based on the image information. The method of claim 1.

4. further comprising identifying one or more item identifiers for the item based on the image information. The method of claim 1.

5. The method of claim 4 , wherein the one or more item identifiers include a location of the item being weighed relative to a weighing area.

6. The method of claim 5 , wherein determining the expected item weight or the expected item identifier comprises detecting whether the location of the item is outside of the weighing area.

7. The method of claim 6 , wherein determining the expected item weight or the expected item identifier comprises estimating a portion of the item that is outside the weighing area.

8. 7. The method of claim 6, wherein determining the expected item weight or the expected item identifier comprises estimating the expected item weight or the expected item identifier based on the item weight or the item identifier and a portion of the item that is inside the weighing area.

9. The method of claim 8 , wherein managing the POS transaction includes using an estimated expected item weight or an estimated expected item identifier.

10. The method of claim 1 , wherein managing the POS transaction includes using the predicted weight or the estimated predicted item identifier.

11. The method of claim 1 , wherein determining the expected item weight or the expected item identifier comprises referencing a table of weights or prices.

12. The method of claim 1 , wherein determining the expected item weight or the expected item identifier comprises a table of the item weight or the item identifier and weights or prices.

13. The method of claim 1 , wherein determining the expected item weight or the expected item identifier comprises a table of weights or prices and a location of the item relative to a weighing area.

14. The method of claim 1 , wherein determining the expected item weight or the expected item identifier comprises estimating the portion of the item that is outside a weighing area and a table of weights or prices.

15. The method of claim 1 , wherein managing the POS transactions comprises referencing a table of average weights or prices.

16. The method of claim 4 , wherein the one or more item identifiers can be used to detect whether two dissimilar items have been weighed together.

17. The method of claim 4 , wherein the one or more item identifiers include a type of item, a number of items in a set, a number of items in a container, a position of the item relative to a weighing area, an item name, or an item size.

18. and communicating the status to a user during the POS transaction. The method of claim 1.

19. 1. A POS system comprising one or more processors, the one or more processors comprising: receiving image information corresponding to a POS transaction including an item being weighed; receiving an indication of an item weight or an indication of an item identifier for the item associated with the POS transaction; determining an expected item weight or an expected item identifier for the item based on the image information; determining that a predetermined condition exists between an item weight and the expected item weight or between an item identifier and the expected item identifier; managing the POS transaction based on the state; and The POS system is configured to:

20. When executed by one or more processors, the one or more processors are receiving image information corresponding to a POS transaction including an item being weighed; receiving an indication of an item weight or an indication of an item identifier for the item associated with the POS transaction; determining an expected item weight or an expected item identifier for the item based on the image information; determining that a predetermined condition exists between an item weight and the expected item weight or between an item identifier and the expected item identifier; managing the POS transaction based on the state; and A non-transitory computer-readable medium having stored thereon computer-executable instructions for causing a

Images