Analyzing sensor data to identify events

The system uses sensors and computer vision to track item transfers by identifying user interactions within a defined volume of interest, addressing inefficiencies in inventory management and enabling seamless payment processing for custom or variable-weight goods.

JP7595767B2Active Publication Date: 2024-12-06AMAZON TECH INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023529063
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-11-30
Filing Date
2021-11-30
Publication Date
2024-12-06
Estimated Expiration
2041-11-30

AI Technical Summary

Technical Problem

Existing systems struggle to accurately identify events, such as the transfer of items between users, within facilities like retail environments, leading to inefficiencies in inventory management and payment processing, especially for custom or variable-weight goods.

Method used

A system utilizing sensors and computer vision algorithms to analyze image data and track user interactions, determining the transfer of items by identifying user hands within a defined volume of interest (VOI) and updating virtual carts accordingly, based on trained classifiers and segmentation maps.

Benefits of technology

Enables accurate and efficient tracking of item transfers, allowing for real-time updates of virtual carts and seamless payment processing, even for custom or variable-weight goods, enhancing inventory management and customer experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595767000001
    Figure 0007595767000001
  • Figure 0007595767000002
    Figure 0007595767000002
  • Figure 0007595767000003
    Figure 0007595767000003
Patent Text Reader

Abstract

The present disclosure is directed to techniques in which a first user (102) in an environment (106) scans a visual indicia, such as a barcode, associated with an item (104) and then hands the item to a second user (110). One or more computing devices (118) may receive an indication of the scan, retrieve image data of the interaction from a camera (116) in the environment (106), identify the user (110) receiving the item, and update a virtual cart (138) associated with the second user (110) to indicate the addition of the item (104).
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. patent application Ser. No. 17 / 106,755, entitled "ANALYZING SENSOR DATA TO IDENTIFY EVENTS," filed Nov. 30, 2020, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Retailers, wholesalers, and other product distributors typically maintain an inventory of various goods that may be ordered, purchased, leased, borrowed, rented, viewed, etc., by business partners or customers. For example, an e-commerce website may maintain inventory in a fulfillment center. When a customer orders a product, the product is selected from the inventory, sent to a packing station, packed, and shipped to the customer. Similarly, a brick-and-mortar store may maintain inventory in a customer-accessible area, such as in a shopping area, where the customer may select products from the inventory and take them to a cashier for purchase, rental, etc. Many brick-and-mortar stores may also maintain inventory in storage areas, fulfillment centers, or other facilities, which may be used to replenish inventory located in the shopping area or to fulfill orders for goods made through other channels (e.g., e-commerce). Other examples of entities that maintain facilities that hold inventory include libraries, museums, rental centers, etc. In each instance, when an item is moved from one location to another, the item is selected from its current location and transferred to the new location, a user may move around the facility, etc. It is often desirable to generate information about events that occur within the facility. [Prior art documents] [Patent documents]

[0003] US10332066 [Brief description of the drawings]

[0004] The detailed description will now be described with reference to the accompanying drawings, in which: In each drawing, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical items or features.

[0005] [Figure 1] 1 illustrates an example architecture in which a first user in an environment scans a visual indicia, such as a barcode, associated with an item and then hands the item to a second user. The architecture also includes a server computing device configured to receive an indication of the scan, retrieve image data of the interaction from a camera in the environment, identify the user receiving the item, and update a virtual cart associated with the second user to indicate the addition of the item. [Figure 2A] Collectively, the figures illustrate an exemplary series of actions in which a first user scans an item and hands the item to a second user, while one or more computing devices analyze image data of an environment near the interaction at the time of the scan to determine a user identifier associated with the second user receiving the item. [Figure 2B] Collectively, the figures illustrate an exemplary series of actions in which a first user scans an item and hands the item to a second user, while one or more computing devices analyze image data of an environment near the interaction at the time of the scan to determine a user identifier associated with the second user receiving the item. [Figure 2C] Collectively, the figures illustrate an exemplary series of actions in which a first user scans an item and hands the item to a second user, while one or more computing devices analyze image data of an environment near the interaction at the time of the scan to determine a user identifier associated with the second user receiving the item. [Figure 3A]1 illustrates an image analysis component that uses machine vision algorithms to generate a segmentation map of a frame of image data. As illustrated, the segmentation map may indicate pixels associated with different objects, such as a background, a user's hand, a product, etc. [Figure 3B] Illustrates an image analysis component that uses the segmentation map and one or more trained classifiers to determine, for each frame of image data, whether the frame contains a hand, and if so, the location of the hand and whether the hand is empty or filled. The image analysis component can use this information to identify a user who acquires an item after it is scanned using a scanning device. [Figure 4A] 10A-10D collectively illustrate a flow diagram of an exemplary process for updating a user's virtual cart data that has received an item scanned by another user using a scanning device. [Figure 4B] 10A-10D collectively illustrate a flow diagram of an exemplary process for updating a user's virtual cart data that has received an item scanned by another user using a scanning device. [Diagram 5] 1 illustrates a flow diagram of another exemplary process for updating virtual cart data for a user receiving a product. [Figure 6] FIG. 1 is a block diagram of an example material handling facility including sensors and an inventory management system configured to use sensor data to generate outputs related to events occurring within the facility. [Figure 7] 1 illustrates a block diagram of one or more servers configured to support the operation of a facility. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The present disclosure is directed to systems and techniques for identifying events occurring within a facility using sensor data generated by sensors within the facility. In one example, the techniques and systems may identify events in response to a first user using a scanning device to scan visual indicia (e.g., a barcode, QR code, etc.) of an item and a second user receiving the item. After identifying the second user receiving the item, the systems and techniques may update a virtual cart associated with the second user to indicate the addition of the item. In some examples, the systems and techniques may be implemented within a "just walk out" retail environment, where a virtual cart for each user is maintained such that the user can retrieve or otherwise receive items and then "just walk out" of the facility such that the cost of the user's virtual cart can be charged to the user's respective pre-stored payment instrument in response to the user exiting the facility.

[0007] In some examples described below, an associate (e.g., employee, etc.) at the establishment may scan visual indicia on an item and then hand the item to a customer at the establishment. In some instances, the item includes a made-to-order item (e.g., customized salad or sandwich, etc.), a variable weight item (e.g., a set amount of seafood or meat, etc.), a variable quantity item (e.g., a set length of quilted fabric), or any other type of item whose cost varies depending on the amount or other parameters (e.g., toppings) of the item that the customer actually receives. Thus, after a customer places an order for one pound of shrimp, one yard of fabric, etc., an associate may package the requested item in the requested amount and use a printing device to print visual indicia associated with the item. For example, the associate may use a scale to measure the amount of the item and use a printer to print a barcode, QR code, etc., that encodes information about the item, such as the item's item identifier, the item's weight / amount, the item's cost, the time of scanning, etc. In some instances, the associate may then affix the visual indicia to the item or to the item's packaging. For example, a printer may print adhesive stickers containing visual indicia, which an interested party may then apply to merchandise or packaging.

[0008] After affixing the visual indicia to the item, the party may then scan the visual indicia using a scanning device. For example, the party may use a barcode scanner, a tablet computing device, or any other device including a camera or other imaging device to identify or otherwise capture information about the visual indicia. The scanning device may generate scan data indicative of the above information, such as the item's identity, the item's weight / quantity, the item's cost, the time of the scan, etc. This scan data may then be transmitted over one or more networks to one or more computing devices that are configured to verify the scan data and then attempt to determine an event associated with the item associated with the scan data. The scan data transmitted to the computing device may include or be accompanied by an identifier of the party signed into the scanning device, an identifier of the scanning device, etc.

[0009] A computing device may receive the scan data and, in response thereto or at a later time, may attempt to determine any events involving the scan data. For example, the computing device may determine an identifier for the scanning device that generated the scan data and may determine a volume of interest (VOI) within the facility associated with the scanning device. That is, each scanning device within the facility may be associated with a respective VOI within the facility, which may include a portion of the environment (e.g., defined in XYZ coordinates) where a customer is likely to interact with (e.g., receive) an item associated with the scan data. For example, the VOI for each scanning device may include an XYZ "box" defined for each scanning device, such as a bounding box that extends a predefined length (e.g., up to the ceiling) over some or all of the countertop on which the scanning device is mounted. In some instances, these VOIs are manually configured for each scanning device within the environment, and in other instances, the VOIs may be determined by analyzing the image data of the scanning devices and the area around these devices using cameras (e.g., overhead cameras) within the facility.

[0010] In either case, upon receiving the scan data, or at some point thereafter, the computing device may determine an identifier for the scanning device used to generate the scan data, which may then be used to determine the VOI of each of the scanning devices. Upon identifying the VOI, the computing device may determine the cameras in the facility that have a field of view (FOV) that includes the VOI. Additionally, the computing device may determine from the scan data the time associated with the scan of the item. Upon identifying one or more cameras within their FOV that have a current VOI, the computing device may retrieve image data generated by this camera (or cameras) closest to the time of the scan. That is, the computing device may retrieve image data that spans the scan of the item (e.g., immediately before and after the scan), image data at the start of the scan or immediately after the scan, etc.

[0011] Upon receiving this image data, the computing device may use one or more trained classifiers to determine any events involving the item associated with the scan. For example, the computing device may first perform a segmentation technique on each frame of image data to identify what is represented in the frame. For example, the computing device may be configured to identify the user, parts of the user (e.g., hands, head, body, arms, etc.), background, countertop, etc., within the frame of image data. These devices may utilize classifiers trained using supervised learning or other techniques to identify predefined objects. In some instances, these classifiers output an indication of pixel values ​​that correspond to different objects, such as an indication that a first pixel in a first location corresponds to the background, a second pixel corresponds to a hand, etc.

[0012] The computing device may use this segmentation map to identify whether one or more hands are present within a spatiotemporal window before and after the scanning of the item. That is, the device may determine that any of the frames of image data that are within a threshold amount of time of the scan include a hand that is within the VOI of the scanning device. If so, the computing device may determine a user identifier of the user whose hand is within the VOI within the threshold amount of time of the scan. In response to identifying this user identifier, the computing device may update a virtual cart associated with the user.

[0013] In some instances, the computing device may make this determination over time, rather than simply based on identifying a hand in a frame of image data. For example, in addition to being trained to identify a user's hand, the classifier may be trained to output a score indicating whether the hand is likely to be filled or empty. Further, the computing device may store an indication of each frame, a first score indicating whether the respective frame includes a hand, and a second score indicating whether the identified hand (if present) is empty or filled. The computing device may also store the location of the identified hand. Some or all of this information may be stored over time, such as the device being able to identify a motion vector indicating how the identified hand moved within the VOI over time.

[0014] For example, after determining a space-time window associated with the scan data, the computing device may analyze the image in space-time by generating feature data associated with a frame of image data and inputting the feature data into one or more trained classifiers. These classifiers may indicate the location of the hand with the image data, an indication of whether the hand is empty or filled (if present), and the location of the hand (if present). In response to identifying an empty hand "entering" the VOI and a filled hand "exiting" the VOI, the computing device may determine that the user associated with the hand has received an item.

[0015] Thus, in response to determining that the user has received the item, the system may invoke or otherwise interact with a location determination component configured to maintain the location of each user identifier within the facility at a given time. That is, when a user enters the facility, the system may assign the user a user identifier (which may, in some cases, be unrelated to personal identification data) and use image data and / or the like to maintain the location of the user identifier within the facility over time. Thus, when the computing device determines that the hand has received the item within a space-time window associated with the scan data, the computing device may determine from the location determination component the user identifier that was present at the location of the item at the time of handover. Upon receiving an indication of the user identifier, the system may update the virtual cart associated with the user identifier to indicate the addition of the item. For example, the user's virtual cart may be updated to indicate that she has received 1.2 pounds of shrimp priced at $9.89.

[0016] In this manner, the technology allows a customer to request a custom or variable weight / size item from an associate, who may have prepared the item, printed visual indicia for the cost of the item, affixed the visual indicia to the item, scanned the visual indicia, and had the item with the requesting user. In response to this, or at some point after this interaction, the systems and technology described herein may analyze image data representative of this interaction to determine which user actually received the item. For example, the systems and technology may analyze a predefined VOI associated with a scanning device within a threshold amount of time to identify the presence of a hand within the VOI, and potentially information about the hand, such as whether it was full or empty, the location of the hand over time, and / or the like. Using this information, it may be determined that the hand, and therefore the user, actually received the item. After making this determination, the systems and technology may then determine which user is associated with the hand, and after making this determination, may update the appropriate user's virtual cart.

[0017] In some cases, the systems and techniques described herein may be implemented in response to a computing device receiving an indication of scan data. That is, in response to a participant (or other user) scanning an item, the techniques may be implemented and, in response, a user identifier associated with the user receiving the item may be determined. In other cases, however, different triggers may cause this determination. For example, the techniques may be implemented in a system that determines the contents of a user's virtual cart in response to the user exiting a facility. In this example, the system may determine a set of candidate users for each potential event in the facility and resolve each event for a particular user in response to the user exiting the environment. For example, if a particular user was near (e.g., within a threshold distance of) the meat counter of the facility when an item was scanned at the meat counter, the user may be marked as a candidate user for the particular event involving the scanned item. When the user exits the store, the system may use the techniques described above and below to determine whether it was indeed the user who received the item, and if so, may update the user's virtual cart at this point. In short, different triggers may cause the execution of the techniques described herein.

[0018] In addition, while the above examples describe using vision-based classifiers to determine the presence (and potentially state and orientation) of a hand within the VOI to determine the user receiving the item, in other examples, one or more other factors may be used to make this determination. For example, a vision algorithm may be used to track an item scanned by a scanning device until it is received by another user within the facility. That is, as the item is scanned, one or more computer vision algorithms may be used to identify the scanned item and track the location (e.g., within the VOI or otherwise) of the scanned item within the image data over time, at least until the item is received by a user different from the user who scanned the item. At this point, the receiving user's virtual cart may be updated.

[0019] Additionally or alternatively, the location of the party (or other user who scanned the item) may be maintained over time. For example, when a party scans an item, one or more computer vision algorithms may be used to identify the party who performed the scan and continue to locate this party over time within the image data until the item is passed on to another user. Again, the virtual cart associated with the receiving user may be updated as a result.

[0020] Additionally, while the above and following discussion includes examples of scanning devices associated with fixed locations, it should be understood that the technology may also be applied to mobile scanning devices, such as an example in which store associates use mobile devices to scan items and hand those items to respective users. For example, when an associate scans an item using a mobile scanning device, the device may provide scan data to one or more computing devices described above. These computing devices may use the identifier of the mobile scanning device to determine the current location of the mobile device and / or associate within the facility. That is, the computing devices may determine the location of the device and / or associate using a tracking component of the system that maintains the current location of customers and associates within the store. This information may then be used to determine one or more cameras (e.g., overhead cameras) that have a FOV of the current location of the mobile scanning device. The computing devices may then obtain image data from these cameras to analyze the VOI around the scanning device. As described above, the VOI may be defined relative to the location of the scanning device. The computing device may then use the techniques introduced above and described in further detail below to identify the customer's hand within the VOI, such as the customer's empty hand entering the VOI and the filled hand exiting the VOI, and the user identifier associated with the hand may then be used to update the corresponding user's virtual cart.

[0021] Finally, while the examples included herein are described with reference to a single item, it should be understood that the technology may also be applied to multiple items. In these instances, an actor may hand a group of items or a container of items (e.g., a bag or box) to a customer after scanning multiple items in succession. Here, the computing device may first determine from the respective scan data that the items were scanned within a threshold amount of time relative to one another. For example, the computing device may determine that the elapsed time between the scan of one item and a subsequent item is less than a threshold amount, and thus the items were scanned in succession. In response to making this determination, the computing device may analyze the VOI for hands using the techniques described above, and associate each of the sequentially scanned items with a determined user identifier associated with the hand. In this way, if an actor scans, for example, five items and then hands the items to a customer (e.g., in a bag), the technology may associate each of the five items with the same customer whose hand is identified in the VOI. Additionally, while the above example describes the computing device determining that a group of items are related based on the items being scanned within a threshold time of each other, in another example, the scanning device may include a participant-selectable control (e.g., an icon) to indicate that multiple items are to be scanned and that the items are for a single customer. Thus, once scan data for multiple items is received by the computing device, the computing device may determine to associate each item with a common user identifier based on the described hand identification and hand tracking techniques.

[0022] The following description describes the use of the technology in a material handling facility. The facilities described herein may include, but are not limited to, warehouses, distribution centers, cross-docking facilities, order fulfillment facilities, packing facilities, shipping facilities, rental facilities, libraries, retail stores, wholesale stores, museums, or other facilities or combinations of facilities for performing one or more functions of material (inventory) handling. In other implementations, the technology described herein may be implemented in other facilities or situations. Specific implementations and embodiments of the present disclosure will now be described more fully below with reference to the accompanying figures, in which various aspects are shown. However, the various aspects may be implemented in many different forms and should not be construed as limited to the implementations described herein. The present disclosure encompasses variations of those embodiments as described herein. Like numbers refer to like elements throughout.

[0023] FIG. 1 illustrates an example architecture 100 in which a first user 102 scans an item 104 in an environment 106 using a scanning device 108. For example, the first user 102 may include a retail establishment associate (e.g., employee, etc.) and may scan a visual indicia associated with the item 104, such as a barcode, a QR code, text, etc., using any type of device capable of generating scan data that encodes or otherwise indicates details of the item. For example, the scanning device 108 may comprise a device configured to read a barcode or a QR code affixed to the item 104. Meanwhile, the item 104 may include a custom or variable weight item in some cases, and thus the first user 102 may generate a visual indicia, such as a barcode, using a scale, a printing device, and / or the like. After physically printing the barcode or the like, the first user may affix (e.g., glue) the visual indicia to the item 104 either before or after scanning the item 104.

[0024] After scanning the item 104 using the scanning device 108, the first user 102 may hand the item 104 to the second user 110, place the item 104 on a counter for the second user 110 to pick up, etc. In either case, the second user 110 reaches his or her hand 112 into a volume of interest (VOI) 114 associated with the scanning device 108 to receive the item 104. The VOI 114 may include a three-dimensional area within the environment 106 associated with the scanning device 108. Although not illustrated, the environment 106 may include multiple scanning devices, each of which may be associated with a respective VOI.

[0025] In some instances, each VOI 114 in the environment 106 may be defined for a respective scanning device, such as the illustrated scanning device 108. For example, when the environment 106 is initially configured with scanning devices and sensors, such as cameras, an association may be stored with each scanning device that is a respective three-dimensional space of the environment 106. This three-dimensional space may include VOIs 114 that may be adjacent to the respective scanning devices 108, may include the scanning devices 108, and / or the like. In one example, the VOI 114 corresponds to the three-dimensional space above a countertop on which the scanning device 108 sits. In another example, the VOI 114 includes a volume defined by a radius centered on the scanning device. Thus, the VOI 114 may include a three-dimensional volume of any shape, such as a sphere, a cube, and the like. Additionally, after an initial VOI is stored in association with a scanner, the VOI may be adjusted over time as interactions occur within the environment. Furthermore, while the above examples describe pre-associating the VOI with the scanner, in other cases the VOI may be dynamically determined as interactions occur within the environment 106.

[0026] As illustrated, the environment 106 may further include one or more cameras, such as camera 116, for generating image data that may be used to identify a user receiving the item, such as identifying a second user 110 receiving the item 104. The environment 106 may include multiple cameras, such as overhead cameras, in-shelf cameras, and / or the like, configured to acquire image data of different and / or overlapping portions of the environment 106. Additionally, an association may be stored between each VOI and one or more cameras having a respective field of view (FOV) that includes some or all of the VOI. For example, assuming that the FOV of the camera 116 includes the VOI 114 of the environment 106, an association may be stored between the illustrated VOI 114 and the illustrated camera 116.

[0027] In response to the scanning device 108 generating the scan data, the scanning device 108 may transmit the scan data to one or more server computing devices 118 via one or more networks 120. The network 120 may represent any combination of one or more wired and / or wireless networks. Meanwhile, the server computing device 118 may be present in the environment, may be present remotely from the environment, and / or may be a combination thereof. As illustrated, the server computing device 118 may include one or more processors 122 and memory 124, which may partially store a location determination component 126, an image analysis component 128, an event determination component 130, and a virtual cart component 132. In addition, memory 124 may store in one or more data stores sensor data 134 received from one or more sensors (e.g., scanning devices, cameras, etc.) in the environment, user data 136 indicating the location of a user identifier within environment 106, virtual cart data (or “cart data”) 138 indicating the contents of each user's virtual cart, and environment data 140 indicating information about sensors and the like in the environment, e.g., the location of the sensors.

[0028] In response to receiving the scan data from the scanning device 108, the server computing device 118 may store the scan data in the sensor data data store 134. In response to receiving this scan data, or in response to another triggering event, such as a second user 110 or another user being detected exiting the environment 106, the event determination component 130 may attempt to determine the outcome of any event involving the scanned item 104. For example, the event determination component may initiate a process to determine personal information of the user who received the item in order to update cart data associated with the identified user to indicate the addition of the item.

[0029] To determine the outcome of an event involving the scanned item 104, the event determination component 130 may direct the image analysis component 128 to analyze image data generated by one or more cameras in the environment having an appropriate VOI within the FOV to identify a user receiving the item 104. Additionally, the event determination component may utilize the location determination component 126, which stores current and past locations of the user identifier within the environment 106 over time, in combination with output from the image analysis component 128 to determine the outcome of the event involving the item 104. After determining the outcome, the event determination component 130 may direct the virtual cart component 132 to update the appropriate virtual cart accordingly.

[0030] First, the image analysis component 128 may receive an indication to analyze image data for a particular VOI in a particular time range. As an example, the event determination component 130 may determine from the scan data where the scan occurred and the time the scan occurred. For example, the scan data received from the scanning device 108 may include the time of the scan and an identifier for the scanning device 108. The event determination component may provide this information to the image analysis component 128 or may otherwise use this information to have the image analysis component analyze the appropriate spatiotemporal window, as described below.

[0031] After receiving a request from the event determination component 130, the image analysis component 128 may determine from the environment data 140 which cameras include the FOV of the VOI 114. That is, the environment data 140 may store an indication of which cameras have which VOIs, or otherwise store an indication of which cameras are used to determine events occurring within a particular VOI. In this example, the image analysis component 128 may determine that the illustrated camera 116 has a FOV that includes the VOI 114. Thus, the image analysis component 128 may retrieve image data from the sensor data data store 134 to run one or more computer vision algorithms on the image data. In some examples, the image analysis component 128 analyzes the image data of the VOI 114 within a time range based at least in part on the time of the scan. For example, the image analysis component 128 may analyze image data starting at the time of the scan and continuing for 30 seconds thereafter, image data starting 15 seconds before the scan and ending one minute later, and / or the like.

[0032] After retrieving the image data, for the determined time range representing the current VOI 114, the image analysis component 128 may use one or more trained classifiers to determine events that occurred within the VOI 114. For example, the trained classifiers may be configured to first receive generated feature data for each frame of the image data and output, for each frame, a segmentation map indicative of predefined objects represented in the respective frame. For example, the image analysis component 128 may utilize the trained classifiers (e.g., via supervised learning) to identify the background, the user, a particular part of the user (e.g., hand, head, arm, body, etc.), one or more products, and / or the like. FIG. 3A, discussed below, illustrates an exemplary segmentation map.

[0033] In addition to generating a segmentation map of individual frames of image data, the image analysis component 128 may utilize one or more trained classifiers configured to use at least the segmentation map to identify events occurring within the VOI 114. For example, the classifiers may be configured to determine whether each frame of image data includes a hand, and if so, the state of the hand, such as "empty" (not holding an item) or "full" (holding an item). In some instances, the trained classifiers may receive feature data generated from individual frames of image data and generate as output a score indicating whether the individual frame includes a hand, the location of any such hand, and a score indicating whether the hand is empty or full. One or more thresholds may be applied to these scores to determine whether each individual frame includes a hand, and if so, whether the hand is empty or full.

[0034] In addition to storing this information for each frame, the image analysis component 128 may determine motion vectors of identified hands in the VOI 114 over time. For example, if the image analysis component 128 determines that an empty hand was detected at a first location in a first frame, an empty hand was detected at a second location in a second subsequent frame, etc., the image analysis component may use this information to determine motion vectors associated with the hand. Additionally, the image analysis component 128 may include one or more trained classifiers configured to determine whether a user received an item, returned an item, etc., based at least in part on these motion vectors and associated information regarding the state of the hand. For example, the image analysis component 128 may have been trained (e.g., using supervised learning) to determine that an empty hand "entering" the VOI 114 and a filled hand "exiting" the VOI 114 represents a user associated with a hand acquiring an item. Thus, the image analysis component 114 may output an indication of "take" or "pick" in this example. In the opposite example, a classifier may be trained to determine that a filled hand entering VOI 114 and an empty hand exiting VOI 114 may represent a return.

[0035] In the illustrated example, the image analysis component 128 may determine (e.g., using one or more trained classifiers) that an empty hand entered the VOI 114 after the time of the scan of the item 104, and that a filled hand subsequently exited the VOI 114. The image analysis component 128 may provide this information and / or related information (e.g., an indication of "take") to the event determination component 130. Additionally, the location determination component 126 may determine the location of the user identifier as the corresponding user moves throughout the environment and store these locations over time as user data 136. For example, when the illustrated second user 110 enters the environment 106, the location determination component 126 may create an identifier associated with this user and store the user's location associated with this user identifier over time. In some instances, the user identifier may be an identifier that does not include personally identifiable information, but rather has no other identifiable tie to the user 110, such that the actual identity of the user 110 is not tracked.

[0036] The event determination component 130 may use this user data 136, in addition to the data received from the image analysis component 128, to determine a user identifier associated with the user 110 who acquired the item 104 (i.e., determine the user 110 associated with the hand identified in the VOI 114). For example, the image analysis component 128 (or the event determination component 128) may have determined that a "picking" of the item 104 occurred at a particular time (e.g., 10:23:55). The location determination component 126 (or the event determination component 130) may determine which user identifier was at the location of the VOI 114 at this particular time and may use this information to determine that the user associated with this user identifier picked up the item 104. In response to this determination, the event determination component 130 may instruct the virtual cart component 132 to update the virtual cart data 138 associated with the particular user identifier, such as the user identifier associated with the user 110. Using the techniques described above, the event determination component can thus instruct the virtual cart component to update the appropriate virtual cart data 138 in response to user 110 receiving item 104 directly from user 102 (e.g., as a hand-to-hand exchange), in response to user 110 taking item 104 from a counter after user 102 places item 104 on the counter, and / or in response to the like.

[0037] 2A-C collectively illustrate an exemplary sequence of operations 200 in which a first user 102, as discussed above in FIG. 1, scans an item 104 and hands the item 104 to a second user 110, while a server computing device 118 analyzes image data of the environment near the interaction at the time of the scan to determine a user identifier associated with the second user 110 who received the item.

[0038] First, the second user 110 or another user may request a particular item, a particular amount of such food, a particular length of fabric, a salad with particular toppings, and / or the like. In response, the first user may prepare a customized item and may print or otherwise generate a physical or digital visual indicia associated with the item, such as a barcode, a QR code, etc. In some instances, the visual indicia may encode information about the item, such as an item identifier, a weight of the item, a length of the item, a quantity of the item, a cost of the item, the time the item was ordered at the time the visual indicia was created, and / or the like. The first user 102 may affix the visual indicia to the item when the visual indicia is physical, or may otherwise associate the visual indicia with the item when the visual indicia is digital or physical.

[0039] After the visual indicia are affixed to or otherwise associated with the item 104, operation 202 represents scanning the item 104 using a scanning device to generate scan data. For example, the first user 102 may use any type of scanning device to scan the visual indicia to generate scan data. As noted above, a system that receives the indication of the scan data may store an association between a particular scanning device and a location within the environment. For example, the system may store an association between a particular VOI and each particular scanning device.

[0040] Act 204 represents transmitting the scan data to one or more computing devices, such as the server computing device 118 described above. In some instances, the scan data includes or is accompanied by additional information, such as an identifier for the scanning device, a time of the scan, and / or the like.

[0041] Act 206 represents generating image data using a camera in the environment. As will be appreciated, in some instances the camera may continually generate this image data to include and identify events in addition to the items scanned in act 202.

[0042] 2B continues illustrating the series of operations 200, including, at operation 208, transmitting the generated image data to one or more computing devices, such as server computing device 118. In some instances, the camera may continually transmit this image data to the computing device, which may analyze the image data to identify events occurring within the environment.

[0043] Operation 210 represents a computing device receiving the scan data determining from the scan data a time associated with the scan and a VOI associated with the scanning device. For example, this operation may include determining a VOI associated with the scanning device stored in the facility data by reading a timestamp of the scan from the scan data and using an identifier of the scanning device as a key. The computing device may then define a spatio-temporal window using the time of the scan and the VOI. For example, as described above, the spatio-temporal window may include a VOI for the spatial portion of the window and a time range of a predefined amount of time (e.g., 10 seconds, 30 seconds, 2 minutes, etc.) for the temporal portion of the window. This spatio-temporal window may be used to analyze the image data to identify a user's hand acquiring an item after scanning.

[0044] Act 212 represents analyzing the VOI from the image data for a threshold amount of time after the time of the scan. For example, this act may include generating feature data associated with the image data and inputting the feature data into a trained classifier configured to identify the user's hand and the state of the user's hand (e.g., full or empty).

[0045] 2C completes the illustration of the series of operations 200, including identifying the user's hand within the VOI within a threshold amount of time from the time of the scan at operation 214. In some instances, the operations include outputting an indication that the trained classifier identified a hand in a frame of image data having a timestamp that is within a defined time range of the spatio-temporal window. In other instances, the operations may include identifying at least one frame within the spatio-temporal window in which the classifier identified an empty hand and at least one subsequent frame within the spatio-temporal window in which the classifier identified a filled hand. In yet other instances, the operations may include generating a motion vector of the identified hand across frames and identifying the empty hand moving further into the VOI and the filled hand moving away from the VOI.

[0046] Operation 216 represents determining a user identifier associated with the user associated with the identified hand. In some instances, this operation may include accessing user data generated by the location determination component to determine which user identifiers were near or at the scanning device and / or VOI during the time of the scan or during a time range defined by the scan.

[0047] Operation 218 represents updating the virtual cart data associated with the user identifier to indicate the addition of the item. In the illustrated example, for example, the user's virtual cart is updated to include the addition of "1.2 pounds of shrimp" at a cost of $12.34. Although this example describes adding an item identifier to the virtual cart, in the event that the user returns an item that has not been scanned, the item identifier may be correspondingly removed from the virtual cart. In these examples, the spatiotemporal window triggered by the item scan may be defined both before and after the scan, and image data generated prior to the scan may be analyzed to identify the return, such as identifying a filled hand entering the VOI and the user's empty hand exiting the VOI.

[0048] FIG. 3A illustrates the image analysis component 128 using a machine vision algorithm to generate a segmentation map 300 of a frame of image data. As shown, the segmentation map 300 may indicate pixels associated with different objects, such as a background, a user's hand, a product, etc. For example, the illustrated segmentation map 300 shows that different regions of a frame of image data are associated with exemplary semantic labels (e.g., "labels"), for example, using one or more trained classifiers 302 of the image analysis component 128. In this example, the semantic labels include background 304, head 306, body 308, arms 310, hands 312, and product (or product in hand) 314, and door 316. Of course, it should be understood that these are merely examples and that any other type of semantic labels may be used. It should also be noted that the classifiers used to generate this exemplary segmentation map 302 may be trained by employing a human user to use computer graphics tools to assign respective semantic labels 304-316 to different regions of the frame. After one or more human users have assigned these semantic labels to a threshold amount of image data, the classifier can be trained to apply the semantic labels to further additional image data.

[0049] In some instances, a first trained classifier of classifiers 302 outputs a segmentation map 300 indicating, for each frame, particular parts of the body (e.g., hands, lower arms, upper arms, head, etc.) and the location of each of these parts. In some instances, this first trained classifier may output a score indicating the likelihood that a particular portion of a frame of image data represents a hand, in addition to the candidate hand contours. The first classifier may also associate each identified hand with an identified head in the image data. This head may be used to determine a user identifier for the user, and thus each hand may be associated with the user identifier.

[0050] 3B illustrates that the image analysis component 128 uses the segmentation map 300 and one or more trained classifiers 302 to determine, for each frame of image data, whether the frame contains a hand, and if so, the location of the hand, and whether it is empty or full. The image analysis component 128 may use this information to identify a user who acquires an item after the item is scanned using a scanning device. For example, the illustrated frames in this example may correspond to frames of image data of the VOI 114 associated with a timestamp that is within a predefined time range of the spatiotemporal window.

[0051] As illustrated, the image analysis component analyzes a first frame 318(1). In this example, the first classifier described above does not identify a hand. However, the first classifier identifies an empty hand in a subsequent frame 318(2). For example, as described above, the classifier may output a score indicating the presence of a hand, a score indicating whether the identified hand is empty or filled, and a location of the identified hand. In addition, the classifier may not only associate the hand with the head, but also associate the hand with a user identifier associated with the head.

[0052] A third exemplary frame 318(3) is determined by the classifier to represent yet another empty hand, while fourth frames 318(4) and 318(5), respectively, are determined by the classifier to represent a filled hand. Additionally, image analysis component 128 may generate one or more motion vectors associated with the hands using the user identifiers and locations of each of the identified hands across frames. For example, image analysis component 128 may use the user identifiers associated with the hands to identify movements of the same hand over time. As illustrated, in this example, image analysis component 128 may identify motion vectors indicating that an empty hand has moved into the VOI and a filled hand has moved away from and / or outside the VOI. Classifier 302 or another classifier may use this information to make a determination that the user associated with the hand did indeed pick up the scanned item.

[0053] 4A-B collectively illustrate a flow diagram of an exemplary process 400 for updating virtual cart data of a user who has received an item scanned by another user using a scanning device. This process 400, as well as other processes discussed herein, may be implemented in hardware, software, or a combination thereof. In the context of software, the described operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more hardware processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. Those skilled in the art will readily appreciate that certain steps or operations illustrated in the figures above may be omitted, combined, or performed in a different order. Any steps or operations may be performed sequentially or in parallel. Furthermore, the order in which the operations are described is not intended to be construed as limiting. Additionally, these processes may be performed by a tote (e.g., a cart, basket, bag, etc.), a server, other computing devices, or a combination thereof.

[0054] Operation 402 represents receiving an indication that a scanner has generated scan data. For example, the event determination component 130 may receive an indication that a particular scanning device in the environment has scanned visual indicia, such as a barcode, a QR code, or the like, associated with an item.

[0055] In operation 404, the event determination component 130, or another component, may determine an item identifier associated with the scanned item. For example, the event determination component 130, or another component, may use the scan data to identify a barcode or the like of the scanned item. In some cases, this operation may occur in response to receiving the scan data, while in other cases, this operation may occur in response to a different trigger, such as a user designated as a candidate user for an event associated with the scan data exiting the environment. Thus, in some cases, this operation may occur after subsequent operations described below.

[0056] Act 406 represents determining a first time associated with the scanner generating the scan data, which may include a time when the scanner generated the scan data, a time when the scanner transmitted the scan data, a time when the computing device received the scan data, and / or the like.

[0057] In operation 408, the image analysis component 128, or another component, may determine a volume of interest (VOI) associated with the scanning device. For example, as described above, the system may store an association between each scanning device in the facility and a respective VOI (e.g., the three-dimensional space of the facility). The image analysis component may determine an identifier for the scanning device that may be included in or associated with the scan data, and may use this scanning device identifier to determine the corresponding VOI.

[0058] Act 410 represents analyzing a first frame of image data that includes the VOI. For example, this act may include image analysis component 128, or another component, identifying a camera whose FOV includes the VOI (e.g., by accessing a data store that stores an association between each camera and the VOI) and receiving from that camera image data generated proximate to the first time determined in act 406. For example, image analysis component 128 may receive image data generated by the camera during a time range starting at, prior to, or immediately after the first time.

[0059] Further, this operation may include analyzing a portion of the image data corresponding to the VOI to determine whether this portion of the image data includes a hand. For example, operation 410 may include suboperation 410(1) in which feature data generated from the image data is input to a classifier. In suboperation 410(2), the classifier may output a first score indicating whether the image data includes a hand and a second score indicating whether the hand is empty or full. In some cases, analyzing the image data includes a two-stage process as described above, namely, first, segmenting each frame of the image data into different predefined objects including the user's hands and head, and second, tracking any identified hand movements across frames. For example, the first classifier may generate a segmentation map that identifies at least one hand and a corresponding head, the latter of which may be used to determine a user identifier for the user. This segmentation information may be output by the first classifier and input to a hand tracking component, for example, to track each hand across frames.

[0060] In this example, the image analysis component 128, or another component, identifies the user's empty hand within the VOI at operation 412. For example, the image analysis component 128, or another component, uses one or more classifiers to determine that an empty hand is present within the VOI after a first time associated with the scan data.

[0061] Operation 414 represents image analysis component 128, or another component, analyzing a second frame of image data including the VOI, the second frame corresponding to a time subsequent to the time associated with the first frame. Again, this operation may include suboperation 414(1) in which feature data generated from the second frame of image data is input to a classifier. At suboperation 414(2), the classifier may output a third score indicative of whether the image data includes a hand and a fourth score indicative of whether the hand is empty or filled.

[0062] 4B continues the illustration of process 400 and includes, at operation 416, identifying a filled hand of the user based on the analysis of the second frame in operation 414. For example, this operation may include receiving an indication from the classifier that the second frame of image data represents a filled hand of the user within the VOI. In some instances, this indication may also indicate that the filled hand is associated with the same user identifier as the empty hand identified in operation 412.

[0063] Finally, operation 418 represents storing the product identifier associated with the product in virtual cart data associated with the user associated with the user identifier. For example, the virtual cart component 132, or another component, may update the appropriate user cart to indicate the addition of the product.

[0064] In some instances, operation 418 may occur in response to determining that a user's hand entered the VOI empty and exited the VOI filled, while in other instances operation 418 may occur in response to identifying the user's hand in the VOI within a time range defined by a first time instant, in response to identifying the user's filled hand in the VOI within the time range, etc. For example, in some instances, a user's virtual cart may be updated in response to identifying a user's hand in the event that a single hand is identified during the time range. However, if multiple hands (corresponding to different user identifiers) are identified within the VOI during the time range, a particular user's virtual cart may be updated in response to identifying the user's empty hand and then identifying the user's filled hand.

[0065] 5 illustrates a flow diagram of another example process 500 for updating virtual cart data for a user receiving an item. Operation 502 represents receiving sensor data generated by a sensor in the environment, the sensor data identifying the item. In some instances, the sensor may comprise a scanning device that generates the scan data in response to scanning visual indicia associated with the item.

[0066] Act 504 represents determining a portion of the environment associated with the sensor. As described above, determining this portion of the environment may include determining a volume of interest (VOI) within the environment and determined relative to the sensor. In some instances, a data store of the described system may store an association between a scanning device and a VOI, an association between a scanning device and a camera with an FOV of the VOI, etc.

[0067] Operation 506 represents receiving image data generated by a camera in the environment, the image data representing a portion of the environment associated with the sensor. For example, this operation may include receiving image data from a camera having a FOV of the VOI. In some instances, the scan data may indicate a first time associated with the sensor data (e.g., the time the sensor generated the sensor data), and thus receiving the image data may include receiving image data from an appropriate camera during a time range based on the first time.

[0068] Operation 508 represents analyzing the image data to identify a user receiving the item. In some instances, this operation may include analyzing image data generated by the camera after the first time and within a threshold amount of time. Additionally, the analysis may include analyzing at least a portion of the image data corresponding to the VOI to identify the user's hand within the VOI. Additionally, this may include analyzing the image data to at least identify, from the first frame of image data, the user's empty hand within the VOI, and from the second frame of image data, the user's filled hand within the VOI. In yet another example, the analysis may include: analyzing at least a portion of a first frame of image data corresponding to the VOI to identify the user's empty hand at a first location within the VOI; analyzing at least a portion of a second frame of image data corresponding to the VOI to identify the user's empty hand at a second location within the VOI; determining a first directional vector based at least in part on the first and second locations; analyzing at least a portion of a third frame of image data corresponding to the VOI to identify the user's filled hand at a third location within the VOI; analyzing at least a portion of a fourth frame of image data corresponding to the VOI to identify the user's filled hand at a fourth location within the VOI; and determining a second directional vector based at least in part on the third and fourth locations. That is, analyzing the image data to identify the user receiving the product may, in some instances, include determining that the user's empty hand has entered the VOI and the user's filled hand has left the VOI.

[0069] Further, as described above, analyzing the image data may include generating a segmentation map to identify one or more hands within the VOI and using this information to determine the user who received the item. For example, this operation may include generating a segmentation map using a first frame of image data, where the segmentation map identifies at least a first set of pixels in the first frame that correspond to the user's hand, inputting first data indicative of the first set of pixels in the first frame that correspond to the user's hand to a trained classifier, and receiving second data as an output of the trained classifier indicating whether the user received the item. Furthermore, in some cases, an item may be identified and tracked within the VOI in addition to the user's hand. In these cases, the item may be identified within the frame of image data and tracked across the frames to identify the item placed in the user's hand. In each of the examples described herein, the user's hand may receive the item from the hand of another user (e.g., an affiliate of the establishment), from a counter where another user placed the item, and / or in any other manner.

[0070] Finally, operation 510 represents updating the virtual cart data associated with the user to indicate the product identifier associated with the product. This operation may include adding information about the product to the appropriate virtual cart, such as the product identifier, the cost of the product, a description of the product, the time the user received the product, and / or the like.

[0071] 6 is a block diagram of an example material handling facility 602 including sensors and an inventory management system configured to use sensor data to generate output related to events occurring within the facility. In some instances, facility 602 corresponds to architecture 100 and / or environment 106 described above.

[0072] However, the following description is merely one illustrative example of an industry and environment in which the technology described herein may be utilized. Material handling facility 602 (or "facility") includes one or more physical structures or areas that may hold one or more items 604(1), 604(2), ..., 604(Q) (generally represented as 604). As used in this disclosure, letters in parentheses, such as "(Q)", indicate an integer result. Items 604 include physical items such as books, medicines, repair parts, electronics, groceries, etc.

[0073] Facility 602 may include one or more areas designated for various functions related to handling inventory. In this illustrative diagram, facility 602 includes a receiving area 606, a storage area 608, and a transition area 610. Receiving area 606 may be configured to receive goods 604 from suppliers or the like for receipt into facility 602. For example, receiving area 606 may include a distribution center where trucks or other cargo transport devices unload goods 604.

[0074] The storage area 608 is configured to store the items 604. The storage area 608 may be organized in a variety of physical configurations. In one embodiment, the storage area 608 may include one or more aisles 612. The aisles 612 may be configured with or defined by inventory locations 614 on one or both sides of the aisles 612. The inventory locations 614 may include one or more of shelves, racks, cases, cabinets, bins, floor locations, or other suitable storage mechanisms for holding or storing the items 604. The inventory locations 614 may be located on another portion of the floor or structure of the facility, or may be movable such that the arrangement of the aisles 612 may be reconfigurable. In some embodiments, the inventory locations 614 may be configured to move independently of an outside vendor. For example, the inventory locations 614 may include powered and motorized racks that are operable by a computing device such that the racks can be moved from one location to another within the facility 602.

[0075] One or more users 616(1), 616(2), ..., 616(U), totes 618(1), 618(2), ..., 618(T) (generally referred to as 618), or other material handling equipment may move within facility 602. For example, users 616 may move about facility 602 to select or place items 604 within various inventory locations 614 and place them on totes 618 to facilitate transport. Each tote 618 is configured to carry or otherwise transport one or more items 604. For example, totes 618 may include baskets, carts, bags, etc. In other embodiments, other intermediaries, such as robots, forklifts, cranes, drones, etc., may move about facility 602 to select, place, or otherwise move items 604.

[0076] One or more sensors 620 may be configured to capture information within facility 602. Sensors 620 within facility 602 may include sensors fixed within the environment (e.g., ceiling mounted cameras) or sensors otherwise carried by users, etc. (e.g., cell phones, tablets, etc.). Sensors 620 may include, but are not limited to, cameras 620(1), weight sensors, radio frequency (RF) receivers, temperature sensors, humidity sensors, vibration sensors, etc. Sensors 620 may be stationary or mobile relative to facility 602. For example, inventory location 614 may include camera 620(1) configured to capture images of the selection or placement of items 604 on a shelf, such as users 616(1) and 616(2) within facility 602. In another example, a floor of facility 602 may include a weight sensor configured to determine the weight of user 616, or other objects on the user.

[0077] During operation of facility 602, sensors 620 may be configured to provide information suitable for tracking how objects are moved or other occurrences within facility 602. For example, a series of images captured by camera 620(1) may show the removal of item 604 from a particular inventory location 614 by one of users 616 and the placement of item 604 at least partially into or within one of totes 618.

[0078] Although the storage area 608 is depicted as having one or more aisles 612, inventory locations 614 for storing items 604, sensors 620, etc., it will be understood that the receiving area 606, the transition area 610, or other areas of the facility 602 may be similarly equipped. Additionally, the arrangement of the various areas within the facility 602 is depicted functionally, rather than diagrammatically. For example, the different receiving areas 606, the storage areas 608, and the transition areas 610 may be dispersed, rather than separated, within the facility 602.

[0079] The facility 602 may include or be coupled to an inventory management system 622, which may implement some or all of the techniques described above with reference to FIGS. 1-5. As described below, the inventory management system 622 may include components of the server 118, illustrated in claim 1 and described above with reference to FIGS. 1-5. For example, the inventory management system may maintain a virtual cart for each user within the facility. The inventory management system may also store records associated with each user indicating the user's personal information, the user's location, and whether the user is eligible to exit the facility with one or more items without performing a manual checkout of the items. The inventory management system may also generate and output notification data to the user indicating whether the user is fully eligible.

[0080] As illustrated, the inventory management system 622 may reside at the facility 602, on a server 118 remote from the facility 602 (e.g., as part of a pre-located server), or a combination thereof. In each instance, the inventory management system 622 is configured to identify interactions and events with and between users 616, devices such as sensors 620, robots, material handling equipment, computing devices, etc., in one or more of the receiving area 606, storage area 608, or transition area 610. As described above, some interactions may further indicate the presence of one or more events 624, or pre-defined activities of interest. For example, the events 624 may include the entry of the user 616 into the facility 602, the stockpiling of the item 604 at the inventory location 614, the selection of the item 604 from the inventory location 614, the return of the item 604 to the inventory location 614, the placement of the item 604 in the tote 618, the movements of the users 616 relative to one another, gestures by the users 616, etc. Other events 624 involving the user 616 may include the user 616 providing authentication information within the facility 602, using a computing device at the facility 602 to verify personal information with the inventory control system 622, etc. Some events 624 may involve one or more other objects within the facility 602. For example, an event 624 may include the movement within the facility 602 of an inventory location 614, such as a counter mounted on wheels. An event 624 may involve one or more of the sensors 620. For example, a change in the operation of the sensor 620, such as a sensor failure, a change in alignment, etc., may be designated as an event 624. Continuing with this example, a movement of the camera 620(1) (such as caused by someone or something hitting the camera 620(1)) (e.g., camera 104) that results in a change in the orientation of the field of view 628 may be designated as an event 624.

[0081] By determining the occurrence of one or more of the events 624, the inventory management system 622 may generate output data 626. This output data 626 may include information about the event 624. For example, if the event 624 includes an item 604 being removed from the inventory location 614, the output data 626 may include an item identifier indicating the particular item 604 that was removed from the inventory location 614 and a user identifier for the user who removed the item.

[0082] The inventory control system 622 may use one or more automated systems to generate the output data 626. For example, the inventory control system may use an artificial neural network, one or more classifiers, or other automated machine learning techniques to process sensor data from one or more sensors 620 to generate the output data 626. For example, the inventory control system may perform some or all of the techniques for generating and utilizing classifiers to identify user activity in image data, as described in detail above. The automated system may operate using probabilistic or non-probabilistic techniques. For example, the automated system may use a Bayesian network. In another example, the automated system may use a support vector machine to generate the output data 626 or the interim results. The automated system may generate confidence level data that provides information indicating the accuracy or reliability with which the output data 626 or the interim data corresponds to the physical world.

[0083] This confidence level data may be generated using a variety of techniques based at least in part on the type of automated system being used. For example, a probabilistic system using a Bayesian network may use a probability assigned to its output as the confidence level. Continuing with the example, the Bayesian network may indicate that there is a 95% probability that the item depicted in the image data corresponds to an item previously stored in memory. This probability may be used as the confidence level for that item, as depicted in the image data.

[0084] In another example, the output from a non-probabilistic technique such as a support vector machine may have a confidence level based on the image data of the product and the distance in the mathematical space that previously stored images of the product were classified in. The greater the distance in this space from a reference point, such as a previously stored image, to the image data captured during the occurrence, the lower the confidence level.

[0085] In yet another example, image data of an object, such as an item 604, a user 616, etc., may be compared to a set of previously stored images. Differences between the image data and the previously stored images may be evaluated, such as differences in shape, color, relative proportions between features in the images, etc. The differences may be expressed in terms of distances in a mathematical space. For example, the color of the object as depicted in the image data and the color of the object as depicted in the previously stored images may be expressed as coordinates in a color space.

[0086] A confidence level may be determined based at least in part on these differences. For example, a user 616 may select an item 604(1), such as a perfume bottle, from an inventory location 614, which is generally cubic in shape. Other items 604 in nearby inventory locations 614 are likely to be primarily spheres. Based on the shape differences between adjacent items (cube vs. sphere) and the shape match with a previously stored image of perfume bottle item 604(1) (cube vs. cube), there is a high confidence level that user 106 picked up perfume bottle item 604(1).

[0087] In some situations, the automated techniques may not be able to generate output data 626 with a confidence level that exceeds a threshold result. For example, the automated techniques may not be able to distinguish which user 616 in a group of users 616 picked up an item 604 from an inventory location 614. In other situations, it may be desirable to provide human verification of the event 624 or human verification of the accuracy of the output data 626. For example, some items 604 may be deemed age-restricted such that they may only be handled by users 616 above a minimum age threshold.

[0088] In instances where human verification is desired, the sensor data associated with the event 624 may be processed to generate query data. This query data may include a subset of the sensor data associated with the event 624. This query data may also include one or more of the one or more interim results or supplemental data as determined by the automated techniques. The subset of sensor data may be determined using information about one or more sensors 620. For example, camera data such as the location of camera 620(1) within facility 602, the orientation of camera 620(1), and the field of view 628 of camera 620(1) may be used to determine whether a particular location within facility 602 is within the field of view 628. The subset of sensor data may include images that may indicate the inventory location 614 or that may indicate that item 604 has been put away. The subset of sensor data may also omit images from other cameras 620(1) that did not have the inventory location 614 within the field of view 628. The field of view 628 may include a scene of a portion within facility 602 around which the sensor 620 may generate sensor data.

[0089] Continuing with the example, the subset of sensor data may include video clips captured by one or more cameras 620(1) having a field of view 628 that includes the item 604. The interim results may include a "best guess" as to which item 604 may have been involved in the event 624. For example, the interim results may include results determined by an automated system that have a confidence level above a minimum threshold.

[0090] The facility 602 may be configured to receive different types of goods 604 from various suppliers and store the goods until a customer orders or retrieves one or more of the goods 604. The general flow of goods 604 through the facility 602 is indicated by arrows in FIG. 2. Specifically, as illustrated in this example, the goods 604 may be retrieved from one or more suppliers, such as manufacturers, distributors, wholesalers, etc., at a receiving area 606. In various implementations, the goods 604 may include manufactured goods, household goods, perishables, or any suitable type of goods 604, depending on the nature of the enterprise operating the facility 602. Receipt of the goods 604 may include one or more events 624 for which the inventory management system 622 may generate output data 626.

[0091] Upon receipt from the supplier at the receiving area 606, the goods 604 may be prepared for storage. For example, the goods 604 may be unpacked or otherwise rearranged. The inventory management system 622 may include one or more software applications executing on a computer system to provide inventory management functions based on events 624 associated with unpacking or rearrangement. These inventory management functions may include maintaining information indicative of type, quantity, condition, cost, location, weight, or any other suitable parameters related to the goods 604. The goods 604 may be stockpiled, managed, or dispensed in terms of countable individual units or multiples, such as packages, cartons, crates, pallets, or other suitable collections. Alternatively, some goods 604, such as bulk products, commodities, etc., may be stored in continuous or arbitrarily divisible quantities that cannot be organized into countable units by nature. Such goods 604 may be managed in terms of measurable quantities, such as length, area, volume, weight, period, duration, or other dimensional characteristics characterized by a unit of measure. Generally speaking, the quantity of item 604 may refer to either a countable number of individual or collective units of item 604, or any measurable amount of item 604, as appropriate.

[0092] After arriving through the receiving area 606, the items 604 may be stored in a storage area 608. In some implementations, similar items 604 may be stored or displayed together in an inventory location 614, such as in a bin, on a shelf, hanging from a pegboard, etc. In this implementation, all items 604 of a given type are stored in one inventory location 614. In other implementations, similar items 604 may be stored in different inventory locations 614. For example, to optimize the search for certain items 604 with low margins and high volume within a large physical facility 602, those items 604 may be stored in several different inventory locations 614 to reduce congestion that may occur at a single inventory location 614. The storage area of ​​the items 604 and their respective inventory locations 614 may include one or more events 624.

[0093] When a customer order specifying one or more items 604 is received or as a user 616 progresses through the facility 602, the corresponding items 604 may be picked or "selected" from the inventory locations 614 that house those items 604. In various embodiments, the selection of items may range from manual selection to fully automated selection. For example, in one embodiment, the user 616 may have a list of desired items 604 and may proceed through the facility 602 to select the items 604 from the inventory locations 614 in the storage area 608 and place those items 604 in a tote 618. In other embodiments, an employee of the facility 602 may select the items 604 using a written or electronic selection list originating from a customer order. These selected items 604 may be placed in a tote 618 as the employee progresses through the facility 602. The selection may include one or more events 624, such as the user 616 moving to the inventory locations 614, removing the items 604 from the inventory locations 614, etc.

[0094] After the items 604 are selected, they may be processed in a transition area 610. The transition area 610 may be any designated area within the facility 602 where the items 604 are transitioned from one location to another or from one entity to another. For example, the transition area 610 may be a packing station within the facility 602. Once the items 604 arrive at the transition area 610, they may be transitioned from the storage area 608 to the packing station. The transitioning may include one or more events 624. Information regarding the transitions may be maintained by the inventory management system 622 using output data 626 associated with the events 624.

[0095] In another example, when the items 604 are leaving the facility 602, the list of items 604 may be obtained and used by the inventory management system 622 to transfer responsibility or control of the items 604 from the facility 602 to another entity. For example, a carrier may accept the items 604 for transportation and the carrier accepts responsibility for the listed items 604. In another example, a customer may purchase or rent the items 604 and remove the items 604 from the facility 602. This purchase or rental may include one or more events 624.

[0096] Inventory management system 622 may access or generate sensor data regarding facility 602 and contents therein, including items 604, users 616, totes 618, etc. This sensor data may be acquired by one or more of sensors 620, data provided by other systems, etc. For example, sensors 620 may include cameras 620(1) configured to acquire image data of a scene within facility 602. This image data may include still images, video, or a combination thereof. This image data may be processed by inventory management system 622 to determine the location of users 616, totes 618, personal information of users 616, etc. As used herein, personal information of a user may refer to a unique identifier of the user (e.g., name, a number associated with the user, a username, etc.), an identifier that distinguishes the user among other users located in the environment, etc.

[0097] The inventory management system 622, or a system coupled thereto, may be configured to identify the user 616 as well as determine other candidate users. In one embodiment, this determination may include comparing the sensor data to previously stored personal information data. For example, the user 616 may be identified by displaying the user's face to a facial recognition system, by presenting a token carrying authentication credentials, by presenting a fingerprint, by scanning a barcode or other type of unique identifier upon entering the facility, etc. The user's 616 personal information may be determined before, during, or after entering the facility 602. Determining the user's 616 personal information may include comparing the sensor data associated with the user 616 in the facility 602 to previously stored user data.

[0098] In some instances, the inventory management system groups users in a facility into respective sessions. That is, the inventory management system 622 may utilize sensor data to determine groups of users that are effectively "together" (e.g., shopping together). In some cases, a particular session may include multiple users who entered the facility 602 together and are likely to navigate the facility together. For example, when a family of two adults and two children enter the facility together, the inventory management system may associate each user with a particular session. Identifying the location of the session in addition to the individual users may help determine the outcome of individual events, given that users within a session may not only individually select, return, or otherwise interact with items, but may also pass items between each other. For example, the child in the above example may select a box of cereal before handing the box to the child's mother, who may place the box in her tote 618. Noting the child and mother as belonging to the same session may increase the chances of successfully adding the box of cereal to the mother's virtual shopping cart.

[0099] By determining the occurrence of one or more events 624 and associated output data 626, the inventory management system 622 may provide one or more services to users 616 of the facility 602. By utilizing one or more human participants to process the inquiry data and then generate response data that may be used to generate the output data 626, the accuracy of the overall system may be improved. The improved accuracy may improve the user experience of the one or more users 616 of the facility 602. In some examples, the output data 626 may be transmitted to one or more servers 118 via a network 630.

[0100] FIG. 7 illustrates a block diagram of one or more servers 118 configured to support the operation of the facility. The servers 118 may be physically present at the facility 602, accessible by a network 630, or a combination of both. The servers 118 do not require end-user knowledge of the physical location and configuration of the systems that deliver the services. Common terms related to the servers 118 may include "on-demand computing," "software as a service (SaaS)," "cloud services," "data center," and the like. The services provided by the servers 118 may be distributed across one or more physical or virtual devices.

[0101] The server 118 may include one or more hardware processors 702 (processors) configured to execute one or more stored instructions. The processors 702 may include one or more cores. The server 118 may include one or more input / output (I / O) interfaces 704 to enable the processor 702, or other parts of the server 118, to communicate with other devices. The I / O interfaces 704 may include Inter-Integrated Circuits (I2C), Serial Peripheral Interface Bus (SPI), Universal Serial Bus (USB) distributed by the USB Implementers Forum, etc.

[0102] The server 118 may also include one or more communication interfaces 706 configured to provide communication between the server 118 and other devices, such as the sensors 620, interface devices, routers, etc. The communication interfaces 706 may include devices configured to couple to a personal area network (PAN), a wired and wireless local area network (LAN), a wired and wireless wide area network (WAN), etc. For example, the communication interfaces 706 may include devices compatible with Ethernet, Wi-Fi™, etc. The server 118 may also include one or more buses or other internal communication hardware or software that enable data transfer between the various modules and components of the server 118.

[0103] The server 118 may also include a power supply 740. The power supply 740 is configured to provide suitable power to operate the components within the server 118.

[0104] The server 118 may further include one or more memories 710. The memory 710 comprises one or more computer readable storage media (CRSM), which may be any one or more of electronic, magnetic, optical, quantum, mechanical, and the like. The memory 710 provides storage of computer readable instructions, data structures, program modules, and other data for operation of the server 118. Although several exemplary functional modules are shown stored within the memory 710, the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC).

[0105] Memory 710 may include at least one operating system (OS) component 712 configured to manage hardware resource devices, such as I / O interface 704, communication interface 708, and provide various services to applications or components executing on processor 702. OS component 712 may implement a variant of the FreeBSD® operating system distributed by the FreeBSD Project, other UNIX® or UNIX-like variants, a variant of the Linux® operating system distributed by Linus Torvalds, the Windows® Server operating system from Microsoft Corporation of Redmond, Washington, USA, or the like.

[0106] One or more of the following components may also be stored within memory 710. These components may run as foreground applications, background tasks, daemons, etc. A communications component 714 may be configured to establish communications with one or more of the sensors 620, one or more of the devices used by the participants, other servers 118, or other devices. These communications may be authenticated, encrypted, etc.

[0107] The memory 710 may store an inventory management system 622. The inventory management system 622 may be configured to provide some or all of the techniques described above with reference to Figures 1-5. For example, the inventory management system 622 may include components to receive scan data, determine events that have occurred with respect to scanned items, update a user's virtual cart, etc.

[0108] Inventory management system 622 may access information stored in one or more data stores 718 in memory 710. Data stores 718 may store information using flat files, databases, linked lists, trees, executable code, scripts, or other data structures. In some implementations, data stores 718, or portions of data stores 718, may be distributed across one or more other devices, including other servers 118, network-attached storage devices, etc. Data stores 718 may include the data stores described above, such as user data 136, environmental data 140, sensor data 134, and cart data 136.

[0109] Data store 718 may also include physical layout data 720. This physical layout data 720 provides a mapping of the physical locations within a physical layout of devices and objects, such as sensors 620, inventory locations 614, etc. The physical layout data 720 may indicate the coordinates within facility 602 of inventory locations 614, sensors 620 within the field of view of that inventory location 614, etc. For example, physical layout data 720 may include camera data including one or more of the location within facility 602 of camera 620(1), the orientation of camera 620(1), an operational status, etc. Continuing the example, physical layout data 720 may indicate the coordinates of camera 620(1), pan and tilt information indicating the direction in which field of view 628 is pointed, whether camera 620(1) is operational or malfunctioning, etc.

[0110] In some implementations, inventory control system 622 may access physical layout data 720 to determine whether a location associated with event 624 is within the field of view 628 of one or more sensors 620. Continuing with the example above, given the location of event 624 within facility 602 and the camera data, inventory control system 622 may determine which camera 620(1) may have generated an image of event 624.

[0111] The product data 722 includes information associated with the products 604. This information may include information indicating one or more inventory locations 614 in which one or more of the products 604 are stored. The product data 722 may also include order data, SKUs or other product identifiers, prices, quantities in stock, weights, expiration dates, images of the products 604, detailed descriptive information, ratings, ratings, etc. The inventory management system 622 may store information associated with its inventory management functions in the product data 722.

[0112] Data store 718 may also include sensor data 134. This sensor data 134 includes information obtained from or based on one or more sensors 620. For example, sensor data 134 may include 3D information about objects in facility 602. As described above, sensors 620 may include camera 620(1), which is configured to obtain one or more images. These images may be stored as image data 726. Image data 726 may include information describing multiple image elements or pixels. Non-image data 728 may include information from other sensors 620, such as input from a microphone, weight sensor, etc.

[0113] User data 730 may also be stored in data store 718. User data 730 may include personal information data, profile information, purchase history, location data, images of users 616, demographic data, etc. Individual users 616, or groups of users 616, may selectively provide user data 730 for use by inventory management system 622. Individual users 616, or groups of users 616, may also permit collection of user data 730 during use of facility 602, or permit access to user data 730 obtained from other systems. For example, users 616 may opt-in to collection of user data 730 to receive enhanced services while using facility 602.

[0114] In some implementations, the user data 730 may include information designating a user 616 for special handling. For example, the user data 730 may indicate that a particular user 616 is associated with an increased number of errors on the output data 626. The inventory management system 622 may be configured to use this information to apply additional scrutiny to events 624 associated with this user 616. For example, events 624 including items 604 having costs or outcomes above a threshold amount may be provided to interested parties for processing, regardless of a determined confidence level in the output data 626, as generated by an automated system.

[0115] The inventory management system 622 may include one or more of the location determination component 124, the identification component 734, the image analysis component 128, the event determination component 130, the virtual cart component 132, and the query component 738, among potentially other components 756.

[0116] The location determination component 124 functions to locate an item or user within the facility's environment so that the inventory management system 622 can assign a particular event to the correct user. That is, the location determination component 124 may assign a unique identifier to a user when the user enters the facility and, with the user's consent, may locate the user throughout the facility 602 for the period the user remains within the facility 602. The location determination component 124 may perform this location determination using sensor data 134, such as image data 726. For example, the location determination component 124 may receive the image data 726 and use facial recognition technology to identify the user from the image. After identifying a particular user within the facility, the location determination component 124 may then locate the user within the image as the user moves throughout the facility 602. Additionally, if the location determination component 124 temporarily "loses" a particular user, the location determination component 124 may again attempt to identify the user within the facility, such as based on facial recognition and / or using other technologies, such as voice recognition.

[0117] Thus, upon receiving an indication of the time and location of the event in question, the location determination component 124 may query the data store 718 to determine which one or more users were at or within a threshold distance of the event's location at the particular time of the event. Additionally, the location determination component 124 may assign different confidence levels to different users, each indicating how likely it is that the corresponding user is in fact the user associated with the event of interest.

[0118] Location determination component 124 may access sensor data 134 to determine this location data for a user and / or item. Location data provides information indicative of a location of an object, such as item 604, user 616, tote 618, etc. The location may be absolute relative to facility 602 or relative to another object or reference point. The term absolute may include latitude, longitude, and elevation relative to a geodetic reference point. The term relative may include a location, such as 25.4 meters (m) along the x-axis and 75.2 m along the y-axis, 5.2 m from inventory location 614 along a 169° heading, as specified by a floor plan of facility 602. For example, location data may indicate that user 616(1) is 25.2 m along aisle 612(1) and standing in front of inventory location 614. In comparison, the relative location may indicate that user 616(1) is 32 cm from tote 618 with a heading of 73° relative to tote 118. The location data may include orientation information, such as which direction user 616 is facing. The orientation may be determined by the relative direction the user's body is facing. In some implementations, the orientation may be relative to the interface device. Continuing with this example, the location data may indicate that user 616(1) is oriented with a heading of 0°, or looking north. In another example, the location data may indicate that user 616 is facing towards the interface device.

[0119] The identification component 734 is configured to identify the object. In one embodiment, the identification component 734 may be configured to identify the item 604. In another embodiment, the identification component 734 may be configured to identify the user 616. For example, the identification component 734 may determine personal information data of the user 616 depicted in the image by processing the image data 726 using facial recognition technology and comparing characteristics in the image data 726 to previously stored results. The identification component 734 may also access data from other sensors 620, such as an RFID reader, an RF receiver, a fingerprint sensor, etc.

[0120] The event determination component 130 is configured to process the sensor data 134 and generate output data 726 using the techniques described above and other techniques. The event determination component 130 may access information stored in a data store 718, including, but not limited to, event description data 742, confidence levels 744, or thresholds 746. In some instances, the event determination component 130 may be configured to implement some or all of the techniques described above with respect to the event determination component 106. For example, the event determination component 130 may be configured to create and utilize an event classifier to identify events (e.g., predefined activities) in the image data, possibly without using other sensor data acquired by other sensors in the environment.

[0121] The event description data 742 includes information indicative of one or more events 624. For example, the event description data 742 may include a predefined profile specifying the movement of the item 604 from the inventory location 614 with the event 624 of “selection”. The event description data 742 may be generated manually or automatically. The event description data 742 may include data indicative of a trigger associated with an event occurring within the facility 602. An event may be determined to occur upon detection of the trigger. For example, sensor data 134, such as a change in weight from a weight sensor 620(6) at the inventory location 614, may trigger detection of an event in which the item 604 is added or removed from the inventory location 614. In another example, the trigger may include an image of a user 616 reaching out towards the inventory location 614. In yet another example, the trigger may include two or more users 616 approaching within a threshold distance of each other.

[0122] The event determination component 130 may process the sensor data 134 using one or more techniques, including, but not limited to, artificial neural networks, classifiers, decision trees, support vector machines, Bayesian networks, and the like. For example, the event determination component 130 may use a decision tree to determine the occurrence of a “selection” event 624 based on the sensor data 134. The event determination component 130 may further use the sensor data 134 to determine one or more interim results 748. The one or more interim results 748 include data associated with the event 624. For example, if the event 624 includes disambiguation of a user 616, the interim results 748 may include a list of personal information of the possible user 616. In another example, if the event 624 includes disambiguation between products 104, the interim results 748 may include a list of possible product identifiers. In some implementations, the interim results 748 may indicate possible actuations. For example, the actions may include the user 616 selecting, placing, moving, or scratching the item 604, providing gesture input, and the like.

[0123] In some implementations, the interim results 748 may be generated by other components. For example, the interim results 748, such as one or more possible identities or locations of the user 616 involved in the event 624, may be generated by the location determination component 124. In another example, the interim results 748, such as items 604 that may have been involved in the event 624, may be generated by the identification component 734.

[0124] The event determination component 130 may be configured to provide a confidence level 744 associated with the determination of the interim result 748. The confidence level 744 provides an indication as to an expected level of accuracy of the interim result 748. For example, a low confidence level 744 may indicate that the interim result 748 has a low probability that it corresponds to the actual environment of the event 624. In comparison, a high confidence level 744 may indicate that the interim result 748 has a high probability that it corresponds to the actual environment of the event 624.

[0125] In some implementations, interim results 748 having a confidence level 744 above a threshold may be considered sufficiently accurate and therefore may be used as output data 626. For example, event determination component 130 may provide interim results 748 indicating three possible products 604(1), 604(2), and 604(3) corresponding to “selection” event 624. The confidence levels 744 associated with possible products 604(1), 604(2), and 604(3) may be 25%, 70%, and 92%, respectively. Continuing with this example, a threshold result may be set such that a confidence level 744 of 90% is considered sufficiently accurate. As a result, event determination component 130 may designate “selection” event 624 as involving product 604(3).

[0126] The query component 738 may be configured to generate query data 750 using at least a portion of the sensor data 134 associated with the event 624. In some implementations, the query data 750 may include one or more of the interim results 748 or the supplemental data 752. The query component 738 may be configured to provide the query data 750 to one or more devices associated with one or more human participants.

[0127] A participant user interface is presented on the participant's respective device. The participant may generate response data 754 by selecting a particular interim result 748, entering new information, indicating that the participant is unable to answer the query, etc.

[0128] The supplemental data 752 includes information associated with the event 624 or information that may be useful in interpreting the sensor data 134. For example, the supplemental data 752 may include a previously stored image of the merchandise 604. In another example, the supplemental data 752 may include one or more image overlays. For example, the image overlays may include graphical user interface elements, such as overlays that depict indicia of an object of interest. These indicia may include highlights, text boxes, arrows, etc., that are superimposed or placed on top of the image data 626 during presentation to an interested party.

[0129] The query component 738 processes the response data 754 provided by one or more participants. This processing may include calculating one or more statistical results associated with the response data 754. For example, the statistical results may include a count of the number of times participants selected a particular interim result 748, a determination of the percentage of participants that selected a particular interim result 748, etc.

[0130] Query component 738 is configured to generate output data 626 based at least in part on response data 754. For example, given that a majority of participants responded with response data 754 indicating that product 604 associated with “select” event 624 was product 604(5), output data 626 may indicate that product 604(5) was selected.

[0131] The query component 738 may be configured to selectively distribute queries to particular participants. For example, some participants may be better suited to answering particular types of queries. Performance data, such as statistical data about the performance of the participants, may be determined by the query component 738 from the response data 754 provided by the participants. For example, information may be maintained indicating the percentage of different queries for which a particular participant selected response data 754 opposite to the majority of participants. In some implementations, test or practice query data 750 with previously known correct answers may be provided to participants for training or quality assurance purposes. The determination of which set of participants to use may be based at least in part on the performance data.

[0132] By using the query component 738, the event determination component 130 may be able to provide a high degree of confidence in the output data 626 accurately representing the event 624. The output data 626 generated by the query component 738 from the response data 754 may again be used to further train the automated systems used by the inventory control system 622. For example, the sensor data 134 and the output data 626 may be provided to one or more of the components of the inventory control system 622 for training process improvements based on the response data 754. Continuing with this example, this information may be provided to an artificial neural network, a Bayesian network, or the like, to further train these systems, thereby improving the confidence level 744 and interim results 748 generated in the future for the same or similar inputs. Finally, as FIG. 7 illustrates, the server 118 may store and / or utilize other data 758.

[0133] The embodiments may be provided as a software program or computer program product, which includes a non-transitory computer-readable storage medium having stored thereon instructions (in compressed or uncompressed form) that can be used to program a computer (or other electronic device) to perform the processes or methods described herein. The computer-readable storage medium may be one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, and the like. For example, the computer-readable storage medium may include, but is not limited to, a hard drive, a floppy disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a magnetic or optical card, a semiconductor memory device, or other type of physical medium suitable for storing electronic instructions. Furthermore, the embodiments may also be provided as a computer program product that includes a transitory machine-readable signal (in compressed or uncompressed form). Examples of machine-readable signals include signals, whether modulated with a carrier or not, that can be configured to be accessed by a computer system or machine that hosts or executes a computer program, including, but not limited to, signals transmitted over one or more networks. For example, a transitory machine-readable signal may include the transmission of software over the Internet.

[0134] Separate instances of these programs may run on or be distributed across any number of separate computer systems. Thus, although particular steps may be described as being performed by particular devices, software programs, processes, or entities, this need not be the case, and various alternative implementations will be appreciated by those skilled in the art.

[0135] Moreover, those skilled in the art will readily recognize that the techniques described above can be utilized in a variety of devices, environments, and contexts. Although the present subject matter has been described in language specific to structural features or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0136] Although the above invention has been described with reference to specific examples, it should be understood that the scope of the invention is not limited to these specific examples. Since other modifications and variations adapted to suit specific operating requirements and environments will be apparent to those skilled in the art, the invention should not be considered limited to the examples selected for purposes of disclosure, but should extend to all modifications and variations that do not depart from the true concept and scope of the invention.

[0137] Embodiments of the present disclosure can be described in view of the following provisions.

[0138] 1. A system comprising: a camera that generates image data of an environment; a scanner that generates scan data indicative of barcodes associated with items in the environment; and one or more computing devices, the one or more computing devices comprising one or more processors and one or more computer readable media storing computer executable instructions that, when executed, cause the one or more processors to: receive an indication that the scanner generated the scan data; determine a first time associated with the scanner generating the scan data; and determine a volume of interest (VOI) associated with the scanner and within the environment, the VOI being represented in the image data. analyzing a first frame of image data, the first frame being associated with a second time after the first time, and identifying the user's empty hand within the VOI based at least in part on analyzing the first frame; analyzing a second frame of image data, the second frame being associated with a third time after the second time, and identifying the user's filled hand within the VOI based at least in part on analyzing the second frame; determining an item identifier associated with an item from the scan data; and storing the item identifier of the item in association with the user's virtual cart.

[0139] 2. The system of clause 1, wherein analyzing a first frame of image data includes inputting first feature data associated with the first frame into a trained classifier and receiving as an output of the trained classifier a first score indicating whether the first frame includes a hand and a second score indicating whether the hand is empty or filled, and analyzing a second frame of image data includes inputting second feature data associated with the second frame into a trained classifier and receiving as an output of the trained classifier a third score indicating whether the second frame includes a hand and a fourth score indicating whether the hand is empty or filled.

[0140] 3. One or more computing devices include one or more processors and one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts including receiving sensor data generated by a sensor in the environment, the sensor data identifying an item; determining a portion of the environment associated with the sensor; receiving image data generated by a camera in the environment, the image data representing the portion of the environment associated with the sensor; analyzing the image data; identifying a hand represented in the image data based at least in part on the analyzing; determining a user identifier associated with the hand; and updating virtual cart data associated with the user identifier to indicate an item identifier associated with the item.

[0141] 4. One or more computing devices as described in clause 3, wherein receiving sensor data generated by the sensor includes receiving scan data generated by a scanning device that scanned visual indicia associated with the item.

[0142] 5. The one or more computing devices of clause 3, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts including receiving data indicative of a first time at which the sensor generated the sensor data, and wherein the analyzing includes analyzing image data generated by the camera after the first time and within a threshold amount of time of the first time.

[0143] 6. One or more computing devices as described in clause 3, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, and analyzing includes analyzing at least a portion of the image data corresponding to the VOI.

[0144] 7. One or more computing devices as described in clause 3, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, analyzing includes analyzing at least a portion of the image data corresponding to the VOI, and identifying includes identifying the hand after the hand enters the VOI.

[0145] 8. One or more computing devices as described in clause 3, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, analyzing includes analyzing at least a portion of a first frame of image data corresponding to the VOI and analyzing at least a portion of a second frame of image data corresponding to the VOI, and identifying includes identifying an empty hand within the VOI based at least in part on analyzing at least a portion of the first frame and identifying a filled hand within the VOI based at least in part on analyzing at least a portion of the second frame.

[0146] 9. The determining portion includes determining a volume of interest (VOI) within the environment relative to the sensor, and the analyzing includes analyzing at least a portion of a first frame of image data corresponding to the VOI, analyzing at least a portion of a second frame of image data corresponding to the VOI, analyzing at least a portion of a third frame of image data corresponding to the VOI, and analyzing at least a portion of a fourth frame of image data corresponding to the VOI, and the identifying includes identifying an empty hand at a first location within the VOI based at least in part on analyzing at least a portion of the first frame, and identifying an empty hand at a second location within the VOI based at least in part on analyzing at least a portion of the second frame, and analyzing at least a portion of the third frame. and identifying a hand filled at a third location within the VOI based at least in part on analyzing at least a portion of the fourth frame; the one or more computer-readable media further storing computer-executable instructions that, when executed, cause the one or more processors to perform acts including determining a first directional vector based at least in part on the first location and the second location, and determining a second directional vector based at least in part on the third location and the fourth location; and wherein the updating includes updating virtual cart data associated with the user identifier based at least in part on the first directional vector and the second directional vector.

[0147] 10. One or more computing devices as described in clause 3, wherein analyzing includes generating a segmentation map using a first frame of image data, the segmentation map identifying at least a first set of pixels in the first frame that correspond to the user's hand, and inputting first data indicative of the first set of pixels in the first frame that correspond to the user's hand into a trained classifier, and identifying includes receiving second data as an output of the trained classifier indicating that the hand has received the item.

[0148] 11. The one or more computing devices described in clause 3, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, analyzing includes analyzing at least a portion of the image data corresponding to the VOI, and identifying includes identifying a hand within the VOI, and the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform actions including identifying an item within the VOI.

[0149] 12. The method includes receiving sensor data generated by a sensor in the environment, the sensor data identifying an item; determining a portion of the environment associated with the sensor; receiving image data generated by a camera in the environment, the image data representing the portion of the environment associated with the sensor; analyzing the image data; identifying a hand represented in the image data based at least in part on the analyzing; determining a user identifier associated with the hand; and updating virtual cart data associated with the user identifier to indicate an item identifier associated with the item.

[0150] 13. The method of clause 12, wherein receiving sensor data generated by the sensor includes receiving scan data generated by a scanning device that has scanned visual indicia associated with the item.

[0151] 14. The method of claim 12, further comprising receiving data indicative of a first time at which the sensor generated the sensor data, and wherein analyzing comprises analyzing the image data generated by the camera after the first time and within a threshold amount of time of the first time.

[0152] 15. The method of claim 12, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, and analyzing includes analyzing at least a portion of the image data corresponding to the VOI.

[0153] 16. The method of claim 12, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, analyzing includes analyzing at least a portion of the image data corresponding to the VOI, and identifying includes identifying the hand after the hand enters the VOI.

[0154] 17. The method of clause 12, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, analyzing includes analyzing at least a portion of a first frame of image data corresponding to the VOI and analyzing at least a portion of a second frame of image data corresponding to the VOI, and identifying includes identifying an empty hand within the VOI based at least in part on analyzing at least a portion of the first frame and identifying a filled hand within the VOI based at least in part on analyzing at least a portion of the second frame.

[0155] 18. The determining portion includes determining a volume of interest (VOI) within the environment relative to the sensor, the analyzing includes analyzing at least a portion of a first frame of image data corresponding to the VOI, analyzing at least a portion of a second frame of image data corresponding to the VOI, analyzing at least a portion of a third frame of image data corresponding to the VOI, and analyzing at least a portion of a fourth frame of image data corresponding to the VOI, and the identifying includes identifying an empty hand at a first location within the VOI based at least in part on analyzing at least a portion of the first frame, and identifying an empty hand at a second location within the VOI based at least in part on analyzing at least a portion of the second frame, and analyzing at least a portion of the third frame. and identifying a hand filled at a fourth location within the VOI based at least in part on analyzing at least a portion of the fourth frame; the one or more computer-readable media further storing computer-executable instructions that, when executed, cause the one or more processors to perform acts including determining a first directional vector based at least in part on the first location and the second location, and determining a second directional vector based at least in part on the third location and the fourth location; and wherein the updating includes updating virtual cart data associated with the user identifier based at least in part on the first directional vector and the second directional vector.

[0156] 19. The method of clause 12, wherein analyzing includes generating a segmentation map using the first frame of image data, the segmentation map identifying at least a first set of pixels in the first frame that correspond to the user's hand; generating includes inputting first data indicative of the first set of pixels in the first frame that correspond to the user's hand into a trained classifier; and identifying includes receiving second data as an output of the trained classifier indicating that the hand has received the item.

[0157] 20. The method of clause 12, wherein determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor, analyzing includes analyzing at least a portion of the image data corresponding to the VOI, and identifying includes identifying a hand within the VOI, and the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform actions including identifying an item within the VOI. ?

Claims

1. One or more computing devices, one or more processors; one or more computer-readable media storing computer-executable instructions; The computer-executable instructions, when executed, cause the one or more processors to: Receiving sensor data generated by sensors in the environment and identifying an item; determining a portion of the environment associated with the sensor by accessing a stored association of the portion of the environment with a sensor in the environment; determining a first camera from a plurality of cameras in the environment having a field of view (FOV) that includes a portion of the environment associated with the sensor; receiving image data representing the portion of the environment generated by the first camera within the environment and associated with the sensor; analyzing the image data; and identifying a hand represented in the image data based at least in part on said analyzing; and determining a user identifier associated with the hand by accessing user data indicative of a user's location within the environment stored over time; updating stored virtual cart data associated with the user identifier, the virtual cart data including a product identifier associated with the product, to add the product identifier associated with the product; and one or more computing devices configured to perform actions including:

2. 10. The one or more computing devices of claim 1 , wherein the receiving of the sensor data generated by the sensor comprises receiving scan data generated by a scanning device that scanned visual indicia associated with the item.

3. The one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts including receiving data indicative of a first time that the sensor generated the sensor data; 2. The one or more computing devices of claim 1, wherein the analyzing comprises analyzing image data generated by the first camera after the first time and within a predefined time of the first time.

4. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; The one or more computing devices of claim 1 , wherein the analyzing comprises analyzing at least a portion of the image data that corresponds to the VOI.

5. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; said analyzing includes analyzing at least a portion of the image data corresponding to the VOI; The one or more computing devices of claim 1 , wherein the identifying comprises identifying the hand after the hand enters the VOI.

6. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; The analyzing step comprises: analyzing at least a portion of the first frame of image data corresponding to the VOI; analyzing at least a portion of the second frame of image data corresponding to the VOI; The identifying step comprises: identifying an empty hand within the VOI based at least in part on the analyzing at least the portion of the first frame; and identifying a hand holding the item within the VOI based at least in part on the analyzing at least the portion of the second frame; and 10. The one or more computing devices of claim 1 .

7. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; The analyzing step comprises: analyzing at least a portion of the first frame of image data corresponding to the VOI; analyzing at least a portion of the second frame of image data corresponding to the VOI; analyzing at least a portion of the third frame of image data corresponding to the VOI; analyzing at least a portion of the fourth frame of image data corresponding to the VOI; The identifying step comprises: identifying an empty hand at a first location within the VOI based at least in part on the analyzing at least the portion of the first frame; and identifying an empty hand at a second location within the VOI based at least in part on the analyzing at least the portion of the second frame; and identifying a hand holding the item at a third location within the VOI based at least in part on the analyzing at least the portion of the third frame; and and identifying a hand holding the item at a fourth location within the VOI based at least in part on the analyzing at least the portion of the fourth frame; The one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to: determining a first direction vector based at least in part on the first location and the second location; determining a second directional vector based at least in part on the third location and the fourth location; 2. The one or more computing devices of claim 1, wherein the updating comprises updating the virtual cart data associated with the user identifier based at least in part on the first directional vector and the second directional vector.

8. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; said analyzing includes analyzing at least a portion of the image data corresponding to the VOI; said identifying includes identifying the hand within the VOI; 10. The one or more computing devices of claim 1 , wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform actions including identifying the product in the VOI.

9. 1. A method comprising: Receiving sensor data generated by sensors in the environment and identifying an item; determining a portion of the environment associated with the sensor by accessing a stored association of the portion of the environment with a sensor in the environment; determining a first camera from a plurality of cameras in the environment having a field of view (FOV) that includes a portion of the environment associated with the sensor; receiving image data representing the portion of the environment generated by the first camera within the environment and associated with the sensor; analyzing the image data; and identifying a hand represented in the image data based at least in part on said analyzing; and determining a user identifier associated with the hand by accessing user data indicative of a user's location within the environment stored over time; and updating virtual cart data stored in memory associated with the user identifier, the virtual cart data including a product identifier associated with the product, to add the product identifier associated with the product.

10. 10. The method of claim 9, wherein the receiving the sensor data generated by the sensor comprises receiving scan data generated by a scanning device that scanned visual indicia associated with the item.

11. 10. The method of claim 9, further comprising receiving data indicative of a first time at which the sensor generated the sensor data, and wherein the analyzing comprises analyzing image data generated by the camera after the first time and within a predefined time of the first time.

12. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; The method of claim 9 , wherein the analyzing comprises analyzing at least a portion of the image data that corresponds to the VOI.

13. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; said analyzing includes analyzing at least a portion of the image data corresponding to the VOI; The method of claim 9 , wherein the identifying comprises identifying the hand after the hand enters the VOI.

14. determining the portion includes determining a volume of interest (VOI) within the environment relative to the sensor; The analyzing step comprises: analyzing at least a portion of the first frame of image data corresponding to the VOI; analyzing at least a portion of the second frame of image data corresponding to the VOI; The identifying step comprises: identifying an empty hand within the VOI based at least in part on the analyzing at least the portion of the first frame; and and identifying a hand holding the item in the VOI based at least in part on the analyzing at least the portion of the second frame.

Citation Information

Patent Citations

  • System, method and program for managing commodities

    JP2018206372A