System and method for identifying second item based on association with first item

Abstract

To provide a system and method for identifying a second item based on an association with a first item.SOLUTION: An item tracking system comprises a plurality of cameras, a memory storing associations between item identifiers of respective items, and a processor. The processor is configured to capture a plurality of first images of a first item and identify a first item identifier of the first item based on the first images. The processor captures a plurality of second images of a second item, generates a cropped image of the second item from each second image, and identifies an item identifier for each cropped image. Based on the associations stored in the memory, the processor determines that an association exists between the first item identifier of the first item and a second item identifier, and assigns the second item identifier to the second item when at least one of the item identifiers corresponding to the cropped images is the second item identifier.SELECTED DRAWING: Figure 29

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. patent application Ser. No. 17 / 455,903, entitled “ITEM LOCATION DETECTION USING HOMOGRAPHIES,” filed Nov. 19, 2021, which is a continuation-in-part of U.S. patent application Ser. No. 17 / 362,261, entitled “ITEM IDENTIFICATION USING DIGITAL IMAGE PROCESSING,” filed June 29, 2021 by Sailesh Bharathwaaj Krishnamurthy et al., both of which are incorporated herein by reference.

[0002] Technical Field FIELD OF THE DISCLOSURE This disclosure relates generally to digital image processing, and more particularly to systems and methods for identifying a second item based on its relationship to a first item. [Background technology]

[0003] Identifying and tracking objects in space presents several technical challenges. For example, when an image contains multiple items, it is computationally intensive to identify various features of the items that can be used to later identify the items in the image. This process may involve identifying individual items in the image and then comparing the features of the items to all items in a database that may contain thousands of items. In addition to being computationally intensive, this process requires a significant amount of time, making it incompatible with real-time applications. This problem becomes more difficult to solve when trying to identify and track multiple items simultaneously. Summary of the Invention

[0004] The disclosed system provides a technical solution to the above technical problems by using a combination of cameras and three-dimensional (3D) sensors to identify and track items placed on a platform. The disclosed system provides several practical applications and technical advantages, including a process of selecting a combination of cameras on an imaging device to capture images of items placed on the platform, identifying items placed on the platform, and assigning the items to users. Requiring a user to scan or manually identify an item creates a bottleneck in the system's ability to quickly identify items. In contrast, the disclosed process can identify items from images of the items and assign items to users without the user having to scan or identify the items. This process provides practical applications of image detection and tracking by improving the system's ability to quickly identify multiple items. These practical applications not only improve the system's ability to identify items, but also improve the underlying network and devices within the network. For example, the disclosed process reduces the time it takes to identify items and assign items to users, increasing the throughput of the image detection process and allowing the system to serve more users. In other words, this process improves hardware utilization without requiring additional hardware resources, increases the number of hardware resources available for other processes, and increases system throughput. Furthermore, these technical improvements enable the expansion of item identification and tracking capabilities described herein.

[0005] In one embodiment, the item tracking system includes an item tracking device configured to detect a trigger event on a platform of the imaging device. The trigger event may correspond to when a user approaches or interacts with the imaging device by placing an item on the platform. The item tracking device is configured to capture depth images of the items on the platform using a 3D sensor and determine an object pose of each item on the platform based on the depth images. The pose corresponds to the location and orientation of the item relative to the platform. The item tracking device is further configured to identify one or more cameras from among a plurality of cameras on the imaging device based on the object pose of each item on the platform. This process allows the item tracking device to select a camera that has the best view of the items on the platform, reducing the number of images processed to identify the items. The item tracking device is further configured to capture images of the items on the platform using the identified cameras and identify the items in the images based on features of the items. The item tracking device is further configured to identify a user associated with the identified items on the platform, identify an account associated with the user, and add the items to the account associated with the user.

[0006] In another embodiment, an item tracking system includes an item tracking device configured to capture a first overhead depth image of a platform using a 3D sensor at a first time instance and capture a second overhead depth image of a first object using the 3D sensor at a second time instance. The item tracking device is further configured to determine that in the second overhead depth image a first portion of the first object is within the region of interest and a second portion of the first object is outside the region of interest. The item tracking device is further configured to capture a third overhead depth image of a second object placed on the platform using the 3D sensor at a third time instance. The item tracking device is further configured to capture a first image of the second object using the camera in response to determining that the first object is outside the region of interest and the second object is within the region of interest relative to the platform.

[0007] In another embodiment, an item tracking system includes an item tracking device configured to identify a first pixel location within a first plurality of pixels corresponding to an item in a first image and apply a first homography to the first pixel location to determine a first (x,y) coordinate. The item tracking device is further configured to identify a second pixel location within a second plurality of pixels corresponding to the item in a second image and apply a second homography to the second pixel location to determine a second (x,y) coordinate. The item tracking device is further configured to determine that a distance between the first (x,y) coordinate and the second (x,y) coordinate is less than or equal to a distance threshold, associate the first plurality of pixels and the second plurality of pixels with a cluster of the item, and output the first plurality of pixels and the second plurality of pixels.

[0008] In another embodiment, an item tracking system includes an item tracking device configured to detect a trigger event corresponding to a user placing a first item on a platform, capture a first image of the first item on the platform using a camera, and input the first image into a machine learning model configured to output a first encoded vector based on features of the first item present in the first image. The item tracking device is further configured to identify a second encoded vector in the encoded vector library that best matches the first encoded vector, and identify a first item identifier associated with the second encoded vector in the encoded vector library. The item tracking device is further configured to identify a user, identify an account associated with the user, and associate the first item identifier with the user's account.

[0009] In another embodiment, an item tracking system includes an item tracking device configured to receive a first encoded vector and receive one or more feature descriptors of a first object. The item tracking device is further configured to remove one or more encoded vectors from the encoded vector library that are not associated with the one or more feature descriptors and identify a second encoded vector in the encoded vector library that best matches the first encoded vector based on the numerical values ​​in the first encoded vector. The item tracking device is further configured to identify a first item identifier associated with the second encoded vector in the encoded vector library and output the first item identifier.

[0010] In another embodiment, an item tracking system includes an item tracking device configured to capture a first image of an item on a platform using a camera and determine a first number of pixels in the first image that correspond to the item. The item tracking device is further configured to capture a first depth image of the item on the platform using a three-dimensional (3D) sensor and determine a second number of pixels in the first depth image that correspond to the item. The item tracking device is further configured to determine that a difference between the first number of pixels in the first image and the second number of pixels in the first depth image is less than a difference threshold, extract a plurality of pixels from the first image that correspond to the item in the first image to generate a second image, and output the second image.

[0011] In another embodiment, an item tracking system includes an item tracking device configured to receive first point cloud data for a first item, identify a first plurality of data points for a first object in the first point cloud data, and extract the first plurality of data points from the first point cloud data. The item tracking device is further configured to receive second point cloud data for the first item, identify a second plurality of data points for the first object in the second point cloud data, and extract the second plurality of data points from the second point cloud data. The item tracking device is further configured to merge the first plurality of data points and the second plurality of data points to generate combined point cloud data, and determine dimensions of the first object based on the combined point cloud data.

[0012] Certain embodiments of the present disclosure may include some, all, or none of these advantages. These advantages and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. [Brief description of the drawings]

[0013] For a more complete understanding of this disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, where like reference numerals represent like parts. [Figure 1] FIG. 1 is a schematic diagram of one embodiment of an item tracking system configured to use digital image processing. [Figure 2A] FIG. 1 is a perspective view of one embodiment of an imaging device. [Figure 2B] 1 is a perspective view of another embodiment of an imaging device including an enclosure. [Figure 2C] 1 is a perspective view of another embodiment of an imaging device including an open enclosure. [Diagram 3] 1 is a flow chart of one embodiment of an item tracking process of the item tracking system. [Figure 4] 1 is an example comparison of depth images of an imaging device platform before and after an item is placed on the platform. [Figure 5A] 1 is an example of an image of an item. [Figure 5B] 1 is an example of an image of an item. [Figure 5C] 1 is an example of an image of an item. [Figure 6] 1 is an embodiment of an item tracking device configured for an item tracking system. [Figure 7] 1 is a flow chart of one embodiment of a hand detection process for triggering item identification. [Figure 8A] 1 is an example of an overhead depth image of the platform of an imaging device without an item being placed on the platform. [Figure 8B] 1 is an example of an overhead depth image of an imaging device platform when a hand is on the platform. [Figure 8C] 1 is an example of an overhead image of an imaging device platform when a hand is on the platform. [Figure 8D] 1 is an example of an overhead depth image of an imaging device platform with an item placed on the platform. [Figure 9]1 is a flow chart of one embodiment of an image cropping process for item identification. [Figure 10A] 1 is an example of a depth image of an item on the platform of an imaging device and a region of interest from an image capturing the item. [Figure 10B] 1 is an example of a region of interest from an image capturing an item superimposed onto a depth image of the item. [Figure 10C] 4 is another example of a depth image of an item on the platform of an imaging device and a region of interest from an image capturing the item. [Figure 10D] 1 is another example of a region of interest from an image capturing an item superimposed onto a depth image of the item. [Figure 11] 1 is a flow diagram of one embodiment of an item location process. [Figure 12A] 1 is an example of a homography for an item tracking system. [Figure 12B] 1 is an example of an inverse homography for a tracking system. [Figure 13A] 2 is an example of a camera captured image of a region of interest of an item on the platform of an imaging device. [Figure 13B] 4 is another example of a camera captured image of a region of interest of an item on the platform of an imaging device. [Figure 13C] 4 is another example of a camera captured image of a region of interest of an item on the platform of an imaging device. [Figure 14] 1 is an example of an overhead view of a platform of an imaging device including an item cluster during item location detection. [Figure 15] 1 is a flowchart of one embodiment of a search space reduction process for an encoded vector library. [Figure 16] This is an example of an encoded vector library. [Figure 17] FIG. 1 is an example of a process for generating a similarity vector between an encoded vector and an entry in an encoded vector library. [Figure 18] 1 is a flow chart of an embodiment of a process for determining item dimensions using point cloud information. [Figure 19] 1 is an example of an overhead view of point cloud data from a 3D sensor of an item on an imaging device platform. [Figure 20] 1 is an example of a perspective view of point cloud data from a 3D sensor of an item on the platform of an imaging device. [Figure 21] FIG. 13 is another example of a perspective view of point cloud data from a 3D sensor of an item on the platform of the imaging device. [Figure 22] An example of combined point cloud data of an item from multiple 3D sensors. [Diagram 23] 1 is a flow chart of one embodiment of a process for tracking items using encoded vectors. [Figure 24] 1 is an example of a graphical user interface displaying detected items on an imaging device platform. [Diagram 25] 1 is another example of a graphical user interface for displaying detected items on an imaging device platform. [Figure 26] 1 is an example of a graphical user interface displaying a confirmation message for deleting an item from an imaging device platform. [Figure 27] FIG. 1 illustrates identifying a second item based on relevance to a first item, in accordance with one or more embodiments of the present disclosure. [Figure 28] FIG. 1 illustrates an association between item identifiers in accordance with one or more embodiments of the present disclosure. [Figure 29] 1 is a flowchart of an example method for identifying a second item based on a relationship with a first item, in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] System Overview FIG. 1 is a schematic diagram of an embodiment of an item tracking system 100 configured to use digital image processing. The item tracking system 100 can use digital image processing to identify an item 204 placed on a platform 202 of an imaging device 102 and assign the item 204 to a particular user. This process allows a user to retrieve an item 204 from a space without scanning or manually identifying the item 204 they wish to retrieve. In one embodiment, the item tracking system 100 can be installed in a space (e.g., a store) such that a shopper does not need to engage in a traditional checkout process. Although this disclosure uses the example of a store, this disclosure contemplates that the item tracking system 100 can be installed and used in any type of physical space (e.g., a room, an office, an outdoor stand, a shopping mall, a supermarket, a convenience store, a pop-up store, a warehouse, a storage center, an amusement park, an airport, an office building, etc.). As an example, the space can be a store that includes a plurality of items 204 available for purchase. The item tracking system 100 may be installed in a store so that shoppers do not have to engage in a traditional checkout process to purchase items from the store. In this example, the store may be a convenience store or grocery store. In other examples, the store may not be a physical building, but rather a physical space or environment in which shoppers can shop. For example, the store may be a "grab and go" pantry at an airport, a kiosk in an office building, an outdoor market at a park, etc. As another example, the space may be a warehouse or supply room that contains multiple items 204 that users can use or rent. In this example, the item tracking system 100 may be installed to allow users to check out parts or equipment themselves. In other examples, the item tracking system 100 may be used for other suitable applications.

[0015] In one embodiment, the item tracking system 100 is comprised of one or more imaging devices 102 and item tracking devices 104 in signal communication with each other via a network 106. The network 106 enables communication between the various components of the item tracking system 100. This disclosure contemplates the network 106 to be any suitable network operable to facilitate communication between the components of the item tracking system 100. The network 106 may include any interconnection system capable of transmitting audio, video, signals, data, messages, or any combination thereof. The network 106 may include any network that is capable of transmitting audio, video, signals, data, messages, or any combination thereof. The network 106 may include any network that is capable of transmitting audio, video, signals, data, messages, or any combination thereof. The network 106 may include any network that is capable of transmitting audio, video, signals, data, messages, or any combination thereof. The network 106 may include any network that is capable of transmitting audio, video, signals, data, messages, or any combination thereof. This may include all or a portion of a Mobile Telecommunications System (MTS) network, a peer-to-peer (P2P) network, a Bluetooth network, a near field communication (NFC) network, a Zigbee network, and / or other suitable networks.

[0016] Imaging Equipment The imaging device 102 is generally configured to capture images 122 and depth images 124 of an item 204 placed on a platform 202 of the imaging device 102. In one embodiment, the imaging device 102 includes one or more cameras 108, one or more three-dimensional (3D) sensors 110, and one or more weight sensors 112. Additional information regarding the hardware configuration of the imaging device 102 is provided in Figures 2A-2C.

[0017] The camera 108 and the 3D sensor 110 are configured to capture an image 122 and a depth image 124, respectively, of at least a portion of the platform 202. The camera 108 is configured to capture an image 122 (e.g., an RGB image) of the item 204. Examples of the camera 108 include, but are not limited to, a camera, a video camera, a webcam, and a printed circuit board (PCB) camera. The 3D sensor 110 is configured to capture a depth image 124, such as a depth map or point cloud data of the item 204. The depth image 124 is composed of a number of pixels. Each pixel in the depth image 124 is composed of depth information that identifies a distance between the 3D sensor 110 and a surface in the depth image 124. Examples of the 3D sensor 110 include, but are not limited to, a depth sensing camera, a time-of-flight sensor, a LiDAR, a structured light camera, or other suitable type of depth sensing device. In some embodiments, the camera 108 and the 3D sensor 110 may be integrated into a single device. In other embodiments, the camera 108 and the 3D sensor 110 may be separate devices.

[0018] The weight sensor 112 is configured to measure the weight of an item 204 placed on the platform 202 of the imaging device 102. For example, the weight sensor 112 may include a transducer that converts an input mechanical force (e.g., weight, tension, compression, pressure, or torque) into an output electrical signal (e.g., current or voltage). As the input force increases, the output electrical signal may increase proportionally. The item tracking device 104 is configured to analyze the output electrical signal to determine a total weight of the item 204 on the weight sensor 112. Examples of the weight sensor 112 include, but are not limited to, a piezoelectric load cell or a pressure sensor. For example, the weight sensor 112 may include one or more load cells configured to transmit an electrical signal indicative of the weight experienced by the load cell. For example, the load cell may generate an electrical current that varies depending on the weight or force experienced by the load cell. The load cell is configured to transmit the generated electrical signal to the item tracking device 104 for processing.

[0019] Item Tracking Device Examples of item tracking device 104 include, but are not limited to, a server, a computer, a laptop, a tablet, or other suitable type of device. In FIG. 1, imaging device 102 and item tracking device 104 are shown as two devices. In some embodiments, imaging device 102 and item tracking device 104 may be integrated into a single device. In one embodiment, item tracking device 104 includes item tracking engine 114 and memory 116. Additional details regarding the hardware configuration of item tracking device 104 are described in FIG. 6. Memory 116 is configured to store item information 118, user account information 120, machine learning models 126, encoded vector library 128, and / or other suitable types of data.

[0020] In one embodiment, the item tracking engine 114 is generally configured to process the images 122 and the depth images 124 to identify items 204 placed on the platform 202 of the imaging device 102 and to associate the identified items 204 with a user. Examples of the operation of the item tracking engine 114 are described in more detail in Figures 3 and 7-26.

[0021] The item information 118 generally includes information associated with multiple items. Examples of the item information 118 include, but are not limited to, price, weight, barcode, item identifier, item number, item features, or other suitable information associated with the item 204. Examples of item features include, but are not limited to, text, logo, brand, color, barcode, pattern, shape, or other suitable type of attribute of the item 204. The user account information 120 includes information of one or more accounts associated with the user. Examples of accounts include, but are not limited to, a customer account, an employee account, a school account, a business account, a financial account, a digital cart, or other suitable type of account. The user account information 120 may be configured to associate user information with an account associated with the user. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, rewards membership information, or other suitable type of information associated with the user, etc. In some embodiments, the item information 118 and / or the user account information 120 may be stored on a device (e.g., a cloud server) external to the item tracking device 104.

[0022] Examples of the machine learning model 126 include, but are not limited to, a multi-layer perceptron, a recurrent neural network (RNN), an RNN long short-term memory (LSTM), a convolutional neural network (CNN), a transformer, or other suitable types of neural network models. In one embodiment, the machine learning model 126 is generally configured to receive an image 122 as an input and output an item identifier based on the provided image 122. The machine learning model 126 is trained using supervised learning training data including different images 122 of the items 204 with corresponding labels (e.g., item identifiers). During the training process, the machine learning model 126 determines weights and bias values ​​that enable the machine learning model 126 to map the images 122 of the items 204 to different item identifiers. Through this process, the machine learning model 126 can identify the items 204 in the images 122. The item tracking engine 114 may be configured to train the machine learning model 126 using any suitable technique as would be understood by one of ordinary skill in the art. In some embodiments, the machine learning model 126 may be stored and / or trained by a device external to the item tracking device 104.

[0023] The encoded vector library 128 generally includes information about items 204 that can be identified by the item tracking device 104. An example of the encoded vector library 128 is shown in FIG. 16. In one embodiment, the encoded vector library 128 includes a number of entries 1602. Each entry 1602 corresponds to a different item 204 that can be identified by the item tracking device 104. Referring to FIG. 16 as an example, each entry 1602 may include an encoded vector 1606 linked to an item identifier 1604 and a number of feature descriptors 1608. The encoded vector 1606 includes an array of numbers. Each number corresponds to and describes a physical attribute of the item 204 (e.g., item type, size, shape, color, etc.). The encoded vector 1606 may be of any suitable length. For example, the size of the encoded vector 1606 may have a length of 1×256, 1×512, 1×1024, or other suitable length. The item identifier 1604 uniquely identifies the item 204. Examples of the item identifier 1604 include, but are not limited to, a product name, a stock keeping unit (SKU) number, an alphanumeric code, a graphical code (e.g., a barcode), or other suitable types of identifiers. Each feature descriptor 1608 describes a physical characteristic of the item 204. Examples of the feature descriptors 1608 include, but are not limited to, an item type 1610, a dominant color 1612, a dimension 1614, a weight 1616, or other suitable types of descriptors that describe physical attributes of the item 204. The item type 1610 identifies a classification of the item 204. For example, the item type 1610 indicates whether the item 204 is a can, a bottle, a box, a fruit, a bag, etc. The dominant color 1612 identifies one or more colors that appear on a surface (e.g., a package) of the item 204. Dimensions 1614 may identify the length, width, and height of item 204. In some embodiments, dimensions 1614 may be listed in ascending order. Weight 1616 identifies the weight of item 204. Weight 1616 may be displayed in pounds, ounces, liters, or other suitable units.

[0024] Imaging device hardware configuration 2A is a perspective view of one embodiment of the imaging device 102. In this example, the imaging device 102 includes a platform 202, a frame structure 206, multiple cameras 108, multiple 3D sensors 110, and a weight sensor 112. The imaging device 102 may be configured as shown in FIG. 2A or in other suitable configurations. In some embodiments, the imaging device 102 may further include additional components, including, but not limited to, lights, a display, and a graphical user interface.

[0025] The platform 202 includes a surface 208 configured to hold a plurality of items 204. In some embodiments, the platform 202 may be integrated with a weight sensor 112. For example, the platform 202 may be positioned on the weight sensor 112, and the weight sensor 112 may measure the weight of the items 204 placed on the platform 202. As another example, the weight sensor 112 may be disposed within the platform 202 and measure the weight of the items 204 placed on the platform 202. In some embodiments, at least a portion of the surface 208 may be transparent. In this case, the camera 108 or scanner (e.g., a barcode scanner) may be disposed below the surface 208 of the platform 202 and configured to capture the image 122 or scan the bottom of the items 204 placed on the platform 202. For example, the camera 108 or scanner may be configured to identify and read the product label and / or barcode (e.g., SKU) of the items 204 through the transparent surface 208 of the platform 202. The platform 202 may be formed from aluminum, metal, wood, plastic, glass, or other suitable materials.

[0026] The frame structure 206 is generally configured to support and position the camera 108 and the 3D sensor 110. In FIG. 2A, the frame structure 206 is configured to position a first camera 108A and a second camera 108C on opposite sides of the imaging device 102 to capture a perspective view of the item 204 on the platform 202. The frame structure 206 is further configured to position a third camera 108D on the rear of the imaging device 102 to capture a perspective view of the item 204 on the platform 202. In some embodiments, the frame structure 206 may further include a fourth camera 108 (not shown) on the front of the imaging device 102 to capture a perspective view of the item 204 on the platform 202. The frame structure 206 may be configured to use any number and combination of the side cameras 108A and 108C, the rear camera 108D, and the front camera 108. For example, one or more of the identified cameras 108 are optional and may be omitted. The perspective image 122 or depth image 124 is configured to capture a side view of the item 204 placed on the platform 202. The frame structure 206 is further configured to position a third camera 108B and a 3D sensor 110 that captures a top or overhead view of the item 204 on the platform 202. The overhead image 122 or depth image 124 is configured to capture an upward facing surface of the item 204 placed on the platform 202. In other examples, the frame structure 206 can be configured to support and position any other suitable number and combination of cameras 108 and 3D sensors 110. The frame structure 206 may be formed of aluminum, metal, wood, plastic, or other suitable material.

[0027] 2B is a perspective view of another embodiment of the imaging device 102 including an enclosure 210. In this configuration, the enclosure 210 is configured to at least partially encase the frame structure 206, the camera 108, the 3D sensor 110, and the platform 202 of the imaging device 102. The frame structure 206, the camera 108, the 3D sensor 110, and the platform 202 may be configured similarly to that described in FIG. 2A. In one embodiment, the frame structure 206 may further include rails or tracks 212 configured to allow the camera 108 and the 3D sensor 110 to be repositioned within the enclosure 210. For example, the cameras 108A, 108C, and 108D may be repositionable along a vertical axis relative to the platform 202 using the rails 212. Similarly, the camera 108B and the 3D sensor 110 may be repositionable along a horizontal axis relative to the platform 202 using the rails 212.

[0028] 2C is a perspective view of another embodiment of the imaging device 102 including an open enclosure 214. In this configuration, the enclosure 214 is configured to at least partially cover the frame structure 206, the camera 108, the 3D sensor 110, and the platform 202 of the imaging device 102. The frame structure 206, the camera 108, the 3D sensor 110, and the platform 202 may be configured similarly to that described in FIG. 2A. In one embodiment, the frame structure 206 may be integrated within the enclosure 214. For example, the enclosure 214 may include an opening 216 configured to accommodate the camera 108 and the 3D sensor 110. In FIG. 2C, the enclosure 214 has a rectangular cross-section with rounded corners. In other embodiments, the enclosure 214 may be configured with a cross-section of any other suitable shape.

[0029] Item Tracking Process 3 is a flow chart of an embodiment of an item tracking process 300 of the item tracking system 100. The item tracking system 100 can use the process 300 to identify an item 204 placed on the platform 202 of the imaging device 102 and assign the item 204 to a particular user. As an example, the item tracking system 100 can use the process 300 in a store to add the item 204 to a user's digital cart for purchase. As another example, the item tracking system 100 can use the process 300 in a warehouse or supply room to allow a user to check out an item. In other examples, the item tracking system 100 can use the process 300 in other suitable types of applications in which the item 204 is assigned or associated with a particular user. This process allows a user to retrieve an item 204 from a space without scanning or otherwise identifying the item 204 they wish to retrieve.

[0030] In operation 302, the item tracking device 104 performs an automatic exclusion of the imaging device 102. During an initial calibration period, the platform 202 may not have an item 204 placed thereon. During this period, the item tracking device 104 uses one or more cameras 108 and 3D sensor 110 to capture a reference image 122 and a reference depth image 124 of the platform 202 without an item 204 placed thereon. The item tracking device 104 can then use the captured image 122 and depth image 124 as reference images to detect when an item has been placed on the platform 202. For example, the item tracking device 104 can use the 3D sensor 110 configured to capture a top or overhead view of the platform 202 to capture a reference depth image 124 of the platform 202 without an item 204 placed thereon. In this example, the captured depth image 124 may include a substantially constant depth value throughout the depth image 124 corresponding to the surface 208 of the platform 202. Later, the item tracking device 104 may detect that the item 204 has been placed on the surface 208 of the platform 202 based on the difference in depth values ​​between the subsequent depth image 124 and the reference depth image 124. As another example, the item tracking device 104 may capture a reference image 122 of the platform when the item 204 is not placed on the platform 202 using a camera 108 configured to capture a top or perspective view of the platform 202. In this example, the captured image 122 includes pixel values ​​corresponding to a scene of the platform when the item 204 is not present on the platform 202. Later, the item tracking device 104 may detect that the item 204 has been placed on the platform 202 based on the difference in pixel values ​​between the subsequent image 122 and the reference image 122.

[0031] In operation 304, the item tracking device 104 determines whether a trigger event is detected. The trigger event corresponds to an event indicative of a user interacting with the imaging device 102. For example, the trigger event may occur when the user approaches the imaging device 102 or places the item 204 on the imaging device 102. As an example, the item tracking device 104 may determine that a trigger event has occurred in response to detecting motion using the 3D sensor 110 or based on a change in the depth image 124 captured by the 3D sensor 110. For example, the item tracking device 104 may detect that the item 204 has been placed on the surface 208 of the platform 202 based on a difference in depth values ​​between the depth image 124 captured by the 3D sensor 110 and the reference depth image 124. Referring to FIG. 4 as an example, FIG. 4 illustrates an example comparison between depth images 124 from an overhead view of the platform 202 of the imaging device 102 before and after placing the item 204 shown in FIG. 2A on the platform 202. Depth image 124A corresponds to a reference depth image 124 captured when item 204 is not placed on platform 202. Depth image 124B corresponds to a depth image 124 captured after item 204 is placed on platform 202. In this example, the colors or pixel values ​​in depth image 124 represent different depth values. In depth image 124A, the depth values ​​of depth image 124A are approximately constant, which means that there is no item 204 on platform 202. In depth image 124B, the different depth values ​​correspond to items 204 (i.e., items 204A, 204B, and 204C) placed on platform 202. In this example, item tracking device 104 detects a trigger event in response to detecting the presence of item 204 on platform 202 based on the difference between depth image 124A and depth image 124B. The item tracking device 104 may also use the images 122 or the depth images 124 to count the number of items 204 that are on the platform 202 .In this example, the item tracking device 104 determines, based on the depth image 124B, that there are three items 204 placed on the platform 202. The item tracking device 104 can later use the determined number of items 204 to check whether it has identified all of the items 204. This process is described in more detail below in operation 312.

[0032] As another example, the item tracking device 104 may determine that a trigger event has occurred in response to detecting movement using the camera 108 or based on a change in the image 122 captured by the camera 108. For example, the item tracking device 104 may detect that an item 204 has been placed on the platform 202 based on a difference in pixel values ​​between the subsequent image 122 and the reference image 122. As another example, the item tracking device 104 may determine that a trigger event has occurred in response to an increase in weight relative to a weight sensor 112 of the imaging device 102. In this case, the increase in weight measured by the weight sensor 112 indicates that one or more items 204 have been placed on the platform 202. In other examples, the item tracking device 104 may use other suitable types of sensors or techniques to detect when a user approaches the imaging device 102 or places an item 204 on the platform 202.

[0033] In response to determining that the item tracking device 104 has not detected a trigger event, the item tracking device 104 remains at operation 304. In this case, the item tracking device 104 determines that the user has not yet interacted with the imaging device 102. The item tracking device 104 remains at operation 304 and continues checking for the trigger event until the user begins to interact with the imaging device 102. In response to determining that the item tracking device 104 has detected a trigger event, the item tracking device 104 proceeds to operation 306. In this case, the item tracking device 104 determines that the user has begun to interact with the imaging device 102. The item tracking device 104 proceeds to operation 306 and begins identifying items placed on the platform 202 of the imaging device 102.

[0034] In operation 306, the item tracking device 104 identifies one or more cameras 108 for capturing images 122 of the items 204 on the platform 202 of the imaging device 102. The item tracking device 104 can identify the cameras 108 for capturing the images 122 of the items 204 based at least in part on the pose (e.g., position and orientation) of the items 204 on the platform 202. The pose of the items 204 corresponds to the location of the items 204 and how the items 204 are positioned relative to the platform 202. Referring to the example of FIG. 2A, a first item 204A and a second item 204C are positioned in a vertical orientation relative to the platform 202. In the vertical orientation, the identifiable features of the items 204 are primarily in a vertical orientation. The cameras 108, such as the cameras 108A and 108C that capture perspective views, may be suitable for capturing images 122 of the identifiable features of the items 204 in a vertical orientation. For example, the item tracking device 104 may select camera 108A to capture an image 122 of item 204A because most of the identifiable features of item 204A, such as branding, text, barcodes, etc., are located on the sides of item 204A and are best viewed using a perspective view of item 204. Similarly, the item tracking device 104 may then select camera 108C to capture an image 122 of item 204C. In this example, a third item 204B is positioned in a horizontal orientation relative to the platform 202. In a horizontal orientation, the identifiable features of item 204 are primarily oriented horizontally. A camera 108, such as camera 108B that captures a top or overhead view, may be suitable for capturing an image 122 of the identifiable features of item 204 in a horizontal orientation. In this case, the item tracking device 104 may select camera 108B to capture an image 122 of item 204B because most of the identifiable features of item 204B are located on the top of item 204B and are best seen using an overhead view of item 204B.

[0035] In one embodiment, the item tracking device 104 can use the depth image 124 to determine the orientation of the item 204 on the platform 202. Referring to FIG. 4 as an example, the depth image 124B corresponds to the overhead depth image 124 captured after placing the items 204 (i.e., items 204A, 204B, and 204C) shown in FIG. 2A on the platform 202. In this example, the item tracking device 104 can use the area corresponding to each item 204 in the depth image 124B to determine the orientation of the items 204. For example, the item tracking device 104 can determine an area 402 corresponding to the item 204A in the depth image 124B. The item tracking device 104 compares the determined area 402 to a predetermined area threshold 614. The item tracking device 104 determines that the item 204 is vertically oriented if the determined area 402 of the item 204 is less than or equal to the predetermined area threshold 614. Otherwise, the item tracking device 104 determines that the item 204 is horizontally oriented if the determined area 402 of the item 204 is greater than the predefined area threshold 614. In this example, the item tracking device 104 determines that items 204A and 204C are vertically oriented because their areas 402 and 406 are less than or equal to the area threshold 614. The item tracking device 104 determines that item 204B is horizontally oriented because its area 404 is greater than the area threshold 614. This determination means that the item tracking device 104 selects a camera 108 (e.g., cameras 108A and 108C) that captures a perspective view of the platform 202 to capture images 122 of items 204A and 204C. The item tracking device 104 selects a camera 108 (eg, camera 108B) that captures a top or overhead view of the platform 202 to capture an image 122 of item 204B.

[0036] In one embodiment, the item tracking device 104 can identify a camera 108 for capturing an image 122 of the item 204 based at least in part on the distance between the item 204 and the camera 108. For example, the item tracking device 104 can generate a homography 608 between the camera 108 and / or the 3D sensor 110 of the imaging device 102. By generating the homography 608, the item tracking device 104 can use the location of the item 204 in the image 122 to determine the physical location of the item 204 relative to the platform 202, the camera 108, and the 3D sensor 110. This allows the item tracking device 104 to use the physical location of the item 204 to determine the distance between the item 204 and each of the camera 108 and the 3D sensor 110. The homography 608 includes coefficients configured to transform between pixel locations in the image 122 or depth image 124 and (x,y) coordinates in a global plane (i.e., physical locations on the platform 202). The item tracking device 104 uses the homography 608 to correlate pixel locations in a particular camera 108 or 3D sensor 110 with physical locations on the platform 202. In other words, the item tracking device 104 uses the homography 608 to determine where an item 204 is physically located on the platform 202 based on pixel locations in an image 122 or depth image 124 from a camera 108 or 3D sensor 110, respectively. Because the item tracking device 104 monitors the platform 202 using multiple cameras 108 and 3D sensors 110, each camera 108 and 3D sensor 110 is uniquely associated with a different homography 608 based on the physical location of the camera 108 or 3D sensor 110 on the imaging device 102. This configuration enables the item tracking device 104 to determine where the item 204 is physically located on the platform 202 based on which camera 108 or 3D sensor 110 the item 204 appears on and the location of the item in the image 122 or depth image 124 captured by that camera 108 or 3D sensor 110.Additional information regarding the generation of homographies 608 and uses of homographies 608 is disclosed in U.S. Pat. No. 11,023,741, entitled “DRAW WIRE ENCODER BASED HOMOGRAPHY” (Attorney Docket No. 090278.0233), which is incorporated by reference herein as if reproduced in its entirety.

[0037] As an example, the item tracking device 104 can determine the physical location of an item on the platform 202 using an image 122 or depth image 124 from a camera 108 or 3D sensor 110 capturing a top or overhead view of the platform 202, respectively. In this example, the item tracking device 104 can determine a pixel location of the item 204 in the image 122 or depth image 124. The item tracking device 104 can then use the homography 608 to determine the physical location of the item 204 relative to the platform 202 based on the pixel location of the item 204. After determining the physical location of the item 204 on the platform 202, the item tracking device 104 can identify which camera 108 is located physically closest to the item 204 and select the identified camera 108. Returning to the example of Figure 2A, the item tracking device 104 selects camera 108A to capture an image 122 of item 204A because camera 108A is closer to item 204A than camera 108C. Similarly, the item tracking device 104 selects camera 108C to capture an image 122 of item 204C because camera 108C is closer to item 204C than camera 108A. This process ensures that the camera 108 with the best view of item 204 is selected to capture an image 122 of item 204.

[0038] In operation 308, the item tracking device 104 captures an image 122 of the item 204 on the platform 202 using the identified camera 108. Here, the item tracking device 104 captures an image of the item 204 using the identified camera 108. With reference to FIGS. 5A, 5B, and 5C as an example, the item tracking device 104 can capture a first image 122A of the item 204A, a second image 122B of the item 204B, and a third image 122C of the item 204C using cameras 108A, 108B, and 108C, respectively. The item tracking device 104 can collect one or more images 122 of each item 204 for processing. By capturing images of the items 204 using a subset of the cameras 108 available in the imaging device 102, the item tracking device 104 can reduce the number of images 122 that are captured and processed to identify the items 204 on the platform 202. This process reduces the search space for identifying the item 204, improving the efficiency and hardware utilization of the item tracking device 104 since the item tracking device 104 processes fewer images 122 to identify the item 204 instead of processing images 122 from all cameras 108 on the imaging device 102 (which may include multiple images 122 of the same item 204). Additionally, the item tracking device 104 also selects the camera 108 that is positioned to capture the features that are most useful in identifying the item 204 based on the orientation and position of the item 204, as described in operation 306. Examples of features include, but are not limited to, text, logos, brands, colors, bar codes, patterns, shapes, or other suitable types of attributes of the item 204.

[0039] Returning to operation 310 of FIG. 3 , the item tracking device 104 identifies items 204 on the platform 202 based on the captured images 122. Here, the item tracking device 104 identifies items 204 in each image 122 based on features of the items 204 in the images 122. As an example, the machine learning model 126 may be a CNN. In this example, the machine learning model 126 includes an input layer, an output layer, and one or more hidden layers. The hidden layer includes at least one convolutional layer. For example, the machine learning model 126 includes the following sequence of layers: an input layer, a convolutional layer, a pooling layer, a convolutional layer, a pooling layer, one or more fully connected layers, and an output layer. Each convolutional layer of the machine learning model 126 uses a set of convolution kernels to extract features from the pixels forming the image 122. In a particular embodiment, the convolutional layers of the machine learning model 126 are implemented in the frequency domain, and the convolution process is performed using a discrete Fourier transform. This may be desirable to reduce the computation time associated with training the machine learning model 126 for use in image classification purposes. For example, by converting to the frequency domain, a fast Fourier transform algorithm (FFT) may be implemented to perform the discrete Fourier transform associated with the convolution. Using an FFT algorithm alone not only significantly reduces computation time when implemented on a single CPU (compared to applying a convolution kernel in the spatial domain), but the FFT algorithm can be parallelized using one or more graphics processing units (GPUs), further reducing computation time. Converting to the frequency domain may also be desirable to ensure that the machine learning model 126 is translation and rotation invariant (e.g., the assignment of an image 122 to an item identifier by the machine learning model 126 should not depend on the location and / or orientation of the item 204 in the image 122 based on the presence of the item 204 in the image 122).

[0040] As another example, the machine learning model 126 may be a supervised learning algorithm. Thus, in certain embodiments, the item tracking device 104 is configured to train the machine learning model 126 to assign the input image 122 to one of a set of predefined item identifiers. The item tracking device 104 may train the machine learning model 126 in any suitable manner. For example, in certain embodiments, the item tracking device 104 trains the machine learning model 126 by providing the machine learning model 126 with training data (e.g., images 122) that includes a set of labels (e.g., item identifiers) attached to the input image 122. As another example, the machine learning model 126 may be an unsupervised learning algorithm. In such an embodiment, the item tracking device 104 is configured to train the machine learning model 126 by providing the machine learning model 126 with a collection of images 122 and instructing the machine learning model 126 to classify the images 122 using item identifiers identified by the item tracking device 104 based on common features extracted from the images 122. The item tracking device 104 may train the machine learning model 126 at any time before inputting the captured images 122 to the machine learning model 126 .

[0041] After training the machine learning model 126, the item tracking device 104 may input each captured image 122 into the machine learning model 126. In response to inputting the image 122 into the machine learning model 126, the item tracking device 104 receives an item identifier for the item 204 from the machine learning model 126. The item identifier corresponds to the item 204 identified in the image 122. Examples of item identifiers include, but are not limited to, an item name, a barcode, an item number, a serial number, or any other suitable type of identifier that uniquely identifies the item 204.

[0042] In some embodiments, the item tracking device 104 can identify an item 204 in an image 122 using one or more image processing techniques without using a machine learning model 126. For example, the item tracking device 104 can use object detection and / or optical character recognition (OCR) to identify text, logos, brands, colors, barcodes, or other features of the item 204 that can be used to identify the item 204. In this case, the item tracking device 104 can process pixels in the image 122 to identify text, colors, barcodes, patterns, or other characteristics of the item 204. The item tracking device 104 can then compare the identified features of the item 204 to a set of features that correspond to different items 204. For example, the item tracking device 104 can extract text (e.g., product names) from the image 122 and compare the text to a set of text associated with different items 204. As another example, the item tracking device 104 can determine a dominant color in the image 122 and compare the dominant color to a set of colors associated with different items 204. As another example, the item tracking device 104 may identify a barcode in the image 122 and compare the barcode to a set of barcodes associated with different items 204. As another example, the item tracking device 104 may identify a logo or pattern in the image 122 and compare the identified logo or pattern to a set of logos or patterns associated with different items 204. In other examples, the item tracking device 104 may identify other suitable types or combinations of features and compare the identified features to features associated with different items 204. After comparing the identified features from the image 122 to the set of features associated with different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found if at least a meaningful portion of the identified features match the features corresponding to the item 204.In response to determining that a meaningful portion of a feature in the image 122 matches a feature of the item 204, the item tracking device 104 can output an item identifier that corresponds to the matching item 204. In other embodiments, the item tracking device 104 can use one or more image processing techniques in combination with the machine learning model 126 to identify the item 204 in the image 122 using any combination of the aforementioned techniques.

[0043] In some embodiments, the item tracking device 104 is configured to output a confidence score 610 indicating the probability of correctly identifying the item 204. For example, the item tracking device 104 can obtain the confidence score 610 from the machine learning model 126 using the determined item identifier. In this example, the machine learning model 126 outputs a confidence score 610 that is proportional to the number of features used or matched in determining the item identifier. As another example, the item tracking device 104 may determine the confidence score 610 based on how well the identified features match the features of the identified item 204. For example, the item tracking device 104 may obtain a confidence score 610 of 50% if half of the text identified in the image 122 matches the text associated with the identified item 204. As another example, the item tracking device 104 may determine to obtain a confidence score 610 of 100% if the barcode in the image 122 matches the barcode of the identified item 204. As another example, the item tracking device 104 may obtain a confidence score 610 of 25% if the dominant color in the image 122 matches the dominant color of the identified item 204. In other examples, the item tracking device 104 may obtain a confidence score 610 based on how well other suitable types or combinations of features match the features of the identified item 204. Other information that may affect the example confidence score 610 includes, but is not limited to, the orientation of the object, the number of items on the platform 202 (e.g., fewer items on the platform 202 are easier to identify than more items on the platform 202), the relative distance between items on the platform (e.g., items spaced apart on the platform 202 are easier to identify than items crowded together on the platform 202), and the like. The item tracking device 104 may compare the confidence score 610 of the identified item 204 to a confidence score threshold 612 to determine whether the item 204 has been identified.The item tracking device 104 may determine that it has not identified an item 204 if the confidence score 610 of the item 204 is less than the confidence score threshold 612. The item tracking device 104 determines that it has identified an item 204 if the confidence score 610 of the item 204 is greater than or equal to the confidence score threshold 612. The confidence score threshold 612 may be set at 90%, 80%, 75%, or other suitable value.

[0044] In operation 312, the item tracking device 104 determines whether it has identified all of the items 204 on the platform 202. For example, the item tracking device 104 may compare the number of items 204 identified from the captured image 122 to the number of items 204 on the platform 202 determined in operation 304. The item tracking device 104 determines that it has identified all of the items 204 on the platform 202 if the number of items 204 identified from the captured image 122 matches the determined number of items 204 on the platform 202. Otherwise, the item tracking device 104 determines that it has not identified at least one of the items 204 if the number of items 204 identified from the captured image 122 does not match the determined number of items 204 on the platform 202.

[0045] In response to determining that the item tracking device 104 has not identified one or more of the items 204 on the platform 202, the item tracking device 104 proceeds to operation 314. In this case, the item tracking device 104 may output a request to a user to reposition one or more items 204 on the platform 202 so that the item tracking device 104 can identify some of the items 204 on the platform. In operation 314, the item tracking device 104 outputs a prompt to reposition one or more items 204 on the platform 202. As an example, one or more items 204 may be blocking the view of the items 204 from one of the cameras 108. In this example, the item tracking device 104 may output a message to a graphical user interface at the imaging device 102 instructing the user to reposition the items 204 on the platform 202. In some embodiments, the item tracking device 104 may also determine the location of one or more items 204 on the platform 202 that it did not identify. For example, the item tracking device 104 may activate a light source above or below the platform 202 to illuminate the unrecognized item 204. In one embodiment, after outputting the message to relocate the item 204 on the platform 202, the item tracking device 104 returns to operation 306 and resumes the process of identifying the item 204 on the platform 202. This process prevents the item tracking device 104 from double-counting the item 204 after relocating it on the platform 202.

[0046] Returning to operation 312, in response to determining that the item tracking device 104 has identified all of the items 204 on the platform 202, the item tracking device 104 proceeds to operation 316. In some embodiments, the item tracking device 104 can verify the accuracy of detection of the identified items 204 based on the weights of the items 204 on the platform 202. For example, the item tracking device 104 can determine a first weight associated with the weights of the identified items 204 based on the item information 118 associated with the identified items 204. For example, the item tracking device 104 can use the item identifiers of the identified items 204 to determine a weight corresponding to each of the identified items 204. The item tracking device 104 can sum the individual weights of the identified items 204 to determine the first weight. The item tracking device 104 can also receive a second weight of the items 204 on the platform 202 from the weight sensor 112. The item tracking device 104 then determines a weight difference between the first weight and the second weight and compares the weight difference to a weight difference threshold. The weight difference threshold corresponds to a maximum weight difference between the first weight and the second weight. If the weight difference exceeds the weight difference threshold, the item tracking device 104 may determine that there is a mismatch between the weight of the item 204 on the platform 202 of the imaging device 102 and the expected weight of the identified item 204. In this case, the item tracking device 104 may output an error message and / or return to operation 306 to resume the item tracking process. If the weight difference is less than or equal to the weight difference threshold, the item tracking device 104 may determine that there is a match between the weight of the item 204 on the platform 202 of the imaging device 102 and the expected weight of the identified item 204. In this case, the item tracking device 104 may proceed to operation 316.

[0047] At operation 316, the item tracking device 104 determines whether a prohibited or restricted item 204 is present on the platform 202. A prohibited or restricted item 204 is an item 204 that the user is not permitted to acquire due to permission restrictions, age restrictions, or other types of restrictions. The item tracking device 104 may compare the item identifier of the identified item 204 to a list of item identifiers of restricted or prohibited items 616. In response to determining that the item 204 matches one of the items on the list of restricted or prohibited items 616, the item tracking device 104 proceeds to operation 318 and outputs a warning or notification indicating that the user is prohibited from acquiring one of the items 204 that is present on the platform 202. For example, the item tracking device 104 may identify the prohibited item 204 and output a warning message using a graphical user interface at the imaging device 102 to prompt the user to remove the prohibited item 204 from the platform 202. As another example, the item tracking device 104 may output a warning message to another user (e.g., an employee) associated with the space identifying the prohibited item 204. In other examples, the item tracking device 104 may output other suitable types of warning messages in response to detecting a prohibited item 204 on the platform 202.

[0048] At operation 320, the item tracking device 104 determines whether the prohibited item 204 has been removed from the platform 202. For example, the item tracking device 104 may use the weight sensor 112 to determine whether the measured weight of the item 204 on the platform 202 has decreased by an amount corresponding to the weight of the prohibited item 204. As another example, the item tracking device 104 may use the camera 108 and / or the 3D sensor 110 to determine whether the prohibited item 204 is still present on the platform 202. In response to determining that the prohibited item 204 is still present on the platform 202, the item tracking device 104 may pause the process 300 and remain at operation 320 until the prohibited item 204 is removed from the platform 202. This process prevents the user from obtaining the prohibited item 204. After the item tracking device 104 removes the prohibited item 204 from the platform 202, the item tracking device 104 may proceed to operation 322.

[0049] Otherwise, in response to determining that the prohibited item 204 is not present on the platform 202, the item tracking device 104 proceeds to operation 322. In operation 322, the item tracking device 104 associates the item 204 with a user. In one embodiment, the item tracking device 104 may identify a user associated with the item 204 on the platform 202. For example, a user may identify themselves using a scanner or card reader at the imaging device 102. Examples of scanners include, but are not limited to, a QR code scanner, a barcode scanner, a Near Field Communication (NFC) scanner, or other suitable types of scanners capable of receiving an electronic code embedded with information that uniquely identifies a person. In another example, a user may identify themselves by providing user information in a graphical user interface at the imaging device 102. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, or other suitable types of information associated with a user.

[0050] The item tracking device 104 uses the user-provided information to identify an account associated with the user and then adds the identified items 204 to the user's account. For example, the item tracking device 104 can use the user-provided information to identify an account in the user account information 120 associated with the user. As an example, the item tracking device 104 can identify a digital cart associated with the user. In this example, the digital cart includes information about items 204 that the user has placed on the platform 202 for purchase. The item tracking device 104 can add the items 204 to the user's digital cart by adding an item identifier for the identified items 204 to the digital cart. The item tracking device 104 can also add other information related to the items 204 to the digital cart. For example, the item tracking device 104 can use the item identifier to look up price information for the identified items 204 from the stored item information 118. The item tracking device 104 can then add price information corresponding to each of the identified items 204 to the user's digital cart.

[0051] After the item tracking device 104 adds the item 204 to the user's digital cart, the item tracking device 104 can trigger or initiate a transaction for the item 204. In one embodiment, the item tracking device 104 can use previously stored information (e.g., payment card information) to complete the transaction for the item 204. In this case, the user can be automatically charged for the item 204 in the digital cart when they leave the space. In another embodiment, the item tracking device 104 can collect information from the user using a scanner or card reader at the imaging device 102 to complete the transaction for the item 204. This process allows the item 204 to be automatically added to the user's account (e.g., digital cart) without the user having to scan or identify the item 204 they want to take home. After adding the item 204 to the user's account, the item tracking device 104 can output a notification or summary to the user that includes information about the item 204 that was added to the user's account. For example, the item tracking device 104 can output the summary to a graphical user interface at the imaging device 102. As another example, the item tracking device 104 can output the summary by sending the summary to an email address associated with the user or to a user device.

[0052] Hardware configuration of the item tracking device 6 is an embodiment of the item tracking device 104 of the item tracking system 100. In one embodiment, the item tracking device 104 may include a processor 602, a memory 116, and a network interface 604. The item tracking device 104 may be configured as shown or in any other suitable configuration.

[0053] Processor The processor 602 includes one or more processors operatively coupled to the memory 116. The processor 602 is any electronic circuit including, but not limited to, a state machine, one or more central processing unit (CPU) chips, logic units, cores (e.g., multi-core processors), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or digital signal processors (DSPs). The processor 602 is a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the foregoing. The processor 602 is communicatively coupled to and in signal communication with the memory 116 and the network interface 604. The one or more processors are configured to process data and can be implemented in hardware or software. For example, the processor 602 is an 8-bit, 16-bit, 32-bit, 64-bit, or other suitable architecture. Processor 602 may include an arithmetic logic unit (ALU) that performs arithmetic and logical operations, processor registers that supply operands to the ALU and store results of ALU operations, and a control unit that fetches instructions from memory and executes instructions by directing the coordinated operation of the ALU, registers, and other components.

[0054] The one or more processors are configured to execute various instructions. For example, the one or more processors are configured to execute item tracking instructions 606 that cause the processor to implement the item tracking engine 114. In this manner, the processor 602 may be a special-purpose computer designed to implement the functionality disclosed herein. In one embodiment, the item tracking engine 114 is implemented using a logic unit, FPGA, ASIC, DSP, or other suitable hardware. The item tracking engine 114 is configured to operate as described in FIG. 1 and FIG. 3. For example, the item tracking engine 114 may be configured to perform the operations of the process 300 described in FIG. 3.

[0055] Memory The memory 116 is operable to store any of the information described above with respect to Figures 1 and 3, as well as any other data, instructions, logic, rules, or code operable to implement the functions described herein when executed by the processor 602. The memory 116 may be comprised of one or more non-transitory computer-readable media, such as computer disks, tape drives, or solid-state drives, and may be used as an overflow data storage device to store programs selected for execution and to store instructions and data read during program execution. The memory 116 may be volatile or non-volatile, and may be comprised of read-only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), dynamic random access memory (DRAM), and static random access memory (SRAM).

[0056] The memory 116 may be operable to store item tracking instructions 606, item information 118, user account information 120, machine learning models 126, images 122, depth images 124, homographies 608, confidence scores 610, confidence score thresholds 612, area thresholds 614, lists of restricted or prohibited items 616, encoded vector libraries 128, and / or other data or instructions. The item tracking instructions 606 may comprise any suitable set of instructions, logic, rules, or code operable to execute the item tracking engine 114. The item information 118, the user account information 120, the machine learning model 126, the image 122, the depth image 124, the homography 608, the confidence score 610, the confidence score threshold 612, the area threshold 614, the list of restricted or prohibited items 616, and the encoded vector library 128 are configured in the same manner as the item information 118, the user account information 120, the machine learning model 126, the image 122, the depth image 124, the homography 608, the confidence score 610, the confidence score threshold 612, the area threshold 614, the list of restricted or prohibited items 616, and the encoded vector library 128 described in Figures 1 to 26, respectively.

[0057] Network Interface The network interface 604 is configured to enable wired and / or wireless communication. The network interface 604 is configured to communicate data between the imaging device 102 and other devices, systems, or domains. For example, the network interface 604 may be an NFC interface, a Bluetooth interface, a Zigbee interface, a Z-wave interface, a radio frequency identification (RFID) interface, a WiFi interface, a LAN interface, a WAN interface, a PAN interface, a modem, a switch, or a router. The processor 602 is configured to transmit and receive data using the network interface 604. The network interface 604 may be configured to use any suitable type of communication protocol as would be understood by one of ordinary skill in the art.

[0058] Hand detection process to trigger item identification 7 is a flow chart of one embodiment of a hand detection process 700 for triggering an item identification process of the item tracking system 100. Using process 700, the item tracking system 100 can detect a trigger event corresponding to when a user places their hand on the platform 202 to place an item 204 on the platform 202. This process allows the item tracking device 104 to detect the presence of a user interacting with the platform 202 and use this to initiate an item detection process, such as processes 300 and 2300 described in Figures 3 and 23, respectively.

[0059] In operation 702, the item tracking device 104 captures a first overhead depth image 124 at a first time using the 3D sensor 110. Here, the item tracking device 104 first captures an overhead depth image 124 of the platform 202 to ensure that no items 204 are placed on the platform 202 and no hands are present on the platform 202, and then periodically checks whether a user's hands are present on the platform 202. The overhead depth image 124 captures the upward facing surfaces of the object and the platform 202. Referring to FIG. 8A as an example, the item tracking device 104 can capture an overhead depth image 124 of the platform 202 using the 3D sensor 110 positioned above the platform 202. Within the overhead depth image 124 of the platform 202, the item tracking device 104 defines a region of interest 802 of the platform 202. The region of interest 802 (outlined in FIGS. 8A-8C ) identifies a predetermined range of pixels in the overhead depth image 124 that corresponds to the surface of the platform 202. The item tracking device 104 uses the defined region of interest 802 to determine whether an item 204 has been placed on the platform 202 or whether a user has positioned their hand on the platform 202. The region of interest 802 is the same predetermined range of pixels in all depth images 124 captured by the 3D sensor 110.

[0060] Returning to operation 704 of FIG. 7 , the item tracking device 104 captures a second overhead depth image 124 at a second time using the same 3D sensor 110. After capturing the first overhead depth image 124, the item tracking device 104 begins periodically capturing additional overhead depth images 124 of the platform 202 to check whether the user's hand has entered the region of interest 802 of the platform 202. The item tracking device 104 may capture additional overhead depth images 124 every second, every 10 seconds, every 30 seconds, or any other suitable time interval. In some embodiments, the item tracking device 104 may capture the second overhead depth image 124 in response to detecting motion near the platform 202. For example, the item tracking device 104 may use a proximity sensor configured to detect motion near the platform 202 before capturing the second overhead depth image 124. As another example, to detect movement, the item tracking device 104 may periodically capture additional overhead depth images 124. In this example, the item tracking device 104 compares the first overhead depth image 124 to subsequently captured overhead depth images 124 and detects movement based on differences between the overhead depth images 124, e.g., the presence of an object.

[0061] At operation 706, the item tracking device 104 determines whether an object is present within the region of interest 802 of the second overhead depth image 124. In one embodiment, the item tracking device 104 determines whether an object is present within the region of interest 802 based on a difference between the first overhead depth image 124 and the second overhead depth image 124. With reference to FIG. 8B as an example, the item tracking device 104 compares the second overhead depth image 124 (shown in FIG. 8B) with the first overhead depth image 124 (shown in FIG. 8A) to identify a difference between the first overhead depth image 124 and the second overhead depth image 124. In this example, the item tracking device 104 detects an object 804 corresponding to a user's hand within the region of interest 802 in the second overhead depth image 124. FIG. 8C shows the corresponding image 122 of an object 804 present in the second overhead depth image 124 .

[0062] Returning to operation 706 of FIG. 7 , in response to determining that an object is not present within the region of interest 802 of the second overhead depth image 124, the item tracking device 104 returns to operation 704. In this case, the item tracking device 104 returns to operation 704 and continues to periodically capture the overhead depth image 124 of the platform 202 to check where the user's hand has entered the region of interest 802 of the platform 202. In response to determining that an object is present within the region of interest 802 of the second overhead depth image 124, the item tracking device 104 proceeds to operation 708. In this case, the item tracking device 104 proceeds to operation 708 to check whether the object in the second overhead depth image 124 corresponds to the user's hand.

[0063] The item tracking device 104 is configured to distinguish between an item 204 placed on the platform 202 and a user's hand. When the user's hand is on the platform 202, the user's hand is typically within the region of interest 802 of the second overhead depth image 124, while the user's arm is outside the region of interest 802 of the second overhead depth image 124. The item tracking device 104 uses these characteristics to verify that the user's hand is on the platform 202, for example, when the user places the item 204 on the platform 202.

[0064] In operation 708, the item tracking device 104 determines that a first portion 806 of a first object (e.g., a user's hand and arm) is within the region of interest 802 of the second overhead depth image 124. Here, the item tracking device 104 confirms that the first portion 806 of the detected object, which corresponds to the user's hand, is within the region of interest 802 of the second overhead depth image 124. Returning to the example of FIG. 8B , the user's hand (shown as portion 806 of object 804) is at least partially within the region of interest 802 of the second overhead depth image 124.

[0065] Returning to operation 710 of FIG. 7, the item tracking device 104 determines that a first portion 806 of the first object (e.g., the user's hand) is within the region of interest 802 of the second overhead depth image 124 while a second portion 808 of the first object (e.g., the user's wrist or arm) is outside the region of interest 802. Returning to the example of FIG. 8B, the user's wrist and arm (shown as portion 808 of object 804) are at least partially outside the region of interest 802 while the user's hand (shown as portion 806 of object 804) is within the region of interest 802 of the second overhead depth image 124. These characteristics allow the item tracking device 104 to determine that the user's hand has been detected in the second overhead depth image 124.

[0066] After detecting the user's hand, the item tracking device 104 begins periodically capturing additional overhead depth images 124 of the platform 202 and checks whether the user's hand has left the region of interest 802 of the platform 202. In operation 712, the item tracking device 104 captures a third overhead depth image 124 using the 3D sensor 110 at a third time point. The item tracking device 104 may capture the additional overhead depth image 124 every second, every 10 seconds, every 30 seconds, or any other suitable time interval. In some embodiments, the item tracking device 104 may capture the third overhead depth image 124 in response to a change or difference in weight on the platform 202. For example, the item tracking device 104 uses the weight sensor 112 to determine a first weight value at a first time point when no item 204 is placed on the platform 202. The item tracking device 104 then uses the weight sensor 112 to determine a second weight value at a time after the user places the item 204 on the platform 202. In this example, the item tracking device 104 detects a weight difference between the first weight value and the second weight value, and then captures a third overhead depth image 124 in response to detecting the weight difference.

[0067] In operation 714, the item tracking device 104 determines whether the first object (i.e., the user's hand) is still present within the region of interest 802 of the third overhead depth image 124. Here, the item tracking device 104 can determine whether the first object is still present within the region of interest 802 based on the difference between the second overhead depth image 124 and the third overhead depth image 124. Referring to the example of FIG. 8D, the item tracking device 104 compares the third overhead depth image 124 (shown in FIG. 8D) with the second overhead depth image 124 (shown in FIG. 8B) to identify the difference between the third overhead depth image 124 and the second overhead depth image 124. In this example, the item tracking device 104 detects that the first object 804 , which corresponds to the user's hand, is no longer within the region of interest 802 of the third overhead depth image 124 .

[0068] Returning to operation 714 of FIG. 7 , in response to determining that the first object 804 is still present within the region of interest 802 of the third overhead depth image 124, the item tracking device 104 returns to operation 712. In this case, the item tracking device 104 returns to operation 712 and continues to periodically check when the user's hand leaves the region of interest 802 of the platform 202. In response to determining that the first object 804 is no longer present within the region of interest 802 of the third overhead depth image 124, the item tracking device 104 proceeds to operation 716. In this case, the item tracking device 104 begins checking for items 204 that the user has placed on the platform 202.

[0069] In operation 716, the item tracking device 104 determines whether the item 204 is within the region of interest 802 of the third overhead depth image 124. When the item 204 is placed on the platform 202, the item 204 is typically entirely within the region of interest 802 of the third overhead depth image 124. The item tracking device 104 uses this characteristic to distinguish between the item 204 placed on the platform 202 and the user's hand. Returning to the example of FIG. 8D, the item tracking device 104 detects that the item 204 is within the region of interest 802 of the third overhead depth image 124.

[0070] Returning to operation 716 of FIG. 7 , in response to determining that the item 204 is not present within the region of interest 802 of the third overhead depth image 124, the item tracking device 104 returns to operation 704. In this case, the item tracking device 104 determines that the user did not place the item 204 on the platform 202. The item tracking device 104 returns to operation 704 and repeats the hand detection process to detect that the user's hand has re-entered the region of interest 802 of the platform 202. In response to determining that the item 204 is present within the region of interest 802 of the third overhead depth image 124, the item tracking device 104 proceeds to operation 718. In this case, the item tracking device 104 proceeds to operation 718 and begins capturing images 122 and / or depth images 124 of the item 204 for further processing, such as item identification.

[0071] At operation 718, the item tracking device 104 captures an image 122 of the item 204 in response to determining that the first object 804 is no longer within the region of interest 802 of the third overhead depth image 124 and that the item 204 is present within the region of interest 802 of the third overhead depth image 124. The item tracking device 104 may use one or more cameras 108 and / or 3D sensors 110 to capture the image 122 or depth image 124, respectively, of the item 204 placed on the platform 202.

[0072] In some embodiments, the item tracking device 104 can capture an image 122 upon detecting a change or difference in weight on the platform 202. For example, the item tracking device 104 can use the weight sensor 112 to determine a first weight value at a first time when no item 204 is placed on the platform 202. The item tracking device 104 can then use the weight sensor 112 to determine a second weight value at a time after a user places the item 204 on the platform 202. In this example, the item tracking device 104 detects a weight difference between the first weight value and the second weight value, and then captures an image 122 in response to detecting the weight difference.

[0073] After capturing an image 122 of the item 204, the item tracking device 104 can identify the item 204 placed on the platform 202 based on the physical attributes of the item 204 present in the captured image 122 using a process similar to processes 300 and 2300 described in Figures 3 and 23, respectively.

[0074] Image cropping process for item identification FIG. 9 is a flow chart of one embodiment of an image cropping process 900 for item identification by the item tracking system 100. The item tracking system 100 can use the process 900 to isolate an item 204 in an image 122. For example, when the camera 108 captures an image 122 of a platform 202, the image 122 may include multiple items 204 placed on the platform 202. To improve accuracy in identifying the items 204, the item tracking device 104 first crops the image 122 to isolate each item 204 in the image 122. Cropping the image 122 generates a new image 122 (i.e., cropped image 122) that includes pixels from the original image 122 that correspond to the items 204. The item tracking device 104 repeats this process to create a set of cropped images 122, each of which corresponds to an item 204.

[0075] In operation 902, the item tracking device 104 captures a first image 122 of an item 204 on a platform 202 using a camera 108. The item tracking device 104 may capture the first image 122 of an item 204 on a platform 202 using a camera 108 that captures an overhead, perspective, or side profile view. As an example, the camera 108 may be configured with an overhead view to capture an upward facing surface of the item 204. As another example, the camera 108 may be configured with a perspective or side profile view to capture a side of the item 204.

[0076] In operation 904, the item tracking device 104 identifies an area of ​​interest 1002 of the item 204 in the first image 122. The area of ​​interest 1002 is comprised of a number of pixels that correspond to the item 204 in the first image 122. An example of the area of ​​interest 1002 is a bounding box. In some embodiments, the item tracking device 104 can use one or more image processing techniques to identify the area of ​​interest 1002 of the item 204 in the first image 122. For example, the item tracking device 104 can use object detection and / or OCR to identify text, logos, brands, colors, barcodes, or other features of the item 204 that can be used to identify the item 204. In this case, the item tracking device 104 can process the pixels in the first image 122 to identify text, colors, barcodes, patterns, or other features of the item 204. The item tracking device 104 can then compare the identified features of the item 204 to a set of features that correspond to different items 204. For example, the item tracking device 104 may extract text (e.g., a product name) from the first image 122 and compare the text to a set of text associated with a different item 204. As another example, the item tracking device 104 may determine a dominant color in the first image 122 and compare the dominant color to a set of colors associated with a different item 204. As another example, the item tracking device 104 may identify a barcode in the first image 122 and compare the barcode to a set of barcodes associated with a different item 204. As another example, the item tracking device 104 may identify a logo or pattern in the first image 122 and compare the identified logo or pattern to a set of logos or patterns associated with a different item 204. In other examples, the item tracking device 104 may identify other suitable types or combinations of features and compare the identified features to features associated with a different item 204.

[0077] After comparing the identified features from the first image 122 to the set of features associated with the different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found if at least a meaningful portion of the identified features match a feature corresponding to the item 204. In response to determining that a meaningful portion of a feature in the first image 122 matches a feature of the item 204, the item tracking device 104 identifies a region of interest 1002 that corresponds to the matching item 204. In other embodiments, the item tracking device 104 may use other suitable types of image processing techniques to identify the region of interest 1002. FIGS. 10A, 10B, 10C, and 10D show examples of regions of interest 1002 of an item 204.

[0078] In operation 906, the item tracking device 104 determines a first number of pixels in the region of interest 1002 that corresponds to the item 204 in the first image 122. Here, the item tracking device 104 counts the number of pixels in the plurality of pixels of the identified region of interest 1002. The number of pixels in the region of interest 1002 is proportional to how well the first item 204 was detected in the first image 122. For example, a larger number of pixels in the region of interest 1002 indicates that a larger portion of the item 204 was detected in the first image 122. Alternatively, a smaller number of pixels in the region of interest 1002 indicates that a smaller portion of the item 204 was detected in the first image 122. In some cases, a smaller number of pixels in the region of interest 1002 may indicate that only a small portion of the item 204 was visible to the selected camera 108 or that the region of interest 1002 was incorrectly identified. The item tracking device 104 proceeds to operation 908 and determines whether the region of interest 1002 was correctly identified.

[0079] In operation 908, the item tracking device 104 uses the 3D sensor 110 to capture a first depth image 124 of the item 204 on the platform. Here, the item tracking device 104 uses the 3D sensor 110 to capture the first depth image 124 with a similar view of the item 204 captured by the camera 108 in operation 902. For example, the item tracking device 104 can use a 3D sensor 110 configured to capture an overhead view of the item 204 when capturing the first image 122 using a camera 108 that captures an overhead view of the item 204. As another example, the item tracking device 104 can use a 3D sensor 110 configured to capture an oblique or side profile view of the item 204 when capturing the first image 122 using a camera 108 that captures an oblique or side profile view of the item 204. In other examples, the item tracking device 104 may use a 3D sensor 110 that has other types of views of the item 204 similar to the view captured in the first image 122. Figures 10A, 10B, 10C, and 10D show examples of the first depth image 124.

[0080] In operation 910, the item tracking device 104 determines a second number of pixels in the first depth image 124 that correspond to the item 204. Here, the item tracking device 104 counts the number of pixels in the first depth image 124 that correspond to the item 204. In some embodiments, the item tracking device 104 can use a depth threshold to distinguish between pixels that correspond to the item 204 and pixels that correspond to other items 204 or the platform 202. For example, the item tracking device 104 can set a depth threshold behind the surface of the item 204 that faces the 3D sensor 110. After applying the depth threshold, the remaining pixels in the first depth image 124 correspond to the item 204. The item tracking device 104 can then count the number of remaining pixels in the first depth image 124 after applying the depth threshold to the first depth image 124.

[0081] In operation 912, the item tracking device 104 determines a difference between the first pixel number and the second pixel number, where the item tracking device 104 calculates the difference between the pixel number of the item 204 from the region of interest 1002 and the pixel number of the item 204 from the first depth image 124 to determine how similar the two values ​​are to each other. For example, the item tracking device 104 may subtract the first pixel number from the second pixel number to determine the difference between the two values. In this example, the item tracking device 104 uses the absolute value of the difference between the two values.

[0082] In operation 914, the item tracking device 104 determines whether the difference is less than or equal to the difference threshold. The distance threshold is a user-defined value that identifies the maximum pixel difference at which the identified region of interest 1002 is considered valid for further processing. An invalid region of interest 1002 means that the difference between the number of pixels of the item 204 in the region of interest 1002 and the number of pixels of the item 204 in the first depth image 124 is too large. An invalid region of interest 1002 indicates that the region of interest 1002 captures a smaller portion of the item 204 than is visible to the camera 108 and the 3D sensor 110. Because the invalid region of interest 1002 captures only a small portion of the item 204, the region of interest 1002 may not be suitable for subsequent image processing after the region of interest 1002 is used to crop the first image 122. 10A as an example of an invalid region of interest 1002, the item tracking device 104 identifies a first region of interest 1002A of an item 204 and the first depth image 124. In this example, the difference between the number of pixels of the item 204 in the region of interest 1002 and the number of pixels of the item 204 in the first depth image 124 is greater than a difference threshold. An example of the first region of interest 1002A overlaid on the item 204 in the first depth image 124 is shown in FIG.

[0083] A valid region of interest 1002 means that the difference between the number of pixels of the item 204 in the region of interest 1002 and the number of pixels of the item 204 in the first depth image 124 is within a predetermined tolerance level (i.e., a difference threshold). Referring to FIG. 10C as an example of a valid region of interest 1002, the item tracking device 104 identifies a second region of interest 1002B of the item 204 and the first depth image 124. In this example, the difference between the number of pixels of the item 204 in the region of interest 1002 and the number of pixels of the item 204 in the first depth image 124 is equal to or less than the difference threshold. An example of the second region of interest 1002B overlaid on the item 204 in the first depth image 124 is shown in FIG. 10D.

[0084] 9, in response to determining that the difference is greater than the difference threshold, the item tracking device 104 returns to operation 904. In this case, the item tracking device 104 discards the current region of interest 1002 and returns to operation 904 to obtain a new region of interest 1002 for the item 204. In response to determining that the difference is less than or equal to the difference threshold, the item tracking device 104 proceeds to operation 916. In this case, the item tracking device 104 proceeds to operation 916 to crop the first image 122 using the identified region of interest 1002.

[0085] In operation 916, the item tracking device 104 crops the first image 122 based on the region of interest 1002. After determining that the region of interest 1002 is a valid additional process, the item tracking device 104 crops the first image 122 by extracting pixels within the region of interest 1002 from the first image 122. By cropping the first image 122, the item tracking device 104 generates a second image 122 that includes the extracted pixels within the region of interest 1002 of the first image 122.

[0086] In operation 918, the item tracking device 104 outputs the second image 122. After generating the second image 122, the item tracking device 104 can output the second image 122 for further processing. For example, the item tracking device 104 can output the second image 122 by inputting or loading the second image 122 into a machine learning model 126 to identify the item 204 using a process similar to the process 2300 described in FIG. 23. As another example, the item tracking device 104 can associate the second image 122 with feature descriptors 1608 (e.g., item type 1610, dominant color 1612, dimensions 1614, weight 1616) of the item 204 using a process similar to the process 2300 described in FIG. 23.

[0087] Item Location Detection Process FIG. 11 is a flow chart of one embodiment of an item location process 1100 of the item tracking system 100. The item tracking system 100 can use the process 1100 to identify groups of images 122 that correspond to the same item 204. The item tracking device 104 typically uses multiple cameras 108 to capture images 122 of items 204 on the platform 202 from multiple viewpoints. This process allows the item tracking device 104 to use redundancy to ensure that all items 204 appear in at least one of the captured images 122. Because each camera 108 has a different physical location and viewpoint on the platform 202, the items 204 appear in different positions in each of the captured images 122. To solve this problem, the item tracking device 104 uses a homography 608 to cluster images 122 of the same item 204 together based on the physical location of each item 204 on the platform 202. This process allows the item tracking device 104 to generate a set of images 122 of each item 204 located on the platform 202 using images 122 captured from multiple camera perspectives.

[0088] Homography Generation The item tracking device 104 is configured to generate and use homographies 608 to map pixels from the camera 108 and 3D sensor 110 to the platform 202. Examples of homographies 608 are described below in Figures 12A and 12B. By generating homographies 608, the item tracking device 104 can use the location of the item 204 in the image 122 to determine the physical location of the item 204 relative to the platform 202, the camera 108, and the 3D sensor 110. This allows the item tracking device 104 to use the physical location of the item 204 to cluster the images 122 and depth images 124 of the item 204 together for processing. Each homography 608 is made up of coefficients configured to convert between pixel locations in the image 122 or depth image 124 and (x,y) coordinates in a global plane (i.e., physical locations on the platform 202). Each image 122 and depth image 124 is made up of a number of pixels. The location of each pixel in the image 122 or depth image 124 is described by its pixel location 1202, which identifies the pixel row and pixel column in which the pixel is located in the image 122 or depth image 124.

[0089] The item tracking device 104 uses the homography 608 to associate pixel locations in a particular camera 108 or 3D sensor 110 with a physical location on the platform 202. In other words, the item tracking device 104 uses the homography 608 to determine where an item 204 is physically located on the platform 202 based on pixel locations 1202 in an image 122 or depth image 124 from a camera 108 or 3D sensor 110, respectively. Because the item tracking device 104 monitors the platform 202 using multiple cameras 108 and 3D sensors 110, each camera 108 and 3D sensor 110 is uniquely associated with a different homography 608 based on the physical location of the camera 108 or 3D sensor 110 on the imaging device 102. This configuration enables the item tracking device 104 to determine where the item 204 is physically located on the platform 202 based on which camera 108 or 3D sensor 110 the item 204 appears in and the location of the item in the image 122 or depth image 124 captured by that camera 108 or 3D sensor 110. In this configuration, the camera 108 and 3D sensor 110 are configured to capture images 122 and depth images 124, respectively, of at least partially overlapping portions of the platform 202.

[0090] 12A , homography 608 is composed of a number of coefficients configured to transform between pixel locations 1202 in image 122 or depth image 124 and physical locations (e.g., (x,y) coordinates 1204) in a global plane corresponding to the top surface of platform 202. In this example, homography 608 is configured as a matrix, and the coefficients of homography 608 are 11 , H 12 , H 13 , H 14 , H 21 , H 22 , H 23 , H 24 , H 31 , H 32 , H 33 , H 34 , H41 , H 42 , H 43 , and H 44 The item tracking device 104 may generate the homography 608 by using the coefficients to define a relationship or function between pixel locations 1202 in the image 122 or depth image 124 and physical locations (e.g., (x,y) coordinates 1204) in the global plane. For example, the item tracking device 104 may use the coefficients to define one or more functions and perform a regression (e.g., least squares regression) to solve for values ​​of the coefficients that project pixel locations 1202 in the image 122 or depth image 124 to (x,y) coordinates 1204 in the global plane. Each (x,y) coordinate 1204 identifies an x ​​value and a y value in the global plane where an item is located on the platform 202. In other examples, the item tracking device 104 may use other suitable techniques to solve for the coefficients of the homography 608. In the example shown in FIG. 5A, the z value of the pixel location 1202 may correspond to a pixel value that represents distance, depth, altitude, or height. In this case, the homography 608 is further configured to convert between pixel values ​​of the depth image 124 and z-coordinates (eg, height or altitude) in the global plane.

[0091] The item tracking device 104 can use the inverse of the homography 608 to project from the (x,y) coordinates 1204 in the global plane to the pixel locations 1202 in the image 122 or depth image 124. For example, the item tracking device 104 receives the (x,y) coordinates 1204 in the global plane of an object. The item tracking device 104 identifies the homography 608 associated with the camera 108 or 3D sensor 110 in which the object is viewed. The item tracking device 104 can then apply the inverse homography 608 to the (x,y) coordinates 1204 to determine the pixel locations 1202 where the object is located in the image 122 or depth image 124. The item tracking device 104 can calculate the inverse of the homography 608 when the homography 608 is represented as a matrix. Referring to FIG. 12B as an example, the item tracking device 104 can perform a matrix multiplication between the (x,y) coordinates 1204 in the global plane and the inverse homography 608 to determine the corresponding pixel location 1202 in the image 122 or the depth image 124.

[0092] Additional information regarding the generation of homographies 608 and uses of homographies 608 is disclosed in U.S. Pat. No. 11,023,741 (Attorney Docket No. 090278.0233), entitled “DRAW WIRE ENCODER BASED HOMOGRAPHY,” which is incorporated by reference herein as if reproduced in its entirety.

[0093] Using homography to determine item position Returning to FIG. 11, after generating the homography 608 of the camera 108 and / or 3D sensor 110, the item tracking device 104 can then use the homography 608 to cluster the image 122 and depth image 124 of the item 204 together for processing. In operation 1102, the item tracking device 104 captures a first image 122 of the item 204 using the first camera 108. The first camera 108 can be configured to capture an upward facing surface and / or a side of the item 204 on the platform 202. Referring to FIG. 13A, the item tracking device 104 captures a first image 1302 of items 204A and 204B that are on the platform 202 using the first camera 108.

[0094] Returning to operation 1104 of FIG. 11 , the item tracking device 104 identifies a first region of interest 1304 of the item 204 in the first image 122. The first region of interest 1304 is comprised of a number of pixels that correspond to the item 204 in the first image 122. One example of the region of interest 1304 is a bounding box. In some embodiments, the item tracking device 104 can use one or more image processing techniques to identify the region of interest 1304 of the item 204 in the first image 122. For example, the item tracking device 104 can use object detection and / or OCR to identify text, logos, brands, colors, barcodes, or any other features of the item 204 that can be used to identify the item 204. In this case, the item tracking device 104 can process the pixels in the image 122 to identify text, colors, barcodes, patterns, or any other features of the item 204. The item tracking device 104 can then compare the identified features of the item 204 to a set of features that correspond to different items 204. For example, the item tracking device 104 may extract text (e.g., a product name) from the image 122 and compare the text to a set of text associated with different items 204. As another example, the item tracking device 104 may determine a dominant color in the image 122 and compare the dominant color to a set of colors associated with different items 204. As another example, the item tracking device 104 may identify a barcode in the image 122 and compare the barcode to a set of barcodes associated with different items 204. As another example, the item tracking device 104 may identify a logo or pattern in the image 122 and compare the identified logo or pattern to a set of logos or patterns associated with different items 204. In other examples, the item tracking device 104 may identify other suitable types or combinations of features and compare the identified features to features associated with different items 204.

[0095] After comparing the identified features from the image 122 to the set of features associated with the different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found if at least a meaningful portion of the identified features match features corresponding to the items 204. In response to determining that the meaningful portion features in the image 122 match features of the items 204, the item tracking device 104 may identify an area of ​​interest 1304 corresponding to the matching item 204. In other embodiments, the item tracking device 104 may identify the area of ​​interest 1304 using other suitable types of image processing techniques. Returning to the example of FIG. 13A, the item tracking device 104 identifies a first area of ​​interest 1304A corresponding to the first item 204A and a second area of ​​interest 1304B corresponding to the second item 204B in the first image 1302.

[0096] Returning to operation 1106 of FIG. 11 , the item tracking device 104 identifies a first pixel location 1202 within a first region of interest 1304. The pixel location 1202 may be any pixel within the first region of interest 1304. In some embodiments, the item tracking device 104 may identify a pixel location 1202 that is closest to the platform 202. For example, the item tracking device 104 may identify a pixel location 1202 that is at the midpoint of the bottom edge of the region of interest 1304. Returning to the example of FIG. 13A , the item tracking device 104 may identify a pixel location 1202A within a first region of interest 1304A of a first item 204A and a pixel location 1202B within a second region of interest 1304B of a second item 204B.

[0097] 11 , the item tracking device 104 applies the first homography 608 to the first pixel locations 1202 to determine first (x,y) coordinates 1204 on the platform 202 of the items 204. For example, the item tracking device 104 identifies a homography 608 associated with the first camera 108 and then applies the identified homography 608 to the pixel locations 1202 of each item 204 to determine the corresponding (x,y) coordinates 1204 on the platform 202.

[0098] In operation 1110, the item tracking device 104 captures a second image 122 of the item 204 using the second camera 108. Here, the item tracking device 104 captures different views of the item 204 on the platform 202 using different cameras 108. The second camera 108 can be configured to capture an upward facing surface and / or a side of the item 204 on the platform 202. Referring to the example of FIG. 13B, the item tracking device 104 captures a second image 1306 of the items 204A and 204B on the platform 202 using the second camera 108. In this example, the second camera 108 is on an opposite side of the platform 202 from the first camera 108. In this example, the first camera 108 captures a first side of the item 204 on the platform 202, and the second camera 108 captures an opposite side of the item 204 on the platform 202. In other examples, the second camera 108 may be located in other suitable locations.

[0099] Returning to operation 1112 of FIG. 11 , the item tracking device 104 identifies a second region of interest 1304 of the item 204 in the second image 122. The second region of interest 1304 includes a second plurality of pixels that corresponds to the item 204 in the second image 122. The item tracking device 104 can repeat the process described in operation 1104 to identify the second region of interest 1304. Returning to the example of FIG. 13B , the item tracking device 104 identifies a third region of interest 1304C that corresponds to the first item 204A and a fourth region of interest 1304D that corresponds to the second item 204B in the second image 1306.

[0100] Returning to operation 1114 of Figure 11, the item tracking device 104 locates the second pixel location 1202 within the second region of interest 1304. Returning to the example of Figure 13B, the item tracking device 104 can identify a pixel location 1202C within a third region of interest 1304C of the first item 204A and a pixel location 1202D within a fourth region of interest 1304D of the second item 204B.

[0101] 11 , the item tracking device 104 applies the second homography 608 to the second pixel locations 1202 to determine second (x,y) coordinates 1204 on the platform 202 of the items 204. Here, the item tracking device 104 identifies a homography 608 associated with the second camera 108 and then applies the identified homography 608 to the pixel locations 1202 of each item 204 to determine the corresponding (x,y) coordinates 1204 on the platform 202.

[0102] The item tracking device 104 may repeat this process for any other suitable number of cameras 108. As another example, referring to FIGURE 13C, the item tracking device 104 uses a third camera 108 to capture a third image 1308 of the items 204 on the platform 202. The item tracking device 104 then identifies the region of interest 1304 and pixel location 1202 for each item 204. In this example, the item tracking device 104 identifies the region of interest 1304E and pixel location 1202E of the first item 204A and identifies the region of interest 1304F and pixel location 1202F of the second item 204B. After determining the pixel locations 1202 of the items 204, the item tracking device 104 then identifies a homography 608 associated with the third camera 108 and applies the identified homography 608 to the pixel locations 1202 of each item 204 to determine the corresponding (x,y) coordinates 1204 on the platform 202.

[0103] Returning to operation 1118 of FIG. 11, the item tracking device 104 determines a distance 1402 between a first (x,y) coordinate 1204 and a second (x,y) coordinate 1204. Referring to FIG. 14 as an example, FIG. 14 shows an overhead view of the platform 202 where the (x,y) coordinates 1204 of each item 204 are projected onto the platform 202. In this example, the (x,y) coordinates 1204A, 1204B, and 1204C are associated with the first item 204A, and the (x,y) coordinates 1204D, 1204E, and 1204F are associated with the second item 204B. The item tracking device 104 is configured to iteratively select pairs of (x,y) coordinates 1204 and determine a distance 1402 between the pair of (x,y) coordinates 1204. In one embodiment, the item tracking device 104 is configured to determine the Euclidean distance between a pair of (x,y) coordinates 1204 .

[0104] Returning to operation 1120 of FIG. 11 , the item tracking device 104 determines whether the distance 1402 is less than or equal to a distance threshold. The distance threshold identifies the maximum distance between a pair of (x,y) coordinates 1204 that are considered members of the same cluster 1404 of items 204. The distance threshold is a user-defined value that can be set to any suitable value. The units of the distance threshold may be inches, centimeters, millimeters, or other suitable units. The item tracking device 104 compares the distance 1402 between the pair of (x,y) coordinates 1204 to the distance threshold to determine whether the distance 1402 between the pair of (x,y) coordinates 1204 is less than the distance threshold.

[0105] In response to determining that the distance 1402 is greater than the distance threshold, the item tracking device 104 terminates the process 1100. In this case, the item tracking device 104 determines that the pair of (x,y) coordinates 1204 are not members of the same cluster 1404 of items 204. In some embodiments, the item tracking device 104 does not terminate the process 1100, but instead selects another pair of (x,y) coordinates 1204 if additional (x,y) coordinates 1204 are available for comparison to the distance threshold.

[0106] In response to determining that the distance 1402 is less than or equal to the distance threshold, the item tracking device 104 proceeds to operation 1122. In this case, the item tracking device 104 determines that the pair of (x,y) coordinates 1204 are members of the same cluster 1404 of the item 204. In operation 1122, the item tracking device 104 associates pixels in the first region of interest 1304 of the first image 122 and pixels in the second region of interest 1304 of the second image 122 with the cluster 1404 of the item 204. Referring to FIG. 14 as an example, the item tracking device 104 can identify a first cluster 1404A of the first item 204A and a second cluster 1404B of the second item 204B. A first cluster 1404A is associated with (x,y) coordinates 1204A, 1024B, and 1204C and regions of interest 1304A, 1304C, and 1304E. A second cluster 1404B is associated with (x,y) coordinates 1204D, 1024E, and 1204F and regions of interest 1304B, 1304D, and 1304F.

[0107] Returning to operation 1124 of FIG. 11 , the item tracking device 104 outputs the pixels within the first region of interest 1304 from the first image 122 and the pixels within the second region of interest 1304 from the second image 122. In one embodiment, the item tracking device 104 crops the captured image 122 by extracting the pixels within the identified region of interest 1304 from the image 122. By cropping the image 122, the item tracking device 104 generates a new image 122 that includes the extracted pixels within the region of interest 1304 of the original image 122. This process allows the item tracking device 104 to generate a set of new images 122 of the item 204, each image including pixels extracted from the identified region of interest 1304 associated with the item 204. The item tracking device 104 can output the new images 122 of the item 204 for further processing. For example, the item tracking device 104 may output the image 122 by inputting or loading the image 122 into a machine learning model 126 and outputting it to identify the item 204 based on physical attributes of the item 204 in the image 122, using a process similar to process 2300 described in FIG.

[0108] In some embodiments, the item tracking device 104 may also associate the identified feature descriptor with the image 122 of the item 204 and output the feature descriptor along with the image 122 of the item 204. For example, in determining the area of ​​interest 1304 of the item 204, the item tracking device 104 may identify an item type of the item 204. In this example, the item tracking device 104 may associate the item type with the area of ​​interest 1304 and output the item type along with the image 122 of the item 204 generated based on the area of ​​interest 1304. As another example, the item tracking device 104 may obtain a weight of the item 204 using a weight sensor 122. In this example, the item tracking device 104 may associate the weight with the area of ​​interest 1304 and output the weight along with the image 122 of the item 204 generated based on the area of ​​interest 1304. In other examples, the item tracking device 104 may be configured to identify and associate other suitable types of feature descriptors with the area of ​​interest 1304 prior to outputting the area of ​​interest 1304.

[0109] Search space reduction process for the encoded vector library 15 is a flow chart of one embodiment of a search space reduction process 1500 for the encoded vector library 128. The item tracking system 100 may use the process 1500 to filter entries 1602 in the encoded vector library 128 to reduce the amount of items 204 that are considered when attempting to identify an item 204 placed on a platform 202. This process reduces the time required to search for a corresponding entry 1602 in the encoded vector library 128 as well as improves the accuracy of the results of identifying the entry 1602 in the encoded vector library 128.

[0110] In operation 1502, the item tracking device 104 obtains feature descriptors 1608 for the item 204. Each of the feature descriptors 1608 describes a physical characteristic or attribute of the item 204. Examples of the feature descriptors 1608 include, but are not limited to, item type 1610, dominant color 1612, dimensions 1614, weight 1616, or any other suitable type of descriptor that describes the item 204. In one embodiment, the item tracking device 104 may obtain the feature descriptors using a process similar to the process described in operation 1104 of FIG. 11. For example, the item tracking device 104 may use object detection and / or OCR to identify text, logos, brands, colors, barcodes, or other features of the item 204 that can be used to identify the item 204. In some embodiments, the item tracking device 104 may determine the dimensions of the item 204 using a process similar to process 1800 described in FIG. 18. The item tracking device 104 may determine the weight of the item 204 using the weight sensor 112. In other embodiments, the item tracking device 104 may use other suitable processes for determining feature descriptors for the item 204.

[0111] In operation 1504, the item tracking device 104 determines whether the feature descriptor 1608 identifies the item type 1610 of the item 204. Here, the item tracking device 104 determines whether information associated with the item type 1610 of the item 204 is available. The item type 1610 identifies a classification of the item 204. For example, the item type 1610 may indicate whether the item 204 is a can, a bottle, a box, a fruit, a bag, etc. In response to determining that the feature descriptor 1608 identifies the item type 1610 of the item 204, the item tracking device 104 proceeds to operation 1506. In this case, the item tracking device 104 filters the encoded vector library 128 using the item type 1610 to reduce the number of entries 1602 in the encoded vector library 128 before attempting to identify the item 204.

[0112] In operation 1506, the item tracking device 104 filters the encoded vector library 128 based on the item type 1610. Referring to FIG. 16 as an example, the encoded vector library 128 is comprised of a number of entries 1602. Each entry 1602 corresponds to a different item 204 that can be identified by the item tracking device 104. Each entry 1602 may be comprised of an encoded vector 1606 linked to an item identifier 1604 and a number of feature descriptors 1608. The encoded vector 1606 is comprised of an array of numbers. Each number corresponds to and describes an attribute of the item 204 (e.g., item type, size, shape, color, etc.). The encoded vector 1606 may be of any suitable length. For example, the size of the encoded vector 1606 may be 1×256, 1×512, 1×1024, or any other suitable length. The item identifier 1604 uniquely identifies the item 204. Examples of item identifiers 1604 include, but are not limited to, product names, SKU numbers, alphanumeric codes, graphical codes (e.g., bar codes), or other suitable types of identifiers. In this example, the item tracking device 104 uses the item type 1610 to filter or remove entries 1602 in the encoded vector library 128 that do not contain the same item type 1610. This process reduces the number of entries 1602 in the encoded vector library 128 that are considered when identifying an item 204.

[0113] Returning to operation 1504 of FIG. 15 , in response to determining that the feature descriptor 1608 does not identify the item type 1610, the item tracking device 104 proceeds to operation 1508. In this case, the item tracking device 104 checks for other types of feature descriptors 1608 that can be used to filter the entries 1602 in the encoded vector library 128. At operation 1508, the item tracking device 104 determines whether the feature descriptor 1608 identifies a dominant color 1612 of the item 204. The dominant color 1612 identifies one or more colors that appear on a surface (e.g., packaging) of the item 204.

[0114] In response to the feature descriptor 1608 identifying a dominant color 1612 of the item 204, the item tracking device 104 proceeds to operation 1510. In this case, the item tracking device 104 proceeds to operation 1510 to reduce the number of entries 1602 in the encoded vector library 128 based on the dominant color 1612 of the item 204. In operation 1510, the item tracking device 104 filters the encoded vector library 128 based on the dominant color 1612 of the item 204. Here, the item tracking device 104 uses the dominant color 1612 to filter out or remove entries 1602 in the encoded vector library 128 that do not contain the same dominant color 1612.

[0115] Returning to operation 1508, in response to determining that the feature descriptor 1608 does not identify a dominant color 1612 of the item 204, the item tracking device 104 proceeds to operation 1512. At operation 1512, the item tracking device 104 determines whether the feature descriptor 1608 identifies dimensions 1614 of the item 204. The dimensions 1614 may identify the length, width, and height of the item 204. In some embodiments, the dimensions 1614 may be listed in ascending order.

[0116] In response to determining that the feature descriptor 1608 identifies the dimensions 1614 of the item 204, the item tracking device 104 proceeds to operation 1514. In this case, the item tracking device 104 proceeds to operation 1514 to reduce the number of entries 1602 in the encoded vector library 128 based on the dimensions 1614 of the item 204. In operation 1514, the item tracking device 104 filters the encoded vector library 128 based on the dimensions 1614 of the item 204. Here, the item tracking device 104 uses the dimensions 1614 to filter out or remove entries 1602 in the encoded vector library 128 that do not include the same dimensions 1614 as the item 204 or that are within a predetermined tolerance of the dimensions 1614 of the item 204. In some embodiments, the dimensions 1614 of the items 204 may be listed in ascending order to facilitate comparison between the dimensions 1614 of the items 204 and the dimensions 1614 of the items 204 in the encoded vector library 128.

[0117] Returning to operation 1512, in response to determining that the feature descriptors 1608 do not specify the dimensions 1614 of the item 204, the item tracking device 104 proceeds to operation 1516. At operation 1516, the item tracking device 104 determines whether the feature descriptors 1608 specify a weight 1616 of the item 204. The weight 1616 specifies a weight of the item 204. The weight 1616 may be expressed in pounds, ounces, liters, or other suitable units.

[0118] In response to determining that the feature descriptor 1608 identifies the weight 1616 of the item 204, the item tracking device 104 proceeds to operation 1518. In this case, the item tracking device 104 proceeds to operation 1518 to reduce the number of entries 1602 in the encoded vector library 128 based on the weight 1616 of the item 204.

[0119] In operation 1518, the item tracking device 104 filters the encoded vector library 128 based on the weight of the item 204. Here, the item tracking device 104 uses the weight 1616 to filter out or remove entries 1602 in the encoded vector library 128 that do not contain the same weight 1616 as the item 204 or that are within a predetermined tolerance of the weight 1616 of the item 204.

[0120] In some embodiments, the item tracking device 104 may repeat a similar process to filter or reduce the number of entries 1602 in the encoded vector library 128 based on other suitable types or combinations of feature descriptors 1608.

[0121] Similar Vectors After filtering the encoded vector library 128 based on the feature descriptors 1608 of the item 204, the item tracking device 104 can generate a similarity vector 1704 of the received encoded vector. The similarity vector 1704 is comprised of an array of numbers 1710, where each number 1710 indicates how similar a value of the received encoded vector 1702 is to a value of an encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 can generate the similarity vector 1704 using a matrix multiplication between the received encoded vector 1702 and the encoded vector 1606 in the encoded library 128. Referring to FIG. 17 as an example, the encoded vector 1606 in the encoded vector library 128 has dimensions M×N, where M is the number of entries 1602 in the encoded vector library 128, e.g., after filtering the encoded vector library 128, and N is the length of each encoded vector 1606, which corresponds to the number of numeric values ​​1706 in the encoded vector 1606. The encoded vector 1702 of the unidentified item 204 has dimensions N×1, where N is the length of the encoded vector 1702, which corresponds to the number of numeric values ​​1708 in the encoded vector 1702. In this example, the item tracking device 104 can generate a similar vector 1704 by performing a matrix multiplication between the encoded vector 1702 and the encoded vector 1606 in the encoded vector library 128. The resulting similar vector 1704 has dimensions N×1, where N is the length of the similar vector 1704, which is the same length as the encoded vector 1702. Each value 1710 in the similarity vector 1704 corresponds to an entry 1602 in the encoded vector library 128 .For example, a first numerical value 1710 in similarity vector 1704 indicates how similar the value of encoded vector 1702 is to the value of encoded vector 1606 in the first entry 1602 of encoded vector library 128, a second numerical value 1710 in similarity vector 1704 indicates how similar the value of encoded vector 1702 is to the value of encoded vector 1606 in the second entry 1602 of encoded vector library 128, and so on.

[0122] After generating the similarity vector 1704, the item tracking device 104 can identify which entry 1602 in the encoded vector library 128 best matches the encoded vector 1702 of the identified item 204. In one embodiment, the entry 1602 associated with the highest numerical value 1710 in the similarity vector 1704 corresponds to the entry 1602 that best matches the encoded vector 1702 of the item 204. After identifying the entry 1602 from the encoded vector library 128 that best matches the encoded vector 1702 of the identified item 204, the item tracking device 104 can then identify the item identifier 1604 associated with the identified entry 1602. Through this process, the item tracking device 104 can determine which item 204 in the encoded vector library 128 corresponds to the unidentified item 204 based on the enclosed vector 1702. The item tracking device 104 then outputs or uses the identified item identifier 1604 in other processes, such as process 2300 described in FIG.

[0123] Item dimensioning process using point cloud information FIG. 18 is a flow chart of one embodiment of an item dimension determination process 1800 using point cloud information. The item tracking system 100 can use the process 1800 to determine the dimensions 1614 of an item 204 placed on a platform 202. This process generally involves first capturing 3D point cloud data of the item 204 using multiple 3D sensors 110, and then combining the 3D point cloud data from all the 3D sensors 110 to generate a more complete point cloud representation of the item 204. After combining the point cloud data from the 3D sensors 110, the item tracking device 104 then determines the dimensions 1614 of the item 204 based on the new point cloud data representation. This process allows the item tracking device 104 to determine the dimensions 1614 of the item 204 without the user having to physically measure the item 204.

[0124] In an operation 1802, the item tracking device 104 uses the overhead 3D sensor 110 to capture point cloud data 1902 of an item 204 on a platform 202. The point cloud data 1902 is comprised of a number of data points 1901 in 3D space. Each data point 1901 is associated with (x, y, z) coordinates that identify the location of the data point 1901 in the 3D space. In general, the point cloud data 1902 corresponds to a surface of an object as seen by the 3D sensor 110. Referring to FIG. 19 as an example, FIG. 19 illustrates an example of point cloud data 1902 captured using the overhead 3D sensor 110. In this example, the 3D sensor 110 is positioned directly above the platform 202 and configured to capture point cloud data 1902 representing an upward facing surface of the item 204 on the platform 202. The 3D sensor 110 captures point cloud data 1902A corresponding to the first item 204 and point cloud data 1902B corresponding to the second item 204.

[0125] Returning to operation 1804 of FIG. 18 , the item tracking device 104 segments the point cloud data 1902 based on clusters 1904 in the point cloud data 1902. In one embodiment, the item tracking device 104 can identify clusters 1904 in the point cloud data 1902 based on distances between data points 1901 in the point cloud data 1902. For example, the item tracking device 104 can use a distance threshold to identify data points 1901 that are members of the same cluster 1904. In this example, the item tracking device 104 can calculate the Euclidean distance between a pair of data points 1901 to determine whether the data points 1901 are members of the same cluster 1904. For example, if a pair of data points 1901 are within a distance threshold of each other, the item tracking device 104 can associate the data points 1901 with the same cluster 1904. If the distance between a pair of data points 1901 is greater than the distance threshold, the item tracking device 104 determines that the data points 1901 are not members of the same cluster 1904. The item tracking device 104 may repeat this process until it identifies one or more clusters 1904 in the point cloud data 1902. In other examples, the item tracking device 104 may cluster the data points 1901 using k-means clustering or other suitable clustering techniques. After identifying the clusters 1904 in the point cloud data 1902, the item tracking device 104 segments the point cloud data 1902 based on the identified clusters 1904. Segmenting the point cloud data 1902 divides the data points 1901 in the point cloud data 1902 into smaller groups of point cloud data 1902 based on the identified clusters 1904. Each cluster 1904 of data points 1901 corresponds to a different item 204 placed on the platform 202.

[0126] In operation 1806, the item tracking device 104 selects a first item 204 from the segmented point cloud data 1902. The item tracking device 104 now identifies one of the items 204 on the platform 202 and begins aggregating point cloud data 1902 from the other 3D sensors 110 associated with the first item 204. The item tracking device 104 can iteratively select each item 204 from the platform 202. Returning to the example of FIG. 19, the item tracking device 104 can select the first item 204 that corresponds to cluster 1904A.

[0127] Returning to operation 1808 of FIG. 18 , the item tracking device 104 identifies a region of interest 1906 of the first item 204 in the point cloud data 1902. The region of interest 1906 identifies an area in 3D space. For example, the region of interest 1906 may define a range of x-values, y-values, and / or z-values ​​in the 3D space. Returning to the example of FIG. 19 , the item tracking device 104 may identify a region of interest 1906A that includes the point cloud data 1902A of the first item 204. In this example, the item tracking device 104 identifies a range of x-values, y-values, and z-values ​​in the 3D space that includes the point cloud data 1902A.

[0128] Returning to operation 1810 of FIG. 18 , the item tracking device 104 extracts point cloud data 1902 from the identified region of interest 1906. Here, the item tracking device 104 identifies and extracts point cloud data 1902 from within the region of interest 1906 of a first item 204. By extracting the point cloud data 1902 within the region of interest 1906, the item tracking device 104 can separate the data points 1901 of the first item 204 in the point cloud data 1902 from the data points 1901 associated with other items 204 on the platform 202. Returning to the example of FIG. 19 , the item tracking device 104 can extract the data points 1901 (i.e., point cloud data 1902A) within the region of interest 1906A from the point cloud data 1902 of all items 204 on the platform 202.

[0129] Returning to operation 1812 of FIG. 18 , the item tracking device 104 selects another 3D sensor 110. After extracting the point cloud data 1902 of the first item 204 from the overhead 3D sensor 110, the item tracking device 104 can repeat the same process to extract additional point cloud data 1902 of the first item 204 from the perspective of the other 3D sensor 110. Each 3D sensor 110 can capture point cloud data 1902 only for the portion of the first item 204 visible to the 3D sensor 110. By capturing point cloud data 1902 from multiple 3D sensors 110 capturing different views of the first item 204, the item tracking device 104 can generate a more complete point cloud data representation of the first item 204. The item tracking device 104 can iteratively select different 3D sensors 110 from the 3D sensors 110 of the imaging device 102.

[0130] In operation 1814, the item tracking device 104 captures point cloud data 1902 using the selected 3D sensor 110. Here, the item tracking device 104 captures the point cloud data 1902 using the selected 3D sensor 110 using a process similar to that described in operation 1802. With reference to FIG. 20 as an example, the item tracking device 104 can select a 3D sensor 110 having a side perspective view of an item 204 on a platform 202. In other words, the selected 3D sensor 110 captures point cloud data 1902 representing a side of the item 204 on the platform 202. In this example, the 3D sensor 110 captures point cloud data 1902C corresponding to the first item 204 and point cloud data 1902D corresponding to the second item 204.

[0131] Returning to operation 1816 of FIG. 18 , the item tracking device 104 identifies a region of interest 1906 corresponding to the first item 204 of the selected 3D sensor 110. In one embodiment, the item tracking device 104 can use the homography 608 to determine the region of interest 1906 of the selected 3D sensor 110 based on the region of interest 1906 identified by the overhead 3D sensor 110. In this case, the item tracking device 104 can identify the homography 608 associated with the selected 3D sensor 110. The homography 608 is configured similarly to that described in FIG. 12A and FIG. 12B. After identifying the homography 608 associated with the 3D sensor 110, the item tracking device 104 uses the homography 608 to transform a range of x, y, and z values ​​in the 3D space associated with the region of interest 1906 of the overhead 3D sensor 110 to a corresponding range of x, y, and z values ​​in the 3D space associated with the selected 3D sensor 110. In other examples, the item tracking device 104 may use any other suitable technique to identify the region of interest 1906 of the first item 204. For example, the item tracking device 104 may use a process similar to that described in operation 1808. Returning to the example of FIG. 20, the item tracking device 104 identifies a region of interest 1906B that includes the point cloud data 1902C of the first item 204. In this example, the item tracking device 104 identifies a range of x, y, and z values ​​in 3D space that includes the point cloud data 1902C.

[0132] Returning to operation 1818 of Figure 18, the item tracking device 104 extracts point cloud data 1902 from the region of interest 1906 corresponding to the first item 204. Here, the item tracking device 104 identifies and extracts point cloud data 1902 from within the identified region of interest 1906 of the first item 204. Returning to the example of Figure 20, the item tracking device 104 may extract data points 1901 within the region of interest 1906B (i.e., point cloud data 1902C) from the point cloud data 1902 of all items 204 on the platform 202.

[0133] Returning to operation 1820 of FIG. 18 , the item tracking device 104 determines whether to select another 3D sensor 110. Here, the item tracking device 104 determines whether to collect additional point cloud data 1902 for the first item 204. In one embodiment, the item tracking device 104 can determine whether to select another 3D sensor 110 based on the amount of collected point cloud data 1902. For example, the item tracking device 104 may be configured to collect point cloud data 1902 from a predetermined number (e.g., three) of 3D sensors 110. In this example, the item tracking device 104 may track how many sets of point cloud data 1902 to collect. Each set of collected point cloud data 1902 corresponds to point cloud data 1902 obtained from a 3D sensor 110. The item tracking device 104 then compares the number of collected sets of point cloud data 1902 to the predetermined number of 3D sensors 110. The item tracking device 104 determines to select another 3D sensor 110 when the number of collected sets of point cloud data 1902 is less than the predetermined number of 3D sensors 110.

[0134] As another example, the item tracking device 104 may determine whether to select another 3D sensor 110 to collect additional point cloud data 1902 based on the number of data points 1901 collected for the first item 204. In this example, the item tracking device 104 may determine the number of data points 1901 acquired from all the extracted point cloud data 1902 of the first item 204. The item tracking device 104 compares the number of acquired data points 1901 to a predetermined data point threshold. The data threshold identifies a minimum number of data points 1901 that need to be collected for the first item 204. The item tracking device 104 may determine to select another 3D sensor 110 if the number of collected data points 1901 is less than the predetermined data point threshold. In other examples, the item tracking device 104 may determine whether to select another 3D sensor 110 to collect additional point cloud data 1902 based on other suitable types of criteria.

[0135] In response to determining to select another 3D sensor, the item tracking device 104 returns to operation 1812. In this case, the item tracking device 104 returns to operation 1812 to select another 3D sensor 110 and acquire additional point cloud data 1902 of the first item 204. Referring to FIG. 21 as an example, the item tracking device 104 may determine to select another 3D sensor 110 having a side perspective view of an item 204 on a platform 202. In this example, the 3D sensor 110 captures point cloud data 1902E corresponding to the first item 204 and point cloud data 1902F corresponding to the second item 204. The item tracking device 104 then identifies a region of interest 1906C that includes the point cloud data 1902E of the first item 204. In this example, the item tracking device 104 identifies a range of x, y, and z values ​​in the 3D space that includes the point cloud data 1902E. After identifying the region of interest 1906C, the item tracking device 104 extracts the data points 1901 (i.e., point cloud data 1902E) within the region of interest 1906C from the point cloud data 1902 of all items 204 on the platform 202. The item tracking device 104 can repeat this process for other selected 3D sensors 110.

[0136] Returning to operation 1820 of FIG. 18 , in response to determining not to select another 3D sensor 110, the item tracking device 104 proceeds to operation 1822. In operation 1822, the item tracking device 104 combines the extracted point cloud data 1902 for the first item 204. Here, the item tracking device 104 merges all of the collected point cloud data 1902 into a single point cloud data 1902 set. By combining the point cloud data 1902 from multiple 3D sensors 110, the item tracking device 104 can generate a more complete point cloud data representation of the first item 204 that can be used to determine the dimensions 1614 of the first item 204. Referring to FIG. 22 as an example, the item tracking device 104 can combine the point cloud data 1902A, 1902C, and 1902E into a single point cloud data 1902G set. The combined point cloud data 1902G includes all data points 1901 from point cloud data 1902A, 1902C, and 1902E.

[0137] Returning to operation 1824 of FIG. 18 , the item tracking device 104 determines the dimensions 1614 of the first item 204 based on the combined point cloud data 1902. In one embodiment, the item tracking device 104 can determine the dimensions 1614 of the item 204 by determining the distance between data points 1901 at the edges of the combined point cloud data 1902. For example, the item tracking device 104 identifies a pair of data points 1901 at either end of the combined point cloud data 1902 and then calculates the distance (e.g., Euclidean distance) between the pair of data points 1901. In this example, the distance between the data points 1901 corresponds to the length 2202, width 2204, or height 2206 of the first item 204. In other examples, the item tracking device 104 can determine the dimensions 1614 of the first item 204 using other suitable techniques. Returning to the example of FIG. 22, the item tracking device 104 can determine the length 2202, width 2204, and height 2206 of the first item 204 based on the combined point cloud data 1902G.

[0138] Returning to operation 1826 of FIG. 18 , the item tracking device 104 determines whether to determine the dimensions 1614 of another item 204. In one embodiment, the item tracking device 104 may be configured to determine the dimensions 1614 of all items 204 that are on the platform 202. In this case, the item tracking device 104 may determine whether it has determined the dimensions 1614 of all items 204 that are on the platform 202. The item tracking device 104 determines the dimensions 1614 of another item 204 if the dimensions 1614 of an item 204 are not yet known and have not yet been determined. In other examples, the item tracking device 104 may determine whether to determine the dimensions 1614 of another item 204 based on other suitable criteria.

[0139] In response to determining to find the dimensions 1614 of another item 204, the item tracking device 104 returns to operation 1806. In this case, the item tracking device 104 returns to operation 1806 to collect point cloud data 1902 of the different item 204. The item tracking device 104 can then repeat the same process of aggregating point cloud data 1902 from multiple 3D sensors 110, combining the point cloud data 1902, and determining the dimensions 1614 of the item 204 based on the combined point cloud data 1902.

[0140] In response to determining not to determine the dimensions 1614 of another item 204, the item tracking device 104 may store the dimensions 1614 of the first item 204. For example, the item tracking device 104 may obtain the item identifier 1604 of the first item 204 and then generate an entry 1602 in the encoded vector library 128 that associates the determined length 2202, width 2204, and height 2206 with the first item 204 as feature descriptors 1608. In some embodiments, the item tracking device 104 may store the length 2202, width 2204, and height 2206 of the first item 204 in ascending order when generating the entry 1602.

[0141] In other embodiments, the item tracking device 104 may output or store the determined length 2202, width 2204, and height 2206 of the first item 204 as feature descriptors 1608 for other processes, such as item identification. For example, the item tracking device 104 may use the feature descriptors 1608 to help identify the first item 204 using a process similar to the process 2300 described in FIG.

[0142] Item Tracking Process Using Encoded Vectors FIG. 23 is a flow chart of one embodiment of an item tracking process 2300 using the encoded vector 1606 of the item tracking system 100. The item tracking system 100 can use the process 2300 to identify an item 204 placed on the platform 202 of the imaging device 102 and assign the item 204 to a particular user. As an example, the item tracking system 100 can use the process 2300 in a store to add the item 204 to a digital cart for a user to purchase. As another example, the item tracking system 100 can use the process 2300 in a warehouse or supply room to check out an item to a user. In other examples, the item tracking system 100 can use the process 2300 in other suitable types of applications where an item 204 is assigned or associated with a particular user. This process allows a user to retrieve an item 204 from a space without having to scan or otherwise identify the item 204 they want to retrieve.

[0143] In operation 2302, the item tracking device 104 performs an automatic exclusion of the imaging device 102. The item tracking device 104 can perform the automatic exclusion using a process similar to the process described in operation 302 of FIG. 3. For example, during an initial calibration period, the platform 202 may not have an item 204 placed on it. During this period, the item tracking device 104 uses one or more cameras 108 and / or 3D sensors 110 to capture a reference image 122 and a reference depth image 124, respectively, of the platform 202 without an item 204 placed on the platform 202. The item tracking device 104 can then use the captured image 122 and depth image 124 as reference images to detect that an item 204 has been placed on the platform 202. The item tracking device 104 can then detect that the item 204 has been placed on the surface 208 of the platform 202 based on the difference in depth values ​​between the subsequent depth image 124 and the reference depth image 124 and / or the difference in pixel values ​​between the subsequent image 122 and the reference image 124.

[0144] In operation 2304, the item tracking device 104 determines whether it detects a hand on the platform 202. In one embodiment, the item tracking device 104 can detect a trigger event corresponding to a user's hand detected on the platform 202 using a process similar to the process 700 described in FIG. 7. For example, the item tracking device 104 can check the difference between the reference depth image 124 and the subsequent depth image 124 to detect an object present on the platform 202. The item tracking device 104 then checks whether the object corresponds to the user's hand or to an item 204 placed on the platform 202. The item tracking device 104 determines that the object is a user's hand if a first portion of the object (e.g., the user's wrist or arm) is outside the region of interest 802 of the platform 202 and a second portion of the object (e.g., the user's hand) is inside the region of interest 802 of the platform 202. If this condition is met, the item tracking device 104 determines that the user's hand is detected on the platform 202. In other examples, the item tracking device 104 can detect whether it detects a user's hand on the platform 202 using a proximity sensor, a motion sensor, or other suitable technology.

[0145] In response to determining that the item tracking device 104 has not detected a user's hand on the platform 202, it remains at operation 2304. In this case, the item tracking device 104 remains at operation 2304 to continue checking for the presence of the user's hand as a triggering event. In response to determining that the item tracking device 104 has detected a user's hand, it proceeds to operation 2306. In this case, the item tracking device 104 uses the presence of the user's hand as a triggering event and proceeds to operation 2306 to begin identifying items 204 that the user has placed on the platform 202.

[0146] In act 2306, the item tracking device 104 performs segmentation using an overhead view of the platform 202. In one embodiment, the item tracking device 104 can perform segmentation using a depth image 124 from the 3D sensor 110 configured with an overhead or perspective view of the item 204 on the platform 202. In this example, the item tracking device 104 captures an overhead depth image 124 of the item 204 placed on the platform 202. The item tracking device 104 can then use a depth threshold to distinguish the platform 202 and the item 204 placed on the platform 202 in the captured depth image 124. For example, the item tracking device 104 may set a depth threshold that is just above the surface of the platform 202. This depth threshold may be determined based on pixel values ​​that correspond to the surface of the platform 202 in the reference depth image 124 captured during the automatic exclusion process of act 2302. After setting the depth threshold, the item tracking device 104 may apply the depth threshold to the captured depth image 124 to filter out or remove the platforms 202 from the depth image 124. After filtering the depth image 124, the remaining clusters of pixels correspond to items 204 placed on the platform 202. Each cluster of pixels corresponds to a different item 204. After identifying the clusters of pixels for each item 204, the item tracking device 104 then counts the number of items 204 placed on the platform 202 based on the number of clusters of pixels present in the depth image 124. This number of items 204 is later used to determine whether all items 204 on the platform 202 have been identified.

[0147] In operation 2308, the item tracking device 104 captures an image 122 of an item 204 on the platform 202. Here, the item tracking device 104 uses multiple cameras 108 to capture multiple images 122 of the item 204 on the platform 202. For example, the item tracking device 104 may capture an image 122 of an overhead view, an oblique view, and / or a side view of the item 204 on the platform 202. The item tracking device 104 may also capture multiple depth images 124 of the item 204 on the platform 202 using one or more 3D sensors 110.

[0148] In operation 2310, the item tracking device 104 generates a cropped image 122 of the item 204 in each image 122. In one embodiment, the item tracking device 104 generates the cropped image 122 of the item 204 based on features of the item 204 present in the image 122. The item tracking device 104 may first identify a region of interest (e.g., a bounding box) of the item 204 based on the detected features of the item 204 present in the image 122, and then crop the image 122 based on the identified region of interest. The region of interest is comprised of a number of pixels that correspond to the item 204 in the captured image 122 or depth image 124 of the item 204 on the platform 202. The item tracking device 104 may use one or more image processing techniques to identify the region of interest of the item 204 in the image 122 based on the features and physical attributes of the item 204. For example, the item tracking device 104 may use object detection and / or OCR to identify text, logos, brands, colors, barcodes, or other features of the item 204 that can be used to identify the item 204. In this case, the item tracking device 104 may process pixels in the image 122 to identify text, colors, barcodes, patterns, or other characteristics of the item 204. The item tracking device 104 may then compare the identified features of the item 204 to a set of features corresponding to different items 204. For example, the item tracking device 104 may extract text (e.g., product name) from the image 122 and compare the text to a set of text associated with different items 204. As another example, the item tracking device 104 may determine a dominant color in the image 122 and compare the dominant color to a set of colors associated with different items 204. As another example, the item tracking device 104 may identify a barcode in the image 122 and compare the barcode to a set of barcodes associated with different items 204. As another example, the item tracking device 104 may identify a logo or pattern in the image 122 and compare the identified logo or pattern to a set of logos or patterns associated with different items 204 .In other examples, the item tracking device 104 may identify other suitable types or combinations of features and compare the identified features to features associated with different items 204 .

[0149] After comparing the identified features of an item 204 to the set of features associated with different items 204, the item tracking device 104 then determines whether a match is found. The item tracking device 104 may determine that a match is found if at least a meaningful portion of the identified features match features corresponding to an item 204. In response to determining that features of the meaningful portion in the image 122 match features of the item 204, the item tracking device 104 may identify a region of interest that corresponds to the matching item 204.

[0150] After identifying the region of interest of the item 204, the item tracking device 104 crops the image 122 by extracting pixels from the image 122 that are within the region of interest of the item 204. By cropping the image 122, the item tracking device 104 generates a second image 122 that includes the extracted pixels within the region of interest of the item 204 from the original image 122. This process allows the item tracking device 104 to generate a new image 122 that includes the items 204 that are located on the platform 202. The item tracking device 104 repeats this process for all items 204 in the captured image 122 and for all captured images 122 of the items 204 that are located on the platform 202. The result of this process is a set of cropped images 122, each corresponding to an item 204 placed on the platform 202.

[0151] In some embodiments, the item tracking device 104 can generate the cropped image 122 of the item 204 using a process similar to process 900 of Figure 9. In some embodiments, operation 2310 is optional and may be omitted. For example, if the item tracking device 104 detects that only one item 204 is placed on the platform 202, operation 2310 may be omitted.

[0152] At operation 2312, the item tracking device 104 obtains an encoding vector 1606 for each item 204. The encoding vector 1606 includes an array of numbers. Each number in the encoding vector 1606 corresponds to and describes an attribute (e.g., item type, size, shape, color, etc.) of the item 204. The encoding vector 1606 may be of any suitable length. The item tracking device 104 obtains the encoding vector 1606 for each item 204 by inputting each of the images 122 (e.g., cropped images 122) from operation 2310 into a machine learning model 126. The machine learning model 126 is configured to output the encoding vector 1606 for the item 204 based on features or physical attributes of the item 204 present in the images 122 of the item 204. Examples of physical attributes include, but are not limited to, item type, size, shape, color, or other suitable types of attributes of the item 204. After inputting the images 122 of the items 204 into the machine learning model 126, the item tracking device 104 receives an encoded vector 1606 for the item 204. The item tracking device 104 repeats this process to obtain an encoded vector 1606 for each item 204 on the platform 202.

[0153] In operation 2314, the item tracking device 104 identifies each item 204 in the encoded vector library 128 based on the corresponding encoded vector 1606. Here, the item tracking device 104 uses the encoded vector 1606 of each item 204 to identify the best matching encoded vector 1606 in the encoded vector library 128. In some embodiments, the item tracking device 104 may first reduce the search space in the encoded vector library 128 before attempting to identify the item 204. In this case, the item tracking device 104 may obtain or identify feature descriptors 1608 for the item 204 using a process similar to the process described in operation 1104 of FIG. 11. Each of the feature descriptors 1608 describes a physical characteristic of the item 204. Examples of feature descriptors 1608 include, but are not limited to, item type 1610, dominant color 1612, dimensions 1614, weight 1616, or other suitable types of descriptors that describe the item 204. The item tracking device 104 may use object detection and / or OCR to identify text, logos, brands, colors, bar codes, or other features of the item 204 that can be used to identify the item 204. The item tracking device 104 may determine the dimensions of the item 204 using a process similar to the process 1800 described in FIG. 18. The item tracking device 104 may determine the weight of the item 204 using the weight sensor 112. In other embodiments, the item tracking device 104 may use other suitable processes to determine the feature descriptors 1608 of the item 204. After obtaining the feature descriptors 1608 of the item 204, the item tracking device 104 may filter or exclude entries 1602 from consideration in the encoded vector library 128 using a process similar to the process 1500 of FIG. 15.After filtering the entries 1602 in the encoded vector library 128, the item tracking device 104 can identify the encoded vector 1606 in the encoded vector library 128 that best matches the encoded vector 1606 of the unrecognized item 204. This process reduces the time required to search for the corresponding entry 1602 in the encoded vector library 128 and improves the accuracy of the identification of the entry 1602 in the encoded vector library 128.

[0154] In one embodiment, the item tracking device 104 identifies the best matching encoded vector 1606 in the encoded vector library 128 by generating a similarity vector 1704 between the encoded vector 1606 of the unrecognized item 204 and the remaining encoded vectors 1606 in the encoded vector library 128. The similarity vector 1704 is an array of numbers 1710, where each number 1710 indicates how similar the value of the encoded vector 1606 of the item 204 is to the value of the encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 can generate the similarity vector 1704 using a process similar to the process described in FIG. 17. In this example, the item tracking device 104 uses a matrix multiplication between the encoded vector 1606 of the item 204 and the encoded vectors 1606 in the encoded vector library 128. Each number 1710 in the similarity vector 1704 corresponds to an entry 1602 in the encoded vector library 128. For example, a first numerical value 1710 in similarity vector 1704 indicates how similar the value in encoded vector 1702 is to the value in encoded vector 1606 of a first entry 1602 of encoded vector library 128, a second numerical value 1710 in similarity vector 1704 indicates how similar the value in encoded vector 1702 is to the value in encoded vector 1606 of a second entry 1602 of encoded vector library 128, and so on.

[0155] After generating the similarity vector 1704, the item tracking device 104 can identify which entry 1602 in the encoded vector library 128 best matches the encoded vector 1606 of the item 204. In one embodiment, the entry 1602 associated with the highest numerical value 1710 in the similarity vector 1704 corresponds to the entry 1602 that best matches the encoded vector 1606 of the item 204. After identifying the entry 1602 from the encoded vector library 128 that best matches the encoded vector 1606 of the item 204, the item tracking device 104 can then identify the item identifier 1604 from the encoded vector library 128 that is associated with the identified entry 1602. Through this process, the item tracking device 104 can identify which item 204 in the encoded vector library 128 matches the item 204 based on its encoded vector 1606. The item tracking device 104 then outputs the identified item identifier 1604 for the identified item 204. For example, the item tracking device 104 may output the identified item identifier 1604 for the identified item 204 by adding the item identifier 1604 to a list of identified items 204 on a graphical user interface. The item tracking device 104 repeats this process for all of the encoded vectors 1606 obtained in operation 2312.

[0156] In act 2316, the item tracking device 104 determines whether it has identified all of the items 204. Here, the item tracking device 104 determines whether the number of identified items 204 matches the number of items 204 detected on the platform 202 in act 2306. The item tracking device 104 determines that it has identified all of the items 204 if the number of identified items 204 matches the number of items 204 detected on the platform 202. Otherwise, the item tracking device 104 determines that it has not identified one or more items 204 if the number of identified items 204 does not match the number of items 204 detected on the platform 202.

[0157] In response to determining that the item tracking device 104 has not identified one or more items 204, the item tracking device 104 proceeds to operation 2318. In this case, the item tracking device 104 proceeds to operation 2318 and prompts the user to identify the one or more items 204 that have not been identified. In operation 2318, the item tracking device 104 outputs a prompt requesting the user to identify one or more items 204 on the platform 202. In one embodiment, the item tracking device 104 may prompt the user to identify an item 204 from among a set of similar items 204. Referring to FIG. 24 as an example, the item tracking device 104 outputs a screen 2400 displaying the detected items 204 (shown as display element 2402) and the items 204 that were not identified. In this example, the screen 2400 displays recommendations (shown as display element 2404) for other similar items 204 if the item 204 was not identified. In one embodiment, the item recommendations may correspond to other items 204 identified using the similarity vector 1704. For example, the item recommendations may include items 204 associated with the second and third highest values ​​in the similarity vector 1704. The user may be provided with user input to select items 204 that were not identified.

[0158] In some embodiments, the item tracking device 104 may prompt the user to scan the items 204 that it did not identify. For example, the item tracking device 104 may instruct the user to scan the barcode of the item 204 using a barcode scanner. In this case, the item tracking device 104 may use a graphical user interface to display a combination of the items 204 detected by the platform 202 and the items 204 manually scanned by the user. Referring to FIG. 25 as an example, the item tracking device 104 may output a screen 2500 that displays the items 204 detected by the platform 202 (shown as display element 2502) and the items 204 manually scanned by the user (shown as display element 2504).

[0159] Returning to operation 2316 of FIG. 23 , in response to determining that all items 204 have been identified, the item tracking device 104 proceeds to operation 2320. In operation 2320, the item tracking device 104 determines whether there are additional items 204 to detect for the user. In some embodiments, the user can provide a user input indicating that they would like to add additional items 204 to the platform 202. In other embodiments, the item tracking device 104 can use the presence of the user's hands removing or adding items 204 from the platform 202 to determine whether there are additional items 204 to detect for the user. In response to determining that there are additional items 204 to detect, the item tracking device 104 returns to operation 2304. In this case, the item tracking device 104 returns to operation 2304 and begins detecting additional items 204 that the user has placed on the platform 202. In response to determining that there are no additional items 204 to detect for the user, the item tracking device 104 proceeds to operation 2322. In this case, the item tracking device 104 proceeds to operation 2322 to associate the detected item 204 with the user.

[0160] Prior to associating the items 204 with the user, the item tracking device 104 may allow the user to remove one or more items 204 from the list of identified items 204 by selecting the items 204 on a graphical user interface. Referring to FIGURE 26 as an example, the item tracking device 104 may receive user input identifying an item 204 to remove from the list of identified items 204 and output a screen 2600 confirming that the user wishes to remove the item 204. This functionality allows the user to edit and finalize the list of detected items 204 that they wish to purchase.

[0161] Returning to operation 2322 of FIG. 23 , the item tracking device 104 associates the item 204 with a user. In one embodiment, the item tracking device 104 may identify the user who placed the item 204 on the platform 202. For example, the user may identify themselves using a scanner or card reader at the imaging device 102. Examples of scanners include, but are not limited to, a QR code scanner, a barcode scanner, an NFC scanner, or other suitable types of scanners capable of receiving an electronic code embedded with information that uniquely identifies an individual. In another example, the user may identify themselves by providing user information on a graphical user interface at the imaging device 102. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, or other suitable types of information associated with a user.

[0162] The item tracking device 104 uses the user-provided information to identify an account associated with the user and adds the identified items 204 to the user's account. For example, the item tracking device 104 can use the user-provided information to identify an account in the user account information 120 associated with the user. As an example, the item tracking device 104 can identify a digital cart associated with the user. In this example, the digital cart includes information about items 204 that the user has placed on the platform 202 for purchase. The item tracking device 104 can add the items 204 to the user's digital cart by adding the item identifiers 1604 of the identified items 204 to the digital cart. The item tracking device 104 can also add other information related to the items 204 to the digital cart. For example, the item tracking device 104 can use the item identifiers 1604 to look up price information for the identified items 204 from the stored item information 118. The item tracking device 104 can then add price information corresponding to each of the identified items 204 to the user's digital cart.

[0163] After the item tracking device 104 adds the item 204 to the user's digital cart, the item tracking device 104 can trigger or initiate a transaction for the item 204. In one embodiment, the item tracking device 104 can use previously stored information (e.g., payment card information) to complete the transaction for the item 204. In this case, the user can be automatically charged for the item 204 in the digital cart when they leave the space. In another embodiment, the item tracking device 104 can use a scanner or card reader at the imaging device 102 to collect information from the user to complete the transaction for the item 204. This process allows the item 204 to be automatically added to the user's account (e.g., digital cart) without the user having to scan or identify the item 204 they want to acquire. After adding the item 204 to the user's account, the item tracking device 104 can output a notification or summary to the user that includes information about the item 204 that was added to the user's account. For example, the item tracking device 104 can output the summary to a graphical user interface at the imaging device 102. As another example, the item tracking device 104 can output the summary by sending the summary to an email address associated with the user or to a user device.

[0164] Identifying items based on their relationship to other items In some cases, the item tracking device 104 may be unable to identify an item 204 placed on the platform 202. In such cases, as described further below, the item tracking device 104 may identify the unidentified item 204 based on a predefined association 2802 (shown in FIG. 28) between the unidentified item 204 and another item 204 on the platform 202 previously identified as part of the same transaction. For example, as shown in FIG. 27, a transaction may include placing a first item 204A (e.g., a one-liter soda bottle) on the platform 202. The item tracking device 104 may correctly identify the first item 204A as a one-liter soda bottle and assign a corresponding item identifier 1604a (shown in FIG. 28 as I2) from the encoded vector library 128. The item tracking device 104 may identify the first item 204A using a process similar to the process 2300 described with reference to FIG. 23. For example, as described with reference to FIG. 29, identifying the first item 204A includes generating a cropped image 2702 of the first item 204A, and the first item 204A is identified based on the cropped image 2702 of the first item 204A. Once the first item 204A is identified, a second item 204B (e.g., a small bag of chips) may be subsequently placed on the platform 202 as part of the same transaction. In one embodiment, the placement of the first item 204A may be referred to as a first interaction of the transaction, and the placement of the second item 204B may be referred to as a second interaction of the same transaction. In some embodiments, the item tracking device 104 may not be able to identify the second item 204B, for example, based on a process similar to the process 2300 described above with reference to FIG. 23. In such a case, as described further below with reference to FIG. 29, the item tracking device 104 may identify the second item 204B based on a predefined association 2802 between the unidentified second item 204B and the previously identified first item 204A.As described with reference to FIG. 29, identifying the second item 204B includes generating a cropped image 2704 of the second item 204B, and the second item 204B is identified based on the cropped image 2704 of the second item 204B.

[0165] In this context, and referring to FIG. 28, the item tracking device 104 stores (e.g., in memory 116) an association 2802 between item identifiers 1604 of items 204 listed in the encoded vector library 128. The association 2802 between two item identifiers 1604 may correspond to any logical association between items 204 associated with the item identifiers 1604. For example, if the item tracking system 100 is installed and used in a store where multiple items 204 are available for purchase, the store may offer a particular promotion when two or more items 204 are purchased together in a single transaction. One example of a promotion may include a free small bag of chips with the purchase of a one liter bottle of soda. Another example of a promotion may include a discounted price or "buy one get one free" when two 16 ounce bottles of soda of a particular brand and / or flavor are purchased in a single transaction. In such a case, a particular promotion involving two or more items 204 may be stored as an association 2802 between the respective item identifiers 1604 (e.g., stored in the encoded vector library 128) of the items 204. Note that the association 2802 between two item identifiers 1604 may include an association between two instances of the same item identifier 1604 associated with the same item 204. For example, if an example promotion involves two or more of the same item 204 (e.g., a discounted price when buying two of the same item 204), the promotion is stored in memory as an association 2802 between two or more instances of the same item identifier 1604 associated with the same item 204.

[0166] As shown in FIG. 28, the associations 2802 are stored (e.g., in memory 116) as part of the encoded vector library 128. As shown in the figure, each of the entries 1602 is associated with an association 2802. Entry 1602a is associated with association-1 (denoted as A1), entries 1602b and 1602c are associated with association-2 (denoted as A2), and entry 1602d is associated with association-3 (denoted as A3). In one example, association-1 may indicate a promotion associated with two or more of the same items 204 having the same item identifier 1604 (denoted as I1) stored in entry 1602a. For example, association-1 may indicate a promotion in which two 16-ounce water bottles of the same brand will receive a discounted price if purchased together as part of the same transaction. In this example, the 16-ounce water bottles are associated with the item identifier 1604 (I1) of entry 1602a. Similarly, association-3 may indicate a promotion associated with two or more of the same items 204 having the same item identifier 1604 (shown as I4) stored in entry 1602d. For example, association-3 may indicate a promotion that includes a discounted price when two 16-ounce bottles of soda of the same brand and / or flavor are purchased together as part of the same transaction. In this example, the 16-ounce bottles of soda are associated with item identifier 1604 (I4) in entry 1602d. For example, association-2 may indicate a promotion associated with two different items 204 having two different item identifiers 1604a (I2) and 1604b (I3) stored in respective entries 1602b and 1602c. For example, association-2 may indicate a promotion that includes a free bag of chips with the purchase of a one-liter bottle of soda. In this example, the one liter bottle of soda may be associated with a first item identifier 1604a (I2) from entry 1602b, and the bag of chips may be associated with a second item identifier 1604b (I3) from entry 1602c.

[0167] 29 illustrates a flowchart of an example method 2900 for identifying a second item 204B based on an association 2802 with a first item 204A in accordance with one or more embodiments of the present disclosure. The method 2900 may be performed by an item tracking device 104, as shown in FIG.

[0168] In operation 2902, the item tracking device 104 detects a first trigger event on the platform 202, the first trigger event corresponding to a placement of a first item 204A on the platform 202. In particular embodiments, the first trigger event may correspond to a user placing the first item 204A on the platform 202.

[0169] As described above, the item tracking device 104 performs the automatic exclusion of the imaging device 102 using a process similar to that described in operation 302 of FIG. 3. For example, during an initial calibration period, the platform 202 may not have an item 204 placed on it. During this period, the item tracking device 104 uses one or more cameras 108 and / or 3D sensors 110 to capture a reference image 122 and a reference depth image 124, respectively, of the platform 202 without the item 204 placed on it. The item tracking device 104 can then use the captured image 122 and depth image 124 as reference images to detect when the item 204 was placed on the platform 202. The item tracking device 104 can then detect that the item 204 was placed on the surface 208 of the platform 202 based on the difference in depth values ​​between the subsequent depth image 124 and the reference depth image 124 and / or the difference in pixel values ​​between the subsequent image 122 and the reference image 122.

[0170] In one embodiment, to detect the first trigger event, the item tracking device 104 can use a process similar to the process 700 described in FIG. 7 to detect a trigger event, such as an event corresponding to a user's hand being detected on the platform 202 and an item 204 being placed on the platform. For example, the item tracking device 104 can check the difference between the reference depth image 124 and the subsequent depth image 124 to detect the presence of an object on the platform 202. The item tracking device 104 then checks whether the object corresponds to the user's hand or to the item 204 placed on the platform 202. The item tracking device 104 determines that the object is a user's hand if a first portion of the object (e.g., the user's wrist or arm) is outside the region of interest 802 of the platform 202 and a second portion of the object (e.g., the user's hand) is inside the region of interest 802 of the platform 202. If this condition is met, the item tracking device 104 determines that the user's hand has been detected on the platform 202. In other examples, the item tracking device 104 can detect whether a user's hand is detected on the platform 202 using a proximity sensor, a motion sensor, or other suitable detection technology. After detecting the user's hand, the item tracking device 104 begins periodically capturing additional overhead depth images 124 of the platform 202 and checks whether the user's hand has left the platform 202. In response to determining that the user's hand is no longer on the platform 202, the item tracking device 104 determines whether a first item 204A is on the platform 202. In response to determining that the first item 204A has been placed on the platform, the item tracking device 104 determines that a first trigger event has occurred and proceeds to identify the first item 204A that the user placed on the platform 202.

[0171] When the first trigger event is detected, the item tracking device 104 performs segmentation using an overhead view of the platform 202. In one embodiment, the item tracking device 104 can perform segmentation using a depth image 124 from a 3D sensor 110 positioned to capture an overhead or perspective view of the item 204 on the platform 202. In this example, the item tracking device 104 captures an overhead depth image 124 of the item 204 placed on the platform 202. The item tracking device 104 can then use a depth threshold to distinguish between the platform 202 and the item 204 placed on the platform 202 in the captured depth image 124. For example, the item tracking device 104 can set a depth threshold just above the surface of the platform 202. This depth threshold can be determined based on pixel values ​​corresponding to the surface of the platform 202 in the reference depth image 124 captured during the automatic exclusion process described above. After setting the depth threshold, the item tracking device 104 can apply the depth threshold to the captured depth image 124 to filter out or remove the platform 202 from the depth image 124. After filtering the depth image 124, the remaining pixel clusters correspond to items 204 placed on the platform 202. Each pixel cluster corresponds to a different item 204. For example, one of the pixel clusters corresponds to the first item 204 placed on the platform 202 as part of the first trigger event detected in operation 2902.

[0172] In operation 2904, in response to detecting a first trigger event, the item tracking device 104 captures multiple first images 122A of a first item 204 placed on the platform 202 using two or more cameras 108.

[0173] As described above, the item tracking device 104 can use multiple cameras 108 to capture multiple first images 122A (shown in FIG. 5A ) of a first item 204 on a platform 202. For example, the item tracking device 104 can capture a first image 122A that includes an overhead view, an oblique view, and / or a side view of the first item 204 on the platform 202.

[0174] In an operation 2906, the item tracking device 104 identifies a first item identifier 1604a associated with the first item 204 based on the plurality of first images 122A.

[0175] The item tracking device 104 can identify the first item 204A using a process similar to the process 2300 described with reference to FIG. 23. For example, the item tracking device 104 can generate a cropped image 2702 of the first item 204A from each first image 122A of the first item 204A captured by the respective camera 108 by isolating at least a portion of the first item 204A from the first image 122A. In other words, the item tracking device 104 generates one cropped image 2702 of the first item 204A based on each first image 122A of the first item 204A captured by the respective camera 108. As shown in FIG. 27, the item tracking device 104 generates three cropped images 2702a, 2702b, and 2702c of the first item 204A from each first image 122A of the first item 204A.

[0176] As described above, in one embodiment, the item tracking device 104 can generate a cropped image 2702 of the first item 204A based on features of the first item 204A present in the first image 122A (e.g., one of the first images 122A). The item tracking device 104 can first identify a region of interest (e.g., a bounding box) 1002 (shown in FIG. 10A ) of the first item 204A based on the detected features of the first item 204A present in the first image 122A, and then crop the first image 122A based on the identified region of interest 1002. The region of interest 1002 includes a number of pixels corresponding to the first item 204A in the captured first image 122A of the first item 204A on the platform 202. The item tracking device 104 can use one or more image processing techniques to identify the region of interest 1002 of the first item 204A in the first image 122A based on the features and physical attributes of the first item 204A. After the item tracking device 104 identifies the region of interest 1002 of the first item 204A, the item tracking device 104 crops the first image 122A by extracting pixels in the region of interest 1002 that correspond to the first item 204A in the first image 122A. By cropping the first image 122A, the item tracking device 104 generates another image (e.g., cropped image 2702) that includes the extracted pixels in the region of interest 1002 of the first item 204A from the original first image 122A. The item tracking device 104 can repeat this process for all of the captured first images 122A of the first item 204A on the platform 202. The result of this process is a set of cropped images 2702 that correspond to the first item 204A placed on the platform 202. In some embodiments, the item tracking device 104 can generate the cropped images 2702 of the first item 204A using a process similar to process 900 of FIG.

[0177] The item tracking device 104 generates an encoding vector 1702 (shown in FIG. 17 ) for each cropped image 2702 of the first item 204A. The encoding vector 1702 is comprised of an array of numbers. Each number in the encoding vector 1702 corresponds to and describes an attribute (e.g., item type, size, shape, color, etc.) of the first item 204A. The encoding vector 1702 may be of any suitable length. The item tracking device 104 generates the encoding vector 1702 of the first item 204A by inputting each of the cropped images 2702 into a machine learning model (e.g., machine learning model 126). The machine learning model 126 is configured to output the encoding vector 1702 of the item 204 based on features or physical attributes of the item 204 present in the image 122 of the item 204. Examples of physical attributes include, but are not limited to, item type, size, shape, color, or other suitable types of attributes of the item 204. After inputting the cropped image 2702 of the first item 204A into the machine learning model 126, the item tracking device 104 receives the encoded vector 1702 of the first item 204A. The item tracking device 104 repeats this process to obtain an encoded vector 1702 for each cropped image 2702 of the first item 204A on the platform 202.

[0178] The item tracking device 104 identifies the first item 204A from the encoded vector library 128 based on the corresponding encoded vector 1702 generated for the first item 204A. Here, the item tracking device 104 uses the encoded vector 1702 of the first item 204A to identify the best matching encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 identifies the best matching encoded vector 1606 in the encoded vector library 128 by generating a similarity vector 1704 (shown in FIG. 17 ) between the encoded vector 1702 generated for the unidentified first item 204A and the encoded vector 1606 in the encoded vector library 128. The similarity vector 1704 is comprised of an array of numerical similarity values ​​1710, where each numerical similarity value 1710 indicates how similar the value of the encoded vector 1702 of the first item 204A is to a particular encoded vector 1606 in the encoded vector library 128. In one embodiment, the item tracking device 104 may generate the similarity vector 1704 using a process similar to that described in FIG. 17. In this example, the item tracking device 104 uses a matrix multiplication between the encoded vector 1702 of the first item 204A and the encoded vector 1606 in the encoded vector library 128. Each numerical similarity value 1710 in the similarity vector 1704 corresponds to an entry 1602 in the encoded vector library 128. For example, a first numerical value 1710 in similarity vector 1704 indicates how similar the value of encoded vector 1702 is to the value of encoded vector 1606 in the first entry 1602 of encoded vector library 128, a second numerical value 1710 in similarity vector 1704 indicates how similar the value of encoded vector 1702 is to the value of encoded vector 1606 in the second entry 1602 of encoded vector library 128, and so on.

[0179] After generating the similarity vector 1704, the item tracking device 104 can identify which entry 1602 in the encoded vector library 128 best matches the encoded vector 1702 of the first item 204A. In one embodiment, the entry 1602 associated with the highest numerical similarity value 1710 in the similarity vector 1704 is the entry 1602 that best matches the encoded vector 1702 of the first item 204A. After the item tracking device 104 identifies the entry 1602 from the encoded vector library 128 that best matches the encoded vector 1702 of the first item 204A, the item tracking device 104 can then identify an item identifier 1604 from the encoded vector library 128 that is associated with the identified entry 1602. Through this process, the item tracking device 104 can determine which item 204 in the encoded vector library 128 corresponds to the unidentified first item 204A based on its encoded vector 1702. The item tracking device 104 then outputs the identified item identifier 1604 for the identified item 204. The item tracking device 104 repeats this process for each encoded vector 1702 generated for each cropped image 2702 (e.g., 2702a, 2702b, and 2702c) of the first item 204A. This process generates a set of item identifiers 1604 corresponding to the first item 204A, which may include multiple item identifiers 1604 corresponding to the multiple cropped images 2702 of the first item 204A. In other words, the item tracking device 104 identifies an item identifier 1604 for each cropped image 2702 of the first item 204A.

[0180] The item tracking device 104 can select one of the multiple item identifiers 1604 identified for the first item 204A based on each of the multiple cropped images 2702 of the first item 204A. For example, the item tracking device 104 can select a first item identifier 1604a associated with the first item 204A based on the multiple item identifiers 1604 identified for the first item 204A based on each of the multiple cropped images 2702 of the first item 204A.

[0181] In one or more embodiments, the item tracking device 104 can input each cropped image 2702 of the first item 204A into a machine learning model configured to determine whether the cropped image 2702 of the first item 204A is a front image 122 of the first item 204A or a back image 122 of the first item 204A. The front image 122 of the first item 204 corresponds to an image 122 of a portion of the first item 204A that includes identifying information (e.g., text, color, logo, pattern, photo, image, etc.) unique to the first item 204A or that can be used to identify the first item 204A. The back image 122 of the first item 204A corresponds to an image 122 of a portion of the first item 204 that does not include identifying information that can be used to identify the first item 204A. The machine learning model may be trained using a dataset that includes known front images 122 and back images of items 204 of the first items 204A identified in the encoded vector library 128. As each cropped image 2702 of the unidentified first items 204A is identified (e.g., tagged) as a front image 122 or a back image 122 of the first items 204A, the item tracking device 104 discards all cropped images 2702 that it identified as back images 122. The item tracking device 104 selects item identifiers 1604 for the unidentified first items 204A from only those item identifiers 1604 that correspond to cropped images 2702 that it identified as front images 122 of the first items 204A. In certain embodiments, after discarding all cut-out images 2702 identified as the back image 122 of the first item 204A, if only one cut-out image 2702 identified as the front image 122 of the first item 204A remains, the item tracking device 104 selects an item identifier 1604 that corresponds to the one remaining cut-out image 2702.If all of the cropped images 2702 of the first item 204A are identified as rear images 122, the item tracking device 104 may display, on a user interface device, item identifiers 1604 corresponding to one or more cropped images 2702 of the first item 204A and prompt the user to select one of the displayed item identifiers 1604. Alternatively, the item tracking device 104 may display instructions on the user's user interface device for flipping or rotating the first item 204A on the platform 202. If the first item 204A is flipped or rotated on the platform 202, the item tracking device 104 performs operations 2902-2906 to re-identify the first item 204A.

[0182] In some cases, multiple cropped images 2702 of a first item 204A may be identified as the front image 122. In such cases, the item tracking device 104 may be configured to select from the item identifiers 1604 an item identifier 1604 that corresponds to the cropped front image 122 of the item 204 based on the similarity value 1710 used to identify the respective item identifier 1604 from the encoded vector library 128. As described above, for each cropped image 2702, the item tracking device 104 selects from the encoded vector library 128 the entry 1602 associated with the highest numerical similarity value 1710 in the similarity vector 1704 generated for the cropped image 2702. The item tracking device 104 then identifies from the encoded vector library 128 the item identifier 1604 associated with the identified entry 1602. Thus, the item identifiers 1604 identified for each cropped image 2702 of the first item 204A correspond to respective similarity values ​​1710 based on which the item identifiers 1604 were selected from the encoded vector library 128 .

[0183] In one embodiment, the item tracking device 104 discards all cropped front images 2702 of the first item 204A that have selected an item identifier 1604 from the encoded vector library 128 based on a numerical similarity value 1710 that is below a threshold similarity value. Since the similarity value 1710 indicates the degree of similarity between the generated encoded vector 1702 for the unidentified first item 204A and a particular encoded vector 1606 in the encoded vector library 128, a lower similarity value 1710 indicates a lower similarity between the generated encoded vector 1702 and the corresponding encoded vector 1606 in the encoded vector library 128. The item tracking device 104 discards all cropped front images 2702 containing item identifiers 1604 selected from the encoded vector library 128 based on numerical similarity values ​​1710 below the threshold similarity value, thereby discarding all cropped images 2702 that are unlikely to correctly identify the first unidentified item 204A. In one embodiment, if the item identifiers 1604 of all cropped front images 2702 of an item 204 were selected from the encoded vector library 128 based on numerical similarity values ​​1710 below the threshold similarity value, the item tracking device 104 displays the item identifiers 1604 on a user interface device and prompts the user to select one of the displayed item identifiers 1604.

[0184] After discarding all cropped front images 2702 containing item identifiers 1604 selected from the encoded vector library 128 based on numerical similarity values ​​1710 below the threshold similarity value, the item tracking device 104 applies a majority rule to select from the item identifiers 1604 those item identifiers 1604 that correspond to the remaining cropped front images 2702 containing item identifiers 1604 selected from the encoded vector library 128 based on numerical similarity values ​​1710 equal to or greater than the threshold similarity value. The majority rule provides that the same item identifier 1604 is selected if the same item identifier 1604 is identified for a majority of the remaining cropped front images 2702 of the unidentified item 204.

[0185] However, if there is no majority of the item identifiers 1604 of the remaining cropped front images, the majority rule cannot be applied. For example, if the same item identifier 1604 has not been identified for the majority of the remaining cropped front images 2702 of the unidentified first item 204, the majority rule cannot be applied. In such a case, the item tracking device 104 compares the two highest numerical similarity values ​​1710 of the remaining cropped front images 2702. If the difference between the highest similarity value and the second highest similarity value is equal to or exceeds the threshold difference, the item tracking device 104 selects the item identifier 1604 corresponding to the highest similarity value. However, if the difference between the highest similarity value and the second highest similarity value is below the threshold difference, the item tracking device 104 displays the item identifiers 1604 corresponding to one or more remaining cropped front images 2702 of the first item 204A on the user interface device and prompts the user to select one of the displayed item identifiers 1604.

[0186] Regardless of the particular method used to identify the first item 204A, the end result of this entire process is that a first item identifier 1604a is identified for the first item 204A.

[0187] In an operation 2908, the item tracking device 104 assigns a first item identifier 1604a to the first item 204A captured in the first image 122A.

[0188] At act 2910, the item tracking device 104 detects a second trigger event on the platform 202, where the second trigger event corresponds to a placement of a second item 204B on the platform 202. In particular embodiments, the second trigger event corresponds to a user placing the second item 204B on the platform 202. The item tracking device 104 can detect the second trigger event similar to detecting the first trigger event described above with reference to act 2902.

[0189] In operation 2912, in response to detecting a second trigger event, the item tracking device 104 captures multiple second images 122B (e.g., as shown in FIG. 5B) of the second item 204B using two or more cameras 108 of the multiple cameras 108.

[0190] In operation 2914, the item tracking device 104 generates a plurality of cropped images 2704 (shown in FIG. 27), where each cropped image (e.g., 2704a, 2704b, 2704c, and 2704d) is associated with a corresponding second image 122B and is generated by editing the corresponding second image 122B to isolate at least a portion of the second item 204B.

[0191] To generate the multiple cropped images 2704 of the second item 204B, the item tracking device 104 can use a method similar to that described above with reference to operation 2906 for generating the cropped image 2702 of the first item 204A based on the first image 122A.

[0192] In operation 2916, for each cropped image 2704 of the second item 204B generated from each second image 122B, the item tracking device 104 identifies an item identifier 1604 based on attributes of the second item 204B in the cropped image 122B.

[0193] The item tracking device 104 may identify the item identifier 1604 of each cropped image 2704 of the second item 204B based on a method similar to that described above with reference to operation 2906 for identifying the item identifier 1604 of each cropped image 2702 of the first item 204A.

[0194] In operation 2918 , the item tracking device 104 accesses (eg, from memory 116 ) the associations 2802 between the item identifiers 1604 of each item 204 .

[0195] In operation 2920, based on the associations 2802 stored in memory 116, the item tracking device 104 identifies an association 2802a between the first item identifier 1604a and the second item identifier 1604b identified for the first item 204A. Based on searching the associations 2802 stored in memory 116, the item tracking device 104 may determine that an association 2802a (e.g., association-2) exists between the first item identifier 1604a of entry 1602b and the second item identifier 1604b of entry 1602c. Following the example above, the first item identifier 1604a of entry 1602b is associated with a one liter bottle of soda, and the second item identifier 1604b of entry 1602c is associated with a small bag of chips.

[0196] At operation 2922, the item tracking device 104 checks whether at least one of the item identifiers 1604 identified for the cropped image 2704 of the second item 204B is the second item identifier 1604b. If none of the item identifiers 1604 identified for the cropped image 2704 of the second item 204B is the second item identifier 1604b, the method 2800 proceeds to operation 2924, where the item tracking device 104 displays the item identifiers 1604 of the cropped image 2704 of the second item 204B on a user interface device and prompts the user to select one of the displayed item identifiers 1604.

[0197] However, if at least one of the item identifiers 1604 identified for the cropped image 2704 of the second item 204B is the second item identifier 1604b, the method 2900 proceeds to operation 2926, where the item tracking device 104 assigns the second item identifier 1604b to the second item 204B captured in the second image 122B. Following the above example, if a first item 204A is assigned a first item identifier 1604a from entry 1602b associated with a one liter bottle of soda, and at least one of the item identifiers 1604 identified for the cropped image 2704 of a second item 204B is a second item identifier 1604b from entry 1602c associated with a small bag of chips, the item tracking device 104 assigns the second item identifier 1604b from entry 1602c to the second item 204B, thus identifying the second item 204 as a small bag of chips.

[0198] In accordance with the second example of Association-1 above, if the first item 204A is assigned the first item identifier 1604(I1) from the entry 1602a associated with a 16 ounce bottle of water, and at least one of the item identifiers 1604 identified for the cropped image 2704 of the second item 204B is also the first item identifier 1604(I1) from the entry 1602a, then the item tracking device 104 assigns the same first item identifier 1604(I1) from the entry 1602a to the second item 204B. In this example, the first item identifier 1604 and the second item identifier 1604 are two different instances of the same item identifier 1604(I1) from the entry 1602a, and the first item 204A and the second item 204B are two different instances of the same item 204, e.g., two different 16 ounce bottles of water.

[0199] In one or more embodiments, the item tracking device 104 applies the association 2802-based logic described above to identify the second item 204B when one or more other methods described above for identifying the first item 204A do not apply or the second item 204B cannot be identified.

[0200] In one embodiment, after generating the cropped image 2704 for each second image 122B of the unidentified second item 204B, the item tracking device 104 inputs each cropped image 2704 of the second item 204 into a machine learning model to determine whether the cropped image 2702 of the second item 204B is the front image 122 of the second item 204B or the back image 122 of the second item 204B. Once each cropped image 2704 of the second item 204B is identified as the front image 122 or the back image 122 of the second item 204B, the item tracking device 104 discards all cropped images 2704 identified as the back image 122. The item tracking device 104 selects the item identifier 1604 of the unidentified second item 204B from only the item identifiers 1604 corresponding to the cropped images 2704 identified as the front image 122. For example, if after discarding all of the cropped images 2704 of the second item 204 identified as the back image 122, only one cropped image 2704 identified as the front image 122 remains, the item tracking device 104 selects the item identifier 1604 that corresponds to the one remaining cropped image 2704. If all of the cropped images 2704 of the second item 204B have been identified as the front image 122, the item tracking device 104 displays the item identifiers 1604 corresponding to the one or more cropped images 2704 on the user interface device and prompts the user to select one of the displayed item identifiers 1604. Alternatively, the item tracking device 104 may display instructions on the user interface device for the user to flip or rotate the second item 204B on the platform 202. Once the second item 204B is flipped or rotated on the platform 202, the item tracking device 104 can perform operations 2910-2916 to re-identify the second item 204B.

[0201] Once multiple cropped images 2704 of the second item 204B are identified as front images 122, the item tracking device 104 selects from the item identifiers 1604 an item identifier 1602 that corresponds to the cropped front image 2704 of the second item 204B based on the similarity value 1710 used to identify each item identifier 1604 from the encoded vector library 128. As described above with reference to the cropped image 2702 of the first item 204B, for each cropped image 2704 of the second item 204B, the item tracking device 104 selects from the encoded vector library 128 an entry 1602 associated with the highest numerical similarity value 1710 in the similarity vector 1704 generated for the cropped image 2704. The item tracking device 104 then identifies from the encoded vector library 128 an item identifier 1604 associated with the identified entry 1602. Thus, the item identifiers 1604 identified for each cropped image 2704 of the second item 204B correspond to respective similarity values ​​1710, and the item identifiers 1604 were selected from the encoded vector library 128 based on these similarity values.

[0202] In one embodiment, the item tracking device 104 discards all cropped front images 2704 of the second item 204B that contain a selected item identifier 1604 from the encoded vector library 128 based on a numerical similarity value 1710 that is below a threshold similarity value. Since the similarity value 1710 indicates the degree of similarity between the generated encoded vector 1702 for the unidentified second item 204B and a particular encoded vector 1606 from the encoded vector library 128, a lower similarity value 1710 indicates a lower similarity between the generated encoded vector 1702 and the corresponding encoded vector 1606 from the encoded vector library 128. The item tracking device 104 discards all cropped front images 2704 containing item identifiers 1604 selected from the encoded vector library 128 based on a numerical similarity value 1710 below the threshold, thereby discarding all cropped images 2704 that are unlikely to correctly identify the unidentified second item 204B. In one embodiment, if the item identifiers 1604 of all cropped front images 2704 of the second item 204B were selected from the encoded vector library 128 based on a numerical similarity value 1710 below the threshold similarity value, the item tracking device 104 displays the item identifiers 1604 on a user interface device and prompts the user to select one of the displayed item identifiers 1604.

[0203] After discarding all cropped front images 2704 that contain an item identifier 1604 selected from the encoded vector library 128 based on a numerical similarity value 1710 below the threshold similarity value, the item tracking device 104 applies a majority rule to select an item identifier 1604 from the remaining cropped front images 2704 that correspond to the item identifiers 1604 that contain an item identifier 1604 selected from the encoded vector library 128 based on a numerical similarity value 1710 that is greater than or equal to the threshold similarity value. The majority rule provides that the same item identifier 1604 is selected if the same item identifier 1604 is identified for a majority of the remaining cropped front images 2704 of the unidentified second item 204B.

[0204] However, the majority rule cannot be applied if there is not a majority of the item identifiers 1604 of the remaining cropped front images 2704. For example, the majority rule cannot be applied if the same item identifier 1604 has not been identified for the majority of the remaining cropped front images 2704 of the unidentified second item 204B. In such a case, the item tracking device 104 compares the two highest numerical similarity values ​​1710 of the remaining cropped front images 2704. If the difference between the highest similarity value and the second highest similarity value is equal to or greater than a threshold difference, the item tracking device 104 selects the item identifier 1604 corresponding to the highest similarity value.

[0205] However, if the difference between the highest similarity value and the second highest similarity value is below a threshold difference, the item tracking device 104 applies the association-based logic described above with reference to operations 2918-2926.

[0206] Although several embodiments are presented in this disclosure, it should be understood that the disclosed systems and methods can be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples should be considered as illustrative and not limiting, and the intention is not to be limited to the details provided herein. For example, various elements or components can be combined or integrated into another system, and certain features can be omitted or not implemented.

[0207] Additionally, the techniques, systems, subsystems, and methods described and illustrated in various embodiments as separate or distinct may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items illustrated or described as coupled or directly coupled or in communication with one another may be indirectly coupled or in communication electrically, mechanically, or otherwise, through some interface, device, or intermediate component. Other examples of changes, substitutions, and alterations may be ascertainable by one of ordinary skill in the art and may be made without departing from the spirit and scope of the present disclosure.

[0208] In order to assist the Patent Office and any readers interested in patents issued based on this application in interpreting the claims appended to this application, applicants note that, unless the words "means" or "step" are expressly used in a particular claim, none of the appended claims are intended to invoke 35 U.S.C. 112(f) as in existence on the filing date of this application.

Claims

1. 1. An item tracking system, comprising: a plurality of cameras, each configured to capture an image of at least a portion of the platform; a memory configured to store an association between item identifiers for each item; a processor communicatively coupled to the plurality of cameras and the memory; The processor, detecting a first trigger event on the platform, the first trigger event corresponding to a placement of a first item on the platform; responsive to detecting the first trigger event, capturing a plurality of first images of the first item on the platform using two or more cameras of the plurality of cameras; identifying a first item identifier associated with the first item based on the plurality of first images; assigning the first item identifier to the first item captured in the first image; detecting a second trigger event on the platform, the second trigger event corresponding to a placement of a second item on the platform; responsive to detecting the second trigger event, capturing a plurality of second images of the second item on the platform using two or more cameras of the plurality of cameras; generating a plurality of cropped images, each cropped image associated with a corresponding second image and generated by editing the corresponding second image to isolate at least a portion of the second item; for each cropped image, identifying an item identifier based on one or more attributes of the second item; accessing the association from the memory; determining an association between the first item identifier and the second item identifier of the first item; detecting that at least one of the identified item identifiers is the second item identifier; in response to detecting that at least one of the identified item identifiers is the second item identifier, and based on the determined association between the first item identifier and the second item identifier, assigning the second item identifier to the second item. Item tracking system.

2. The processor further comprises: For each cropped image generated for each second image, inputting the cropped image into a machine learning model, the machine learning model being configured to output whether the cropped image is a back image of an item or a front image of an item; obtaining an output from the machine learning model indicating whether the cropped image is a back image of an item or a front image of an item; and tagging the cropped image as a back image or a front image based on the output; The item tracking system of claim 1 , wherein two or more of the cropped images are tagged as front images.

3. the memory is further configured to store an encoded vector library; the encoded vector library includes a plurality of encoded vectors, each encoded vector describing one or more attributes of a particular item and associated with an item identifier for the particular item; The processor, generating a first encoding vector of the cropped image, the first encoding vector describing one or more attributes of the first item based on the cropped image; comparing the first encoded vector to the encoded vectors in the encoded vector library; selecting a second encoded vector from the encoded vector library that most closely matches the first encoded vector, a numerical similarity value indicating a similarity between the first encoded vector and the selected second encoded vector; and 3. The item tracking system of claim 2, further configured to identify the item identifier for each cropped image by identifying the item identifier associated with the second encoded vector in the encoded vector library.

4. The item tracking system of claim 3 , wherein the processor is further configured to determine that a plurality of the cropped second images are tagged as front images.

5. The processor, responsive to determining that the plurality of cropped second images are tagged as foreground images, determining a first set of item identifiers identified for each of the plurality of cropped second images from the plurality of item identifiers based on a similarity value equal to or greater than a threshold similarity value; and 5. The item tracking system of claim 4, further configured to determine that for a majority of the plurality of cropped second images, the same item identifier was not identified from the first set of item identifiers.

6. The processor, In response to determining that the same item identifier from the first set of item identifiers was not identified for a majority of the plurality of cropped second images, determining a third item identifier from the first set based on a highest one of the similarity values ​​corresponding to the first set of item identifiers; determining a fourth item identifier from the first set based on a second highest similarity value among the similarity values ​​corresponding to the item identifiers in the first set for the second cropped second image; and The item tracking system of claim 5 , further configured to determine that a difference between the highest similarity value and the second highest similarity value is less than a threshold difference.

7. the first item and the second item are identified by the same item identifier; The item tracking system of claim 1 , wherein the first item identifier and the second item identifier are two instances of the same item identifier.

8. 1. A method of identifying an item, the method comprising: detecting a first trigger event on a platform, the first trigger event corresponding to a placement of a first item on the platform; responsive to detecting the first trigger event, capturing a plurality of first images of the first item on the platform using two or more cameras of a plurality of cameras; identifying a first item identifier associated with the first item based on the plurality of first images; assigning the first item identifier to the first item captured in the first image; detecting a second trigger event on the platform, the second trigger event corresponding to a placement of a second item on the platform; responsive to detecting the second trigger event, capturing a plurality of second images of the second item on the platform using two or more cameras of the plurality of cameras; generating a plurality of cropped images, each cropped image associated with a corresponding second image and generated by editing the corresponding second image to isolate at least a portion of the second item; for each cropped image, identifying an item identifier based on one or more attributes of the second item; accessing from a memory an association between item identifiers for each item; determining an association between the first item identifier and a second item identifier of the first item; detecting that at least one of the identified item identifiers is the second item identifier; in response to detecting that at least one of the identified item identifiers is the second item identifier, and based on an identified association between the first item identifier and the second item identifier, assigning the second item identifier to the second item. method.

9. For each cropped image generated for each second image, inputting the cropped image into a machine learning model, the machine learning model configured to output whether the cropped image is a back image of an item or a front image of an item; obtaining an output from the machine learning model indicating whether the cropped image is a back image of an item or a front image of an item; and tagging the cropped image as a back image or a front image based on the output. The method of claim 8 , wherein two or more of the cropped images are tagged as foreground images.

10. the memory is configured to store an encoded vector library; the encoded vector library includes a plurality of encoded vectors, each encoded vector describing one or more attributes of a particular item and associated with an item identifier for the particular item; Identifying the item identifier for each cropped image includes: generating a first encoding vector of the cropped image, the first encoding vector describing one or more attributes of the first item based on the cropped image; comparing the first encoded vector to encoded vectors in the library of encoded vectors; selecting a second encoded vector from the encoded vector library that most closely matches the first encoded vector, a numerical similarity value indicating a similarity between the first encoded vector and the selected second encoded vector; and 10. The method of claim 9, further comprising identifying, within the encoded vector library, the item identifier associated with the second encoded vector.

11. The method of claim 10 , further comprising determining that a plurality of the cropped second images are tagged as foreground images.

12. responsive to determining that the plurality of cropped second images are tagged as foreground images, determining a first set of item identifiers identified for each of the plurality of cropped second images from the plurality of item identifiers based on the similarity value being equal to or greater than a threshold similarity value; The method of claim 11 , further comprising: determining that for a majority of the plurality of cropped second images, the same item identifier was not identified from the first set of item identifiers.

13. In response to determining that the same item identifier from the first set of item identifiers was not identified for a majority of the plurality of cropped second images, determining a third item identifier from the first set based on a highest one of the similarity values ​​corresponding to the item identifiers in the first set for the first cropped second image; determining a fourth item identifier from the first set based on a second highest similarity value among the similarity values ​​corresponding to the item identifiers in the first set for the second cropped second image; and The method of claim 12 , further comprising determining that a difference between the highest similarity value and the second highest similarity value is less than a threshold difference.

14. the first item and the second item are identified by the same item identifier; The method of claim 8 , wherein the first item identifier and the second item identifier are two instances of the same item identifier.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to: detecting a first trigger event on a platform, the first trigger event corresponding to a placement of a first item on the platform; responsive to detecting the first trigger event, capturing a plurality of first images of the first item on the platform using two or more cameras of a plurality of cameras; identifying a first item identifier associated with the first item based on the plurality of first images; assigning the first item identifier to the first item captured in the first image; detecting a second trigger event on the platform, the second trigger event corresponding to a placement of a second item on the platform; responsive to detecting the second trigger event, capturing a plurality of second images of the second item on the platform using two or more cameras of the plurality of cameras; generating a plurality of cropped images, each cropped image associated with a corresponding second image and generated by editing the corresponding second image to isolate at least a portion of the second item; for each cropped image, identifying an item identifier based on one or more attributes of the second item; accessing from the memory an association between item identifiers for each item; determining an association between the first item identifier and a second item identifier of the first item; detecting that at least one of the identified item identifiers is a second item identifier; in response to detecting that at least one of the identified item identifiers is the second item identifier, and based on the determined association between the first item identifier and the second item identifier, assigning the second item identifier to the second item. Non-transitory computer-readable medium.

16. The instructions cause the processor to: For each cropped image generated for each second image, inputting the cropped image into a machine learning model, the machine learning model being configured to output whether the cropped image is a back image of an item or a front image of an item; obtaining an output from the machine learning model indicating whether the cropped image is a back image of an item or a front image of an item; and tagging the cropped image as a back image or a front image based on the output; The non-transitory computer readable medium of claim 15 , wherein two or more of the cropped images are tagged as foreground images.

17. the memory is configured to store an encoded vector library; the encoded vector library includes a plurality of encoded vectors, each encoded vector describing one or more attributes of a particular item and associated with an item identifier for the particular item; Identifying the item identifier for each cropped image includes: generating a first encoding vector of the cropped image, the first encoding vector describing one or more attributes of the first item based on the cropped image; comparing the first encoded vector to the encoded vectors in the encoded vector library; selecting a second encoded vector from the encoded vector library that best matches the first encoded vector, a numerical similarity value indicating a similarity between the first encoded vector and the selected second encoded vector; and 20. The non-transitory computer-readable medium of claim 16, further comprising identifying the item identifier associated with the second encoded vector within the encoded vector library.

18. 20. The non-transitory computer readable medium of claim 17, wherein the instructions further cause the processor to determine that a plurality of the cropped second images are tagged as foreground images.

19. The instructions cause the processor to: in response to determining that the plurality of cropped second images are tagged as foreground images, determining, from a plurality of item identifiers, a first set of identified item identifiers for each of the plurality of cropped second images based on a similarity value being equal to or greater than a threshold similarity value; and 20. The non-transitory computer readable medium of claim 18, further comprising: determining that for a majority of the plurality of cropped second images, a same item identifier was not identified from the first set of item identifiers.

20. The instructions cause the processor to: In response to determining that the same item identifier from the first set of item identifiers was not identified for the majority of the plurality of cropped second images, determining a third item identifier from the first set based on a highest one of the similarity values ​​corresponding to the item identifiers in the first set for the first cropped second image; determining a fourth item identifier from the first set based on a second highest similarity value among the similarity values ​​corresponding to the item identifiers in the first set for the second cropped second image; and 20. The non-transitory computer readable medium of claim 19, further comprising: determining that a difference between the highest similarity value and the second highest similarity value is less than a threshold difference.

Images