Multi-sensor perception for resource tracking and quantification

Abstract

Disclosed herein are embodiments of a system, method, and computer program product for multi-sensor perception for resource tracking and quantification. One embodiment operates by receiving resource information indicating a resource identifier, a location of the resource, and an amount of the resource. Based on sensor data and the resource identifier received from a first sensing device, a resource removed from the location of the resource and placed at a predefined location can be tracked. Based on depth information from sensor data received from a second sensing device, an amount of the resource placed at the predefined location can be determined. A notification indicating that additional resources matching the resource should be stopped from being removed from the location of the resource and placed at the predefined location can be generated based on a match between the amount of the resource and the amount of the resource placed at the predefined location.

Description

Technical Field

[0001] Economic growth in industries such as e-commerce has revealed unexpected inventory management challenges. Inventory inaccuracies cause problems such as the delivery of inaccurate resources (e.g., items, products, components, etc.), which affect customer service levels and erode profit margins. Pallets are commonly used daily to package, transport, and manage resources / products. Conventionally, the pallet construction process, which requires entities (e.g., product / resource pickers, warehouse workers, autonomous / semi-autonomous robots / devices, etc.) to manually build each pallet based on purchase orders, requires a significant amount of labor. Entities often have to manually count and track resource / product identifiers (e.g., stock keeping unit (SKU), etc.) when resources / products are placed on pallets. Manual processes are often error-prone, resulting in inaccurate pallets being sent to consumers, businesses, etc. This may prompt costly corrective measures. Additionally, inadequately packaged pallets can cause injury and / or physical damage from falling resources / products.

Brief Description of the Drawings

[0002] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and enable one of ordinary skill in the art to make and use the present disclosure.

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

[0003] This specification provides exemplary embodiments of systems, apparatuses, devices, methods, computer program products for multi-sensor perception for resource tracking and quantification, and / or combinations and sub-combinations thereof. Resources can include, for example, items, products, elements, packages, devices, etc. According to some aspects, a computing device (e.g., a modular device, a stand-alone device, an Internet-of-Things (IoT) device, a warehouse management device, a resource management device, etc.) can be configured with and / or receive resource information indicating respective identifiers, respective locations, and respective quantities for each of a plurality of resources, such as resources in a warehouse environment. The resource information may include, for example, purchase order information, warehouse slot mapping information, etc. The computing device can include one or more sensing devices (e.g., an imaging device, a camera, a Light Detection and Ranging (LIDAR) sensor, a depth detection device, etc.) arranged to capture sensor data (e.g., video, still images, image data, depth information, etc.) indicating resources in its vicinity. The one or more sensing devices may capture the entire resource acquisition process, including resource selection / picking and resource placement in a predefined area (e.g., on a pallet, etc.). For example, the computing device can detect, track, and count resources while they are placed on a pallet and identify discrepancies with the resource information. The computing device can generate a notification that facilitates the correction / verification of any identified discrepancy between the resources placed on the pallet and the resource information. The notification can include an audible notification (e.g., text-to-speech, recorded / computerized voice, chime, alarm, etc.), a text notification (e.g., a text message sent to a mobile device of a user, product / resource picker, warehouse worker, etc.), a visual notification (e.g., a notification displayed on a user interface, etc.), etc.

[0004] Embodiments of systems, apparatuses, devices, methods, computer program products for multi-sensor perception for resource tracking and quantification, and / or combinations and sub-combinations thereof overcome various technical problems associated with conventional systems. For example, conventionally, inventory and / or resource management processes such as pallet construction processes in warehouses, etc., require a great deal of labor for each entity (e.g., product / resource pickers, warehouse workers, autonomous / semi-autonomous robots / devices, etc.) to manually identify and count the resources placed on the pallet in order to manually construct each pallet according to a purchase order. The manual process results in inaccurate pallet construction, a degraded user experience, losses in revenue due to inaccuracies in inventory and inaccuracies in product shipping / sales. For example, industry analysts recommend that in the United States, there are always more than 2 billion pallets in circulation, and resource inaccuracies in pallet construction result in an average annual loss of more than $40 million in revenue. Conventional inventory and / or resource management processes cannot accurately identify, track, and quantify resources, for example, in a warehouse environment. For example, conventional inventory and / or resource management processes that use tracking devices (e.g., RFID transmitters, NFC transmitters, etc.) that require radio frequency identification (RFID), near field communication (NFC), are complex and cumbersome, require expensive tracking devices to be attached to each resource in the inventory, and can output inaccurate information due to wireless interference, unclear components, etc.

[0005] Embodiments of systems, apparatuses, devices, methods, computer program products for multi-sensor perception for resource tracking and quantification, and / or combinations and sub-combinations thereof overcome problems related to accurately identifying, tracking, and quantifying internal resources, such as in a warehouse setting, by using sensor data (e.g., high-resolution imaging data, depth detection data) in combination with a trained analysis module (e.g., a machine learning model, a recognition model, etc.). The trained analysis module uses not only resource identification information but also resource-specific depth and location information from sensor data to significantly reduce the likelihood of misidentifying resources and significantly improve the accuracy of resource tracking, counting, and placement compared to conventional systems that implement manual or technical methods such as RFID tracking, NFC tracking, etc. (resulting in avoiding damage to resources and / or hazards to entities, etc.). Since the trained analysis module uses sensor data captured from multiple views and representing different types of information related to the resources, the resources can be accurately identified, tracked, and quantified.

[0006] Embodiments of systems, apparatuses, devices, methods, computer program products for multi-sensor perception for resource tracking and quantification described herein, and / or combinations and sub-combinations thereof improve computer vision techniques for identifying, tracking, and quantifying resources, such as within a warehouse environment, while eliminating procedures such as resource scanning and logging. Embodiments of systems, apparatuses, devices, methods, computer program products for multi-sensor perception for resource tracking and quantification described herein, and / or combinations and sub-combinations thereof enable and / or facilitate an accurate and efficient pallet building process using sensor data for the field of inventory assessment and resource management. For example, notifications of resource discrepancies generated in real time facilitate rapid pallet correction / verification and, thus, improve the accuracy of each pallet build while also improving the speed at which an accurate pallet of resources can be constructed. These and other technical advantages are described herein.

[0007] Figure 1 shows a block diagram of an exemplary system 100 for multi-sensor perception for resource tracking and quantification, according to some aspects. System 100 can include a computing device 102 (e.g., a modular device, a stand-alone device, an Internet of Things (IoT) device, a warehouse management device, a resource management device, etc.) and a mobile unit 104 (e.g., a pallet truck, a forklift, a stock cart, a vehicle, a vehicle that is fully or partially autonomous, etc.).

[0008] According to some aspects, the computing device 102 can be attached to the mobile unit 104 to facilitate accurate and efficient tracking of the location and amount of resources, such as resources included within a pallet building process, via multi-sensor perception. For example, the computing device 102 and the mobile unit 104 can operate, e.g., within a warehouse environment, to reduce and / or eliminate discrepancies between requested / ordered resources and picked / loaded resources on a pallet.

[0009] According to some aspects, computing device 102 may include interface module 106. Interface module 106 may be any interface for presenting information such as resource information to the user and / or receiving information from the user. The resource information may include, for example, purchase order information, warehouse slot mapping information, and the like. Interface module 106 may include a graphical user interface (GUI) and / or a communication interface such as a web browser (e.g., Microsoft Internet Explorer®, Mozilla Firefox®, Google Chrome®, Apple Safari®, etc.). Other software, hardware, and / or interfaces may be used to provide communication between computing device 102, a user (e.g., a product / resource picker, a warehouse worker, etc.), other computing devices 102, a warehouse management system, and the like.

[0010] According to some aspects, interface module 110 may include one or more input devices and / or components such as, for example, a keyboard, a pointing device (e.g., a computer mouse, a remote control), a microphone, a joystick, a tactile input device (e.g., a touch screen, a glove, etc.). According to some aspects, through interaction with the input device and / or component, the user may view, access, request, and / or navigate information related to resources such as purchase order information, warehouse slot mapping information, which indicates identifiers (e.g., SKU, universal product code (UPC), unique identifier, etc.), locations (e.g., warehouse inventory location, shelf, aisle, bin, etc.), and / or quantities (e.g., purchase order quantity, inventory quantity, required quantity, etc.) of resources 120 (e.g., beverages, inventory / inventory resources, products, etc.) in a warehouse, a resource storage area, within a facility, and the like.

[0011] According to some aspects, computing device 102 may include memory module 108. According to some aspects, memory module 108 may be loaded with and / or configured using purchase order information, warehouse slot mapping information, etc. that indicate identifiers (such as SKUs, UPCs, unique identifiers, etc.), locations (such as warehouse inventory locations, shelves, aisles, bins, etc.), and / or quantities (such as purchase order quantities, inventory quantities, demand quantities, etc.) of resources (such as beverages, inventory / stock resources, products, etc.) in a warehouse. Although not shown, according to some aspects, computing device 102 may receive purchase order information, warehouse slot mapping information, etc. that indicate identifiers, locations, and / or quantities of resources in a warehouse, etc. from devices (such as cloud-based devices, service provider devices, computing devices, etc.), networks (such as private networks, public networks, the Internet, etc.), systems (such as warehouse management systems, etc.). Computing device 102 may receive and store resource information (such as via memory module 108, etc.) by any method or technique.

[0012] According to some aspects, computing device 102 may include a sensing module 110. According to some aspects, to facilitate multi-sensor perception and track the location and amount of resources, computing device 102 may receive different types of data / information from different sensing devices of sensing module 110. The different types of data / information received from the sensing devices of sensing module 110 may be analyzed separately, collectively, etc. According to some aspects, sensing module 110 may include one or more sensing devices such as a camera (e.g., a stereo camera, a high-resolution camera, a video camera, a smart camera, a power-over-ethernet (PoE) camera, etc.), a LIDAR sensor, an infrared sensor, a position detection sensor, a depth sensor, etc. For example, according to some aspects, the configuration of sensing module 110 may include three cameras such as two stereo cameras and one LIDAR camera. According to some aspects, sensing module 110 may include any number and / or type of sensing devices. Sensing module 110 may capture sensor data (e.g., video, still images, spatial / depth information, etc.) that provides a real-time representation and / or a real-world representation of resources within the field of view (and / or detection) of sensing module 110. For example, sensing module 110 may capture the entire resource acquisition process, including resource selection / picking and resource placement onto a pre-defined area (e.g., on a pallet, etc.).

[0013] For example, computing device 102 can track each resource among a plurality of resources at its respective location, removed from its respective location and placed at a predefined location, based on sensor data received from at least one sensing device and its respective identifier. According to some aspects, the predefined location can be on a pallet. For example, the resource can be a box of beverages (or any other product), a package, etc., and computing device 102 can track the resource when it is removed from a location within the warehouse and placed on the pallet. According to some aspects, the pallet can be placed on a mobile unit (e.g., a pallet truck, a forklift, a stock cart, etc.) and / or attached to the mobile unit. According to some aspects, the sensing device can include one or more of a camera (e.g., a stereo camera, a high-resolution camera, a video camera, a smart camera, a PoE camera, etc.), a LIDAR sensor, an infrared sensor, a position detection sensor, etc.

[0014] FIG. 2A shows an example of system 100 of FIG. 1. As shown in FIG. 2A, computing device 102 (e.g., sensing module 110) can use sensor data to track resources 202 and 204 during a resource acquisition process (including resource selection / picking and placement of resources 202 and 204 within a predefined area such as pallet 208) performed by entity 206 (e.g., a product / resource picker, a warehouse worker, an autonomous / semi-autonomous robot / device, etc.).

[0015] Computing device 102 can generate and / or display instructions for resource selection that are displayed via interface module 106. For example, as shown in FIG. 2A, interface module 106 can display the respective amounts (e.g., "pick up 5", "pick up 3", etc.) and identifiers (e.g., SKU) of resources 202 and 204 that entity 206 selects and places on palette 208. According to some aspects, computing device 102 can generate instructions for resource selection that are output via an audible notification and / or transmitted to a user device of entity 206 (e.g., a mobile device, a smart device, a computing terminal, etc.).

[0016] According to some aspects, computing device 102 tracks each of the resources removed from a location designated for resources such as shelves 220 and 220, wherein for each of resources 202 and 204 removed from their respective locations (e.g., shelves 220 and 220), it is determined that the type of the resource corresponds to the type of the resource indicated by its respective identifier (e.g., SKU indicated by interface module 106). For example, computing device 102 can track each resource removed from its respective location (e.g., resources 202 and 204 removed from shelves 220 and 220) and placed at a predefined location such as palette 208 by determining, based on object recognition performed by a trained machine learning model, that the type of the resource corresponds to the type of the resource indicated by its respective identifier for each of the resources removed from their respective locations (e.g., shelves 220 and 220).

[0017] For example, returning to FIG. 1, according to some aspects, computing device 102 may include an analysis module 112. The analysis module 112 may be configured for object detection, object tracking, etc. According to some aspects, the analysis module 112 may use artificial intelligence and / or machine learning, such as image / object recognition, to identify resources depicted by one or more of a plurality of images, such as video frames, still images, etc., included in the sensor data. For example, FIG. 2A shows an interface module 106 that displays an image of an entity 206 that selects / picks resources 224 (such as resources 202 and 204) being tracked by image / object recognition (shown by bounding box 230, etc.) to identify resources depicted by one or more of a plurality of images, such as video frames, still images, etc., included in the sensor data. According to some aspects, the analysis module 112 can determine / detect resources in the imaging / sensor data using one or more object identification and / or tracking algorithms.

[0018] According to some aspects, the analysis module 112 may include a trained machine learning model, computer vision techniques, and the like. The analysis module 112 may determine, for example, that each resource (e.g., resources 202 and 204, etc.) is placed on the palette 208 based on object tracking applied to the sensor data received from the sensing module 110. For example, the analysis module 112 may identify resources (e.g., resources 202 and 204, etc.) within a first image among a plurality of images received from the sensor data and generate a detection result. The analysis module 112 can track the resources by analyzing a second image among the plurality of images and the detection result from the first image. To do so, in order to consider the movement of the resources from one or more positions within the first image determined from the detection result, it adjusts one or more locations of one or more bounding boxes associated with the resources within the second image. According to some aspects, the analysis module 112 may implement any object tracking method and / or technique.

[0019] According to some aspects, the analysis module 112 can determine the amount of resources (such as resources 202 and 204, etc.) placed on the pallet 208 based on the depth information extracted from the sensor data, based on the match between the depth value indicated by the depth information and the depth value associated with the amount of each resource type. For example, FIG. 2B shows an exemplary screen of the interface module 106. The computing device 102 can store resource information that details how many resources of one resource type are represented within a defined area based on the determined depth of the resource placement (such as the stacking depth). The imaging module 106 may display an image of the resources placed on the pallet 208. The analysis module 112 can identify a resource type such as the resource 202 and display a bounding box 240 around the designated area on the pallet 208 for the resource. Information describing the contents of the bounding box 240 may also be displayed. For example, as shown, an indication that the bounding box 240 surrounds the resource 202 and a depth value (such as 2 feet, etc.) describing the distance, such as from the sensing module 110 to the surface of the resource 202, may be displayed. The analysis module 112 can determine the amount of the resource 202 based on the match between the indicated depth value and the depth value associated with the amount of the resource 202 indicated by the resource information stored by the memory module 108.

[0020] According to some aspects, the analysis module 112 uses object recognition and / or object tracking to determine the orientation of resources (such as resource 202 and 204, etc.) placed on a pallet (such as pallet 208, etc.), ensuring proper loading of the pallet and avoiding damage to resources (such as resource 202 and 204, etc.) and / or entities (such as entity 206, etc.). According to some aspects, as shown in FIG. 2B, a resource such as resource 260 (or resource 202) can be placed on pallet 208 without optimized positioning. For example, the analysis module 112 may detect that the shape of resource 260 extends outside the bounding box 240. The extension of resource 260 outside the bounding box 240 can make the resources placed / packaged on pallet 208 in a non-uniform, unstable, or non-optimized spatial arrangement state. The analysis module 112 may detect any inconsistent and / or misaligned resources, such as resource 260 that extends outside the bounding box 240, and generate a notification to inform entity 206. The notification may cause entity 206 to repack, exchange, reorient, and / or optimize the packaging of resource 260 placed on pallet 208.

[0021] FIG. 3 is an exemplary system 300 for training an analysis module 112 for multi-sensor perception for resource tracking and quantification, according to some embodiments. FIG. 3 will be described with reference to FIGS. 1 and 2. According to some aspects, the analysis module 112 can be trained to automatically detect, track, and count resources placed on a pallet. The analysis module 112 can be trained to identify any discrepancies found between the resource information and the resources placed on a pallet (e.g., pallet 208 in FIG. 2A) and / or any other predefined location (e.g., resources 202 and 204 in FIG. 2A). The system 300 can be configured to use machine learning techniques to train at least one machine learning-based classifier 330 (e.g., a software model, a neural network classification layer, etc.) configured to classify features extracted from sensor data, such as sensor data received from the sensing module 110 of FIG. 1. The machine learning-based classifier 330 can be trained to classify features extracted from sensor data to identify resources and determine information about the resources, such as resource type, location, optimal stacking orientation, inconsistent resources when stacked and / or oriented together, resource count, etc., based on, for example, an analysis of one or more training data sets 310A - 310N.

[0022] The one or more training data sets 310A - 310N can include labeled reference data, such as labeled resource types (e.g., resources of various shapes, bottles, cans, bowls, boxes, etc.) and / or labeled pallet packing scenarios (e.g., optimized stacking based on resource shape / design, resource location, optimal stacking orientation, inconsistent resources when stacked and / or oriented together, etc.). The labeled reference data can include any number of feature sets (e.g., labeled data identifying features extracted from sensor data).

[0023] The labeled reference data can be stored in one or more databases. Data for multi-sensor perception for resource tracking and quantification (e.g., sensor data, resource information, etc.) can be randomly assigned to a training data set or a test data set. According to some aspects, the assignment of data to a training data set or a test data set may not be completely random. In this case, one or more criteria can be used during the assignment, such as ensuring that the same resource type, the same pallet packing scenario, different resource types, different pallet packing scenarios, etc. can be used in each of the training data set and the test data set. Generally, any suitable method can be used to assign data to a training data set or a test data set.

[0024] The analysis module 112 can train the machine learning-based classifier 330 by extracting a feature set from the labeled reference data according to one or more feature selection techniques. According to some aspects, the analysis module 112 can further define a feature set obtained from the labeled reference data by applying one or more feature selection techniques to the labeled reference data in one or more training data sets 310A - 310N. The analysis module 112 can extract the feature set from the training data sets 310A - 310N in various ways. The analysis module 112 can perform feature extraction multiple times, using different feature extraction techniques each time. In some cases, different machine learning-based classification models 340 can be generated using the feature sets generated using various techniques respectively. According to some aspects, the feature set with the highest quality metrics can be selected for use in training. The analysis module 112 can construct one or more machine learning-based classification models 340A - 340N configured to determine and / or predict resource types, pallet packing scenarios, etc. using the feature set.

[0025] According to some aspects, the training data sets 310A - 310N, and / or the labeled reference data may be analyzed to determine any dependencies, associations, and / or correlations between resource types, pallet packing scenarios, etc. in the training data sets 310A - 310N, and / or the labeled reference data. As used herein, the term "feature" may refer to any characteristic of a data resource that can be used to determine whether the data resource falls into one or more specific categories. For example, the features described herein may include resource type, pallet packing scenario, and / or any other characteristic.

[0026] According to some aspects, a feature selection technique can include one or more feature selection rules. The one or more feature selection rules can include determining which features in the labeled reference data occur more than a threshold number of times in the labeled reference data and identifying these features that meet the threshold as candidate features. For example, a feature that occurs two or more times in the labeled reference data can be considered a candidate feature. Features that occur less than two times can be excluded from consideration as features. According to some aspects, a single feature selection rule may be applied to feature selection, or multiple feature selection rules may be applied to feature selection. According to some aspects, the feature selection rules can be applied in a cascading manner, where the feature selection rules are applied in a specific order and applied to the results of previous rules. For example, the feature selection rules can be applied to the labeled reference data to generate information (e.g., indication of resource type, indication of pallet packing scenario, etc.) that can be used for multi - sensor perception for resource tracking and quantification. The final list of candidate features can be analyzed according to additional features.

[0027] According to some aspects, the analysis module 112 can generate information (e.g., indication of resource type, indication of pallet packing scenario, etc.) that can be used for multi-sensor perception for resource tracking and quantification, and the operation can be based on a wrapper method. The wrapper method can be configured to use a subset of features to train a machine learning model using that subset of features. Based on inferences derived from a previous model, features can be added to and / or removed from the subset. The wrapper method includes, for example, forward feature selection, backward feature elimination, recursive feature elimination, combinations thereof, etc. According to some aspects, forward feature selection can be used to identify one or more candidate resource types, pallet packing scenarios, etc. Forward feature selection is an iterative method that starts with no features in the machine learning model. In each iteration, as long as adding a new variable improves the performance of the machine learning model, the feature that best improves the model is added. According to some aspects, backward elimination can be used to identify one or more candidate resource types, pallet packing scenarios, etc. Backward elimination is an iterative method that starts with all features in the machine learning model. In each iteration, the least important feature is removed as long as the removal does not result in a lack of improvement. According to some aspects, recursive feature elimination can be used to identify one or more candidate resource types, pallet packing scenarios, etc. Recursive feature elimination is a greedy optimization algorithm aimed at finding the best performing subset of features. Recursive feature elimination creates an iterative model and in each iteration, it retains the best performing feature or the worst performing feature. Recursive feature elimination constructs the next model with the remaining features until all features are exhausted. Next, recursive feature elimination ranks the features based on the order of elimination.

[0028] According to some aspects, one or more candidate resource types, pallet packing scenarios, etc. can be determined by an embedded method. The embedded method has the characteristics of a filter and a wrapper method. The embedded method includes, for example, the Least Absolute Shrinkage and Selection Operator (LASSO) that implements a penalty function to reduce overfitting, and ridge regression. For example, LASSO regression performs L1 regularization by adding a penalty corresponding to the absolute value of the coefficient magnitude, and ridge regression performs L2 regularization by adding a penalty corresponding to the square of the coefficient magnitude.

[0029] According to some aspects, one or more candidate resource types, pallet packing scenarios, etc. can be determined by an ensemble method. The ensemble method can combine the outputs from separate classification models to generate a final decision regarding the candidate resource type, pallet packing scenario, etc. According to some aspects, the ensemble method can include thresholding logic that evaluates each classification from one classification model to weight the influence of the classification model on the final classification result.

[0030] After the analysis module 112 generates the feature set, the analysis module 112 can generate a machine learning-based prediction model 340 based on the feature set. The machine learning-based prediction model can refer to a complex mathematical model for data classification generated using machine learning techniques. For example, this machine learning-based classifier can include a map of support vectors representing boundary features. As an example, the boundary features can be selected from the feature set and / or represent the highest-ranked features in the feature set.

[0031] According to some aspects, the analysis module 112 can build machine learning-based classification models 340A - 340N for determining and / or predicting resource types, palette packing scenarios, etc. using the training data sets 310A - 310N and / or feature sets extracted from labeled reference data. According to some aspects, the machine learning-based classification models 340A - 340N may be combined into a single machine learning-based classification model 340. Similarly, the machine learning-based classifier 330 may represent a single classifier including one or more machine learning-based classification models 340, and / or multiple classifiers each including one or more machine learning-based classification models 340. According to some aspects, the machine learning-based classifier 330 may also include each of the training data sets 310A - 310N, and / or each feature set extracted from the training data sets 310A - 310N and / or extracted from labeled reference data. Although shown separately, the analysis module 112 may include the machine learning-based classifier 330.

[0032] The features extracted from the imaging data can be combined using a classification model trained with the following machine learning approaches: discriminant analysis, decision trees, nearest neighbor (NN) algorithms (e.g., k-NN model, replicator NN model, etc.), statistical algorithms (e.g., Bayesian network, etc.), clustering algorithms (k-means method, mean shift method, etc.), neural networks (e.g., reservoir network, artificial neural network, etc.), support vector machine (SVM), logistic regression algorithm, linear regression algorithm, Markov model, or chain, principal component analysis (PCA) (e.g., for linear models), multi-layer perceptron (MLP) ANN (e.g., for non-linear models), replicated reservoir network (e.g., for non-linear models, usually for time series), random forest classification, combinations thereof, etc. The resulting machine learning-based classifier 330 can include decision rules, or mappings, that use sensor data to determine and / or predict resource types, pallet packing scenarios, etc. The resulting machine learning-based classifier 330 can include a family of composite scaled object detection models trained using the imaging data and can include simple functions for test time augmentation (TTA), model ensemble, hyperparameter evolution, and data export.

[0033] Sensor data, and the machine learning-based classifier 330, can be used to determine and / or predict resource types, pallet packing scenarios, etc. for test samples within a test dataset. For example, the result of each test sample can include a confidence level corresponding to the likelihood or probability that the corresponding test sample accurately determines and / or predicts a resource type, pallet packing scenario, etc. The confidence level can be a value from 0 to 1 representing the likelihood that the determined / predicted resource type, pallet packing scenario, etc. matches the calculated value. Multiple confidence levels can be provided for each test sample and each candidate (approximate) resource type, pallet packing scenario, etc. The candidate resource type, pallet packing scenario, etc. with the highest performance can be determined by comparing the results obtained for each test sample with the resource type, pallet packing scenario, etc. calculated for each test sample. Generally, the candidate resource type, pallet packing scenario, etc. with the highest performance will have results that closely match the calculated resource type, pallet packing scenario, etc. The candidate resource type, pallet packing scenario, etc. with the highest performance can be used for multi-sensor perception for resource tracking and quantification operations.

[0034] FIG. 4 is a flowchart illustrating an exemplary training method 400 for generating a machine learning classifier 330 using an analysis module 112, according to some embodiments. The analysis module 112 can implement a machine learning-based classification model 340 with supervised, unsupervised, and / or semi-supervised (e.g., reinforcement-based) learning. The method 400 shown in FIG. 4 is an example of a supervised learning method, and variations of this example of the training method will be described later. However, other training methods can be implemented by analogy to train unsupervised and / or semi-supervised (predictive) machine learning models. The method 400 can be executed by processing logic that can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps may be required to implement the disclosure provided herein. Further, as will be understood by those skilled in the art, some of the steps can be executed simultaneously or in an order different from that shown in FIG. 4.

[0035] Referring to FIGS. 1-3, method 400 will be described. However, method 400 is not limited to the embodiments of these figures.

[0036] At 410, the analysis module 112 determines (e.g., accesses, receives, obtains, etc.) sensor data and / or resource information. The sensor data and / or resource information can include one or more data sets, and each data set is associated with a resource type, a pallet packing scenario, etc.

[0037] At 420, the analysis module 112 generates a training data set and a test data set. According to some aspects, the training data set and the test data set can be generated by indicating resource types, pallet packing scenarios, and the like. According to some aspects, the training data set and the test data set can be generated by randomly assigning resource types, pallet packing scenarios, and the like to either the training data set or the test data set. According to some aspects, assigning sensor data and / or resource information as training samples or test samples may not be completely random. According to some aspects, only labeled reference data regarding specific features extracted from specific sensor data (e.g., an optimized pallet packing scenario, etc.) can be used to generate the training data set and the test data set. According to some aspects, most of the labeled reference data extracted from sensor data and / or resource information can be used to generate the training data set. For example, 75% of the labeled reference data for determining resource types, pallet packing scenarios, etc. extracted from imaging data can be used to generate the training data set, and 25% can be used to generate the test data set. Any method or technique can be used to create the training data set and the test data set.

[0038] At 430, the analysis module 112 determines (e.g., extracts, selects, etc.) one or more features that can be used by, for example, a classifier (e.g., a software model, a classification layer of a neural network, etc.) to label features extracted from various sensor data and / or resource information. The one or more features can include indications such as resource types, pallet packing scenarios, and the like. According to some aspects, the analysis module 112 can determine a set of training reference features from the training data set. The features of the sensor data and / or resource information can be determined by any method.

[0039] At 440, the analysis module 112 trains one or more machine learning models, for example, using one or more features. In some aspects, the machine learning models can be trained using supervised learning. In some aspects, other machine learning techniques, including unsupervised learning and semi-supervised learning, can also be employed. The machine learning models trained at 440 can be selected based on different criteria (e.g., how close the predicted resource types, pallet packing scenarios, etc. are to the actual resource types, pallet packing scenarios, etc.) and / or the data available in the training dataset. For example, machine learning classifiers can have problems with various degrees of bias. In some aspects, two or more machine learning models can be trained.

[0040] At 450, the analysis module 112 optimizes, improves, and / or cross-validates the trained machine learning models. For example, the data in the training dataset and / or the test dataset can be updated and / or modified to include more labeled data indicating various resource types, pallet packing scenarios, etc.

[0041] At 460, the analysis module 112 selects one or more machine learning models to build a prediction model (e.g., a machine learning classifier, a prediction engine, etc.). The prediction model can be evaluated using a test dataset.

[0042] At 470, the analysis module 112 runs the prediction model to analyze the test dataset and generate classification values and / or prediction values.

[0043] At 480, the analysis module 112 evaluates the classification value and / or prediction value output by the prediction model and determines whether such a value has reached the desired accuracy level. The performance of the prediction model can be evaluated in many ways based on the number of true positives, false positives, true negatives, and / or false negatives of a plurality of data points indicated by the prediction model. For example, the false positives of the prediction model can refer to the number of times the prediction model has incorrectly predicted and / or determined the resource type, pallet packing scenario, etc. Conversely, the false negatives of the prediction model can refer to the number of times the machine learning model has incorrectly predicted and / or determined the resource type, pallet packing scenario, etc. when the resource type, pallet packing scenario, etc. actually predicted and / or determined actually matches the actual resource type, pallet packing scenario, etc. True negatives and true positives can refer to the number of times the prediction model has accurately predicted and / or determined the resource type, pallet packing scenario, etc. Related to these measurements are the concepts of recall and precision. Generally, recall refers to the ratio of true positives to the sum of true positives and false negatives, and quantifies the sensitivity of the prediction model. Similarly, precision refers to the ratio of true positives to the sum of true positives and false positives.

[0044] At 490, the analysis module 112 outputs the prediction model (and / or the output of the prediction model). For example, the analysis module 112 can output the prediction model when such a desired accuracy level is reached. The output of the prediction model can terminate the training phase.

[0045] According to some aspects, if the desired accuracy level is not reached, at 490, the analysis module 112 can execute the next iteration of the training method 400 starting from 410, including variant forms such as considering a larger-scale collection of sensor and / or resource information.

[0046] FIG. 5 shows a flowchart of an exemplary method 500 for multi-sensor perception for resource tracking and quantification, according to some aspects. Method 500 can be executed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps may be required to implement the disclosure provided herein. Further, as will be understood by those skilled in the art, some of the steps may be executed simultaneously or in an order different from the order shown in FIG. 5.

[0047] Referring to FIGS. 1-4, method 500 will be described. However, method 500 is not limited to the aspects of these figures. A computer-based system (e.g., system 100, etc.) can facilitate multi-sensor perception for resource tracking and quantification.

[0048] At 510, computing device 102 receives resource information indicating an identifier, a resource location, and a resource quantity. The resource information may include, for example, purchase order information, warehouse slot mapping information, and the like. For example, according to some aspects, computing device 102 may load and / or be configured using an identifier (e.g., SKU, UPC, unique identifier, etc.), a location (e.g., warehouse inventory location, shelf, aisle, bin, etc.), and / or a quantity (e.g., purchase order quantity, inventory quantity, required quantity, etc.) of a resource and / or a plurality of resources (e.g., beverages, inventory / inventory resources, products, etc.), such as purchase order information, warehouse slot mapping information. According to some aspects, computing device 102 may receive, from a device (e.g., a cloud-based device, a service provider device, a computing device, etc.), a network (e.g., a private network, a public network, the Internet, etc.), a system (such as a warehouse management system, etc.), purchase order information, warehouse slot mapping information, etc., indicating an identifier, a location, and / or a quantity of a resource and / or a plurality of resources. Computing device 102 may receive the resource information by any method or technique.

[0049] At 520, computing device 102 tracks resources that are removed from their location and placed at a predefined location at the location of the resources. For example, computing device 102 may track resources that are removed from their location and placed at a predefined location based on sensor data and resource identifiers received from a first sensing device. According to some aspects, the predefined location may be on a pallet. For example, the resources may be boxes, packages, etc. of beverages (or any other product), and computing device 102 may track the resources when they are removed from any location within the warehouse and placed on the pallet. According to some aspects, the pallet may be placed on a mobile unit (e.g., a pallet truck, forklift, stock cart, etc.) and / or attached to the mobile unit. According to some aspects, the sensing device may include one or more of a camera (e.g., a stereo camera, high-resolution camera, video camera, smart camera, PoE camera, etc.), a LIDAR sensor, an infrared sensor, a position detection sensor, etc.

[0050] According to some aspects, computing device 102 tracks resources that are removed from their location and placed at a predefined location by determining that a certain resource type corresponds to the resource type indicated by the respective identifier. For example, computing device 102 may track resources that are removed from their location and placed at a predefined location by determining, based on object recognition performed by a trained machine learning model, that the resource type corresponds to the resource type indicated by the resource identifier. The trained machine learning model may identify the resource type based on the correspondence between the feature classification data and the features extracted from the sensor data received from the first sensing device. The trained machine learning model may identify the resource based on the correspondence between the resource type and the resource type indicated by the resource identifier.

[0051] Based on object tracking applied to sensor data received from, for example, a first sensing device, computing device 102 determines that a resource is located within an area of a pre-defined location. For example, a trained machine learning model may determine that a resource is located within an area of a pre-defined location based at least on a difference between a position of a resource shown in a first image extracted from sensor data received from a first sensing device and a position of a resource shown in a second image extracted from sensor data received from the first sensing device.

[0052] In some aspects, computing device 102 causes, via a user interface, an indication of a resource located within each area of a pre-defined location to be displayed. The indication of a resource located within each area of a pre-defined location may include, for example, a value associated with the resource, an image of the resource, a graphical representation of the resource, and the like.

[0053] At 530, computing device 102 determines an amount of a resource removed from a location of the resource and placed at a pre-defined location. For example, computing device 102 may determine an amount of a resource among a plurality of resources removed from a location of the resource and placed at a pre-defined location based on depth information indicated by sensor data received from a second sensing device.

[0054] According to some aspects, the computing device 102 determines the amount of resources placed at a predefined location removed from the location of the resources by causing an indication of the resources placed within an area of the predefined location associated with the resource type of the resources to be displayed. The computing device 102 determines the amount of resources based on a match between each depth value indicated by depth information for each resource removed from its respective location and a depth value associated with the amount of each resource type.

[0055] At 540, the computing device 102 determines a match between the amount of resources and the amount of resources removed from the location of the resources and placed at the predefined location. For example, the computing device 102 may access a storage device storing the amount of resources and determine that the value indicated by the amount of resources matches / corresponds to the amount of resources removed from the location of the resources and placed at the predefined location.

[0056] At 550, computing device 102 generates a notification indicating that additional resources matching the resource should stop being removed from the location of the resource and placed in a predefined location. For example, computing device 102 generates a notification indicating that additional resources matching the resource should stop being removed from the location of the resource and placed in a predefined location based on a match between the amount of the resource and the amount of the resource being removed from the location of the resource and placed in a predefined location. The notification can include an audible notification (e.g., text-to-speech, recorded / computerized voice, chime, alarm, etc.), a text notification (e.g., a text message sent to a mobile device of a user, product / resource picker, warehouse worker, entity, etc.), a visual notification (e.g., a notification displayed on a user interface, etc.). The notification can cause an entity, for example, to stop additional resources matching the resource from being removed from the location of the resource and placed in a predefined location. According to some aspects, the notification can be an instruction that directs an entity to stop removing additional resources matching the resource from the location of the resource and placing them in a predefined location. According to some aspects, the notification can, for example, indicate in real time the amount of additional resources remaining in the location of the resource, in inventory, etc.

[0057] According to some aspects, the notification can include one or more control signals that cause an entity, such as an autonomous robot / device, to stop removing additional resources matching the resource from the location of the resource and placing them in a predefined location.

[0058] According to some embodiments, method 500 may further include the computing device 102 generating a notification indicating a discrepancy for one of the plurality of resources based on a discrepancy between the respective amount of that resource and the amount of the resource removed from each location and placed at a predefined location. The notification may include an audible notification (e.g., text-to-speech, recorded / computerized voice, chime, alarm, etc.), a text notification (e.g., a text message sent to a mobile device of a user, product / resource picker, warehouse worker, entity, etc.), a visual notification (e.g., a notification displayed on a user interface, etc.), and the like.

[0059] FIG. 6 is an exemplary computer system useful in implementing various embodiments. The various embodiments can be implemented using one or more known computer systems, such as the computer system 600 shown in FIG. 6. One or more computer systems 600 can be used to implement, for example, any of the embodiments discussed herein, as well as combinations and sub-combinations thereof. According to some embodiments, the computing device 102 of FIG. 1 (and / or other devices / components described herein) can be implemented using the computer system 600. According to some embodiments, methods 400 and 500 can be implemented using the computer system 600.

[0060] The computer system 600 can include one or more processors, such as a processor 604 (also referred to as a central processing unit (CPU)). The processor 604 can be connected to a communication infrastructure or bus 606.

[0061] The computer system 600 can also include user input / output devices 602, such as a monitor, keyboard, pointing device, etc., which can communicate with the communication infrastructure or bus 606 through the user input / output devices 602.

[0062] One or more processors 604 can be a graphics processing unit (GPU). In one embodiment, the GPU can be a processor that is a dedicated electronic circuit designed to process mathematically intensive applications. The GPU can have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

[0063] Also, computer system 600 can also include a main memory, or primary memory 608, such as a random access memory (RAM). The main memory 608 can include one or more levels of cache. The main memory 608 can store control logic (i.e., computer software) and / or data therein.

[0064] Also, computer system 600 can include one or more secondary storage devices, or memories 610. The secondary memory 610 can include, for example, a hard disk drive 612 and / or a removable storage device or drive 614. The removable storage drive 614 can be a floppy disk drive, magnetic tape drive, compact disk drive, optical storage device, tape backup device, and / or other storage device / drive.

[0065] Removable memory drive 614 can interact with removable memory unit 618. Removable memory unit 618 can include a computer-usable or readable memory device in which computer software (control logic) and / or data is stored. Removable memory unit 618 can be a floppy disk, magnetic tape, compact disk, DVD, optical memory disk, and other computer data storage devices. Removable memory drive 614 can read from and / or write to removable memory unit 618.

[0066] Secondary memory 610 can include other means, devices, components, apparatuses, and / or other approaches for making a computer program, and / or other instructions, and / or data accessible to computer system 600. Such means, devices, components, apparatuses, and / or other approaches can include, for example, removable memory unit 622 and interface 620. Examples of removable memory unit 622 and interface 620 can include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and / or other removable memory units and associated interfaces.

[0067] Furthermore, computer system 600 may include a communication, or network interface 624. By means of communication interface 624, computer system 600 can communicate and interact with any combination of external devices, external networks, external entities, etc. (collectively and individually referred to by reference numeral 628). For example, by means of communication interface 624, computer system 600 can communicate with external device, or remote device 628 via communication path 626, which can be wired and / or wireless (or a combination thereof) and can include any combination such as LAN, WAN, Internet, etc. Control logic and / or data can be transmitted between computer system 600 via communication path 626.

[0068] Also, computer system 600 can be, by way of some non-limiting examples, a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet, a smartphone, a smartwatch or other wearable, an appliance, a part of the Internet of Things, and / or an embedded system, or any combination thereof.

[0069] Computer system 600 can be a client or server that accesses or hosts any application and / or data through any delivery paradigm, including but not limited to: remote or distributed cloud computing solutions, local or on-premises software (an "on-premises" cloud-based solution), a "X as a service" model (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (SaaS), managed software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (IaaS), etc.), and / or a hybrid model that includes any combination of the foregoing examples or other services or delivery paradigms.

[0070] Data structures, file formats, and schemas applicable in computer system 600 can be derived from standards including, but not limited to, JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or other functionally similar representations, either alone or in combination. Alternatively, an original data structure, format, and / or schema can be used alone or in combination with known or open standards.

[0071] In some embodiments, a tangible non-transitory device or article of manufacture comprising a tangible non-transitory computer-usable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 600, main memory 608, secondary memory 610, and removable storage units 618, 622, and tangible articles of manufacture embodying any of the foregoing combinations. Such control logic, when executed by one or more data processing devices (such as computer system 600), can operate such data processing devices as described herein.

[0072] Based on the teachings contained in this disclosure, it will be apparent to those skilled in the art how to create and use embodiments of this disclosure using data processing devices, computer systems, and / or computer architectures other than those shown in FIG. 6. Specifically, the embodiments can operate using implementations of software, hardware, and / or operating systems other than those described herein.

[0073] It should be understood that the "Detailed Description" section, and not any other section, is intended to be used to interpret the claims. The other sections may describe one or more, but not all, exemplary embodiments contemplated by the inventor(s), and are in no way intended to limit the present disclosure and the appended claims.

[0074] Additionally and / or alternatively, in this disclosure, exemplary embodiments are described for exemplary fields and uses, but it should be understood that the present disclosure is not limited thereto. Other embodiments and modifications thereof are possible and are within the scope and spirit of the present disclosure. For example, without limiting the generality of this paragraph, the embodiments are not limited to the software, hardware, firmware, and / or entities illustrated in the figures and / or described herein. Further, the embodiments (whether explicitly described herein or not) have significant utility in fields and uses beyond the examples described herein.

[0075] One or more portions of the above implementations may include software. Software is a general term meaning certain functions and their relationships. The boundaries of these functional building blocks are arbitrarily defined herein for convenience of explanation. Alternative boundaries may be defined as long as the certain functions and relationships (or their equivalents) are properly implemented. Also, alternative embodiments may implement functional blocks, steps, operations, methods, etc. using an order different from that described herein.

[0076] References to "one embodiment", "an embodiment", "an exemplary embodiment", or similar phrases in this specification indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments necessarily include the particular feature, structure, or characteristic. Further, such phrases do not necessarily refer to the same embodiment. Additionally, when describing a particular feature, structure, or characteristic in relation to one embodiment, it is within the knowledge of those skilled in the art to incorporate such feature, structure, or characteristic into other embodiments, whether or not explicitly recited or described herein. Moreover, some embodiments may be described using the terms "coupled" and "connected" along with their derivatives. These terms are not necessarily intended to be synonyms of each other. For example, some embodiments may be described using the terms "connected" and / or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other but still cooperate or interact with each other.

[0077] The breadth and scope of the present disclosure should not be limited by any of the above exemplary embodiments, but should be defined only by the following claims and their equivalents.

Claims

Claim 1 A computer-implemented method for multi-sensor perception for resource tracking and quantification, comprising: receiving resource information indicating a resource identifier, a location of the resource, and an amount of the resource; tracking, at the location of the resource, a resource removed from the location of the resource and disposed at a predefined location based on sensor data received from a first sensing device and the resource identifier; determining an amount of the resource removed from the location of the resource and disposed at the predefined location based on depth information indicated by sensor data received from a second sensing device; determining a match between the amount of the resource and the amount of the resource removed from the location of the resource and disposed at the predefined location; generating a notification indicating that additional resources matching the resource should be stopped from being removed from the location of the resource and disposed at the predefined location based on the match between the amount of the resource and the amount of the resource removed from the location of the resource and disposed at the predefined location; A computer-implemented method comprising the above steps. Claim 2 Tracking the resource removed from the location of the resource and disposed at the predefined location further comprises: identifying a type of the resource based on a correspondence between feature classification data and features extracted from the sensor data received from the first sensing device; identifying the resource based on a correspondence between the type of the resource and a type of the resource indicated by the resource identifier; determining that the resource is disposed in an area of the predefined location based at least on a difference between a position of the resource shown in a first image extracted from the sensor data received from the first sensing device and a position of the resource shown in a second image extracted from the sensor data received from the first sensing device; displaying an indication of the resource disposed in the area of the predefined location; The method according to claim 1, further comprising the above steps. Claim 3 The indication of the resource disposed within the area of the pre-defined location includes at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource, according to the method described in claim 2.

4. Determining the amount of the resource removed from the location of the resource and disposed at the pre-defined location includes displaying an indication of the resource disposed within the area of the pre-defined location associated with the resource type of the resource; and determining the amount of the resource based on a match between the depth value indicated by the depth information and the depth value associated with the amount of the resource type. The method according to claim 1 further includes the above steps.

5. The notification to stop additional resources matching the resource from being removed from the location of the resource and disposed at the pre-defined location includes at least one of an audible notification, a text notification, or a visual notification, according to the method described in claim 1.

6. The method according to claim 1 further includes generating a notification indicating the discrepancy based on a discrepancy between the amount of the resource and the amount of the resource removed from the location of the resource and disposed at the pre-defined location.

7. The method according to claim 6, wherein the notification indicating the discrepancy includes at least one of an audible notification, a text notification, or a visual notification.

8. A non-transitory computer-readable medium storing instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for multi-sensor perception to track the location and amount of a resource, the operations including receiving resource information indicating a resource identifier, a location of the resource, and an amount of the resource; tracking, at the location of the resource, the resource removed from the location of the resource and disposed at the pre-defined location based on sensor data received from a first sensing device and the resource identifier; and determining the amount of the resource removed from the location of the resource and disposed at the pre-defined location based on depth information indicated by sensor data received from a second sensing device. Determining a match between the amount of the resource and the amount of the resource removed from the location of the resource and placed at the predefined location; Generating a notification indicating that additional resources matching the resource should be stopped from being removed from the location of the resource and placed at the predefined location based on the match between the amount of the resource and the amount of the resource removed from the location of the resource and placed at the predefined location; A non-transitory computer-readable medium including the above.

9. Tracking the resource removed from the location of the resource and placed at the predefined location includes: Identifying the type of the resource based on a correspondence between feature classification data and features extracted from the sensor data received from the first sensing device; Identifying the resource based on a correspondence between the type of the resource and the type of the resource indicated by the resource identifier; Determining that the resource is placed in the area of the predefined location based at least on a difference between the position of the resource shown in a first image extracted from the sensor data received from the first sensing device and the position of the resource shown in a second image extracted from the sensor data received from the first sensing device; Displaying an indication of the resource placed in the area of the predefined location; The non-transitory computer-readable medium according to claim 8, further including the above.

10. The indication of the resource placed within the area of the predefined location includes at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource, according to the non-transitory computer-readable medium of claim 9.

11. Determining the amount of the resource removed from the location of the resource and placed at the predefined location includes: Displaying an indication of the resource placed within the area of the predefined location associated with the resource type of the resource; Determining the amount of the resource based on a match between the depth value indicated by the depth information and the depth value associated with the amount of the resource type; The non-transitory computer-readable medium according to claim 8, further comprising **Claim 12** The notification that stops an additional resource matching the resource from being removed from the location of the resource and placed at the predefined location includes at least one of an audible notification, a text notification, or a visual notification, the non-transitory computer-readable medium according to claim 8. **Claim 13** The non-transitory computer-readable medium according to claim 8, further comprising generating a notification indicating the discrepancy based on a discrepancy between the amount of the resource and the amount of the resource removed from the location of the resource and placed at the predefined location. **Claim 14** The notification indicating the discrepancy includes at least one of an audible notification, a text notification, or a visual notification, the non-transitory computer-readable medium according to claim 13. **Claim 15** A memory and At least one processor coupled to the memory and configured to perform operations for multi-sensor perception to track the location and amount of a resource, the operations being Receiving resource information indicating a resource identifier, a location of the resource, and an amount of the resource; Tracking, at the location of the resource, the resource removed from the location of the resource and placed at a predefined location based on sensor data received from a first sensing device and the resource identifier; Determining the amount of the resource removed from the location of the resource and placed at the predefined location based on depth information indicated by sensor data received from a second sensing device; Determining a match between the amount of the resource and the amount of the resource removed from the location of the resource and placed at the predefined location; Generating a notification indicating that an additional resource matching the resource should be stopped from being removed from the location of the resource and placed at the predefined location based on the match between the amount of the resource and the amount of the resource removed from the location of the resource and placed at the predefined location; A system comprising. **Claim 16** Tracking the resource removed from the location of the resource and placed at the predefined location is Identifying the type of the resource based on a correspondence relationship between the feature classification data and the features extracted from the sensor data received from the first sensing device; Identifying the resource based on a correspondence relationship between the type of the resource and the type of the resource indicated by the resource identifier; Determining that the resource is arranged in the area of the predefined location based at least on a difference between the position of the resource shown in a first image extracted from the sensor data received from the first sensing device and the position of the resource shown in a second image extracted from the sensor data received from the first sensing device; Causing an indication of the resource arranged in the area of the predefined location to be displayed; The system according to claim 15, further comprising.

17. The indication of the resource arranged in the area of the predefined location includes at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource, according to the system of claim 16.

18. Determining the amount of the resource removed from the location of the resource and arranged at the predefined location includes: Causing an indication of the resource arranged in the area of the predefined location associated with the resource type of the resource to be displayed; Determining the amount of the resource based on a match between the depth value indicated by the depth information and the depth value associated with the amount of the resource type; The system according to claim 15, further comprising.

19. The notification for stopping an additional resource matching the resource from being removed from the location of the resource and arranged at the predefined location includes at least one of an audible notification, a text notification, or a visual notification, according to the system of claim 15.

20. The operation includes: The system according to claim 15, further comprising generating a notification indicating the discrepancy based on a discrepancy between the amount of the resource and the amount of the resource removed from the location of the resource and arranged at the predefined location.

Images