Multi-sensor awareness for resource tracking and quantization

Abstract

Systems, methods, and computer program product embodiments for multi-sensor perception for resource tracking and quantification are disclosed herein. One embodiment operates by receiving resource information indicating a resource identifier, a resource location, and a number of resources. Based on sensor data and a resource identifier received from a first sensing device, a resource removed from a resource location and placed at a predefined location may be tracked. The number of resources placed at the predefined location may be determined based on depth information from sensor data received from the second sensing device. A notification may be generated based on a match between the number of resources and the number of resources placed at the predefined location indicating that additional resources matching the resources should stop being removed from the resource location and placed at the predefined location.

Description

Background Art

[0001] Economic growth in industries such as e-commerce has revealed unexpected inventory management challenges. Inventory inaccuracies cause problems such as inaccurate resource (e.g., items, products, components, etc.) delivery, which affect customer service levels and erode profit margins. Pallets are often used to package, transport, and manage resources / products. Traditionally, the pallet building process that requires entities (e.g., product / resource pickers, warehouse workers, autonomous / semi-autonomous robots / equipment, etc.) to manually build each pallet based on a purchase order is labor-intensive. When resources / products are placed on pallets, entities usually have to manually count and track resource / product identifiers (e.g., stock keeping units (SKUs), etc.). Manual processes are often error-prone and result in inaccurate pallets being sent to consumers, businesses, etc. This can prompt expensive corrective measures. In addition, poorly packaged pallets may cause injury and / or physical harm due to falling resources / products. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and enable those skilled in the relevant art to make and use the present disclosure.

[0003] Figure 1 An example system for multi-sensor awareness for resource tracking and quantification according to some aspects is shown.

[0004] Figure 2A An example system for multi-sensor awareness for resource tracking and quantification according to some aspects is shown.

[0005] Figure 2B An example display of a multi-sensor aware user interface for resource tracking and quantification according to some aspects is shown.

[0006] Figure 3 An example system for training an analysis module for multi-sensor awareness that can be used for resource tracking and quantification is shown in accordance with some aspects.

[0007] Figure 4 A flow chart illustrating an example training method for generating a machine learning classifier to classify sensor data for multi-sensor sensing for resource tracking and quantification according to some aspects.

[0008] Figure 5 A flow diagram of an example method of multi-sensor awareness for resource tracking and quantification in accordance with some aspects is shown.

[0009] Figure 6A schematic block diagram of an exemplary computer system is shown in which the described aspects may be implemented. DETAILED DESCRIPTION

[0010] Provided herein are example systems, devices, apparatuses, methods, computer program product implementations, and / or combinations and sub-combinations thereof for multi-sensor perception of resource tracking and quantification. Resources may include, for example, articles, products, components, packaging, equipment, and the like. According to some aspects, a computing device (e.g., a modular device, a stand-alone device, an Internet of Things (IoT) device, a warehouse management device, a resource management device, and the like) may be configured with and / or receive resource information indicating a corresponding identifier, a corresponding location, and a corresponding quantity of each resource in a plurality of resources (such as resources within a warehouse setting, and the like). The resource information may include, for example, purchase order information, location mapping information, and the like. The computing device may include one or more sensing devices (e.g., imaging devices, cameras, light detection and ranging (LIDAR) sensors, depth detection devices, and the like) positioned to capture sensor data (e.g., video, static images, image data, depth information, and the like) indicating resources in its vicinity. One or more sensing devices may capture the entire resource retrieval process, including resource selection / picking and placement of resources in a predefined area (e.g., on a pallet, and the like). For example, a computing device may detect, track, and count resources when they are placed on a pallet, and identify any differences with the resource information. The computing device may generate a notification to facilitate correction / verification of any identified discrepancies between the resources placed on the pallet and the resource information. The notification may include an audible notification (e.g., text-to-speech, recorded / computerized voice, beep, alarm, etc.), a text notification (e.g., a text message sent to a mobile device of a user, product / resource picker, warehouse worker, etc.), a visual notification (e.g., a notification displayed on a user interface, etc.), etc.

[0011] The system, device, apparatus, method, computer program product implementation scheme and / or its combination and sub-combination of multi-sensor perception for resource tracking and quantification overcomes various technical problems associated with conventional systems. For example, traditionally, inventory and / or resource management processes (such as pallet building processes in warehouses, etc.) are labor-intensive-requiring each entity (e.g., product / resource pickers, warehouse workers, autonomous / semi-autonomous robots / equipment, etc.) to manually build each pallet according to a purchase order by manually identifying and counting the resources placed on the pallet. The manual process leads to inaccurate pallet building, reduced user experience, loss of revenue due to inaccuracy in inventory and product shipment / sales, etc. For example, industry analysts suggest that there are more than 2 billion pallets in circulation in the United States at any time, and resource inaccuracies in pallet building cause an average of more than $40 million in revenue losses each year. Conventional inventory and / or resource management processes cannot accurately identify, track and quantify resources within, for example, warehouse settings, etc. For example, conventional inventory and / or resource management processes that use radio frequency identification (RFID), near field communication (NFC), etc., which require tracking devices (e.g., RFID transmitters, NFC transmitters, etc.), are cumbersome, require expensive tracking devices to be attached to each resource in the inventory, and may output inaccurate information due to wireless interference, obscure components, etc.

[0012] The system, apparatus, device, method, computer program product implementation scheme and / or combinations and sub-combinations thereof for multi-sensor perception of resource tracking and quantification overcomes problems associated with accurately identifying, tracking and quantifying resources within, for example, a warehouse setting, etc., by using sensor data (e.g., high-resolution imaging data, depth detection data) in conjunction with a trained analysis module (e.g., a machine learning model, a recognition model, etc.). The trained analysis module uses not only resource identification information, but also resource-specific depth and location information from the sensor data to greatly reduce the possibility of resource misidentification and significantly improve the accuracy of resource, tracking, counting and placement (to avoid damage to resources and / or harm to entities, etc.) relative to conventional systems that implement manual methods or technical methods (such as RFID tracking, NFC tracking, etc.). Because the trained analysis module uses sensor data captured from multiple views and representing different types of information related to a resource, the resource can be accurately identified, tracked and quantified.

[0013] The systems, devices, equipment, methods, computer program product embodiments and / or combinations and sub-combinations of multi-sensor perception for resource tracking and quantification described herein provide improvements to computer vision technology to identify, track and quantify resources within, for example, warehouse settings, while eliminating the need for processes such as scanning and recording resources. The systems, devices, equipment, methods, computer program product embodiments and / or combinations and sub-combinations of multi-sensor perception for resource tracking and quantification described herein reach the field of inventory assessment and resource management by using sensor data to implement and / or promote accurate and efficient pallet building processes. For example, notifications of resource differences generated in real time facilitate accelerated pallet correction / verification—which in turn improves the accuracy of each pallet build, while also improving the rate at which accurate pallets of resources can be built. These and other technical advantages are described herein.

[0014] Figure 1 A block diagram of an example system 100 for multi-sensor sensing for resource tracking and quantification according to some aspects is shown. The system 100 can include a computing device 102 (e.g., a modular device, a standalone device, an Internet of Things (IoT) device, a warehouse management device, a resource management device, etc.) and a mobile unit 104 (e.g., a pallet truck, a forklift, a stock truck, a vehicle, a fully or partially autonomous vehicle, etc.).

[0015] According to some aspects, computing device 102 may be attached to mobile unit 104 to facilitate accurate and efficient tracking of the location and quantity of resources (such as resources included in a pallet building process) via multi-sensor sensing. For example, computing device 102 and mobile unit 104 may operate, for example, within a warehouse setting or the like to reduce and / or eliminate discrepancies between requested / ordered resources and picked / put away resources.

[0016] According to some aspects, computing device 102 may include interface module 106. Interface module 106 may be any interface for presenting information to a user and / or receiving information from a user (such as resource information, etc.). Resource information may include, for example, purchase order information, inventory location mapping information, etc. Interface module 106 may include a graphical user interface (GUI) and / or a communication interface, such as a web browser (e.g., Microsoft Internet Explorer). ® 、Mozilla Firefox ® , Google Chrome ® , Apple Safari ® Other software, hardware, and / or interfaces may be used to provide communications between the computing device 102 and users (e.g., product / resource pickers, warehouse workers, etc.), other computing devices 102, warehouse management systems, etc.

[0017] According to some aspects, the interface module 110 may include one or more input devices and / or components, such as, for example, a keyboard, a pointing device (e.g., a computer mouse, a remote control), a microphone, a joystick, a tactile input device (e.g., a touch screen, a glove, etc.), etc. According to some aspects, interaction with the input devices and / or components may enable a user to view, browse, access, request, and / or navigate to information about resources, such as identifiers (e.g., SKUs, universal product codes (UPCs), unique identifiers, etc.) indicating resources 120 (e.g., beverages, inventory / stock resources, products, etc.) within a warehouse, resource storage area, facility, etc., purchase order information indicating locations (e.g., warehouse inventory locations, shelves, aisles, bins, etc.) and / or quantities (e.g., purchase order quantities, inventory quantities, requested quantities, etc.), location mapping information, etc.

[0018] According to some aspects, the computing device 102 may include a storage module 108. According to some aspects, the storage module 108 may be loaded and / or configured with purchase order information, bin location mapping information, etc. indicating identifiers (e.g., SKUs, UPCs, unique identifiers, etc.), locations (e.g., warehouse inventory locations, shelves, aisles, bins, etc.), and / or quantities (e.g., purchase order quantities, inventory quantities, requested quantities, etc.) of resources (e.g., beverages, inventory / stock resources, products, etc.) within a warehouse, etc. Although not shown, according to some aspects, the computing device 102 may receive purchase order information, bin location mapping information, etc. indicating identifiers, locations, and / or quantities of resources within a warehouse, etc. from a device (e.g., a cloud-based device, a service provider device, a computing device, etc.), a network (e.g., a private network, a public network, the Internet, etc.), a system (a warehouse management system, etc.), etc. The computing device 102 may receive and store (e.g., via the storage module 108, etc.) the resource information by any method or technique.

[0019] According to some aspects, computing device 102 may include sensing module 110. According to some aspects, to facilitate multi-sensor sensing to track the location and quantity of resources, computing device 102 may receive different types of data / information from different sensing devices of sensing module 110. The different types of data / information received from the sensing devices of sensing module 110 may be analyzed individually, collectively, etc. According to some aspects, sensing module 110 may include one or more sensing devices, such as cameras (e.g., stereo cameras, high-definition cameras, video cameras, smart cameras, Power over Ethernet (PoE) cameras, etc.), LIDAR sensors, infrared sensors, location detection sensors, depth sensors, etc. For example, according to some aspects, an arrangement of sensing module 110 may include three cameras, such as two stereo cameras and one LIDAR camera. According to some aspects, sensing module 110 may include any number and / or type of sensing devices. Sensing module 110 may capture sensor data (e.g., video, static images, spatial / depth information, etc.) that provides real-time and / or real-world representations of resources within the field of view (and / or detection) of sensing module 110. For example, the sensing module 110 may capture the entire resource retrieval process, including resource selection / picking and resource placement in a predefined area (eg, on a pallet, etc.).

[0020] For example, the computing device 102 may track each resource removed from a corresponding position and placed at a predefined position at a corresponding position of each of the plurality of resources based on sensor data and corresponding identifiers received from at least one sensing device. According to some aspects, the predefined position may be on a pallet. For example, a resource may be a box, package, or the like of a beverage (or any other product), and the computing device 102 may track the resource as it is removed from a position in a warehouse and placed on a pallet. According to some aspects, the pallet may be placed on and / or attached to a mobile unit (e.g., a pallet truck, a forklift, a stock truck, etc.). According to some aspects, the sensing device may include one or more of a camera (e.g., a stereo camera, a high-definition camera, a video camera, a smart camera, a PoE camera, etc.), a LIDAR sensor, an infrared sensor, a position detection sensor, and the like.

[0021] Figure 2A Shows Figure 1 An example of a system 100 is shown in FIG. Figure 2A As shown, the computing device 102 (e.g., the sensing module 110 ) can use sensor data to track resources 202 and 204 during a resource retrieval process performed by an entity 206 (e.g., a product / resource picker, a warehouse worker, an autonomous / semi-autonomous robot / device, etc.), including resource selection / picking and placement of the resources 202 and 204 in a predefined area such as a pallet 208 .

[0022] Computing device 102 may generate and / or display instructions for resource selection displayed via interface module 106. For example, Figure 2A As shown, the interface module 106 can display the quantity (e.g., "Pick 5", "Pick 3", etc.) and identifier (e.g., SKU) of each of the resources 202 and 204 that the entity 206 is to select and place on the pallet 208. According to some aspects, the computing device 102 can generate instructions for resource selection, which are output via a sound notification and / or sent to a user device (e.g., a mobile device, a smart device, a computing terminal, etc.) of the entity 206.

[0023] According to some aspects, computing device 102 tracks each of the resources removed from a designated location of the resources (such as shelves 220 and 220) by determining, for each of the resources 202 and 204 removed from the corresponding location (e.g., shelves 220 and 220, etc.), that the type of the resource corresponds to the type of the resource indicated by the corresponding identifier (e.g., the SKU indicated by interface module 106, etc.). For example, computing device 102 can track each resource (e.g., resources 202 and 204, etc.) removed from the corresponding location (e.g., shelves 220 and 220, etc.) and placed in a predefined location such as pallet 208 by determining, for each of the resources removed from the corresponding location, that the type of the resource corresponds to the type of the resource indicated by the corresponding identifier based on object recognition performed by a trained machine learning model.

[0024] For example, return to Figure 1 According to some aspects, computing device 102 may include analysis module 112. Analysis module 112 may be configured for object detection, object tracking, etc. According to some aspects, analysis module 112 may use artificial intelligence and / or machine learning (such as image / object recognition) to identify resources depicted by one or more of a plurality of images (such as video frames, still images, etc.) included with the sensor data. For example, Figure 2A Interface module 106 is shown displaying an image of entity 206 selecting / picking a resource 224 (e.g., resources 202 and 204, etc.) tracked by image / object recognition (illustrated by bounding box 230, etc.) to identify the resource depicted by one or more of a plurality of images (such as video frames, still images, etc.) included with the sensor data. According to some aspects, analysis module 112 may use one or more object recognition and / or tracking algorithms to determine / detect the resource in the imaging / sensor data.

[0025] According to some aspects, analysis module 112 may include a trained machine learning model, computer vision technology, etc. Analysis module 112 may determine that each resource (e.g., resources 202 and 204, etc.) is placed on pallet 208, for example, based on object tracking applied to sensor data received from sensing module 110. For example, analysis module 112 may identify resources (e.g., resources 202 and 204, etc.) in a first image of a plurality of images received from sensor data and generate detection results. Analysis module 112 may track the resources by analyzing a second image of the plurality of images and the detection results from the first image and by adjusting one or more positions of one or more bounding boxes associated with the resources in the second image to account for the movement of the resources from one or more positions in the first image determined according to the detection results. According to some aspects, analysis module 112 may implement any object tracking method and / or technology.

[0026] According to some aspects, analysis module 112 may use depth information extracted from sensor data to determine the quantity of resources (e.g., resources 202 and 204, etc.) placed on pallet 208 based on a match between a depth value indicated by the depth information and a depth value associated with a quantity of a corresponding resource type. For example, Figure 2B An example screen of the interface module 106 is shown. The computing device 102 may store resource information detailing how many resources of a resource type are represented within a defined area based on a determined depth (e.g., stacking depth) of the resource arrangement. The imaging module 106 may display an image of the resources placed on the pallet 208. The analysis module 112 may identify a resource type, such as a resource 202, and display a bounding box 240 of the resource around a specified area on the pallet 208. Information describing the contents of the bounding box 240 may be displayed. For example, as shown, an indication that the bounding box 240 surrounds the resource 202 and a depth value (e.g., 2 ft., etc.) describing the distance from the sensing module 110 to the surface of the resource 202 may be displayed. The analysis module 112 may determine the number of resources 202 based on a match between the indicated depth value and a depth value associated with the number of resources 202 indicated by the resource information stored by the storage module 108.

[0027] According to some aspects, analysis module 112 can use object recognition and / or object tracking to determine the orientation of resources placed on a pallet (e.g., pallet 208, etc.) to ensure proper loading of the pallet and avoid damage to resources (e.g., resources 202 and 204, etc.) and / or entities (entity 206, etc.). According to some aspects, as Figure 2BAs shown, resources such as resource 260 (or resource 202) can be placed on pallet 208 without optimizing positioning. For example, analysis module 112 can detect that the shape of resource 260 extends outside bounding box 240. The extension of resource 260 outside bounding box 240 may cause the resources placed / packed on pallet 208 to be unbalanced, unstable, or not in an optimized spatial arrangement. Analysis module 112 can detect any misaligned and / or misplaced resources, such as resource 260 extending outside bounding box 240, and generate a notification to notify entity 206. The notification can cause entity 206 to repack, replace, redirect, and / or optimize the packaging of resources 260 placed on pallet 208.

[0028] Figure 3 is an example system 300 for training the analysis module 112 for multi-sensor awareness for resource tracking and quantification according to some embodiments. Figure 1 To Figure 2 Description Figure 3 According to some aspects, the analysis module 112 can be trained to automatically detect, track, and count resources placed on a pallet. The analysis module 112 can be trained to identify the difference between resource information and the resources placed on the pallet (e.g., Figure 2A pallet 208, etc.) and / or any other resource at a predefined location (e.g. Figure 2A System 300 may use machine learning techniques to train at least one machine learning based classifier 330 (e.g., a software model, a neural network classification layer, etc.) that is configured to classify the data from the sensor data (e.g., such as from the Figure 1 The machine learning based classifier 330 can classify features extracted from the sensor data (e.g., sensor data received by the sensing module 110). The machine learning based classifier 330 can be trained to classify features extracted from the sensor data, for example, based on an analysis of one or more training data sets 310A-310N. The machine learning based classifier 330 can classify features extracted from the sensor data to identify resources and determine information about the resources, such as resource type, placement location, optimal stacking orientation, misaligned resources when stacked and / or oriented together, resource counts, etc.

[0029] One or more training data sets 310A-310N may include labeled baseline data, such as labeled resource types (e.g., resources of various shapes, bottles, cans, bowls, boxes, etc.) and / or labeled pallet packaging scenarios (e.g., optimized stacking based on resource shape / design, resource placement location, optimal stacking orientation, misaligned resources when stacked and / or oriented together, etc.). The labeled baseline data may include any number of feature sets (labeled data that identifies features extracted from sensor data, etc.).

[0030] The baseline data of the mark can be stored in one or more databases. The data (for example, sensor data, resource information, etc.) of the multi-sensor perception for resource tracking and quantification can be randomly assigned to the training data set or the test data set. According to some aspects, the data may not be completely random to be assigned to the training data set or the test data set. In this case, one or more criteria can be used during the assignment, such as ensuring that similar resource types, similar pallet packaging scenarios, different resource types, different pallet packaging scenarios, etc. can be used in each of the training data set and the test data set. Generally speaking, any suitable method can be used to assign data to the training data set or the test data set.

[0031] The analysis module 112 can train the machine learning based classifier 330 by extracting a feature set from the labeled baseline data according to one or more feature selection techniques. According to some aspects, the analysis module 112 can also define a feature set obtained from the labeled baseline data by applying one or more feature selection techniques to the labeled baseline data in one or more training data sets 310A-310N. The analysis module 112 can extract feature sets from the training data sets 310A-310N in various ways. The analysis module 112 can perform feature extraction multiple times, using different feature extraction techniques each time. In some cases, the feature sets generated using different techniques can each be used to generate different machine learning based classification models 340. According to some aspects, the feature set with the highest quality metric can be selected for training. The analysis module 112 can use the feature set to build one or more machine learning based classification models 340A-340N, which are configured to determine and / or predict resource types, pallet packaging scenarios, etc.

[0032] According to some aspects, the training data sets 310A-310N and / or the labeled baseline data can be analyzed to determine any dependencies, associations, and / or correlations between resource types, pallet packaging scenarios, and / or the like in the training data sets 310A-310N and / or the labeled baseline data. As used herein, the term "feature" can refer to any characteristic of a data resource that can be used to determine whether the data resource falls into one or more specific categories. For example, the features described herein can include resource types, pallet packaging scenarios, and / or any other characteristics.

[0033] According to some aspects, feature selection techniques may include one or more feature selection rules. One or more feature selection rules may include determining which features in the tagged baseline data appear in the tagged baseline data more than a threshold number of times and identifying those features that meet the threshold as candidate features. For example, any feature that appears greater than or equal to 2 times in the tagged baseline data may be considered a candidate feature. Any feature that appears less than 2 times may not be considered as a feature. According to some aspects, a single feature selection rule may be applied to select features or multiple feature selection rules may be applied to select features. According to some aspects, feature selection rules may be applied in a cascaded manner, wherein the feature selection rules are applied in a specific order and applied to the results of previous rules. For example, feature selection rules may be applied to the tagged baseline data to generate multi-sensor perception information that can be used for resource tracking and quantification (e.g., an indication of resource type, an indication of a pallet packaging scenario, etc.). The final list of candidate features may be analyzed based on additional features.

[0034] According to some aspects, the analysis module 112 may generate multi-sensor sensed information (e.g., an indication of resource type, an indication of pallet packaging scenario, etc.) that can be used for resource tracking, and the quantization operation may be based on a wrapper method. The wrapper method may be configured to use a feature subset and use the feature subset to train a machine learning model. Based on inferences drawn from a previous model, features may be added and / or deleted from the subset. The wrapper method includes, for example, forward feature selection, reverse feature elimination, recursive feature elimination, combinations thereof, and the like. According to some aspects, forward feature selection may be used to identify one or more candidate resource types, pallet packaging scenarios, and the like. Forward feature selection is an iterative method that does not start with features in a machine learning model. In each iteration, the features that best improve the model are added until the addition of new variables does not improve the performance of the machine learning model. According to some aspects, reverse elimination may be used to identify one or more candidate resource types, pallet packaging scenarios, and the like. Reverse elimination is an iterative method that starts with all features in a machine learning model. In each iteration, the least important features are removed until no improvement is observed in the removal of features. According to some aspects, recursive feature elimination can be used to identify one or more candidate resource types, pallet packaging scenarios, and the like. Recursive feature elimination is a greedy optimization algorithm that aims to find the best performing feature subset. Recursive feature elimination repeatedly creates models and retains the best or worst performing features at each iteration. Recursive feature elimination constructs the next model with the remaining feature window until all features are exhausted. Recursive feature elimination then ranks the features based on their elimination order.

[0035] According to some aspects, one or more candidate resource types, pallet packaging scenarios, etc. can be determined according to an embedded method. The embedded method combines the qualities of the filter method and the wrapper method. The embedded method includes, for example, the least absolute shrinkage and selection operator (LASSO) and ridge regression that implement a penalty function to reduce overfitting. For example, LASSO regression performs L1 regularization, which adds a penalty equal to the absolute value of the coefficient size, and ridge regression performs L2 regularization, which adds a penalty equal to the square of the coefficient size.

[0036] According to some aspects, one or more candidate resource types, pallet packaging scenarios, etc. can be determined according to an ensemble approach. The ensemble approach can combine outputs from separate classification models to produce a final decision regarding the candidate resource type, pallet packaging scenario, etc. According to some aspects, the ensemble approach can include thresholding logic that evaluates each classification from a classification model to weight the impact of the classification model on the final classification result.

[0037] After the analysis module 112 generates the feature set, the analysis module 112 may generate a machine learning-based prediction model 340 based on the feature set. The machine learning-based prediction model may refer to a complex mathematical model for data classification generated using machine learning techniques. For example, the machine learning-based classifier may include a mapping of support vectors representing boundary features. For example, the boundary feature may be selected from the feature set and / or represent the highest-ranking feature in the feature set.

[0038] According to some aspects, the analysis module 112 can use the feature sets extracted from the training data sets 310A-310N and / or the labeled baseline data to build machine learning-based classification models 340A-340N to determine and / or predict resource types, pallet packaging scenarios, etc. According to some aspects, the machine learning-based classification models 340A-340N can be combined into a single machine learning-based classification model 340. Similarly, the machine learning-based classifier 330 can represent a single classifier containing a single or multiple machine learning-based classification models 340 and / or multiple classifiers containing a single or multiple machine learning-based classification models 340. According to some aspects, the machine learning-based classifier 330 can also include each training data set in the training data sets 310A-310N and / or each feature set extracted from the training data sets 310A-310N and / or extracted from the labeled baseline data. Although shown separately, the analysis module 112 can include the machine learning-based classifier 330.

[0039] Features extracted from the imaging data may be combined in a classification model trained using a machine learning approach such as discriminant analysis; decision trees; nearest neighbor (NN) algorithms (e.g., k-NN models, replicator NN models, etc.); statistical algorithms (e.g., Bayesian networks, etc.); clustering algorithms (e.g., k-means, mean shift, etc.); neural networks (e.g., reservoir networks, artificial neural networks, etc.); support vector machines (SVMs); logistic regression algorithms; linear regression algorithms; Markov models or chains; principal component analysis (PCA) (e.g., for linear models); multilayer perceptron (MLP) ANN (e.g., for nonlinear models); replicated reservoir networks (e.g., for nonlinear models, typically for time series); random forest classification; combinations thereof, etc. The resulting machine learning-based classifier 330 may include decision rules or mappings that use sensor data to determine and / or predict resource types, pallet packaging scenarios, etc. The resulting machine learning-based classifier 330 may include a family of compound scaled object detection models trained using imaging data, and may include simple functions for test-time enhancement (TTA), model integration, hyperparameter evolution, and data output.

[0040] Sensor data and machine learning-based classifier 330 can be used to determine and / or predict resource types, pallet packaging scenarios, etc. of test samples in the test data set. For example, the results of each test sample can include a confidence level corresponding to the possibility or probability that the corresponding test sample accurately determines and / or predicts the resource type, pallet packaging scenario, etc. The confidence level can be a value between zero and one, which represents the possibility that the determined / predicted resource type, pallet packaging scenario, etc. are consistent with the calculated value. Multiple confidence levels can be provided for each test sample and each candidate (approximate) resource type, pallet packaging scenario, etc. The highest performance candidate resource type, pallet packaging scenario, etc. can be determined by comparing the results obtained for each test sample with the resource type, pallet packaging scenario, etc. calculated for each test sample. Generally speaking, the highest performance candidate resource type, pallet packaging scenario, etc. will have a result that closely matches the calculated resource type, pallet packaging scenario, etc. The highest performance candidate resource type, pallet packaging scenario, etc. can be used for multi-sensor perception of resource tracking and quantification operations.

[0041] Figure 4 is a flow chart illustrating an example training method 400 for generating a machine learning classifier 330 using an analysis module 112, according to some aspects. The analysis module 112 can implement a supervised, unsupervised, and / or semi-supervised (eg, boosting-based) machine learning-based classification model 340. Figure 4The method 400 shown is an example of a supervised learning method; variations of this example of a training method are discussed below, however, other training methods may be similarly implemented to train unsupervised and / or semi-supervised machine learning (prediction) models. The method 400 may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps may be required to perform the disclosure provided herein. In addition, as will be appreciated by those of ordinary skill in the art, some of these steps may be performed simultaneously, or in conjunction with one another. Figure 4 The order shown may be different from the order in which it is performed.

[0042] Reference Figures 1 to 3 Method 400 is described. However, method 400 is not limited in all aspects to those figures.

[0043] At 410 , the analysis module 112 determines (eg, accesses, receives, retrieves, etc.) sensor data and / or resource information. The sensor data and / or resource information may include one or more data sets, each data set associated with a resource type, a pallet packaging scenario, and the like.

[0044] In 420, analysis module 112 generates training data set and test data set. According to some aspects, training data set and test data set can be generated by indicating resource type, pallet packaging scene, etc. According to some aspects, training data set and test data set can be generated by randomly assigning resource type, pallet packaging scene, etc. to training data set or test data set. According to some aspects, sensor data and / or resource information can be assigned as training sample or test sample and may not be completely random. According to some aspects, only the baseline data of the mark of the specific feature extracted from specific sensor data (for example, optimized pallet packaging scene, etc.) can be used to generate training data set and test data set. According to some aspects, the baseline data of most of the marks extracted from sensor data and / or resource information can be used to generate training data set. For example, 75% of the baseline data of the mark for determining resource type, pallet packaging scene, etc. extracted from imaging data can be used to generate training data set, and 25% can be used to generate test data set. Any method or technology can be used to create training data set and test data set.

[0045] At 430, the analysis module 112 determines (e.g., extracts, selects, etc.) one or more features that can be used, for example, by a classifier (e.g., a software model, a classification layer of a neural network, etc.) to label features extracted from various sensor data and / or resource information. The one or more features may include an indication of a resource type, a pallet packaging scenario, etc. According to some aspects, the analysis module 112 may determine a set of training baseline features from a training data set. Features of the sensor data and / or resource information may be determined by any method.

[0046] In 440, the analysis module 112 uses one or more features to train one or more machine learning models, for example. According to some aspects, supervised learning can be used to train the machine learning model. According to some aspects, other machine learning techniques can be used, including unsupervised learning and semi-supervised learning. The machine learning model trained in 440 can be selected based on different criteria (e.g., how close the predicted resource type, pallet packaging scenario, etc. are to the actual resource type, pallet packaging scenario, etc.) and / or the data available in the training data set. For example, a machine learning classifier may suffer from different degrees of deviation. According to some aspects, more than one machine learning model can be trained.

[0047] At 450, the analysis module 112 optimizes, improves and / or cross-validates the trained machine learning model. For example, the data used for the training dataset and / or the test dataset can be updated and / or modified to include more labeled data indicating different resource types, pallet packaging scenarios, etc.

[0048] At 460, the analysis module 112 selects one or more machine learning models to build a predictive model (eg, a machine learning classifier, a prediction engine, etc.) The predictive model may be evaluated using a test data set.

[0049] At 470 , the analysis module 112 executes the predictive model to analyze the test data set and generate classification values ​​and / or prediction values.

[0050] In 480, analysis module 112 evaluates the classification value and / or the predicted value output by the prediction model to determine whether such value has achieved the expected accuracy level.Can be based on a plurality of true positives, false positives, true negatives and / or false negatives classification of a plurality of data points indicated by the prediction model to evaluate the performance of the prediction model in a variety of ways.For example, the false positive of the prediction model can refer to the number of times that the prediction model incorrectly predicts and / or determines the type of resource, pallet packaging scene, etc.On the contrary, the false negative of the prediction model can refer to the number of times that the machine learning model incorrectly predicts and / or determines the type of resource, pallet packaging scene, etc., and in fact, the predicted and / or determined type of resource, pallet packaging scene, etc. are matched with the actual type of resource, pallet packaging scene, etc.True negative and true positive can refer to the number of times that the prediction model correctly predicts and / or determines the type of resource, pallet packaging scene, etc.Related to these measurements is the concept of recall and precision.Usually, recall refers to the ratio of the sum of true positive and true positive and false negative, and this ratio quantifies the sensitivity of the prediction model. Similarly, precision is the ratio of true positives to the sum of true positives and false positives.

[0051] At 490, the analysis module 112 outputs the prediction model (and / or the output of the prediction model). For example, the analysis module 112 may output the prediction model when such a desired level of accuracy is achieved. The output of the prediction model may end the training phase.

[0052] According to some aspects, when a desired level of accuracy is not achieved, at 490 , analysis module 112 can perform a subsequent iteration of training method 400 initiated at 410 , with variations, such as, for example, considering a larger set of sensor and / or resource information.

[0053] Figure 5 A flow chart of an example method 500 for multi-sensor sensing for resource tracking and quantification according to some aspects is shown. The method 500 may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps may be required to perform the disclosure provided herein. In addition, as will be understood by a person of ordinary skill in the art, some of these steps may be performed simultaneously or in parallel. Figure 5 The order shown may be different from the order in which it is performed.

[0054] Reference Figures 1 to 4 Method 500 is described. However, method 500 is not limited in aspects of these figures. A computer-based system (eg, system 100, etc.) can facilitate multi-sensor perception for resource tracking and quantification.

[0055] At 510, computing device 102 receives resource information indicating an identifier, a location of a resource, and a quantity of a resource. The resource information may include, for example, purchase order information, location mapping information, and the like. For example, according to some aspects, computing device 102 may be loaded and / or configured with purchase order information indicating an identifier (e.g., SKU, UPC, unique identifier, etc.), location (e.g., warehouse inventory location, shelf, aisle, bin, etc.), and / or quantity (e.g., purchase order quantity, inventory quantity, requested quantity, etc.) of a resource and / or multiple resources (e.g., beverages, inventory / stock resources, products, etc.), location, location mapping information, and the like. According to some aspects, computing device 102 may receive purchase order information indicating an identifier, location, and / or quantity of a resource and / or multiple resources, location mapping information, and the like from a device (e.g., a cloud-based device, a service provider device, a computing device, etc.), a network (e.g., a private network, a public network, the Internet, etc.), a system (a warehouse management system, etc.), and the like. Computing device 102 may receive resource information by any method or technique.

[0056] In 520, computing device 102 tracks resources removed from resource locations and placed at predefined locations at resource locations. For example, computing device 102 may track resources removed from resource locations and placed at predefined locations based on sensor data and resource identifiers received from a first sensing device. According to some aspects, the predefined location may be on a pallet. For example, a resource may be a box, package, or the like of a beverage (or any other product), and computing device 102 may track the resource when the resource is removed from any location in a warehouse and placed on a pallet. According to some aspects, a pallet may be placed on and / or attached to a mobile unit (e.g., a pallet truck, a forklift, a stock truck, etc.). According to some aspects, a sensing device may include one or more of a camera (e.g., a stereo camera, a high-definition camera, a video camera, a smart camera, a PoE camera, etc.), a LIDAR sensor, an infrared sensor, a position detection sensor, and the like.

[0057] According to some aspects, computing device 102 tracks resources removed from resource locations and placed at predefined locations by determining that the type of the resource corresponds to the type of the resource indicated by the corresponding identifier. For example, computing device 102 may track resources removed from resource locations and placed at predefined locations by determining that the type of the resource corresponds to the type of the resource indicated by the resource identifier based on object recognition performed by a trained machine learning model. The trained machine learning model may identify the type of resource based on a correspondence between feature classification data and features extracted from sensor data received from a first sensing device. The trained machine learning model may identify the resource based on a correspondence between the type of the resource and the type of the resource indicated by the resource identifier.

[0058] The computing device 102 determines that the resource is placed in the area of ​​the predefined location, for example, based on object tracking applied to the sensor data received from the first sensing device. For example, the trained machine learning model can determine that the resource is placed in the area of ​​the predefined location based at least on a difference between a location of the resource indicated in a first image extracted from the sensor data received from the first sensing device and a location of the resource indicated in a second image extracted from the sensor data received from the first sensing device.

[0059] According to some aspects, computing device 102 causes an indication of resources placed in a corresponding area of ​​a predefined location to be displayed via a user interface. The indication of resources placed in a corresponding area of ​​a predefined location may include, for example, a value associated with the resource, an image of the resource, a graphical representation of the resource, etc.

[0060] In 530, computing device 102 determines the number of resources removed from the resource location and placed at the predefined location. For example, computing device 102 may determine the number of resources among the plurality of resources removed from the resource location and placed at the predefined location based on depth information indicated by sensor data received from the second sensing device.

[0061] According to some aspects, computing device 102 determines the number of resources removed from resource locations and placed at predefined locations by causing a display of indications of resources placed in an area of ​​predefined locations associated with a resource type of the resources. For each resource removed from the corresponding location, computing device 102 determines the number of resources based on a match between a corresponding depth value indicated by the depth information and a depth value associated with the number of the corresponding resource type.

[0062] In 540, computing device 102 determines a match between the number of resources and the number of resources removed from the resource location and placed at the predefined location. For example, computing device 102 may access a storage storing the number of resources and determine that the value indicated by the number of resources matches / corresponds to the number of resources removed from the resource location and placed at the predefined location.

[0063] In 550, the computing device 102 generates a notification indicating that the additional resources matching the resource should stop being removed from the resource location and placed at the predefined location. For example, the computing device 102 generates a notification based on the matching between the number of resources and the number of resources removed from the resource location and placed at the predefined location, indicating that the additional resources matching the resource should stop being removed from the resource location and placed at the predefined location. The notification may include a sound notification (e.g., text to speech, recorded / computerized voice, beep, alarm, etc.), a text notification (e.g., a text message sent to a mobile device of a user, a product / resource picker, a warehouse worker, an entity, etc.), a visual notification (e.g., a notification displayed on a user interface, etc.), etc. The notification may cause the additional resources matching the resource to stop being removed from the resource location and placed at the predefined location by the entity, for example. According to some aspects, the notification may be an instruction to instruct the entity to stop removing the additional resources matching the resource from the resource location and placing them at the predefined location. According to some aspects, the notification may indicate in real time the number of additional resources remaining, for example, at the resource location, in the inventory, etc.

[0064] According to some aspects, the notification may include one or more control signals that cause an entity such as an autonomous robot / device or the like to stop removing additional resources that match the resource from the resource location and place them at the predefined location.

[0065] According to some aspects, method 500 may also include computing device 102 generating, for a resource in the plurality of resources, a notification indicating the mismatch based on a mismatch between a corresponding quantity of the resource and a quantity of the resource removed from the corresponding location and placed at the predefined location. The notification may include an audible notification (e.g., text-to-speech, a recorded / computerized voice, a beep, an alarm, etc.), a text notification (e.g., a text message sent to a mobile device of a user, a product / resource picker, a warehouse worker, an entity, etc.), a visual notification (e.g., a notification displayed on a user interface, etc.), etc.

[0066] Figure 6 is an example computer system that can be used to implement various embodiments. For example, one or more well-known computer systems (such as Figure 6 The computer system 600 shown in the figure can be used to implement various embodiments. One or more computer systems 600 can be used, for example, to implement any of the embodiments discussed herein and their combinations and sub-combinations. According to some aspects, Figure 1 The computing device 102 (and / or any other device / component described herein) may be implemented using the computer system 600. According to some aspects, the computer system 600 may be used to implement the methods 400 and 500.

[0067] Computer system 600 may include one or more processors (also referred to as central processing units or CPUs), such as processor 604. Processor 604 may be connected to a communications infrastructure or bus 606.

[0068] The computer system 600 may also include user input / output devices 602 , such as a monitor, keyboard, pointing device, etc., which may communicate with the communication infrastructure 606 through a user input / output interface 602 .

[0069] One or more of processors 604 may be a graphics processing unit (GPU). In an embodiment, a GPU may be a processor of a dedicated electronic circuit designed to process mathematically intensive applications. A GPU may have a parallel structure that efficiently processes large blocks of data in parallel, such as mathematically intensive data commonly used in computer graphics applications, images, videos, etc.

[0070] The computer system 600 may also include a main memory or primary storage 608, such as a random access memory (RAM). The main memory 608 may include one or more levels of cache. The main memory 608 may store control logic (ie, computer software) and / or data.

[0071] The computer system 600 may also include one or more secondary storage devices or memories 610. The secondary storage 610 may include, for example, a hard disk drive 612 and / or a removable storage device or drive 614. The removable storage drive 614 may be a floppy disk drive, a magnetic tape drive, an optical disk drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0072] The removable storage drive 614 can interact with a removable storage unit 618. The removable storage unit 618 may include a computer usable or readable storage device having computer software (control logic) and / or data stored thereon. The removable storage unit 618 may be a floppy disk, a magnetic tape, an optical disk, a DVD, an optical storage disk, and / or any other computer data storage device. The removable storage drive 614 can read from and / or write to the removable storage unit 618.

[0073] Secondary memory 610 may include other means, devices, components, tools, and / or other methods for allowing computer programs and / or other instructions and / or data to be accessed by computer system 600. Such means, devices, components, tools, and / or other methods may include, for example, a removable storage unit 622 and an interface 620. Examples of removable storage unit 622 and interface 620 may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface.

[0074] The computer system 600 may also include a communication or network interface 624. The communication interface 624 may enable the computer system 600 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively represented by reference numeral 628). For example, the communication interface 624 may allow the computer system 600 to communicate with an external or remote device 628 via a communication path 626, which may be wired and / or wireless (or a combination thereof) and may include any combination of a LAN, a WAN, the Internet, etc. Control logic and / or data may be transmitted to and from the computer system 600 via the communication path 626.

[0075] The computer system 600 may also be any of a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet computer, a smart phone, a smart watch or other wearable device, an appliance, part of the Internet of Things, and / or an embedded system, to name a few non-limiting examples, or any combination thereof.

[0076] Computer system 600 can be a client or server accessing or hosting any application and / or data through any delivery paradigm, which delivery paradigm includes but is not limited to remote or distributed cloud computing solutions; local or on-premises software ("on-premises" cloud-based solutions); "as a service" models (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (SaaS), management software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (IaaS), etc.); and / or a hybrid model including any combination of the foregoing examples or other services or delivery paradigms.

[0077] Any applicable data structures, file formats, and schemes in the computer system 600 may be derived from standards including, but not limited to, JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations, alone or in combination. Alternatively, proprietary data structures, formats, and / or schemes may be used exclusively or in combination with known or open standards.

[0078] In some embodiments, a tangible non-transitory device or article of manufacture, which may also be referred to herein as a computer program product or program storage device, includes a tangible non-transitory computer usable or readable medium having control logic (software) stored thereon. This includes, but is not limited to, computer system 600, main memory 608, auxiliary memory 610, and removable storage units 618 and 622, and tangible articles of manufacture embodying any combination of the foregoing. When executed by one or more data processing devices (such as computer system 600), such control logic may cause such data processing devices to operate as described herein.

[0079] Based on the teachings contained in this disclosure, it will be apparent to those skilled in the relevant art how to use the Figure 6 The embodiments of the present disclosure may be made and used with data processing devices, computer systems, and / or computer architectures other than those shown. Specifically, the embodiments may operate together with software, hardware, and / or operating system implementations other than those described herein.

[0080] It should be understood that the detailed description section, rather than any other section, is intended to be used to interpret the claims. Any section may present one or more but not all exemplary embodiments contemplated by the inventors, and is therefore not intended to limit the present invention and the appended claims in any way.

[0081] Additionally and / or alternatively, although the present disclosure describes exemplary embodiments of exemplary fields and applications, it should be understood that the present disclosure is not limited thereto. Other embodiments and modifications thereof are possible and are within the scope and spirit of the present disclosure. For example, and without limiting the generality of this paragraph, the embodiments are not limited to the software, hardware, firmware, and / or entities shown in the drawings and / or described herein. In addition, the embodiments (whether or not explicitly described herein) have significant utility for fields and applications outside the examples described herein.

[0082] One or more parts of the above-mentioned specific implementations may include software. Software is a general term for the meaning of its characteristic functions and their relationships. For the convenience of description, the boundaries of these functional building blocks are arbitrarily defined herein. As long as the specific functions and relationships (or their equivalents) can be properly performed, alternative boundaries can be defined. In addition, alternative embodiments can use an order different from the order described herein to perform functional blocks, steps, operations, methods, etc.

[0083] "One embodiment", "embodiment", "exemplary embodiment" or similar phrases mentioned herein indicate that the described embodiments may include specific features, structures or characteristics, but each embodiment may not necessarily include specific features, structures or characteristics. Moreover, such phrases do not necessarily refer to the same embodiment. In addition, when describing specific features, structures or characteristics in conjunction with the embodiments, whether or not explicitly mentioned or described herein, one or more technicians in the relevant fields are able to incorporate such features, structures or characteristics into other embodiments. In addition, some embodiments may be described using the expressions "connect" and "connect" and their derivatives. These terms are not necessarily intended to be synonyms for each other. For example, the terms "connect" and / or "connect" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "connect" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0084] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A computer-implemented method for multi-sensor sensing of resource tracking and quantification, the method comprising: receiving resource information indicating a resource identifier, a resource location, and a resource quantity; tracking, at the resource location, a resource that is removed from the resource location and placed at a predefined location based on sensor data received from a first sensing device and the resource identifier; determining, based on depth information indicated by sensor data received from a second sensing device, a quantity of the resource removed from the resource location and placed at the predefined location; determining a match between the quantity of the resource and the quantity of the resource removed from the resource location and placed at the predefined location; as well as Based on the match between the number of resources and the number of resources removed from the resource locations and placed at the predefined locations, a notification is generated indicating that additional resources matching the resources should stop being removed from the resource locations and placed at the predefined locations.

2. The method of claim 1, wherein tracking the resource removed from the resource location and placed at the predefined location further comprises: identifying a type of the resource based on a correspondence between feature classification data and features extracted from the sensor data received from the first sensing device; identifying the resource based on a correspondence between the type of the resource and the type of the resource indicated by the resource identifier; determining that the resource is placed in an area of ​​the predefined location based at least on a difference between a location of the resource indicated in a first image extracted from the sensor data received from the first sensing device and a location of the resource indicated in a second image extracted from the sensor data received from the first sensing device; as well as An indication of the resource placed in the area of ​​the predefined location is caused to be displayed. 3 . The method of claim 2 , wherein the indication of the resource placed in the area of ​​the predefined location comprises at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource.

4. The method of claim 1 , wherein determining the amount of the resource removed from the resource location and placed at the predefined location further comprises: causing display of an indication of the resource placed in an area of ​​the predefined location associated with a resource type of the resource; as well as Based on a match between a depth value indicated by the depth information and a depth value associated with the quantity of the resource type, the quantity of the resource is determined.

5. The method of claim 1, wherein the notification causing additional resources matching the resource to cease to be removed from the resource location and placed at the predefined location comprises at least one of an audible notification, a textual notification, or a visual notification.

6. The method according to claim 1, further comprising: Based on a mismatch between the number of resources and the number of resources removed from the resource location and placed at the predefined location, a notification is generated indicating the mismatch. 7 . The method of claim 6 , wherein the notification indicating the mismatch comprises at least one of an audible notification, a textual notification, or a visual notification.

8. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for multi-sensor sensing to track the location and amount of a resource, the operations comprising: receiving resource information indicating a resource identifier, a resource location, and a resource quantity; tracking, at the resource location, a resource that is removed from the resource location and placed at a predefined location based on sensor data received from a first sensing device and the resource identifier; determining, based on depth information indicated by sensor data received from a second sensing device, a quantity of the resource removed from the resource location and placed at the predefined location; determining a match between the quantity of the resource and the quantity of the resource removed from the resource location and placed at the predefined location; as well as Based on the match between the number of resources and the number of resources removed from the resource locations and placed at the predefined locations, a notification is generated indicating that additional resources matching the resources should stop being removed from the resource locations and placed at the predefined locations.

9. The non-transitory computer readable medium of claim 8, wherein tracking the resource removed from the resource location and placed at the predefined location further comprises: identifying a type of the resource based on a correspondence between feature classification data and features extracted from the sensor data received from the first sensing device; identifying the resource based on a correspondence between the type of the resource and the type of the resource indicated by the resource identifier; determining that the resource is placed in an area of ​​the predefined location based at least on a difference between a location of the resource indicated in a first image extracted from the sensor data received from the first sensing device and a location of the resource indicated in a second image extracted from the sensor data received from the first sensing device; as well as An indication of the resource placed in the area of ​​the predefined location is caused to be displayed.

10. The non-transitory computer-readable medium of claim 9, wherein the indication of the resource placed in the area of ​​the predefined location comprises at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource.

11. The non-transitory computer readable medium of claim 8, wherein determining the number of the resources removed from the resource location and placed at the predefined location further comprises: causing display of an indication of the resource placed in an area of ​​the predefined location associated with a resource type of the resource; as well as Based on a match between a depth value indicated by the depth information and a depth value associated with the quantity of the resource type, the quantity of the resource is determined.

12. The non-transitory computer-readable medium of claim 8, wherein the notification that causes additional resources matching the resource to cease to be removed from the resource location and placed at the predefined location comprises at least one of an audible notification, a textual notification, or a visual notification.

13. The non-transitory computer readable medium of claim 8, further comprising: Based on a mismatch between the number of resources and the number of resources removed from the resource location and placed at the predefined location, a notification is generated indicating the mismatch.

14. The non-transitory computer-readable medium of claim 13, wherein the notification indicating the mismatch comprises at least one of an audible notification, a textual notification, or a visual notification.

15. A system, comprising: Memory; and At least one processor coupled to the memory and configured to perform operations for multi-sensor sensing to track the location and amount of resources, the operations comprising: receiving resource information indicating a resource identifier, a resource location, and a resource quantity; tracking, at the resource location, a resource that is removed from the resource location and placed at a predefined location based on sensor data received from a first sensing device and the resource identifier; determining, based on depth information indicated by sensor data received from a second sensing device, a quantity of the resource removed from the resource location and placed at the predefined location; determining a match between the quantity of the resource and the quantity of the resource removed from the resource location and placed at the predefined location; and Based on the match between the number of resources and the number of resources removed from the resource locations and placed at the predefined locations, a notification is generated indicating that additional resources matching the resources should stop being removed from the resource locations and placed at the predefined locations.

16. The system of claim 15, wherein tracking the resource removed from the resource location and placed at the predefined location further comprises: identifying a type of the resource based on a correspondence between feature classification data and features extracted from the sensor data received from the first sensing device; identifying the resource based on a correspondence between the type of the resource and the type of the resource indicated by the resource identifier; determining that the resource is placed in an area of ​​the predefined location based at least on a difference between a location of the resource indicated in a first image extracted from the sensor data received from the first sensing device and a location of the resource indicated in a second image extracted from the sensor data received from the first sensing device; as well as An indication of the resource placed in the area of ​​the predefined location is caused to be displayed.

17. The system of claim 16, wherein the indication of the resource placed in the area of ​​the predefined location comprises at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource.

18. The system of claim 15, wherein determining the amount of the resource removed from the resource location and placed at the predefined location further comprises: causing display of an indication of the resource placed in an area of ​​the predefined location associated with a resource type of the resource; as well as Based on a match between a depth value indicated by the depth information and a depth value associated with the quantity of the resource type, the quantity of the resource is determined.

19. The system of claim 15, wherein the notification that causes additional resources matching the resource to cease to be removed from the resource location and placed at the predefined location comprises at least one of an audible notification, a textual notification, or a visual notification.

20. The system of claim 15, the operations further comprising: Based on a mismatch between the number of resources and the number of resources removed from the resource location and placed at the predefined location, a notification is generated indicating the mismatch.

Images