Methods and Apparatus for Machine Learning Systems for Edge Computer Vision and Active Reality

Abstract

The method includes receiving an image frame of an inventory. The method includes locating control points used to determine a spatial search within the image frame and detecting a plurality of store units within the spatial search using a machine learning model. Each store unit is associated with a unit type from a plurality of unit types. The method includes calculating a number of store units from the plurality of store unit counts for each unit type from the plurality of unit types from the detected plurality of store units based on a depth analysis. Each store unit count includes a total number of store units associated with each store unit type. The method includes determining a replenishment status for each unit type based on the number of store units per unit type and automatically generating a replenishment request based on the replenishment status.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and benefit of U.S. Provisional Patent Application No. 63 / 411,341, entitled "Methods And Apparatus For Machine Learning System For Inventory Management Using Edge Computer Vision And Active Reality," filed September 29, 2022, which is incorporated by reference in its entirety.

[0002] The present disclosure relates generally to the fields of computer vision and machine learning. In particular, the present disclosure is directed to methods and apparatus related to machine learning systems for edge computer vision and active reality. [Background technology]

[0003] Inventory management relies heavily on human-based computation and action. Some known systems use cameras installed within warehouses or retail spaces to enable humans to efficiently manage item inventory on a large scale. Furthermore, some known systems require strict organization and a single appearance for each item, without distinguishing between packaging for items that share the same stock-keeping unit (SKU) or various storage methods for items. Furthermore, some known systems use fixed cameras with computer vision, which may require multiple cameras, which are limited to specific areas and fields of view and have multiple blind spots. Additionally, such known technologies often rely on manual data entry, which is cumbersome, time-consuming, and potentially introduces inaccuracies.

[0004] Therefore, there is a need for computer vision and machine learning systems for predictive and real-time inventory management. Summary of the Invention

[0005] In one or more embodiments, an apparatus for inventory management using edge computer vision and active reality includes a processor of a user device and memory operably coupled to the processor. The memory stores instructions for causing the processor to receive multiple image frames of inventory from a sensor operably coupled to the processor and identify control points used to determine a spatial search within the multiple image frames. The memory further stores instructions for causing the processor to detect multiple store units in the spatial search using a machine learning model. Each store unit from the multiple store units is associated with a unit type from a multiple unit types. The instructions include instructions for causing the processor to calculate a store unit count from the multiple store unit counts for each unit type from the multiple unit types from the multiple store units detected based on the depth analysis. Each store unit count includes a total number of store units associated with each store unit type. The instructions further include instructions for causing the processor to identify duplicate store units associated with the duplicate store units and remove the duplicate store units from the store unit counts associated with the duplicate store units. The instructions further store instructions for causing the processor to determine a replenishment status of each unit type from the plurality of unit types based on the number of stored units per unit, and automatically generate a replenishment request based on the replenishment status.

[0006] In one or more embodiments, a method includes receiving a plurality of image frames of an inventory from a sensor operably coupled to a processor of a user device. The method further includes locating control points used to determine the spatial search in the plurality of image frames. The method further includes detecting a plurality of store units in the spatial search with a machine learning model. Each store unit from the plurality of store units is associated with a unit type from a plurality of unit types. The method further includes calculating a store unit number from the plurality of store unit numbers for each unit type from the plurality of unit types from the detected plurality of store units based on a depth calculation. The method further includes generating a digital model from the plurality of digital models. Each digital model from the plurality of digital models is overlaid around a different store unit from the plurality of store units. The method further includes determining duplicate store units based on at least one overlap between areas surrounding one or more digital models. The method further includes updating the store unit numbers associated with the duplicate store units.

[0007] In one or more embodiments, a non-transitory processor-readable medium stores instructions that, when executed by a processor, cause the processor to receive, from a sensor, detection of a first control point for determining a first spatial search of a first inventory. The processor further causes the processor to detect the first plurality of store units to calculate a number of store units from the first plurality of store unit numbers using a machine learning model and based on a depth calculation of the first spatial search. Each store unit number from the first plurality of store unit numbers is associated with a unit type from the plurality of unit types. The processor further causes the processor to generate a digital model from the first plurality of digital models, the digital model being superimposed around each store unit from the first plurality of store units. The processor further causes the processor to store the first inventory data in a database such that the first plurality of digital models are hidden. The processor further causes the processor to receive, from the sensor, detection of a second control point for determining a second spatial search of a second inventory. The processor further causes the processor to detect the second plurality of store units to calculate a store unit number from the second plurality of store unit numbers using the machine learning model and based on the depth calculation of the second spatial search, where each store unit number from the second plurality of store unit numbers is associated with a unit type from the plurality of unit types. The processor further causes the processor to generate a digital model from the second plurality of digital models, the digital model being superimposed around each store unit from the second plurality of store units. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a user device for inventory control, according to one embodiment. [Figure 2] FIG. 1 is a block diagram of a system for inventory control using edge computer vision and active reality, according to one embodiment. [Figure 3]FIG. 1 is a block diagram of a machine learning system for edge computer vision and active reality, according to one embodiment. [Figure 4] FIG. 1 is an illustration of spatial searching using active reality, according to one embodiment. [Figure 5] FIG. 1 is a flow diagram of a method for inventory control using edge computer vision and active reality, according to one embodiment. [Figure 6] FIG. 1 is a flow diagram of a method for a machine learning system to determine overlap, according to one embodiment. [Figure 7] FIG. 1 is a flow diagram of a method for a machine learning system for edge computer vision and active reality, according to one embodiment. [Figure 8] 1 is an example screenshot of inventory with active reality icons, according to one embodiment. [Figure 9] 1 is an example screenshot of an inventory with an active reality digital model superimposed on a storage unit, according to one embodiment. [Figure 10] 1 is an example screenshot of an inventory with an active reality digital model superimposed on a storage unit, according to one embodiment. [Figure 11] 1 is an example screenshot of an inventory with an active reality digital model superimposed on the potential location of a storage unit, according to one embodiment. [Figure 12] 1 is an example screenshot of a menu list, according to one embodiment. [Figure 13] 1 is an example screenshot of an inventory with an active reality digital model superimposed on a storage unit, according to one embodiment. [Figure 14] 1 is an example screenshot of an inventory with an active reality digital model showing volume, according to one embodiment. [Figure 15] 1 is a screenshot of an inventory management dashboard, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] FIG. 1 is a block diagram of a system 100 for inventory management using edge computer vision and active reality, according to one embodiment. The system 100 includes a user device 101 and an inventory 116. The inventory 116 may include (or be) any physical storage location for multiple storage units, such as, for example, items, goods, merchandise, materials, and / or products. The inventory 116 may also include a warehouse, closet, freezer, retail space, and / or any location for storing items. The inventory 116 may include multiple stock keeping units (SKUs). An SKU may also be referred to as a "storage unit." An SKU may be a unique code including letters and / or numbers that identifies characteristics for each item and / or storage unit in the inventory 116, such as, for example, manufacturer, brand, style, color, size, type, and / or product. In some examples, the inventory 116 may include a storage unit identifier for each storage unit, such as, for example, a label, logo, and / or barcode. Inventory 116 may include multiple items of the same type (e.g., cans of the same type of coffee beans, bags of coffee powder, packages of straws, cups of the same size, etc.). Inventory 116 may include storage units of different storage types, which may be based on size and / or type of packaging. For example, storage types may include bottles, small items, large items, medium-sized boxes, large-sized boxes, large bags, and / or jars. Storage units may include any goods and / or materials, such as milk cartons, bags of coffee powder, cups, and / or boxes of potato chips. Inventory 116 may also include multiple storage units, and each storage unit may also be associated with a unit type. The unit type may include (or be) the name and / or product of the storage unit. In some implementations, multiple storage units may have the same unit type. For example, if a storage unit may be a "ketchup bottle," the unit type of that storage unit may be "ketchup" or "ketchup bottle."In some cases, inventory 116 may store storage units that are the same product or item (e.g., ketchup), and the common product, article, and / or material that the storage units share is a unit type. For example, a storage unit may have a storage type that is bottles and a unit type that is ketchup. In some implementations, the multiple storage units may include a subset of the multiple storage units. In such implementations, each storage unit may be associated with a respective unit type from the multiple unit types. For example, one subset of storage units may be associated with ketchup, and each storage unit in the subset associated with ketchup bottles is one ketchup bottle. In some examples, each storage unit in the subset of storage units is substantially identical and / or the same article / material.

[0010] The user device 101 may be a computing device including a processor 104 and memory 108 that communicate with each other and with other components via a bus (not shown). The bus may include any of several types of bus structures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of a variety of bus architectures. The user device 101 may include, for example, a computer workstation, a terminal computer, a server computer, a laptop computer, a mobile / handheld device (e.g., a tablet computer, a smartphone, a smartwatch, smart glasses, a headset, etc.), any device capable of executing a sequence of instructions that specify operations to be performed by such a device, and / or any combination thereof. The user device 101 may also include multiple computing devices and / or other user devices that can be used to implement a specially configured set of instructions to cause one or more devices to perform any one or more aspects and / or methodologies disclosed herein. The user device 101 may include a computer vision device, an active reality device, an augmented reality device, etc.

[0011] The user device 101 may include a network interface (not shown). A network interface device, such as a network interface, may be used to connect the user device 101 to one or more various networks and one or more remote devices connected thereto. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of networks may include a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, building, campus, or other geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider's data network and / or voice network), and / or a direct connection between two computing devices. The user device 101 may utilize wired and / or wireless modes of communication.

[0012] The user device 101 includes one or more sensors 112. The sensor(s) 112 may include, for example, a charge-coupled device (CCD), an active pixel sensor (APS), and / or a digital image sensor, such as any digital image sensor fabricated with metal-oxide semiconductor (MOS), complementary metal-oxide semiconductor (CMOS), N-type metal-oxide semiconductor (NMOS), Live MOS, etc. In some implementations, the sensor(s) 112 may include a depth sensor, such as, for example, a time-of-flight (TOF) sensor. The sensor(s) 112 may also include a camera, such as, for example, an ultra-wide-angle camera, a wide-angle camera, a telephoto camera, a monochrome camera, and / or a macro camera. The sensor(s) 112 may also include a light detection and ranging (LIDAR) sensor. The sensor(s) 112 may be used to scan and / or capture multiple image frames of the inventory 116 and storage units of the inventory 116. The sensor(s) 112 can capture and process image frames in substantially real time. A user can operate the user device 101 to control where the sensor(s) 112 are capturing and / or generating image frames. The sensor(s) 112 can also be used to capture the height of stacked storage units, such as cups, as described in further detail herein. In some cases, the sensor(s) can be configured to capture image frames in substantially real time at various locations and / or positions. For example, in some implementations, the sensor(s) 112 can be positioned to point at the inventory 116 at multiple different angles (and in multiple different spatial searches). In some cases, the user device 101 including the sensor(s) 112 can be mobile (rather than in a fixed location) so that the sensor(s) 112 can be configured to identify objects from image frames captured at various positions and / or angles.Although shown in FIG. 1 as being part of the user device 101, in some embodiments the sensor(s) 112 can be separate from the user device 101 but communicatively coupled to the user device 101.

[0013] The processor 104 can be or include, for example, a hardware-based integrated circuit (IC) or any other suitable processing device configured to run and / or execute a set of instructions or code. For example, the processor 104 can be a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic array (PLA), a complex programmable logic device (CPLD), and / or a programmable logic controller (PLC), etc. In some implementations, the processor 104 can be configured to run any of the methods and / or portions of methods described herein.

[0014] The memory 108 can store the machine learning model 124, the SKU training data 128, the control points 132, the storage unit number 136, the digital model 140, and the spatial label 144. The memory 108 can be or include, for example, a random access memory (RAM), a memory buffer, a hard drive, a read-only memory (ROM), and / or an erasable programmable read-only memory (EPROM). In some examples, the memory can store one or more software programs and / or code, which can include, for example, instructions that cause the processor 104 to perform one or more processes and / or functions. In some implementations, the memory 108 can include an expandable storage unit that can be used incrementally. In some implementations, the memory 108 can be a portable memory (e.g., a flash drive and / or a portable hard disk, etc.) that can be operably coupled to the processor 104. The memory 108 can include various components (e.g., machine-readable media), including, but not limited to, random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within user device 101, such as during start-up, may be stored in memory 108. Memory 108 may further include any number of program modules, including, for example, an operating system, one or more application programs, other program modules, program data, and / or any combination thereof.

[0015] The memory 108 may store instructions that cause the processor 104 to generate a digital model 140 of an object captured in an image frame via the sensor(s) 112 and store the digital model 140 in the memory 108. The digital model may include (or be) a real-time (or substantially real-time) virtual representation of a real-world physical object, such as, for example, inventory 116, a wall, a floor, a shelf, a rack, and / or a storage unit. The digital model 140 may be stored in the memory 108 and may also be presented on the display 148 in a form of active reality and / or augmented reality.

[0016] The memory 108 stores instructions that cause the processor 104 to capture multiple image frames via the sensor(s) 112 and detect multiple store units in the inventory 116. Representations of the image frames and the detected store units can be stored in the memory 108. The memory 108 further stores instructions that cause the processor 104 to detect, via the sensor(s) 112, a spatial search 120 that includes the inventory 116. The spatial search 120 can include (or be) a representation and / or virtual representation of a zone, boundary, border, and / or area within the space in which the inventory 116 is located. Similarly, the spatial search 120 can define a layout of a space that holds and / or contains the inventory 116. The processor 104 can store coordinates of the spatial search 120 in the memory 108 to recognize and / or predict where the inventory 116 is to be located and / or the area that will be searched to identify the inventory 116. In some implementations, the spatial search 120 can be static and serve as a point of interest in an image frame for the sensor(s) 112 to capture information about the inventory 116. Coordinates and / or parameters of the spatial search 120 can be stored in the memory 108, allowing the processor 204 to filter, ignore, and / or discard areas outside the spatial search 120 to reduce computational load. Similarly, in some implementations, content within the area of ​​the spatial search 120 can be processed while content outside the area of ​​the spatial search 120 can be discarded and / or ignored. In some implementations, the spatial search 120 can be manually modified by a user. In some implementations, the spatial search 120 can be automatically modified (e.g., if new inventory is identified outside the area of ​​the spatial search 120). In some implementations, the spatial search 120 can include a planogram for the inventory 116.

[0017] The control points 132 may include (or be) representations and / or placeholders for barcodes (e.g., quick response (QR) codes), predetermined identifiers, predetermined identification indicia, floors, ceilings, and / or walls, etc., that define the location of the storage unit within the inventory 116. The control points 132 may be static control points within an image frame that generally illustrates the location of the storage unit. For example, the static control points may remain in the same position in virtual space and may be used by the machine learning model 124 as reference points for generating a virtual representation of the inventory 116 and / or the area surrounding the inventory 116, including a virtual representation of the storage unit. The control points 132 may also include (or be) representations of physical objects that store, hold, contain, and / or maintain the storage unit, such as, for example, racks, where the control points 132 include representations of lower and upper shelves, dividers, columns, barriers, beams, and / or frames, etc. Similarly, control points 132 can serve as static representations of inventory-containing objects (e.g., racks, shelves, etc.). In some implementations, control points 132 can be manually set by a user (e.g., barcodes). In some implementations, control points 132 can also be temporary. In some implementations, control points 132 can also be digital models 140, where control points 132 are substantially real-time virtual representations of static objects such as, for example, walls, racks, shelves, floors, or ceilings. Multiple control points from multiple locations, inventories, and / or warehouses can be stored in memory 108.

[0018] In some implementations, control points 132 can be moved within the virtual space within spatial search 120 via user input on a touchscreen, which can also function as the display 148 of user device 101. For example, a user can tap on the touchscreen where a control point of interest is located and drag across the touchscreen to the desired location in the virtual space within spatial search 120 as shown on display 148. In another example, a user can point and tap on the touchscreen, with the tapped touchscreen location representing a virtual representation of the desired location for the control point to be set. A user can also use the touchscreen to remove control points and / or place new control points throughout the virtual space within spatial search 120 and / or inventory 116.

[0019] The machine learning model 124 may include, for example, a supervised machine learning model and / or an unsupervised machine learning model. The machine learning model 124 and / or the user device 101 may include and / or enable computer vision. The machine learning model 124 may include a convolutional neural network (CNN), a recurrent neural network (RNN), and / or any neural network or other suitable machine learning model. The machine learning model 124 may include an end-to-end machine learning model. An end-to-end machine learning model may learn the steps between an initial input stage (e.g., image frames) and an output result (e.g., identification of the digital model 140, identification of the store unit, store unit number 136, etc.). The machine learning model 124 may be trained to identify the store unit and / or calculate the store unit number 136. In some implementations, the machine learning model 124 may enable edge computer vision by processing image frames in proximity to the inventory 116.

[0020] The machine learning model 124 can be configured to use the image frames as input to detect multiple store units and / or calculate a store unit count from a plurality of store unit counts 136 of each unit type from a plurality of unit types. In some cases, the store unit count can include a total count for each store unit associated with the same unit type from a plurality of unit types. In some cases, the store unit count can include a total count for each store unit in a subset of store units associated with a unit type. The machine learning model 124 can be trained using SKU training data 128. In some implementations, the SKU training data 128 includes an augmented store unit digital model correlated with store unit identification information. The memory 108 can store instructions that cause the processor 104 to continuously, sporadically, and / or periodically provide data (e.g., SKU training data) to the machine learning model 124 to generate a robust and / or trained machine learning model. In some implementations, the machine learning model 124 can be synthetically trained. For example, the machine learning model 124 can be trained in virtual and real-world training environments using store units.

[0021] In some implementations, the sensor(s) 112 can identify and classify storage units having various shapes. For example, a bag of rice can be free-form based on how the bag is handled. In some implementations, the sensor(s) 112 can determine the identity of a storage unit by scanning an identifier (e.g., a label, a logo, etc.) on the storage unit. In some cases, the sensor(s) 112 can scan and determine the dimensions (e.g., height, width, length, etc.) of the storage unit without using LIDAR. The machine learning model 124 can determine the number of storage units 136 using data including the storage unit dimensions, the storage unit shape, and / or the storage unit label. Data including the storage unit dimensions can be included in SKU training data 128 for training the machine learning model 124. In some cases, the sensor(s) 112 can also scan the storage unit to determine the fill level of a container (e.g., a can, a bottle, a box, etc.), as described in further detail herein.

[0022] The augmented storehouse unit digital model can include a virtual and / or real-time representation, model, shape, and / or layout of the storehouse unit that the machine learning model 124 can be trained to recognize. The augmented storehouse unit digital model can also include a scanned physical identifier (e.g., a storehouse unit identifier), such as a label, photo, or logo physically located on the storehouse unit. The identification information of the storehouse unit can be used to count the number of other similar storehouse units (e.g., other milk cartons). In some implementations, the augmented storehouse unit digital model can be obtained from a digital model 140 stored in memory 108 from previously processed image frames of a previously identified storehouse unit. In some implementations, memory 108 can store the storehouse unit identifier to reduce the computational burden in identifying the storehouse unit digital model 140 and calculating the total number of each storehouse unit of each unit type in inventory 116. For example, the machine learning model 124 may use optical character recognition (OCR) to read and / or recognize a storage unit identifier on a storage unit and / or text on the storage unit (e.g., a barcode, keyword, product number, etc.) and compare the text and / or storage unit identifier to text and / or storage unit identifiers stored in memory 108 and / or a database. In some cases, the storage unit may include a box containing multiple storage units, such as a box of milk cartons. For example, the machine learning model 124 may use OCR to read and / or identify text on the box. The text may indicate the number of milk cartons in the box, and the machine learning model 124 may use that number to generate (or update) a storage unit count for the milk cartons. In some cases, after performing OCR to read and / or identify text captured on a storage unit, shelf, wall, label, etc., the processor 104 may identify patterns, such as images and / or characters, that form words, phrases, and / or brand logos, etc., and search memory 108 and / or a database to determine the identity of the storage unit.For example, the machine learning model 124 may be further trained to match certain combinations of characters, words, and / or images to particular unit types (e.g., milk cartons, bottles, cups, etc.).

[0023] In some implementations, the machine learning model 124 can alternatively and / or additionally identify store units based on the shape and / or form of the store unit identifier. For example, the sensor(s) 112 (e.g., LIDAR) can detect that store units may include a shape, size, form, label, logo, and / or image specific to a particular unit type and can count the number of store units sharing the same shape, size, form, label, logo, and / or image. In some implementations, the machine learning model 124 can generate a planogram of the inventory 116. The planogram can be used as a map that the machine learning model 124 uses to identify store units and / or generate digital models 140 for the store units. This is, at least in part, to allow the machine learning model 124 to better predict and / or identify store units and / or digital models 140 of store units in substantially real time.

[0024] In some implementations, memory 108 stores instructions that cause processor 104 to detect, via machine learning model 124, multiple storage units within an area generally indicated by spatial search 120. In some cases, storage units may be placed on their side, flat, upside down, etc. In some cases, multiple storage units may be stacked on top of each other, placed behind each other, and / or placed next to each other on a shelf, etc. Machine learning model 124 may be configured to detect each storage unit area, which may include multiple spatial storage units, via shape analysis. do.After the store units and / or areas in which they are located are detected, the machine learning model 124 can also determine the depth of each store unit to identify each store unit to be counted to generate the store unit count 136. For example, the shape of the store unit in the image frames may vary because the sensor(s) 112 capture image frames of the store unit from different angles, positions, and / or locations. The machine learning model 124 can calculate different measurements of the same store unit in the 3D space of the spatial search 120 and / or inventory 116 and can determine that the store unit across multiple image frames from different angles is the same store unit.

[0025] In some implementations, the multiple store units do not have a predefined orientation. For example, a store unit including a cereal box can be facing the sensor(s) 112. In such an example, for example, a store unit identifier, such as a logo, is captured by the sensor(s) 112. In some examples, the cereal box can be turned upside down to present its barcode to the sensor(s) 112. In such a case, the sensor(s) 112 can scan the barcode. Thus, in some implementations, the sensor(s) 112 can identify multiple different store unit identifiers (e.g., logos and barcodes). In some examples, the cereal box can be laid flat. The machine learning model 124 can detect and / or identify the store unit in multiple orientations, configurations, and / or angles, etc. The machine learning model 124 can also detect the store unit without a predefined field of view of the sensor(s) 112. For example, the sensor(s) 112 may capture image frames of the store unit from one field of view and also from a different field of view. The machine learning model 124 may correctly detect and / or identify the storage unit despite different fields of view from the sensor(s) 112.

[0026] In some implementations, the machine learning model 124 can identify and / or confirm the identity of the storage unit based on the shape of the digital model 140 generated based on depth analysis via LIDAR. In some cases, cups can be stored in stacks, the user device 101 can capture and calculate the height of the stacked cups, and the machine learning model 124 can determine the number of storage units for the cups based on the height of the stacked cups and the height of a single cup.

[0027] Alternatively or additionally, user device 101 may include a second machine learning model (not shown in FIG. 1 ) that is different from machine learning model 124. In such implementations, a first machine learning model (e.g., machine learning model 124) may be used to identify and / or detect storage units, and a second model may be used to calculate storage unit count 136 using the storage unit identification results as input. In some examples, the first machine learning model may also generate digital model 140 for the identified storage units using image frames as input.

[0028] The store unit number 136 may include (or be) a numeric value representing the total number of each unit type in the inventory 116 (e.g., number of milk cartons, number of coffee powder packets, number of straws, etc.). In some cases, the store unit number 136 may include duplicate store units and / or inaccurate counts of different unit types (e.g., cartons of creamer may be counted as cartons of milk). The memory 108 may store instructions that cause the processor 104 to detect duplicate counted store units and / or duplicate counts and remove duplicate counted store units in the store unit number 136 for a unit type. In some implementations, for example, the memory 108 stores instructions that cause the processor 104 to calculate the store unit number 136 by detecting the height of stacked store units via the sensor(s) 112 and / or by detecting the depth of the store units via the sensor(s) 112, as described in further detail herein. In some implementations, the height and / or depth of a storage unit may be identified by looking up the height and / or depth of the storage unit, shelf, and / or storage rack, etc. in a database.

[0029] The spatial labels 144 may include a digital and / or substantially real-time virtual representation of a unit type identifier for each unit type and / or a subset of store units associated with each unit type, and their designated locations for storage using the control points 132 and / or in the spatial search 120. In some implementations, the spatial labels 144 may be static within the augmented reality / active reality space. Specifically, store units detected within the designated locations generally indicated by the spatial labels 144 are counted toward the number of store units associated with the spatial label. In some implementations, the memory 108 stores instructions that cause the processor 104 to identify a spatial label for each subset of store units from the subset of multiple store units from the plurality of store units and calculate, via the machine learning model 124, a number of store units for each subset of store units associated with each unit type. For example, sensor(s) 112 may be used to detect storage units in an image frame, digital models 140 of the storage units, and spatial labels 144 of the storage units, and via machine learning model 124, a total number of storage units per unit type may be counted to generate storage unit count 136. In some examples, machine learning model 124 may use digital models 140 and spatial labels 144 to count the total number of storage units associated with each unit type to generate storage unit count 136. In some implementations, machine learning model 124 may also determine the identity of a storage unit in a specified location of a spatial label from multiple spatial labels 144, regardless of the angle at which the storage unit is placed (e.g., upside down, flat, sideways, etc.).

[0030] For example, the processor 104 may determine (e.g., based on a spatial label, a barcode, etc.) that the top shelf of a rack is designated for milk cartons. The memory 108 may store a representation of the milk cartons (e.g., spatial label 144), which may also include control points (e.g., control points 132) on the top shelf. This may allow the machine learning model 124 to count detected storage units on the top shelf as milk cartons without using additional sensors to scan the physical labels and / or shapes of each storage unit, reducing computational load, for example. The spatial labels 144 may be configured, updated, modified, moved to different locations, etc. In some implementations, the spatial labels 144 may also be real-time virtual representations that appear at locations within the inventory 116 designated for each unit type and / or for a subset of storage units associated with each unit type. In some implementations, the spatial labels 144 may be viewed on the display 148 in a form of active reality and / or augmented reality. In some implementations, the control points 132 can be manually set (eg, virtually and / or physically, for example, using barcodes on racks).

[0031] In some implementations, the user device 101 can optionally include an inventory management system (not shown in FIG. 1 ). The inventory management system can include software programs and / or code capable of generating a digital dashboard for organizing the store unit count 136. The inventory management system can also include interactive features that allow a user to analyze, manage, and / or view store units, unit types, and / or store unit counts 136, etc. The inventory management system can be executed by the processor 104 and can allow a user to manually, for example, rename store unit counts 136, unit types, and / or order store units, etc. In some implementations, the memory 108 can store instructions that cause the processor 104 to generate a store unit report. In some implementations, the inventory management system can allow a user to make changes based on the store unit report. The store unit report may include any information describing the store unit, the store unit count 136, the inventory 116, the spatial search 120, the coordinates of the spatial search 120, the control points 132, the coordinates of the control points 132, the digital model 140, the spatial label 144, the store unit identifier, and / or the icon 152.

[0032] In some implementations, the memory 108 can also store a refill status. The refill status can include (or be) an alert, signal, and / or sign indicating that the number of store units associated with a unit type is low. In some implementations, a low number indicates that a subset of store units associated with the unit type is low, out of stock, and / or in need of refilling. For example, the refill status can include an alert that milk cartons are low and / or empty. The refill status can include a sign that there is no imminent need to refill the milk cartons. In some implementations, the refill status can generate an alert based on the number of store units 136 falling below a refill threshold. The refill threshold can include a minimum and / or maximum value for the number of store units 136, and an alert can be generated indicating that refilling of the unit type is desired when the number of store units 136 falls below or exceeds the refill threshold. In some cases, the refill threshold can be different for each unit type. In some implementations, the memory 108 stores instructions that cause the processor 104 to perform replenishment based on generated / triggered alerts for one or more unit types. In some implementations, performing replenishment can include automatically ordering multiple store units for one or more store units indicated for replenishment. In some implementations, performing replenishment can include just-in-time (JIT) delivery of store units needing replenishment and / or triggered for replenishment.

[0033] The user device 101 may include a display 148. The display 148 may include (or be) an electronic device with a screen used for presenting and / or displaying information and images, such as a plurality of icons 152, a substantially real-time feed 156, and / or an inventory dashboard 160. The display 148 may include, for example, a monitor, such as an LED monitor, an OLED monitor, and / or an AMOLED monitor. In some implementations, the display 148 may include (or be) a touchscreen that receives user input via touching the display 148.

[0034] The icons 152 for the spatial search 120 can include three-dimensional (3D) objects, such as spheres, cubes, or prisms, and serve as virtual representations for the objects identified in the image frames. In some implementations, the icons 152 can be based on the storage unit, the storage type of the storage unit, and / or the unit type from multiple unit types. In some implementations, the icons 152 can also include different storage type colors based on the storage type of each storage unit. Certain icons 152 that can be presented on the display 148 are described in more detail with respect to FIGS. 8-14. In some implementations, the icons 152 can be visual representations of the digital model 140 for the objects identified in the image frames. In some implementations, the icons 152 can also include storage unit labels for each unit type and / or a subset of storage units associated with the unit type. In some implementations, the icons 152 for the storage unit labels can include natural language identifiers, letters, abbreviations, and / or the like. In some implementations, the icons 152 can also include numeric labels for the number of storage units of each unit type. For example, at a control point designated for a cup, the icon for that control point may include the label "CUP," which appears hovering over an area designated for cup storage units. Each cup may be associated with a green sphere, where green is designated for small-sized storage object units. In some implementations, any other color or notation may be displayed. The memory 108 may store instructions that cause the processor 104 to generate icons 152 for the control points 132 and spatial labels for each unit type from multiple unit types.

[0035] The display 148 can present a substantially real-time feed 156. The substantially real-time feed 156 can include a real-time display of the environment being captured by the sensor(s) 112, including icons 152 that are continuously generated and / or modified based on movement of the sensor(s) 112 and / or the user device 101. For example, a user operating the user device 101 can point the sensor(s) 112 at inventory 116 that includes boxes of bagged coffee. The substantially real-time feed 156 can present on the display 148 in substantially real time (e.g., with little or no perceptible delay) the actual environment as seen by the sensor(s) 112, as well as icons 152 for objects (e.g., boxes of bagged coffee) and associated information for the objects (e.g., store unit labels, store unit numbers, etc.).

[0036] The display 148 may also present an inventory dashboard 160. The inventory dashboard may include (or be) a list of unit types, a number of stock units 136, and / or a replenishment status by unit type, etc. The inventory dashboard 160 is described in more detail with respect to FIG. 15 . In some implementations, the inventory management system may also include the inventory dashboard 160. In such implementations, the inventory management system and the inventory dashboard 160 may include (or be) a user interface and / or a user experience platform.

[0037] In some implementations, the user device 101 can also be used for gamification purposes, such as based on icons 152, digital models 140, and / or inventory dashboards 160. The system 100 can also be integrated into existing camera systems. For example, the processor 104 can generate icons 152, such as 3D coins and / or 3D toys, scattered throughout a virtual space generally represented by a spatial search 120 that can be located by pointing the sensor(s) 112 at different locations in or around the inventory 116 as the user moves around the inventory 116. The 3D coins / toys can also be hidden behind physical objects (e.g., behind a first storage unit with other storage units stacked behind it, on a top shelf, under a bottom shelf, behind a wall, etc.). In some cases, the processor 104 may generate a temporary icon at a virtual location in the inventory 116 to indicate that a store unit should be placed there and / or is missing from that store unit group (e.g., an area and / or space label 144 designated for a particular unit type).

[0038] 2 is a block diagram of a system 200 for inventory management using edge computer vision and active reality, according to one embodiment. System 200 includes user device 101, management device 170, network 190, and server 180 of FIG. 1. In some implementations, some of the functions and / or processes described as being executed and / or performed on user device 101 with respect to FIG. 1 may be executed and / or performed by server 180 and / or management device 170.

[0039] The server 180 may include a processor 182 operably coupled to a memory 184 that stores instructions for execution by the processor 182. The processor 182 of the server 180 may be or include, for example, a hardware-based integrated circuit (IC) or any other suitable processing device configured to run and / or execute a set of instructions or code. For example, the processor 182 may be a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic array (PLA), a complex programmable logic device (CPLD), and / or a programmable logic controller (PLC), etc. In some implementations, the processor 104 may be configured to run any of the methods and / or portions of methods described herein.

[0040] The memory 184 of the server 180 may be or include, for example, random access memory (RAM), a memory buffer, a hard drive, read-only memory (ROM), and / or erasable programmable read-only memory (EPROM). In some examples, the memory may store one or more software programs and / or code, which may include, for example, instructions that cause the processor 182 to perform one or more processes and / or functions. In some implementations, the memory 184 may include an expandable storage unit that can be used incrementally. In some implementations, the memory 108 may be portable memory (e.g., a flash drive and / or a portable hard disk, etc.) that may be operably coupled to the processor 182. The memory 184 may include various components (e.g., machine-readable media), including, but not limited to, random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system (BIOS), containing basic routines that help to transfer information between elements within the server 180, such as during startup, may be stored in the memory 184. Memory 184 may further include any number of program modules including, for example, an operating system, one or more application programs, other program modules, program data, and / or any combination thereof.

[0041] The server 180 may include (or be) a hardware device that provides functionality to devices on and / or connected to the network 190, such as the user device 101 and the management device 170. In some implementations, the server 180 may include (or be) a remote device that can process requests from the user device 101. For example, the user device 101 may capture and transmit image frames to the server 180 for identification and counting. The memory 184 of the server 180 may store instructions that cause the processor 182 of the server 180 to identify storage units and / or calculate the number of storage units from the received image frames (e.g., using one or more machine learning models as described herein). The memory 184 may store instructions that cause the processor 182 to transmit the storage unit identification results and / or the number of storage units to the user device 101 over the network 190.

[0042] In some implementations, server 180 can process multiple requests from multiple user devices (similar to user device 101) connected to network 190. In some implementations, server 180 can perform any other processes and / or functions described herein as being performed by user device 101. In some implementations, multiple user devices can capture image frames of the same inventory in the same warehouse. Server 180 can be configured to perform de-duplication of duplicately counted store units, even when multiple user devices are used to detect and / or count store units from the same inventory. In some implementations, server 180 can process the image frames captured by each user device and use spatial and / or depth analysis to determine which store units have been counted multiple times and update the store unit counts associated with the duplicate store units accordingly.

[0043] The management device 170 can be connected to the network 190 to communicate with the user devices 101 and / or the server 180. In some implementations, the management device 170 can include (or be) a smartphone, tablet, PC, laptop, or the like used to manage the system 200 and its processes. In some implementations, multiple user devices can be connected to the management device 170. The management device 170 can include a processor 172, a memory 174, a display 176, and / or peripheral(s) 178 operatively coupled to each other. The memory 174 of the management device 170 stores instructions for execution by the processor 172. The processor 172 of the management device 170 can be or include, for example, a hardware-based integrated circuit (IC) or any other suitable processing device configured to run and / or execute a set of instructions or code. For example, the processor 172 may be a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic array (PLA), a complex programmable logic device (CPLD), and / or a programmable logic controller (PLC), etc. In some implementations, the processor 104 may be configured to perform any of the methods and / or portions of methods described herein.

[0044] The memory 174 of the management device 170 can be or include, for example, random access memory (RAM), a memory buffer, a hard drive, read-only memory (ROM), and / or erasable programmable read-only memory (EPROM). In some examples, the memory can store one or more software programs and / or code, which can include, for example, instructions that cause the processor 172 to perform one or more processes and / or functions. In some implementations, the memory 184 can include an expandable storage unit that can be used incrementally. In some implementations, the memory 174 can be portable memory (e.g., a flash drive and / or a portable hard disk, etc.) that can be operably coupled to the processor 172. The memory 108 can include various components (e.g., machine-readable media), including, but not limited to, random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system (BIOS), containing basic routines that help to transfer information between elements within the management device 170, such as during startup, can be stored in the memory 174. Memory 174 may further include any number of program modules including, for example, an operating system, one or more application programs, other program modules, program data, and / or any combination thereof.

[0045] The display 176 of the management device 170 may include any display as described throughout this disclosure. The display 148 may include (or be) an electronic device with a screen used for displaying information and images, such as the plurality of icons 152, the substantially real-time feed 156, and / or an inventory dashboard. The display 148 may include, for example, an LED monitor, an OLED monitor, an AMOLED monitor, etc. In some implementations, the display 148 may include (or be) a touchscreen for receiving user input via touching the display 148.

[0046] The peripheral(s) 178 may include, for example, a mouse, keyboard, trackpad, and / or speakers. The management device 170 may be used to manage inventory. For example, a user operating the management device 170 may manually set a predefined threshold for replenishment status (e.g., a replenishment threshold) for each subset of store units associated with a unit type from the plurality of unit types in inventory. The user operating the management device 170 may also manually update the number of store units per unit type. In some implementations, the management device 170 may receive the number of store units in inventory from the user device 101 and / or the server 180. The management device 170 may also compare the number of store units with a predefined threshold and / or automatically order units if the count is below the predefined threshold (e.g., a replenishment threshold). The management device 170 may present the data on an inventory dashboard to the user operating the user device 101 to present the number of store units and the number of unit types.

[0047] In some implementations, the system 200 may enable application programming interface (API) integration with multiple devices, such as the user device 101, the management device 170, and / or the server 180, to facilitate communication of information and / or data over the network 190.

[0048] 3 is a block diagram of a machine learning system 300 for edge computer vision and active reality, according to one embodiment. System 300 may include a user device 301, other user devices 311, a database 391, a server 380, and a network 390 that enables communication between user device 301, other user devices 311, and / or database 391, etc. User device 301 may include a sensor (not shown in FIG. 3 ) operably coupled to user device 301 and configured to scan and capture images or video of inventory items 316. In some cases, a user operating user device 301 may capture multiple inventory items within a storage location (e.g., a warehouse, a storage closet, etc.) or multiple inventory items from multiple storage locations.

[0049] Inventory 316 may be similar to inventory 116 of FIG. 1. Sensors of user device 301 may be configured to capture and analyze information found within a set of boundaries, such as, for example, spatial search 320. Spatial search 320 may include (or be) a representation and / or virtual representation of a zone, boundary, border, and / or area within a space in which inventory 316 is located. Spatial search 320 may define the layout of a space that holds and / or contains inventory 316. Spatial search 320 may be similar to spatial search 120 of FIG. 1. In some cases, spatial search 320 may be established by sensing one or more barcodes (e.g., quick response (QR) codes), predetermined identifiers, corners, walls, and / or floors where inventory 316 is located. In some cases, sensors may capture spatial search 320 through depth calculations as a user touches the walls, floors, and / or ceilings surrounding inventory 316. In some cases, the sensor can capture spatial search 320 by recognizing a particular inventory (e.g., a particular shelf, a particular horizontal row, etc.) by a control point (e.g., a QR code, landmarks, etc.) and using the control point (e.g., the unique ID of the QR code) to retrieve depth or dimensional information about the particular inventory from database 391. In some implementations, the sensor can capture storage units that are located in various locations and at various orientations within inventory 316 and / or the warehouse. For example, if a box of straws falls off a shelf, the sensor can scan the box and determine that the box is a box of straws.

[0050] The user device 301 can be or include a computing device operated by a user. The user device 301 can be structurally and / or functionally similar to the user device 101 of FIG. 1. In some implementations, the user device 301 can include a display 348 configured to present to the user a substantially real-time feed 356 of images or video captured by a sensor, icons 352 overlaid and / or positioned within the active / augmented reality of the inventory 316, an inventory dashboard 360, product listings 362, and / or buttons 364, etc.

[0051] The icons 352 may be or include active reality / augmented reality objects, such as, for example, 3D digital representations that virtually and visually highlight, identify, and / or enhance each store unit from a set of store units in the inventory 316. For example, a carton of milk may be overlaid with a rectangular prism icon that generally covers a portion of the carton. In some cases, the rectangular prism may be various shades, opaque, semi-transparent, and / or colored to allow a user viewing the display 348 to easily identify the carton. The icons 352 may appear and / or disappear appropriately based on sensor activity visualized via a substantially real-time feed 356. The inventory dashboard may include (or be) a list of unit types, number of store units 336, and / or replenishment status by unit type, etc. The inventory dashboard 360 is described in further detail with respect to FIG. 15 . In some cases, the icons 352 may be a visual representation of the digital model 340.

[0052] The processor 382 stores coordinates for the control points of the inventory 316 in the spatial search 320. 384 to know and / or predict where inventory 316 is expected to be located and / or the area that will be searched to identify inventory 316. In some implementations, spatial search 320 can be static (e.g., configured to be predefined to a particular location and / or coordinates within the 3D world). In some cases, spatial search 320 can also indicate an area within spatial search 320 via a particular point and / or coordinates to identify and focus information within the area where sensor(s) 312 are indicated. sky The coordinates and / or parameters of the spatial search 320 can be stored in memory 308 and / or database 391. This allows processor 304 to filter, ignore, and / or discard areas outside of spatial search 320 to reduce computational load. Similarly, in some implementations, image content capturing areas within spatial search 320 can be processed while image content capturing areas outside of spatial search 320 can be discarded and / or ignored. In some implementations, the spatial search 320 area can be manually modified by a user. In some implementations, the spatial search 320 area can be automatically modified via user input (e.g., touching display 348 or pressing button 364) (e.g., if new inventory is recognized outside the spatial search 320 area).

[0053] Product list 362 may be a menu that a user can interact with to modify the data of icon 352. For example, icon 352 may include a description of the store unit, a description of inventory 316, and / or the number of store units. A user can interact with product list 362 to change the description and name and / or modify the count. Product list 362 is described in more detail with respect to FIG. 12. Button 364 may be an input / output interface that allows a user to modify the data of the store unit and / or icon 352.

[0054] Database 391 may be or include a local database, a cloud database, a Standard Query Language (SQL) database, a relational database, etc. Database 391 may be configured to collect, store, and / or record data captured by sensors and / or data processed by server 380. Database 391 may store data such as first inventory data 392 and / or second inventory data 394. For example, database 391 may be configured to store and distinguish between storage units from different inventories (e.g., warehouses, stores, shelves, etc.). In some cases, different shelves or warehouses may store the same type of storage units. Database 391 may store data regarding which inventory (e.g., shelf or warehouse) stores how many storage units of a particular type. As an example, if two shelves both store boxes of cups, database 391 may record how many boxes of cups each shelf stores. If a user removes a box of cups from one inventory, after rescanning, database 391 may record data that that inventory has one fewer box of cups, while the other inventory maintains the same number of boxes of cups. In some implementations, database 391 may store specific details such as, for example, which inventory a particular storage unit is stored in, which shelf within the inventory a particular storage unit is stored on, the location of a particular storage unit when stacked on top of other storage units, the quantity of each unit type, and / or remaining inventory space.

[0055] In some cases, a user can use a sensor on the user device 301 to capture an image of the inventory 316 and can choose to record data (e.g., "checkout") of the inventory 316, which can be stored as first inventory data 392. The user can then point the sensor at a different inventory and press a button to record data for the different inventory, which can be stored as second inventory data 394. In some cases, the first inventory data 392 and the second inventory data 394 can include the same or different storage units.

[0056] The server 380 may be or may include a computing device configured to analyze and / or process data captured by the sensors. In some cases, the server 380 may be located remotely from the database 391 and / or the user device 301. The server may include a processor 382, ​​a network interface 386, and / or a memory 384 that stores instructions to be executed by the processor 382. The network interface 386 may connect the server 380 to a network 390 to communicate with the user device 301 and / or the database 391.

[0057] Processor 382 may be structurally and / or functionally similar to processor 104 or processor 182 of FIGS. 1 and 2, respectively. Memory 384 may be structurally and / or functionally similar to memory 108 and memory 184 of FIGS. 1 and 2, respectively. Memory 384 may store machine learning models 324, SKU training data 328, control points 332, store unit counts 336, digital models 340, spatial labels 344, collision zones 326, SKU status 330, and / or filters 334, etc. Memory 384 may be or include, for example, random access memory (RAM), a memory buffer, a hard drive, read-only memory (ROM), and / or erasable programmable read-only memory (EPROM), etc. In some examples, memory may store one or more software programs and / or code, which may include, for example, instructions that cause processor 382 to perform one or more processes and / or functions, etc. In some implementations, memory 384 may include an expandable storage unit that can be used incrementally. In some implementations, memory 384 may be portable memory (e.g., a flash drive and / or a portable hard disk, etc.) that can be operably coupled to processor 382. Memory 384 may include various components (e.g., machine-readable media), including, but not limited to, random-access memory components, read-only components, and any combination thereof. In one example, a basic input / output system (BIOS), containing basic routines that help to transfer information between elements within server 380, such as during startup, may be stored in memory 384. Memory 384 may further include any number of program modules, including, for example, an operating system, one or more application programs, other program modules, program data, and / or any combination thereof.

[0058] The machine learning model 324 can be structurally and / or functionally similar to the machine learning model 124 of FIG. 1 . The machine learning model 324 can be configured to detect a plurality of store units using images captured by a sensor of the user device 301 as input and / or calculate a number of store units from a plurality of store unit numbers 336 of each unit type from a plurality of unit types. The machine learning model 324 can be trained using SKU training data 328. In some implementations, the SKU training data 328 includes labeled training data including store unit digital models correlated to and / or labeled with store unit identifiers. The memory 308 can store instructions that cause the processor 304 to continuously, sporadically, and / or periodically provide data (e.g., SKU training data) to the machine learning model 324 to generate a robust and / or trained machine learning model. In some implementations, the machine learning model 324 can be synthetically trained. For example, the machine learning model 324 can be trained in virtual and real-world training environments using store units.

[0059] In some implementations, the machine learning model 324 can alternatively and / or additionally identify store units based on the shape and / or form of the store unit identifier. For example, a sensor in the user device 301 can detect that a store unit may include a shape, size, form, label, logo, barcode, and / or image specific to a particular unit type and can count the number of store units that share the same shape, size, form, label, logo, barcode, and / or image. In some implementations, the machine learning model 324 can generate a planogram of the inventory 316. The planogram can be used as a map that the machine learning model 324 uses to identify store units and / or generate digital models 340 for the store units. This is, at least in part, to allow the machine learning model 324 to better predict and / or identify store units and / or digital models 340 of store units in substantially real time.

[0060] In some implementations, the machine learning model 324 can be configured to determine the depth of a storage unit in the inventory 316. For example, the milk cartons can be arranged so that the sensor primarily captures the frontmost milk cartons. The machine learning model 324 can determine how many milk cartons are in a row of milk cartons using depth calculations, LIDAR data, and / or data stored on shelf depths and correlated based on control points (e.g., barcodes), etc. In some cases, the machine learning model 324 can also determine the depth of a storage unit based on a user interacting with a storage unit that may be located further back and / or behind other storage units, identifying (e.g., touching) a wall, and / or identifying (e.g., touching) a portion of the inventory 316 (e.g., a shelf), etc.

[0061] In some implementations, memory 308 stores instructions that cause processor 304 to detect, via machine learning model 324, multiple storage units within the area generally indicated by spatial search 320. In some cases, storage units may be placed on their side, flat, upside down, etc. In some cases, multiple storage units may be stacked on top of each other on a shelf, placed behind one another, and / or next to one another, etc. Machine learning model 324 may be configured to detect each storage unit area, which may include multiple spatial storage units, via shape analysis. Machine learning model 324 may calculate different measurements of the same storage unit in the 3D space of spatial search 320 and / or inventory 316 and may determine that the storage unit across multiple image frames from different angles is the same storage unit.

[0062] In some implementations, the machine learning model 324 can identify and / or confirm the identity of the storage unit based on the shape of the digital model 340 generated based on depth analysis via LIDAR. In some cases, cups can be stored in stacks, the user device 301 can capture and calculate the height of the stacked cups, and the machine learning model 324 can determine the number of storage units for the cups based on the height of the stacked cups and the height of a single cup.

[0063] In some implementations, the machine learning model 324 can be configured to determine different form factors of a store unit. For example, a store unit such as chicken may be stored in various forms, such as fried chicken, baked chicken, frozen chicken, and / or raw chicken. The machine learning model 324 can identify the different forms and process these different forms accordingly (e.g., count, report weight, etc.). The machine learning model 324 can also determine the quantity of the store unit based on the reported weight of the box, container, and / or bag.

[0064] Store unit count number 336 may include (or be) a numeric value representing the total number of each unit type in inventory 316 (e.g., number of milk cartons, number of coffee powder packets, number of straws, etc.). In some cases, store unit count 336 may include duplicate store units and / or inaccurate counts of different unit types (e.g., cream cartons may be erroneously included in the count as milk cartons). Memory 308 may store instructions that cause processor 304 to detect duplicate counted store units and / or duplicate counts and remove duplicate counted store units in store unit count 336 for a unit type. In some aspects, for example, memory 308 stores instructions that cause processor 304 to calculate store unit number 336 by detecting the height of stacked store units via sensor(s) 312 and / or by detecting the depth of store units via sensor(s) 312.

[0065] The space label 344 may include a digital and / or substantially real-time virtual representation of a unit type identifier for each unit type and / or a subset of store units associated with each unit type, and its designated storage location for storage using the control point 332 and / or in the space search 320. In some implementations, the space label 344 may be static within the augmented reality / active reality space. Specifically, store units detected within the designated location generally indicated by the space label 344 are counted toward the store unit count associated with the space label. In some cases, the control point 332 may include a quick response (QR) code indicating a location within the inventory 316 that will be designated for a particular unit type of store unit. The QR code for a particular store unit may include and / or reference database information regarding the number of store units that can be stacked in a vertical column in the inventory 316, which may be used to determine the depth of the inventory 316 for counting the store units. In some cases, the QR code may also provide a reference point for the location of the user device 301 sensor, allowing the sensor to identify and / or determine the characteristics of that location (e.g., shelf size) and where to search for inventory based on this reference.

[0066] The digital model 340 may include a substantially real-time virtual representation of a real-world physical object, such as, for example, inventory 316, walls, floors, shelves, racks, and / or storage units. The digital model 340 may be stored in memory 308 and may also be presented on a display 348 in a form of active reality and / or augmented reality. In some cases, the digital model 340 may include spatial labels 344 and / or icons 352. In some implementations, the processor 382 (via the machine learning model 324) may generate a 3D world using the digital model 340.

[0067] Collision zone 326 can be or include data indicating an area around a digital model of a storage unit so that when another storage unit (or a digital model of another storage unit) is detected within the collision zone, server 380 can be alerted to the presence of a potentially overlapping storage unit. For example, if the collision zones of two digital models of two storage units overlap by more than a certain amount (or predetermined threshold), an error may have occurred. For example, the digital model for one of the two storage units may have been incorrectly generated, positioned, and / or aligned. Because real-world objects do not overlap, overlap between the digital models 340 of the objects may indicate an anomaly, such as overlapping digital models. In some implementations, collision zone 326 can be modified by the user.

[0068] The SKU status 330 can indicate multiple states of a store unit. In some cases, the SKU status 330 can be represented as a digital model 340, which is described in more detail with respect to FIG. 14. The status of the SKU can include the state of the store unit. For example, a container of sour cream may be 50% full, which can be recorded as the SKU status. The SKU status can also include the state of the store unit, such as frozen, liquid, thawed, fried, and / or baked.

[0069] The filter 334 may include a user-definable filter for analyzing stock units in the inventory 316. For example, a user may set the filter to 5 feet. In such an example, if the sensor is more than 5 feet away from the inventory 316, the processor 382 may hide the digital model 340. Setting a distance filter may enable the user to create a 3D world for each digital model of inventory while reducing the computational burden on the processor 382 when rendering / generating the digital model 340 and / or storing the data. In other words, by populating the 3D world with the digital model 340 when the sensor is within a predefined distance from the inventory 316, the processor 382 may accurately and efficiently process and present the data on the display 348 of the inventory 316 while ignoring other nearby inventory (greater than the predefined distance). In some cases, by limiting the generation of the digital model 340 and / or the analysis of the data for the inventory 316 to a certain distance, the processor 382 may generate a digital model 340 that is most relevant to the user. In some cases, moving further away from the inventory 316 and outside the filter can cause the digital model 340 of the inventory 316 to disappear so as not to obstruct the real-time feed 356 and display 348 of the user device 301. Moving closer to the inventory 316 and within the distance set by the filter can cause the digital model 340 to reappear. In some implementations, depending on the orientation of the sensor capturing images or video of the inventory 316, the processor 382 can hide the 3D world at different positions, angles, and / or distances to avoid visual clutter. For example, if the sensor is to the side or behind the inventory's control point (e.g., positioned at a predetermined angle relative to the front of the inventory), the processor 382 can hide the digital model 340 and / or counting results. When the sensor is again positioned in front of the control point (within the predetermined viewing angle), the processor can present the digital model 340 and / or counting results.Even if the digital model 340 of the store unit is not displayed, the data for the digital model 340 may be stored in the database 391.

[0070] In some implementations, other user devices 311 (structurally and functionally similar to user device 301) can scan inventory 316 (or other inventory) and determine spatial search 321 from the perspective of the sensor(s) of the other user devices 311 (e.g., operated by other users). In some cases, multiple user devices, including user device 301 and other user devices 311, can simultaneously or sequentially scan inventory 316 to obtain and record information (e.g., digital model 340, control points 332, store unit count 336, collision zone 326, space label 344, SKU status 330, etc.) from different positions and angles. This is at least in part to efficiently scan inventory from multiple sensors and at multiple angles to accurately identify store units, determine collision zone 326, and / or calculate store unit count 336, etc. In other words, the user device 301 and the other user devices 311 can operate synchronously to capture inventory 316 data, such that the digital model 340 and / or storage unit count 336 captured by the user device 301 in the spatial search 320 can also be synchronously presented on the display of the other user devices 311, and vice versa. In some implementations, the processor 382 of the server 380 can receive data from both the user device 301 and the other user devices 311 and, based on the positioning of the user device 301 and the other user devices 311 (e.g., based on control points), process the data to remove duplicates and / or update inventory counts, etc.

[0071] FIG. 4 is an illustration of a spatial search 400 captured by one or more sensors (not shown in FIG. 4 ) of a user device (e.g., user device 101 of FIGS. 1 and 2 ) using active reality, according to one embodiment. The spatial search 400 may correspond to the spatial search 120 described with respect to FIG. 1 . As shown in FIG. 4 , the spatial search 400 may include representations and / or coordinates of locations and / or areas relative to inventory 402. The spatial search 400 may also include a plurality of control points 404 and spatial labels 408 represented as icons. In some implementations, the control points 404 and spatial labels 408 may function as calibration points. In some implementations, the spatial labels 408 may also function as location histories of associated unit types. Specifically, for example, the control points 404 and spatial labels 408 may remain in their original positions within the spatial search 400 and / or maintain the same coordinates within the spatial search 400 even when the field of view of one or more sensors changes due to movement of the one or more sensors.

[0072] Inventory 402 may include (or be) any physical storage location for multiple storage units, such as, for example, items, goods, merchandise, materials, and / or products. Inventory 402 may also include a warehouse, closet, freezer, retail space, and / or any location for storing items. Inventory 402 may include multiple storage units. In some implementations, the storage units may include a unique code, including letters and / or numbers, that identifies characteristics for each item and / or storage unit in inventory 402, such as, for example, the manufacturer, brand, style, color, size, type, product, etc. (associated with the code in the database). In some examples, inventory 402 may include a storage unit identifier for each storage unit, such as, for example, a label, logo, and / or barcode. Inventory 402 may include multiple items of the same type (e.g., cans of the same type of coffee beans, bags of coffee powder, packs of straws, cups of the same size, etc.). Inventory 402 may include storage units of different storage types. In some implementations, the storage type may be based on size, type of packaging, etc. For example, storage types may include bottles, small items, large items, medium boxes, large boxes, large bags, and / or jars, etc. Inventory 402 may also include multiple storage units. Each storage unit may be associated with one unit type from multiple unit types. A unit type may refer to a group of storage units of the same item, article, material, and / or product. A unit type may include (or be) the name and / or product of the storage unit. In some implementations, multiple storage units may have the same unit type. For example, a storage unit may be a "ketchup bottle," and the unit type of that storage unit may be "ketchup" or "ketchup bottle." In some cases, inventory 402 may store storage units that are the same product or item (e.g., ketchup). The common product, article, and / or material shared by those storage units may be a unit type.For example, a storage unit may have a storage type that is bottles and a unit type that is ketchup. In some implementations, the plurality of storage units may include a plurality of subsets of storage units. Each storage unit may be associated with a respective unit type from the plurality of unit types. For example, one subset of storage units may be associated with ketchup, and each storage unit in the subset associated with ketchup bottles is one ketchup bottle. In some examples, each storage unit in the subset of storage units is identical and / or the same item / material.

[0073] The spatial search 400 may include coordinates and / or a virtual representation of an area surrounding the inventory 402. In some implementations, the area may include walls, a floor, and / or a ceiling, etc. The area may also include real-world physical objects, including the inventory 402 housing storage units, such as shelves, racks, and / or storage units. The spatial search 400 may include a number of control points 404. As shown in FIG. 4 , the control points 404 may include, for example, walls, floor supports of a rack, the top of a rack, the middle of a rack, the bottom of a rack, a barcode (e.g., a QR code), a label, etc. 、The control points 404 may include coordinates and / or virtual representations of physically static objects, such as landmarks and / or identifiers. In some implementations, the processor can use the control points 404 to generate a map of the layout of real-world physical objects found in the spatial search 400. In some implementations, the control points 404 can be moved within the virtual space within the spatial search 400 via user input on a touchscreen, which can also function as a display for a user device (not shown in FIG. 4 ). For example, a user can tap on the touchscreen where a control point of interest is located and drag across the touchscreen to a desired location in the virtual space within the spatial search 400 as shown on the display. In another example, a user can point and tap on the touchscreen, with the tapped touchscreen location representing a virtual representation of the desired location of the control point to be set. The user can also use the touchscreen to remove control points and / or place new control points throughout the virtual space within the spatial search 400 and / or inventory 402.

[0074] The spatial label 408 may include coordinates and / or a substantially real-time virtual representation of a unit type identifier for each unit type and its designated location for storage using the control points 404 and / or in the spatial search 400. For example, a spatial label may be reserved for a milk carton. In some implementations, a detected storage unit that falls within a generally indicated range of the spatial label for the milk carton may be counted as a milk carton. In some implementations, the detection of the storage unit may be confirmed via shape analysis, text analysis, OCR, etc. The control points 404 and spatial label 408 may be calibrated and / or adjusted by the user. In other words, the user may customize the virtual environment associated with the inventory screenshot. For example, the user may customize the shape, size, color, placement, and / or transparency of an icon. In some implementations, the icon may be a non-fungible token (NFT).

[0075] FIG. 5 is a flow diagram of a method 500 for inventory management via a user device using edge computer vision and active reality, according to one embodiment. At 502, method 500 includes capturing image frames of inventory. In some implementations, capturing image frames may include receiving, at a processor of the user device, multiple image frames of the inventory from a sensor. The sensor may include multiple sensors, such as, for example, a charge-coupled device (CCD), an active pixel sensor (APS), and / or any digital image sensor fabricated with metal-oxide semiconductor (MOS), complementary metal-oxide semiconductor (CMOS), N-type metal-oxide semiconductor (NMOS), and / or Live MOS, etc. In some implementations, the sensor may include a depth sensor, such as, for example, a time-of-flight (TOF) sensor. The sensor may also include a camera, such as, for example, an ultra-wide-angle camera, a wide-angle camera, a telephoto camera, a monochrome camera, and / or a macro camera. The sensor may also include a light detection and ranging (LIDAR) sensor. In some implementations, method 500 can include scanning and / or capturing multiple image frames of inventory and multiple store units in substantially real time. In some implementations, a user can operate a user device to control where the sensor is capturing images and / or generating image frames. The sensor can also be used to capture the height of stacked store units, such as cups, as described in further detail herein.

[0076] At 504, method 500 includes locating a control point. In some implementations, locating a control point can include a location control point used to determine the spatial search within the image frame. In some implementations, the control point can include a barcode, an icon, a landmark, an identifier, a particular shelf, and / or other identification that can be used to orient the sensor. In some implementations, locating the control point can include generating an icon representing the control point and displaying the icon on a display on the device. In some implementations, method 500 can also include generating an icon for the control point from the plurality of control points for display on a display of the user device. The spatial search can include coordinates and / or a virtual representation of an area surrounding the inventory. In some implementations, the area can include walls, floors, and / or ceilings, etc. The area can also include real-world physical objects, including inventory housing storage units, such as shelves, racks, and / or storage units. The spatial search can include multiple control points.

[0077] At 506, method 500 includes detecting a storage unit. In some implementations, detecting a storage unit can include detecting multiple storage units in a spatial search via a machine learning model. In some implementations, each storage unit from the multiple storage units is associated with a unit type from a multiple storage unit types. For example, the inventory can include storage units of different storage types. In some implementations, the storage type can be based on size, type of packaging, and / or the like. For example, storage types can include bottles, small items, large items, medium boxes, large boxes, large bags, and / or jars, and the like. In some implementations, 506 can include identifying a spatial label for each unit type from the multiple unit types from the storage unit. In some implementations, identifying the multiple unit types can include generating an icon for the storage unit and / or a spatial label for each unit type from the multiple unit types for display on the user device. The unit type can include (or be) the name and / or product of the storage unit. In some implementations, the multiple storage units can have the same unit type. For example, a storage unit can be a "ketchup bottle," and the unit type for that storage unit can be "ketchup" or "ketchup bottle." In some cases, inventory can store storage units that are the same product or item (e.g., ketchup). The common product, item, and / or material that those storage units share can be a unit type. For example, a storage unit can have a storage type that is bottle and a unit type that is ketchup. In some implementations, multiple storage units can include multiple subsets of storage units. Each storage unit can be associated with each unit type from multiple unit types.For example, one subset of storage units may be associated with ketchup, and each storage unit in the subset associated with ketchup bottles is one ketchup bottle. In some instances, each storage unit in the subset of storage units is identical and / or the same item / material.

[0078] At 508, method 500 includes calculating a number of storage units. In some implementations, calculating the number of storage units may include calculating a number of storage units from the plurality of storage unit numbers for each unit type from the plurality of unit types from the plurality of storage units detected based on the depth analysis. Each storage unit number may include a total number of storage units associated with each unit type. In some implementations, the depth analysis may be performed via a sensor, such as a LIDAR sensor. For example, the sensor may detect storage units, such as ketchup bottles, that are present within the spatial search and / or focus on real-world physical objects generally represented by a plurality of control points. A machine learning model may be trained to detect multiple ketchup bottles stacked behind a first bottle located closest to the sensor to calculate the total number of ketchup bottles (e.g., calculate the number of ketchup bottles using the known depth of each ketchup bottle and the sum of the depths from the control points identified by the sensor). In some implementations, the control points may indicate and / or identify the depth of a shelf, rack, container, etc. For example, the control points can be barcodes (e.g., QR codes) that encode identifiers that can associate (e.g., in a database) a particular shelf with various characteristics (e.g., dimensions) of that shelf. Thus, after scanning the barcode, shelf characteristics (including depth) can be identified and used to calculate the total number of store units. In some implementations, method 500 can include calculating the number of store units based on the space labels. In such implementations, the machine learning model detects and / or counts the store units within and / or schematically indicated by the space labels.

[0079] In some implementations, method 500 can include detecting store units within a spatial search based on storage type, identifying, via a sensor, one store unit identifier from a set of store unit identifiers located on each store unit from the set of store units, and calculating, via a machine learning model, a number of store units per unit type based on the set of store unit identifiers. The store unit identifiers can include physical identifiers located on the store units, such as barcodes, identification numbers, labels, and / or logos. In some cases, method 500 can include performing optical character recognition (OCR), which can include converting image frames of the store unit identifiers into natural language, such as text in a machine-readable format. In some implementations, method 500 can include calculating, via a machine learning model, a number of store units per unit type based on the set of store unit identifiers.

[0080] In some implementations, method 500 includes recording in a memory and / or database a set of store unit identifiers associated with a first set of store units and the control points, and identifying, via a machine learning model, a second set of store units based on the set of store unit identifiers stored in memory based on the control points. For example, the machine learning model can be trained to read the store unit identifiers of the store units and accurately predict identification results for the store units and / or other store units scanned by the sensor.

[0081] In some implementations, method 500 can include calculating the number of store units per unit type by detecting each unit type and filtering out unit types that are not intended to be counted (e.g., located on a different shelf, rack, etc.). For example, method 500 can include detecting a first group of store units of a first unit type and filtering out a second group of store units of the second unit type (and / or any other group of store units) before calculating the number of store units for a second store unit and / or any other number of store units for any other unit type.

[0082] At 510, method 500 includes determining whether duplicate store units exist. In some cases, determining whether duplicate store units exist can include identifying duplicate store units based on a plurality of store unit counts. For example, a machine learning model can detect a store unit, such as a ketchup bottle, and count the store unit as both a ketchup bottle and a mustard bottle. Alternatively or additionally, the machine learning model can generate a digital model for each detected store unit, and as the machine learning model counts the total number of each unit type, the machine learning model can detect store units that have been incorrectly counted from the store unit counts for each unit type. In some implementations, spatial search 400 can include identifying duplicate store units, ensuring that duplicate store units are not counted more than once for two or more store unit counts from the plurality of store unit counts. In some cases, a store unit may be counted more than once if a single user device or two or more user devices capture multiple image frames of the store unit from different angles, perspectives, and / or positions, etc.

[0083] At 512, the method includes removing duplicates in the store unit count. Removing duplicates in the store unit count can include excluding the duplicate store unit from a store unit count associated with the duplicate store unit. For example, if a duplicate store unit (e.g., a ketchup bottle counted as a mustard bottle) is identified, the machine learning model can detect that the ketchup bottle was used, increasing the store unit count for the unit type (e.g., mustard bottle) and remove the duplicate store unit (e.g., a ketchup bottle counted as a mustard bottle) from the store unit count for the mustard bottle.

[0084] At 514, method 500 includes determining whether the store unit needs replenishment. In some cases, determining whether the store unit needs replenishment includes determining a replenishment status for each unit type from the plurality of unit types based on a number of store units per unit type. The replenishment status may include (or be) an alert, signal, sign, etc., indicating that the number of unit types is low (e.g., below a predetermined threshold). The low number may indicate that the unit type is low, out of stock, and / or needs replenishment. For example, the replenishment status may include an alert that milk cartons are low and / or empty. The replenishment status may include a sign indicating that there is no imminent need to replenish the milk cartons. In some implementations, method 500 may include generating an alert based on the number of store units being below a replenishment threshold. The replenishment threshold may include a minimum and / or maximum value for the number of store units. In some implementations, if the number of store units is below or above the replenishment threshold, an alert may be generated indicating that replenishment of the unit type and its associated store units is desirable. In some cases, replenishment thresholds for different unit types may vary. In some cases, replenishment thresholds may be manually modified. In some implementations, replenishment thresholds may be automatically updated and / or modified based on sales data, inventory history, etc.

[0085] At 516, method 500 includes automatically replenishing the store units. The automatic replenishing can include automatically generating a replenishing request based on the replenishing status. The replenishing request can include a request for replenishment of unit types identified as desiring replenishment based on a replenishment threshold. In some implementations, method 500 can include performing the replenishment based on alerts generated / triggered for one or more unit types. In some implementations, performing the replenishment can include automatically ordering multiple store units for one or more store units requiring replenishment. In some implementations, method 500 and its steps can be performed automatically.

[0086] In some implementations, a user device can download a software application to implement the functions and processes of method 500 described herein. This software application can be downloaded onto multiple user devices.

[0087] 6 is a flow diagram of a method 600 for a machine learning system for determining duplicates, according to one embodiment. At 602, the method 600 includes receiving image frames of inventory. In some cases, receiving the image frames may include receiving the image frames of the inventory from a sensor (e.g., a camera) operably coupled to a processor of a user device.

[0088] At 604, method 600 includes locating static control points. In some implementations, locating static control points may include locating control points used to determine the spatial search within the image frame. In some implementations, for example, such static control points may include barcodes (e.g., QR codes), predefined landmarks, etc.

[0089] At 606, the method 600 includes detecting the store units. In some implementations, detecting the store units can include detecting the store units in a spatial search with a machine learning model. Each store unit can be associated with a unit type.

[0090] At 608, the method 600 includes calculating a number of store units. In some implementations, calculating the number of store units may include calculating a number of store units from a set of store unit numbers for each unit type based on a depth calculation.

[0091] In some cases, method 600 may include identifying sets of vertical rows of store units of the same unit type. For example, an inventory may include shelves made up of horizontal rows of store units, and these store units may be organized as vertical rows within the horizontal rows. Method 600 may include calculating, for each vertical row, the number of store units in that vertical row via a depth calculation. Method 600 may further include generating, for each vertical row, an icon representing the number of store units for that vertical row. Method 600 may include calculating, for each unit type, the number of store units based on a sum of the counts of the store units per vertical row for that unit type.

[0092] In some embodiments, the method 600 may include focusing, via a sensor, on an icon representing a count of store units for a vertical column, and allowing a user to manually update the count of store units for the vertical column in response to focusing on the icon for a predetermined period of time.

[0093] At 610, method 600 may include generating a digital model. In some implementations, generating a digital model may include generating a digital model from the set of digital models such that each digital model is overlaid around a different storage unit from the set of storage units. In some cases, method 600 may further include, for each subset of digital models from the set of digital models that is associated with a unit type, generating a digital label to be overlaid on the subset of digital models. The digital label may include a description of the subset of digital models. In some implementations, method 600 may include focusing, via a sensor, on a digital label of a subset of digital models from the plurality of digital models and allowing a user to manually update the digital label in response to focusing on the digital label for a predetermined period of time.

[0094] At 612, method 600 may include determining whether a spatial overlap exists between the digital models. In some cases, this may include determining a particular overlap based on at least an overlap between areas surrounding one or more digital models. If an overlap is determined, a duplicate store unit may be identified as existing.

[0095] At 614, the method includes updating the store unit count. In some implementations, updating the store unit count can include updating the store unit count associated with the duplicate store unit. For example, the store unit identified as a duplicate can be removed from the store unit count.

[0096] Although not shown in FIG. 6 , in some cases, method 600 may include (1) determining a replenishment status of unit types based on the number of store units for each unit type and (2) planogram compliance for each unit type via a depth calculation. Method 600 may further include automatically generating a replenishment request for each unit type from the plurality of unit types based on the replenishment status and planogram compliance. In some cases, the depth calculation may be based on stored depth values ​​associated with each unit type and control point. In some cases, planogram compliance may be set by a supplier of store units. For example, a dispensing device may have shelves assigned with a particular type of store unit and a desired quantity of store units. Method 600 may include determining the replenishment status based on the desired quantity of store units assigned via planogram compliance. In some cases, rather than a user counting store units individually, missing store units may be automatically identified based on a scan of a location on a shelf intended to store a particular quantity (or desired quantity) of store units and the remaining space at that location.

[0097] In some implementations, a user device can download a software application for implementing the functions and processes of method 600 described herein. This software application can be downloaded onto multiple user devices. Such user devices can be structurally and / or functionally similar to user device 101 of FIGS. 1 and 2 and can be operatively and / or communicatively coupled to other user devices, management device 170, and / or server 180 via network 190 (see, e.g., FIG. 2 ). Thus, multiple user devices can scan inventory, and server 180 can process scan results from multiple user devices. This allows server 180 to remove duplicate scan results, reconcile scan results across user devices, etc.

[0098] 7 is a flow diagram of a method 700 for a machine learning system for edge computer vision and active reality, according to one embodiment. At 705, method 700 includes receiving, from a sensor, a detection of a first control point for determining a first spatial search of a first inventory. In some implementations, for example, such a first control point may include a barcode (e.g., a QR code), a predefined landmark, an identifier or other identifying indicia indicating a first area (e.g., a first shelf, a first rack, etc.) for the inventory.

[0099] At 710, the method 700 includes detecting a first plurality of store units to calculate a store unit number from the first plurality of store unit numbers by the machine learning model and based on the depth calculation of the first spatial search, wherein each store unit number from the first plurality of store unit numbers is associated with a unit type from the plurality of unit types.

[0100] At 715, method 700 includes generating a digital model from the first plurality of digital models, the digital model being superimposed around each storage unit from the first plurality of storage units. Such digital model may be presented to a user in an augmented reality display. Further, such digital model may be used to identify duplicates.

[0101] At 720, method 700 includes storing the first inventory data in a database such that the first plurality of digital models are hidden. Similarly, in some implementations, when a user stores and / or "checks out" a particular inventory, shelf, rack, and / or area, the digital models are no longer displayed in the augmented reality display.

[0102] At 725, method 700 includes receiving, from the sensor, detection of a second control point for determining a second spatial search for the second inventory. In some implementations, for example, such a second control point may include a barcode (e.g., a QR code), a predefined landmark, an identifier or other identifying indicia indicating a second area for the inventory (e.g., a second shelf, a second rack, etc.).

[0103] At 730, the method 700 includes detecting a second plurality of store units to calculate a store unit number from the second plurality of store unit numbers by the machine learning model and based on the depth calculation of the second spatial search, wherein each store unit number from the second plurality of store unit numbers is associated with a unit type from the plurality of unit types.

[0104] At 735, method 700 includes generating a digital model from the second plurality of digital models, the digital model being superimposed around each store unit from the second plurality of store units. In some cases, the first inventory data includes metadata for the first plurality of store units, a number of the first plurality of store units, and the first plurality of digital models. In some cases, method 700 may include storing the second inventory data in a database, the second inventory data including metadata for the second plurality of store units, a number of the second plurality of store units, and the second plurality of digital models. In some cases, method 700 may further include distinguishing between the first plurality of store units and the second plurality of store units having the same unit type.

[0105] In some implementations, a user device can download a software application for implementing the functions and processes of method 700 described herein. This software application can be downloaded onto multiple user devices. Such user devices can be structurally and / or functionally similar to user device 101 of FIGS. 1 and 2 and can be operatively and / or communicatively coupled to other user devices, management device 170, and / or server 180 via network 190 (see, e.g., FIG. 2). Thus, multiple user devices can scan inventory, and server 180 can process scan results from multiple user devices. This allows server 180 to remove duplicate scan results, reconcile scan results between user devices, etc.

[0106] FIG. 8 is an example screenshot 800 of inventory captured by a sensor with an active reality icon, according to one embodiment. The active reality (or augmented reality) icon can be matched with other icons as described herein and can include (be) a digital model 804, a representation placed on a store unit, and / or a real-time virtual representation. The active reality icon can also include a store unit label 808. In some implementations, the store unit label 808 can be an identifier for multiple store units in the space label 802 for an associated unit type (e.g., bottled cinnamon powder). The active reality icon can also include a store unit number 812 on the digital model 804 and / or the store unit label 808. In some implementations, the store unit number 812 indicates the total number of store units in the space label 802 and / or the total number of store units having a store unit identifier associated with the store unit label 808. In some implementations, the placement of the icon can be adjusted substantially in real time based on the orientation of the sensor. For example, a sensor can capture image frames of the inventory at multiple angles, and the icon can be adjusted to stay near the spatial storage unit. In some implementations, based on the physical boundaries of the inventory, multiple spatial storage units can be appropriately spaced to form multiple horizontal rows of spatial storage units, as shown in screenshot 800.

[0107] In some implementations, store units can be stacked in multiple ways, and a machine learning model in a user device including a processor operably coupled to a sensor can detect the store units regardless of how they are stacked and / or arranged. For example, as a user navigates the space where the inventory is located while operating the user device, the icon can remain in its original position if the store unit is located in image frames captured from different angles. The machine learning model can also detect and count the store unit if it is facing the front of the sensor or if the side of the store unit is facing the sensor. The machine learning model can generate different calculations regarding the store unit detected via shape analysis based on whether the sensor capturing the image frames of the inventory is facing the front or the side of the inventory. As the sensor capturing the image frames moves around the inventory, the machine learning model can also show store unit data (e.g., icons), such as the store unit number 812, digital model 804, and store unit label 808 for a particular unit type, while hiding store unit data and / or icons of other store units associated with different unit types. For example, if a user wants to see the unit of storage data for cinnamon powder, the machine learning model may display the digital model 804, the unit of storage label 808, and / or the number of storage units 812 for the cinnamon powder, but not other inventory items. Additionally, in some cases, if the sensor captures an image frame from the right side of the inventory, such that a group of peanut butter jars are in front of the cinnamon powder and obscure it, the machine learning model 124 of FIG. 1 or FIG. 2 may filter out and / or hide the unit of storage data and / or icon for the cinnamon powder and display the unit of storage data and / or icon for the peanut butter jar to indicate that the cinnamon powder is behind the peanut butter jar.The machine learning model can recognize that in 3D space, the cinnamon powder is located within spatial label 802 or within the spatial label to the left of the spatial label for the peanut butter jar when viewed from the front of the inventory, thereby recognizing that the cinnamon powder is located behind the peanut butter when viewed from the right side of the inventory. In another example, the machine learning model 124 can hide interfering objects, such as the peanut butter jar, when detecting the cinnamon powder by reducing the visibility of the storage unit data and / or icon for the peanut butter jar.

[0108] FIG. 9 is an example screenshot 900 of inventory with an active reality digital model overlaid on a storeroom unit, according to one embodiment. Screenshot 900 can include a substantially real-time feed of inventory captured by a sensor (e.g., a camera) on a user device. As shown in FIG. 9, screenshot 900 can include a digital model (e.g., 909) overlaid on the storeroom unit. This digital model can include a three-dimensional shape similar to the storeroom unit on which the digital model is overlaid. In some cases, this digital model can encompass similar dimensions to the storeroom unit on which the digital model is overlaid. As shown in FIG. 9, screenshot 900 can include a digital model, such as a spatial label 904, that includes a description of the storeroom unit on which the spatial label is overlaid. As shown in FIG. 9, screenshot 900 can include a digital model, such as an icon 903 representing the storeroom unit number of a storeroom unit associated with the same unit type as that labeled by the spatial label. As shown in FIG. 9, screenshot 900 can include a digital model of a control point 901 that can serve as a reference point in a 3D world defined by a machine learning model. The digital model, including the control points 901, spatial labels 904, and icons 903, can be an object in active reality and / or augmented reality.

[0109] 9, screenshot 900 includes buttons such as edit item button 910, add button 912, and subtract button 913. As shown in FIG. 9, screenshot 900 also includes a reticle 909 that a user can point at via a sensor on a user device. For example, a user can point the sensor so that reticle 909 is focused on space label 904. After reticle 909 has been focused on space label 904 for a predetermined amount of time, the user can click edit item button 910 via a touch operation. 0 The user may select the reticle 909 to allow the user to change the description of the space label 904. In some cases, the space label 904 may be mislabeled, allowing the user to correct the error. In some cases, the user may focus the reticle 909, via a sensor, on an icon 903 representing a store unit number associated with the space label 904. After focusing on the icon for the store unit number icon 903 for a predetermined period of time, the user may use the add button 912 or the subtract button 913 to allow the store unit number icon 903 stored in the database to be updated. If the database recorded an incorrect store unit number, the user may correct the error. The user may use the reticle 909 to focus on a component of inventory, which may cause the processor to add the digital model, control points 901, the icon representing the store unit number, and the space label to a display (e.g., a user interface) on the user device, as shown in FIG. 9 .

[0110] After the user scans relevant objects in inventory, the user can press a checkout button 914 to capture and record the data and / or any changes made by the user in a database. In some cases, when the user presses the checkout button 914, the digital model, including the control points, space labels, and icons representing the number of store units, can disappear and / or not appear. In some cases, pressing the checkout button 914 allows the user to then proceed to scan a different inventory. Before pressing the checkout button 914 (e.g., while still scanning), the digital model of the control points 901, the icons representing the number of store units, and the space labels can be presented in a brighter and / or more vibrant color configuration to indicate a pending and / or active status. In some implementations, if the number of store units is inaccurate, the user can change the statistics on the user device. In some cases, the user can rescan the inventory to correctly scan the store units. In some implementations, the user can tap (or tap multiple times) on the display of the user device to send a signal to the processor or server of the user device to indicate that an error has occurred. The processor can further collect training data for false positives (e.g., duplicate storage units) from inventory scans or manual changes to train the machine learning model. For example, a user can tap the display of the user device multiple times over a period of time (indicating the user correcting multiple errors), during which the processor can transmit a signal indicating an error. The processor can collect training data for false positives based on the pattern of taps on the display to train the machine learning model and improve its accuracy. In some implementations, errors identified by the user can be confirmed and / or verified by the user, thereby providing verified training data.

[0111] In some implementations, a user may install an application that enables sensors on the user device to scan store units in inventory and record information about the store units in inventory. In some cases, the user may terminate the application on the user device (e.g., voluntarily or involuntarily) while the user is using the user device and sensors to acquire data about the store units. In some implementations, data (e.g., store counts, digital model 902, control points 901, coordinates of digital model 902 and control points 901, etc.) may be continuously stored as it is acquired (e.g., in local memory, on a server, in a networked database, etc.) so that when the user resumes the application, the stored data may reappear on the user device's display and / or be re-added, thereby allowing the user to continue scanning the inventory. In some implementations, for example, an SQLite database may be implemented on the user device to maintain persistence.

[0112] FIG. 10 is another example screenshot 1000 of an exemplary inventory with an active reality digital model superimposed on a store unit, according to one embodiment. Screenshot 1000 includes a digital model including icons representing space labels, control points, and / or store unit numbers. As shown in FIG. 10, some of the icons 1022 representing store unit numbers and control points (e.g., QR codes 1012) are displayed in a different style (e.g., monotone colors, different translucencies, etc.) to indicate an inactive state. For example, as shown in FIG. 10, icon 1022 representing store unit numbers has a different hue and / or color compared to icon 903 in FIG. 9, such that the different hue and / or color of icon 1022 in FIG. 10 indicates that the store unit (and associated data) is “checked out,” while icon 903 in FIG. 9 indicates a pending “checkout.”

[0113] When a user "checks out" an inventory scan, the digital model may change from a colored state (e.g., FIG. 9) to a grayed-out state (e.g., FIG. 10) to indicate that this storage unit has already been scanned. In some cases, the user may modify the inventory data for the checked-out inventory to scan additional storage units that may have been missed. In some implementations, after the user makes manual changes and / or corrections, the machine learning model may be automatically further trained with the changes and / or corrections to produce better and more accurate results.

[0114] As shown in FIG. 10 , the inventory can include a QR code 1012 that a user can scan using a sensor to determine inventory data associated with the QR code 1012. For example, the QR code can include and / or point to (e.g., include a reference in a database) information about the inventory (e.g., depth, location, volume, SKU, etc.). In some cases, different locations (e.g., shelves) in the inventory can be associated with QR codes that can indicate assigned unit types. In some cases, the QR code 1012 can be associated with information about the inventory associated with the inventory data (e.g., type of store unit, number of store units, store unit condition, etc.). In some cases, the QR code 1012 can be associated with information describing the dimensions of the inventory, allowing a machine learning model to determine the shelf depth of the inventory and count store units stacked in horizontal and vertical columns. This depth can also allow the machine learning model to determine the maximum and / or desired capacity of the inventory. In some implementations, the QR code 1012 can be associated with information about the location of each spatial label associated with the QR code 1012. For example, different inventory with different QR codes may contain the same storage unit but be located in different locations or on different shelves. Because the QR code 1012 can be unique to the inventory, a processor (or server) in the user device can easily track, determine, and / or identify the storage unit without having to redetect the storage unit for different inventory. In some implementations, the QR code 1012 can be scannable when the user is within a predetermined distance (e.g., set via parameters). In some cases, the parameters can be set to allow the user to scan the QR code 1012 when positioned directly in front of the QR code 1012, but not diagonally to the side.

[0115] In some implementations, the processor can implement periodic automatic replenishment (PAR) to automatically submit and / or prepare replenishment requests for low or empty store units. For example, the QR code 1012 can include information about planogram compliance of various store units from various suppliers, indicating the desired and / or predetermined capacity and / or inventory of a particular store unit. Based on the desired and / or predetermined capacity as defined by the planogram compliance(s), the system can determine whether a unit type is low on inventory and needs to be replenished. For example, a user can use a user device including a sensor to scan the inventory to identify a store unit that is missing from the inventory. In some cases, the user can initially scan the inventory to record a baseline of the entire inventory, including the store unit. The user can rescan the inventory, which allows the processor to identify differences between the initial scan results and subsequent scan results to determine anomalies (e.g., a missing store unit, a new store unit, a misplaced store unit, etc.). The processor can dynamically implement PAR via sensors (e.g., shelf depth, storage unit size, empty space, etc.) to determine the amount of storage units missing per unit type and / or generate replenishment requests to replenish according to various planogram compliances.

[0116] FIG. 11 is an example screenshot 1100 of inventory with an active reality digital model superimposed at the potential location of a store unit, according to one embodiment. In some cases, a store unit may be removed from inventory while inventory data is not updated to indicate a change in the number of store units based on the store unit being removed. As shown in FIG. 11 , a digital model 1109 of a box store unit may be visible at a location in the inventory, indicating that the location is designated for a store unit of the same type as that box. In some cases, a digital model 1109 without a store unit present may indicate a store unit that is missing. In some implementations, this may indicate that the processor may not have had an opportunity to rescan and determine that the store unit is absent.

[0117] 12 is an example screenshot 1200 from a menu list, according to one embodiment. The screenshot may include a menu containing a list of store units for a given store and / or inventory. This menu may allow a user to modify the count of each type of store unit. This may allow a user to manually update the store unit count via a user device.

[0118] 13 is another example screenshot 1300 of inventory with an active reality digital model superimposed on a store unit, according to one embodiment. As shown in FIG. 13, screenshot 1300 can include tracked store units of stacked cups in layers, which can be recorded and stored in a database to determine the status of the store unit (e.g., full, low on stock, etc.). As shown in FIG. 13, stacked cups counted and / or identified for different SKUs can be indicated with different icons. FIG. 14 is another example screenshot 1400 of an inventory with an active reality digital model showing capacity, according to one embodiment. As shown in FIG. 14 , a sensor on a user device can scan inventory, such as bins (e.g., at a buffet and / or salad bar). Each storage unit can be represented as a pyramidal digital model positioned elevated from the storage unit. If the inventory is difficult to view, the digital model within the pyramidal shape can be oriented toward the storage unit. In some cases, a processor can generate a digital model representing the capacity and / or status of the storage unit. As shown in FIG. 14 , storage units that are less than half full (e.g., 42%) can be so indicated. For example, the capacity and / or depth of each bin can be identified (e.g., based on information associated with the associated control points and / or based on depth calculations). The amount of inventory in each bin can be calculated based on depth analysis and displayed as a percentage. This allows for inventory management even with open bins.

[0119] FIG. 15 is a screenshot 1500 of an inventory management dashboard, according to one embodiment. The inventory management dashboard may include a list of unit types 1502, a vertical column for the number of store units per unit type 1504 (also referred to as “count results”), a vertical column for the maximum and / or desired number of store units per unit type 1506 (also referred to as “PAR”), and / or a vertical column for the order status per unit type 1508 (also referred to as “order”). In some cases, the maximum number of store units 1506 may be calculated automatically and / or dynamically based on inventory depth calculations, store unit size, and / or dimensional data from QR codes, etc. The number of store units 1504 vertical column may include the current amount of store units of the associated unit type. The number of PAR store units 1506 vertical column may include the capacity per unit type and / or the preferred capacity per unit type. The order status 1508 vertical column may include the number of replenishment requests and / or the number of store units to be ordered to replenish inventory. In some implementations, a unit type 1502 that includes a pending order for replenishment can be associated with a first identification (e.g., red), and a unit type 1502 that does not have a pending order for replenishment can be associated with a second identification (e.g., yellow).

[0120] In some implementations, the inventory management system can generate reports tailored to different customers. For example, one customer may prefer to receive reports of inventory in units such as pounds, kilograms, and / or ounces. In some cases, another customer may prefer a report that includes the status of the storage unit (e.g., frozen chicken, thawed chicken, fried chicken, etc.). These reports can be customizable. In some implementations, reports can be generated based on information collected by a sensor using object character recognition (OCR). For example, the sensor can scan and read labels, nutritional information, and / or images on the storage unit. Information captured by the sensor using OCR can be stored in a database, making it searchable and usable to generate reports.

[0121] In some implementations, the replenishment of a unit type and / or the prediction of the occurrence of a replenishment request for a unit type can be based on external factors such as, for example, weather, season, day of the week, and / or external events. For example, even if the replenishment request could be fulfilled, the fulfillment of the replenishment request may be affected by future weather, planned events, supply chain issues, etc. In some examples, a unit type needing replenishment may trigger a replenishment request, in which case the machine learning model can predict when to automatically order replenishment for the unit type needing replenishment based on the external factors. For example, the machine learning model can predict that shipping times for a unit type will be longer during the winter season, thereby generating a replenishment request earlier and / or before the unit type runs out of stock.

[0122] The systems and methods described herein allow a user to define a three-dimensional virtual world from their inventory and provide accurate count results for items identified within that inventory. Specifically, a person tasked with counting inventory often checks the shelves and / or inventory, recognizes an item (e.g., a tetra pack of lemonade), counts the number of items behind that item (e.g., the number of lemonades), and records that number (e.g., on paper or in an input application). A similar process can be used for barcode counting. A person can either scan a first barcode (e.g., of an item) and then count the number of items behind the first item, or manually scan each item. Such count results can then be provided to an inventory management system. The systems and methods described herein automate, optimize, and / or improve these processes. Specifically, as described herein, computer vision can be used to identify items (e.g., a tetra pack of lemonade). Spatial intelligence can collect information about a space (e.g., depth, location of objects in front, etc.), define three-dimensional objects and / or models, and automatically count the number of items. The counting results can then be displayed in front of the items in adaptive reality and / or augmented reality. By defining a three-dimensional object in the three-dimensional world for each item, the system can define a space for each item, which fills the space in the three-dimensional world and provides an accurate counting result for the items. Similarly, the systems and methods described herein can enable a system to replicate and / or implement human spatial perception used to count items.

[0123] It should be noted that any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices utilized as user computing devices for electronic documents, one or more server devices, such as document servers) programmed in accordance with the teachings herein. The above-described aspects and implementations employing software and / or software modules may also include appropriate hardware to assist in implementing the machine-executable instructions of the software and / or software modules.

[0124] Examples of computing devices include, but are not limited to, e-book readers, computer workstations, terminal computers, server computers, mobile devices (e.g., tablet computers, smartphones, etc.), web appliances, network routers, network switches, network bridges, any machine capable of executing a sequence of instructions that specify operations to be performed by the machine, and any combination thereof. As an example, a computing device may include and / or be included in a kiosk.

[0125] All combinations of the foregoing concepts and additional concepts described herein are contemplated as being part of the subject matter disclosed herein (provided that such concepts are not mutually inconsistent). Terms explicitly employed herein, and that may also be used in any disclosure incorporated by reference, should be given the meaning that is most consistent with the particular concepts disclosed herein.

[0126] These drawings are primarily for illustrative purposes and are not intended to limit the scope of the subject matter described herein. These drawings are not necessarily to scale, and in some instances, various aspects of the subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference numerals generally refer to like features (e.g., functionally and / or structurally similar elements).

[0127] The entirety of this application (including the cover, title, headings, background, summary, brief description of the drawings, detailed description, embodiments, abstract, drawings, appendices, and the like) illustrates, by way of example, various embodiments in which the embodiments may be practiced. The advantages and features of this application are merely representative examples of embodiments and are not intended to be exhaustive and / or exclusive. Rather, they are presented to aid in the understanding and teaching of embodiments and are not representative of all embodiments. Thus, it is to be understood that other embodiments may be utilized and functional, logical, operational, organizational, structural, and / or topological modifications may be made without departing from the scope of the present disclosure. As such, all examples and / or embodiments are considered non-limiting throughout this disclosure.

[0128] It is understood that the logical and / or topological structure of any combination of any program components (collections of components), other components, and / or any present feature set illustrated in the drawings and / or throughout this specification is not limited to a fixed order and / or arrangement of operations; rather, any orders disclosed are exemplary, and all equivalents regardless of order are contemplated by this disclosure.

[0129] The term "automatically" is used herein to qualify actions that occur without direct input or prompting by an external source, such as a user. Actions that occur automatically may occur periodically, sporadically, in response to a detected event (e.g., a user logging in), or according to a predetermined schedule.

[0130] The term "determining" encompasses a wide variety of actions, and thus "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), and / or ascertaining, etc. "Determining" can also include receiving (e.g., receiving information) and accessing (e.g., accessing data in a memory), etc. "Determining" can also include resolving, selecting, choosing, establishing, etc.

[0131] Unless expressly stated otherwise, the phrase "based on" does not mean "based only on." In other words, the phrase "based on" describes both "based only on" and "based at least on."

[0132] The term "processor" should be interpreted broadly to encompass a general purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some contexts, a "processor" may refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term "processor" may refer to a combination of processing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration.

[0133] The term "memory" should be interpreted broadly to encompass any electronic component capable of storing electronic information. 「 memory 」 The term can refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. Memory is considered to be in electronic communication with a processor if the processor can read information from and / or write information to the memory. Memory that is integral to a processor is in electronic communication with the processor.

[0134] The terms "instructions" and "code" should be interpreted broadly to include any type of computer-readable description. For example, the terms "instructions" and "code" can refer to one or more programs, routines, subroutines, functions, procedures, etc. "Instructions" and "code" can include a single computer-readable statement or multiple computer-readable statements.

[0135] Some embodiments described herein relate to computer storage products with non-transitory computer-readable media (which may also be referred to as non-transitory processor-readable media) bearing instructions or computer code thereon for performing various computer-implemented operations. The computer-readable media (or processor-readable media) is non-transitory in the sense that it does not include a propagated signal itself (e.g., a propagating electromagnetic wave carrying information over a transmission medium such as space or a cable). The medium and computer code (which may also be referred to as code) may be designed and configured for a specific purpose(s). Examples of non-transitory computer-readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as compact disks / digital video disks (CDs / DVDs), compact disk-read-only memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules; and hardware devices specifically configured to store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), read-only memory (ROM), and random access memory (RAM) devices. Other embodiments described herein relate to computer program products that may include, for example, instructions and / or computer code described herein.

[0136] Some embodiments and / or methods described herein may be implemented by software (executed on hardware), hardware, or a combination thereof. Hardware modules may include, for example, general-purpose processors, field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs). Software modules (executed on hardware) may be expressed in various software languages ​​(e.g., computer code), including C, C++, Java™, Ruby, Visual Basic™, and / or other object-oriented, procedural, or other programming languages ​​and development tools. Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions such as those generated by a compiler, code used to create web services, and files containing high-level instructions executed by a computer using an interpreter. For example, embodiments may be implemented using an imperative programming language (e.g., C, Fortran, etc.), a functional programming language (e.g., Haskell, Erlang, etc.), a logic programming language (e.g., Prolog), an object-oriented programming language (e.g., Java, C++, etc.), or other suitable programming language and / or development tool. Additional examples of computer code include, but are not limited to, control signals, encryption code, and compression code.

[0137] Various concepts may be embodied as one or more methods, at least one example of which is provided. Actions performed as part of the methods may be ordered in any suitable manner. Thus, embodiments may be configured in which actions are performed in an order different from that illustrated, including performing some actions simultaneously even though shown as sequential actions in an exemplary embodiment. In other words, it should be understood that such features are not necessarily limited to a particular order of execution, but rather to any number of threads, processes, services, and / or servers, etc., that may be executed serially, asynchronously, concurrently, parallelly, simultaneously, and / or synchronously, etc., in a manner consistent with this disclosure. Thus, some of these features may be mutually inconsistent in that they cannot exist simultaneously in a single embodiment. Similarly, some features may be applicable to some aspects of the invention but not to other aspects.

[0138] The advantages, embodiments, examples, functional, characteristic, logical, operational, organizational, structural, topological, and / or other aspects of the present disclosure, as defined by the embodiments, should not be construed as limiting the disclosure or equivalents thereto. Depending on the particular needs and / or characteristics of individual and / or enterprise users, database configurations and / or relational models, data types, data transmission and / or network frameworks, and / or syntactic structures, etc., various embodiments of the techniques disclosed herein can be implemented in a manner that allows for great flexibility and customization, as described herein.

[0139] All definitions herein and as used herein should be understood to supersede any dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0140] As used herein, in certain embodiments, the term "about" or "approximately," when preceding a numerical value, indicates a range of plus or minus 10% of that value. Where a range of values ​​is provided, unless the context clearly dictates otherwise, it is understood that each intermediate value, in increments of 1 / 10 of the unit of the lower limit between the upper and lower limits of that range, and any other stated value or intermediate value within that stated range, is encompassed within the disclosure. The upper and lower limits of these smaller ranges, which may independently be included within those smaller ranges, are also encompassed within the disclosure, unless there is any specifically excluded limit within the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also encompassed within the disclosure.

[0141] As used herein, "substantially in parallel" can refer to events occurring simultaneously when adjusting for processing-related delays (such as computational delays, transmission delays, etc.), or can refer to events that overlap in time.

[0142] As used herein, "substantially in real time" may refer to an event occurring shortly after the antecedent event, taking into account processing-related delays (e.g., computational delays, transmission delays, etc.).

[0143] As used herein, in the specification and embodiments, the indefinite articles "a" and "an" should be understood to mean "at least one" unless clearly indicated to the contrary.

[0144] As used herein, in the specification and embodiments, the phrase "and / or" should be understood to mean "either one or both" of the elements so conjoined. That is, in some cases, both elements are present, and in other cases, only one or the other is present. Multiple elements listed with "and / or" should be construed in the same manner, i.e., meaning "one or more" of the elements so conjoined. Other elements other than the elements specifically identified by the "and / or" phrase can optionally be present, whether related to those specifically identified elements or not. Thus, as a non-limiting example, a reference to "A and / or B," when used in conjunction with open-ended expression such as "comprising," can, in one embodiment, refer to only A (optionally including elements other than B); in another embodiment, refer to only B (optionally including elements other than A); in yet another embodiment, refer to both A and B (optionally including other elements), etc.

[0145] As used herein, in the specification and embodiments, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted inclusively, i.e., including at least one of a number or list of elements, and further including one or more, optionally including additional items not listed. Only terms that clearly indicate a different meaning, such as "only one of" or "exactly one of," or, when used in embodiments, "consisting of," shall refer to the inclusion of exactly one element of a number or list of elements. Generally, as used herein, the term "or" shall be interpreted as indicating exclusive alternatives (i.e., "one or the other, but not both") only when preceded by terms indicating exclusivity, such as "any of," "one of," "only one of," or "exactly one of." When used in embodiments, the term "consisting essentially of" shall have its ordinary meaning as used in the field of patent law.

[0146] As used herein, in the specification and embodiments, the phrase "at least one," when referring to a list of one or more elements, should be understood to mean at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of every element specifically listed in the list of elements, nor does it exclude any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether related to those specifically identified elements or not. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B," or, equivalently, "at least one of A and / or B") can refer in one embodiment to at least one that optionally includes two or more As and no Bs (and optionally includes elements other than B); in another embodiment, to at least one that optionally includes two or more Bs and no As (and optionally includes elements other than A); in yet another embodiment, to at least one that optionally includes two or more As, and at least one that optionally includes two or more Bs (and optionally includes other elements); etc.

[0147] In embodiments, and throughout the specification above, all transitional phrases such as "comprising," "including," "carrying," "having," "containing," "involving," "holding," and / or "composed of" shall be understood to be open-ended, i.e., to mean "including, but not limited to." Only the transitional phrases "consisting of" and "consisting essentially of" shall be closed or semi-closed transitional phrases, respectively, as defined in the U.S. Manual of Patent Examining Procedures, Chapter 2111.03.

Claims

1. a processor of the user device; a memory operably coupled to the processor, the memory configured to cause the processor to: receiving a plurality of image frames of the inventory from a sensor operably coupled to the processor; locating control points used to determine a spatial search in the plurality of image frames; detecting, with a machine learning model, a plurality of storage units in the spatial search, wherein each storage unit from the plurality of storage units is associated with a unit type from a plurality of unit types; calculating a number of store units from a plurality of store unit numbers for each unit type from the plurality of unit types from the plurality of store units detected based on the depth calculation; Identifying duplicate store units based on the plurality of store unit counts; removing the duplicate store unit from the store unit count associated with the duplicate store unit; determining a replenishment status for each unit type from the plurality of unit types based on the number of store units for each unit type; the memory storing instructions for automatically generating a replenishment request based on the replenishment status; An apparatus comprising:

2. The memory may be configured to: generating a plurality of icons for each storage unit from the plurality of storage units for display on the user device via augmented reality, the plurality of icons comprising: a storage unit label for each unit type from the plurality of unit types; a number of said store units for each unit type from said plurality of unit types; a memory type color based on a memory type from a plurality of memory types for each unit type from the plurality of unit types; 10. The apparatus of claim 1, further storing instructions for causing the user device to display the icons displayed in front of the storage units in the image frames.

3. The apparatus of claim 2 , wherein the memory stores instructions that cause the processor to generate the control points and the plurality of icons for the spatial search.

4. The device of claim 2 , wherein the memory stores instructions that cause the processor to adjust the placement of the plurality of icons based on the orientation of the sensor in substantially real time.

5. 4. The apparatus of claim 3, wherein the memory storing instructions causing the processor to generate the plurality of icons for the control points further comprises instructions causing the processor to generate a plurality of icons for spatial labels for each subset of storage units associated with each unit type.

6. The memory storing instructions causing the processor to calculate the number of store units for each unit type from the plurality of unit types further comprises: identifying the spatial label for each subset of store units from a plurality of subsets of store units from the plurality of store units; 6. The apparatus of claim 5, further comprising instructions for causing the machine learning model to calculate the number of store units for each subset of store units associated with each unit type.

7. The memory storing instructions for causing the processor to find the plurality of storage units in the spatial search further comprises: identifying, via the sensor, a store unit identifier from a plurality of store unit identifiers provided on each store unit from the plurality of store units; 10. The device of claim 1, further comprising instructions for causing the machine learning model to calculate the number of store units per unit type based on the store unit identifiers from the plurality of store unit identifiers.

8. the plurality of store units is a first plurality of store units; The memory may be configured to: recording in the memory the plurality of store unit identifiers and control points associated with the first plurality of store units; 8. The apparatus of claim 7, further storing instructions for identifying a second plurality of store units based on the control points and based on the plurality of store unit identifiers stored in the memory.

9. 2. The apparatus of claim 1, wherein the memory storing instructions causing the processor to calculate the number of store units for each store unit from the plurality of store units further comprises instructions causing the processor to automatically calculate the number of store units for each store unit from the plurality of store units.

10. 2. The apparatus of claim 1, wherein the memory storing instructions for causing the processor to determine the replenishment status of each unit type from the plurality of unit types based on the number of store units further comprises instructions for causing the processor to automatically determine the replenishment status of each unit type from the plurality of unit types.

11. The apparatus of claim 1 , wherein the sensor is not fixed and is configured to capture the plurality of image frames substantially in real time.

12. The device of claim 1 , wherein the control point is a Quick Response (QR) code.

13. The memory may be configured to: generating a digital inventory dashboard based on the number of store units and the replenishment status of each unit type from the number of unit types; The apparatus of claim 1 , further storing instructions for causing the digital inventory dashboard to be displayed on the user device.

14. The apparatus of claim 1 , wherein the sensor is a LIDAR sensor.

15. 2. The apparatus of claim 1, wherein the memory storing instructions for causing the processor to detect the plurality of storage units in the spatial search using the machine learning model further includes instructions for identifying each storage unit, wherein each unit does not have a predefined orientation.

16. 10. The apparatus of claim 1, wherein the memory stores instructions that cause the processor to train a machine learning model using a training set to generate a trained machine learning model, the training set including an augmented storehouse digital model that correlates to identification information of the storehouse.

17. the unit type is a first unit type, 2. The apparatus of claim 1, wherein the memory stores instructions that cause the processor to filter out a subset of store units associated with a second unit type before calculating the number of store units for the first unit type.

18. receiving a plurality of image frames of the inventory from a sensor operably coupled to a processor of the user device; locating control points used to determine a spatial search in the plurality of image frames; detecting, with a machine learning model, a plurality of store units in the spatial search, wherein each store unit from the plurality of store units is associated with a unit type from a plurality of unit types; calculating a number of store units from a plurality of store unit numbers for each unit type from the plurality of unit types from the plurality of store units detected based on the depth calculation; generating a digital model of a plurality of digital models, each digital model from the plurality of digital models being superimposed around a different store unit from the plurality of store units; determining overlapping storage units based at least on an overlap between areas surrounding one or more digital models; updating the store unit count associated with the duplicate store unit; A method comprising:

19. calculating the number of store units, identifying a plurality of vertical rows of store units of the same unit type from the plurality of unit types; calculating, via a depth calculation, for each vertical row from the plurality of vertical rows, a number of store units in that vertical row; generating, for each vertical column from the plurality of vertical columns, an icon from a plurality of icons representing a number of store units for that vertical column; calculating, for each unit type from the plurality of unit types, the number of store units based on a sum of the number of store units for each column from the plurality of columns for that unit type; 20. The method of claim 18, comprising:

20. focusing via the sensor on the icon representing the number of store units for a vertical row; allowing a user to manually update the number of store units for the vertical column in response to focusing on the icon for a predetermined period of time; 20. The method of claim 19, further comprising:

21. generating, for each subset of digital models from the plurality of digital models, the subset of digital models being associated with a unit type from the plurality of unit types, a digital label to be overlaid on the subset of digital models, the digital label including a description of the subset of digital models; focusing via the sensor on the digital labels of a subset of digital models from the plurality of digital models; enabling a user to manually update the digital label in response to focusing on the digital label for a predetermined period of time; 20. The method of claim 18, further comprising:

22. (1) determining a replenishment status for each unit type from the plurality of unit types based on the number of store units for that unit type; and (2) determining planogram compliance for each unit type from the plurality of unit types via the depth calculation. automatically generating a replenishment request based on the replenishment status for each unit type from the plurality of unit types and the planogram compliance; 20. The method of claim 18, further comprising:

23. The method of claim 18 , wherein the depth calculation is based on stored depth values ​​associated with each unit type from the plurality of unit types and the control point.

Images