Apparatus for automating inventory confirmation, as well as an automatic inventory confirmation system and method

Abstract

An apparatus for automating an inventory procedure for items stored on a shelf in a closed environment includes a mobile mechanical device having a movable appendage including a camera and additional sensory modalities. The camera and additional sensory modalities are used to position the movable appendage to take camera images of the items on the shelf from many different perspectives. The camera vision and additional sensory modalities may be used to rotate, lower and raise the movable appendage to position the appendage on and along the side of the items on the shelf. A context of the mobile mechanical device is determined and edge AI computing retrieves an AI context-specific model based on the context. The AI ​​context-specific model may be downloaded from a cloud service. The edge AI computing uses the AI ​​context-specific model to identify and count the items on the shelf.

Description

Technical Field

[0001] The present disclosure relates to computers and computer applications, and more particularly, to a computer-implemented method and system for inventory control, and in particular to performing inventory verification operations using robots.

Background Art

[0002] Systems are available for maintaining an inventory of products in a store, typically by maintaining a count of specific product stock-keeping units (SKUs). Counts of various items are maintained in a database, which may also store other attributes of the products. However, it is often necessary to compare the count of items stored in the inventory with the physical count of the objects in the store. This process of reconciling the physical inventory with the logical inventory stored in the database is manual and time-consuming.

[0003] There are existing tools for automating the counting of various types of objects. For example, there are applications on mobile phones that can be used to scan tags and labels on physical objects and store the tags and labels in an inventory verification system. However, such applications require manual operation to cover the physical store, which is time-consuming and costly.

[0004] It is possible to use a camera on a robot to scan objects existing outside, fly a drone, or take a photo of an external container using satellite imagery. These are useful and applicable when the items to be counted can be clearly identified from a vantage point where the camera has a good view. However, there are many cases where the vantage point of the camera is blocked from the complete view of the image. As an example, a satellite cannot easily estimate how many containers are stacked on top of each other. Cameras operating in the visible electromagnetic spectrum are blocked by non-transparent objects, while cameras operating in the ultraviolet electromagnetic spectrum can be partially blocked by some object depending on its position.

[0005] When attempting to examine the inventory inside a closed space using either a stationary camera or a moving camera, there are numerous problems to overcome. For example, in a grocery store, there will be obstacles to the view for many objects. The camera on a mobile robot (or a stationary camera at a single location) may be able to see the first object in a shelf, but will not be able to see the object behind the first object in the same shelf. This makes it impossible to automatically count the items.

[0006] A store can deploy many types of robots and attach motor assemblies and cameras to the robots. The robots can have cameras or other sensors and can move around the store. However, even if the robots are good at some tasks such as detecting spills in the aisles or performing other types of visual observations, the state-of-the-art robots cannot overcome the problems caused by the view of the products on the shelves being obstructed, and thus cannot perform well in counting the inventory of various items among the stocked goods.

[0007] Considering such problems, at present, there is no good method for examining the inventory of physical assets in a closed space that is small or can easily count objects that are usually arranged in such a way that their view is blocked by any type of stationary or mobile camera. Summary of the Invention

[0008] In one embodiment, an apparatus for automating an inventory verification procedure for items stored on shelves within a closed environment includes a mobile robotic device having at least one movable appendage, the at least one movable appendage including at least one camera and at least one additional sensory modality. The apparatus includes a control module that uses the at least one camera and the at least one additional sensory modality to position the at least one movable appendage to capture camera images of items on the shelves within the closed environment from a plurality of different viewpoints, and an edge AI computing module. The edge AI computing module uses context information to dynamically search for an AI context-specific model based on the context. The edge AI computing module uses the AI context-specific model to identify and count the items on the shelves. In one embodiment, the control module uses camera vision and at least one additional sensory modality to position the appendage above and along the sides of the items on the shelves. In one embodiment, the AI context-specific model is downloaded from a cloud hosting service. In one embodiment, the AI context model is used to determine the context of the mobile robotic device. In one embodiment, the movable appendage can be rotated about its longitudinal axis and lowered and raised along the vertical dimension of the mobile robotic device. In one embodiment, the item count is uploaded to an inventory management system.

[0009] One embodiment of a computer-implemented method for automating an inventory verification procedure for items stored on a shelf within a closed environment provides a mobile robotic device comprising a movable appendage having a sensory modality including a camera, controls the sensory modality to position the appendage to image items from a number of different viewpoints, determines the context of the mobile robotic device, retrieves an AI context-specific model based on the context, and uses the AI context-specific model to identify and count the items. In some embodiments, the method also includes positioning the appendage over and along the sides of the item, downloading the AI context-specific model from a cloud service, determining the context of the mobile robotic device using the AI context model, rotating the appendage about a vertical axis, lowering and raising the appendage along a vertical dimension of the mobile robotic device, and uploading the item count to an inventory management system.

[0010] A system may also be provided that includes one or more processors operable to perform one or more of the methods described herein.

[0011] A computer-readable storage medium may also be provided that stores a program of machine-executable instructions to perform one or more of the methods described herein.

[0012] Other features as well as structures and operations of various embodiments are described in detail below with reference to the accompanying drawings. In the drawings, like reference numerals indicate identical or functionally similar elements.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0014] In a closed environment such as a retail store, warehouse, or order processing center, there are multiple racks, and each rack consists of multiple shelves on which various items are stacked. The environment typically has several aisles, and each aisle has a set of shelves with items stacked on top of each other. As a result, when a human or machine assigned to check the inventory of items looks at the shelves and products, only the top product, or at most the first few products stacked on the shelf, will be visible. An inventory checker or machine relying solely on vision will not be able to see or count products located deep behind the first object, and thus will not be able to determine how many products are on the shelf.

[0015] In one embodiment, an apparatus and system are disclosed that automate the physical inventory checking of items stored in a closed environment and overcome the problem of products being blocked from view. In one embodiment, a mobile mechanical device, such as a robot that can move around within a closed space, is equipped with at least one camera mounted on at least one attachment. In one embodiment, the at least one attachment also comprises one additional sensory modality, such as a microphone, touch sensor, or weight sensor, or a combination thereof. In one embodiment, the mobile robot uses camera vision, additional sensory modalities, and position information to position the robot at various locations and at various angles, and to position the camera on the attachment, such that the robot can see and count all the objects on the shelf. The modalities can also include a laser, LiDAR, structured light (active stereo) sensor, passive stereo sensor, scanner, infrared sensor, acoustic sensor, radio frequency identification receiver, or some combination thereof.

[0016] By using multiple modalities, the robotic appendage can properly position itself, adjust the positions of cameras and other sensors, and accurately count all the objects on the shelf. To avoid the problem of items being blocked from the field of view, microphones and touch sensors are used to position the appendage at various locations together with the camera. As an example, the robot can move and position the camera deep into the shelf and position the appendage above and along the sides of the item to obtain views of the object from multiple vantage points using sound and touch. From these vantage points, the robot can accurately count the number of items present in the shelf, thereby automatically checking the physical inventory. As a result, the devices and systems can easily populate the inventory management system and compare the physical inventory with the logical inventory.

[0017] In one embodiment, the robot includes a computer processing system that analyzes images and information obtained from other modalities to count items. In one embodiment, the computer processing system uses artificial intelligence (AI) programs such as various machine learning and deep learning programs. Many different types of AI models are required to perform the analysis and counting for the purpose of accurately counting many types of items in an environment such as a store, warehouse, processing center, etc. Since each product has different images and different types of boxes, such AI models should be product-specific. Having a single AI model that can detect and count all types of products is extremely complex to design and will have a very high error rate when there are a large number of objects.

[0018] To address this problem, in one embodiment, a robot computer processing system is configured as an edge AI computing system. In the edge AI computing system, all models for counting and detecting products are stored in a cloud-hosted service or a central service, but all calculations using the models are performed by the processing system of the robot 12. The central service for AI model hosting has a map / floor plan of the store and can determine what types of products are stored at each aisle location. Using the location of the products within the store floor plan to know what types of products are stored in each aisle, the processing system of the robot determines the context for its operation. Then, based on the determined context, the robot processing system downloads the correct AI context-specific model for counting and detecting the products stored in that aisle. When the robot moves to a different aisle or the context changes (e.g., the set of products changes in the middle of an aisle), the processing system 22 searches for the correct set of AI models using the new context.

[0019] In one embodiment, as shown in FIG. 1, an automatic inventory counting system 10 includes a robot 12. FIG. 1 is a front upright view of the robot 12. The robot 12 has a main body 14 and a movement drive part 16 that enables the robot 12 to move around the environment, move to each aisle, move up and down the aisle, and face the shelves on either side of the aisle. In one embodiment, the movement drive part 16 has omnidirectional wheels 18. In one embodiment, the main body 14 may include at least one camera 20. The embodiment shown in FIG. 1 has two cameras 20 within the main body 14. The robot 12 may also include a computer processing system 22 configured to control the movement of the robot and perform data processing related to automatically checking the physical inventory. An example of such a computer processing system is disclosed in FIG. 8 described below.

[0020] In one embodiment, the robot 12 has at least one movable appendage 24. The embodiment shown in FIG. 1 has two appendages 24 attached to the main body 14. In one embodiment, the appendage 24 can be arranged to rotate with respect to the main body 14. In one embodiment, each appendage 24 is attached to the main body 14 by a rotatable connection 26. The appendage 24 is shown in a translation position parallel to the side surface of the main body 14. In one embodiment, the rotatable connection 26 may be foldable, whereby the appendage 24 can be folded inwardly against the side surfaces 25 and 27 of the main body 14 to minimize the width footprint of the appendage 24 and, if necessary, the robot 12 can be easily navigated within the passageway.

[0021] In one embodiment, the appendage 24 can be lowered or raised to various positions between the upper end 21 and the lower end 23 of the main body 14. In one embodiment, the rotatable connection 26 can be slidable along the side surfaces 25 and 27 of the main body 14. Three different operating positions of the appendage 24 are shown in FIG. 2. In position A, the connection 26 is located near the upper end of the main body 14 and the appendage 24 rotates 90° from the translation position. In position B, the appendage 24 moves along the track 28 to a position midway between the upper end 21 and the lower end 23 of the main body 14. In position C, the appendage 24 moves along the track 28 to a position within the track 28 near the lower end 23 of the main body 14.

[0022] As shown in FIG. 3, in one embodiment, each appendage 24 may include a touch sensor 30, a microphone 32, and an appendage camera 34. The touch sensor 30 is shown as being located at an end 36 of the appendage 24, and after the appendage 24 rotates 90° from the moving position, the touch sensor 30 will be located farthest from the robot main body. In one embodiment, the appendage 24 may include a gripping mechanism and a weight sensor for picking up items and counting the number of items using weight. The microphone 32 and the appendage camera 34 are exposed on a surface 38 of the appendage 24 that will face downward toward the floor after the appendage 24 rotates 90° from the moving position.

[0023] To examine the physical inventory of items on a shelf in a closed environment, robot 12 will move up and down each aisle according to a floor plan of the environment controlled by a navigation program installed on processing system 22. When robot 12 positions itself in front of each shelf in the aisle, robot 12 raises its appendage 24 and moves the appendage inside the shelf. For example, as shown in FIG. 4, appendage 24 is positioned above the upper ends of items 40 stacked on top of each other on shelf 42. With appendage 24 positioned at the upper end of item 40, as a result, camera 34 faces the upper end of item 40 and camera 34 can acquire a digital image of item 40 that can be analyzed to count item 40. In one embodiment, one or more of appendages 24 can also be positioned between two rows of items on the shelf. The appendage then rotates about its vertical axis, such that the modality sensor, camera 34, and microphone 32 face the side of the item and camera 34 can acquire a digital image of the side of item 40. To handle objects of various sizes in the shelf, appendage 24 can be raised or lowered as robot 12 moves down the aisle. Based on a signal obtained from touch sensor 30, robot 12 can determine that appendage 24 has contacted or collided with an obstacle. The sound captured by microphone 32 can be analyzed and it can be determined that appendage 24 has contacted or collided with an obstacle. For example, based on a signal from touch sensor 24, robot 12 can determine that appendage 24 has reached the back of the shelf. The robot can then rotate the appendage 180° to acquire an image of the items on the other side of the row as robot 12 retracts from the back of the shelf. The touch signal from touch sensor 30 can also be used to control the robot to avoid contacting or dragging any item on the shelf and the acoustic signal from microphone 32. In an alternative embodiment, LIDAR-based sensing can be used to determine when the appendage has reached the back of the shelf.

[0024] Robot 12 analyzes images taken from camera 34 within attachment 24 and front camera 20 within main body 14 to determine the type of product and count the products. This counting can be done by executing an image processing algorithm on the images. Other techniques for counting items or measuring the quantity of items may also be used. For example, touch sensor 30 can be used, and by using the pressure on the touch sensor, the spring load stack of the items can be counted and the number of items can be counted. When touch sensor 30 includes a weight sensor on the attachment, the attachment picks up the items and uses the weight to count the number of items. Robot 12 then moves to another position within the passageway or to the next passageway to count another set of objects.

[0025] FIG. 5 is a flowchart of an embodiment of a processing system 22 that controls the robot 12 to move around a store aisle and count all of the products on the shelves. In step S1, after the robot 12 is controlled to move to a specific aisle according to a navigation program, the robot 12 acquires an image from the camera 20 within the main body 14, and then the processing system 22 analyzes the view of the aisle from the front of the robot 12. In step S2, based on the stored floor plan and the type of product in the aisle determined from the signal acquired from the camera 20, the processing system 22 determines which position of the attachment 24 to use to count the items in that specific aisle. In step S3, the system checks whether all of the attachment positions determined in step S2 have been analyzed. If the determination in step S3 is no, the method proceeds to step S4. In step S4, the attachment 24 is activated and moved to a first position in the shelf using various modalities on the attachment 24 including the touch sensor 30 and the microphone 32, and the camera 34 is positioned so as to be able to acquire an image of the item on the shelf. In step S5, the processing system 22 determines the field of view detected by the robot 12 and the attachment 24. In one embodiment, in step S5, the processing system 22 controls the camera to acquire images from both the front view camera 20 and the attachment camera 34. The processing system 22 determines whether the attachment camera 34 is positioned to view the side view of the item or the top view of the item. The processing system controls the camera 34 to acquire an image of the item. Then, the acquired image is analyzed in step S6, and a count of the items on the shelf at that specific attachment position is obtained. Then, the item count is transferred to an inventory management system and the information within the inventory management system is updated.

[0026] In step S5, after an image is acquired at a specific attachment position, the method also returns to step S3 and determines that images have been acquired at all the positions determined in step S2. If all the positions are not completed, as described above, the process repeats steps S4 to S6 until all the attachment positions in its first passage are completed. When it is determined in step S3 that all the attachment positions in its first passage are completed, the process proceeds to step S7 and the robot 12 moves to the next passage. In step S8, the system determines whether all the passages are completed. If all the passages are not completed (No in S8), the method returns to step S1 and steps S1 to S8 are repeated until all the passages are completed. When all the passages are completed in step S8, the method ends in step S9.

[0027] The analysis of the images for counting items in step S6 requires the use of an artificial intelligence (AI) program. There are many types of items in environments such as stores, warehouses, processing centers, etc. To create a good counting model and identify products, the robot 12 requires many different types of AI models to perform the analysis and counting. Since each product has different images and different types of boxes, such AI models are specific to the product. Having a single AI model that can detect and count all types of products is extremely complex to design and, with a large number of objects, the error rate becomes very high.

[0028] To address this issue, in one embodiment, the processing system 22 is configured as an edge AI computing system. Edge computing is a distributed computing paradigm that brings computing and data storage closer to devices such as the processing system 22 of the robot 12 where data is being collected. Compared to relying on a central location such as the cloud, edge computing enables real-time data to be free from bandwidth and latency issues that affect app performance. By moving the computing to the network edge, long-distance communication between the client and the server is reduced here. Edge AI computing is running an AI algorithm locally on a hardware device that uses edge computing, and the AI algorithm is based on data created on the device without requiring any connection. This enables data to be processed in less than a few milliseconds, thereby providing real-time information to the user. The AI processing uses a deep learning model, and with edge AI computing, data can be created before being sent to a remote location for further analysis.

[0029] In one embodiment of the edge AI computing system, all models for counting and detecting products are stored in a cloud-hosted service or a central service, but all calculations using the models are performed by the processing system 22 of the robot 12. The central service for AI model hosting has a map / floor plan of the store and can determine what types of products are stored at each aisle location. Using the product positions within the store floor plan to know what types of products are stored in each aisle, the processing system 22 of the robot 12 determines the context for its operation. Then, based on the determined context, the robot processing system 22 downloads the correct AI model for counting and detecting the products stored in that aisle. When the robot moves to a different aisle or the context changes (e.g., the set of products changes in the middle of an aisle), the processing system 22 uses the new context to search for the correct set of AI models.

[0030] An illustration of the cloud-hosted system is shown in FIG. 6. The processing system 22 of the robot 12 communicates with the AI model service 50 via a communication network 52 such as, for example, a local area network, a wide area network, the Internet, the cloud, etc. The AI model service 50 includes one or more counter models 54, one or more type models 56, and one or more context models 58. The processing system 22 of the robot 12 also communicates with the central inventory management service 60 via the communication network 52.

[0031] FIG. 7 is a flowchart of an embodiment of the operation of a context-specific AI model search system that enables the robot 12 to perform more accurate counting. At step S20, the context of the robot 12 is determined. In one embodiment, the processing system 22 determines the context for the operation of the processing system 22 based on the position of the robot within the store floor plan and what types of products are stored in each aisle. In one embodiment, the context is determined by user input. In one embodiment, the processing system 22 downloads a context model 58 from the AI model service 50 that will be used to predict the context of the robot 12. The context model determines what the context of the robot is. The context model takes inputs such as the position of the robot, the direction of the robot (e.g., which rack the robot is looking at), etc., and predicts the context. The context may also include a list of modalities that the robot may wish to use (e.g., using visual and weight sensors, or only one of them). There may be multiple contexts and multiple context models. For example, when the robot is within the gaming section of the store, the robot may wish to use a different model to predict the context of the robot, and when the robot is within the grocery section, the robot may use a different context model.

[0032] In step S22, the processing system 22 determines the context-specific model to be downloaded from the AI model service 50 based on the context. In one embodiment, the context model 58 determines the AI type model or the AI counter model or both that are required for the robot to count items. The context-specific model may include the type model 56 and the counter model 54. In one embodiment, based on the context determined in step S20, which includes the position of the robot 12 within the store 44 and the type of product that is expected to be counted on the shelf at that particular position, the processing system 22 selects an appropriate type model 56 for predicting the type of product to be counted. The type model predicts the type of product (e.g., the number of SKUs) based on sensor inputs such as the camera 34. In some embodiments, there may be multiple type models 56, for example, a type model that determines the type based on a vision algorithm for recognizing boxes, a type model that determines the type based on a vision algorithm optimized for the shape of cans, etc. The type model 56 to be used is determined by the context predicted by the context model 58. The counter model downloaded in step S22 is based on the type of product to be counted, which is determined by either the context model 58 or the type model 56. For example, if the context of the robot 12 is that the robot 12 is within aisle 10 of the store and aisle 10 contains over-the-counter pain relievers, the robot 12 downloads an AI counter model 54 that can count the bottles of pain relievers. If the context of the robot 12 is that the robot 12 is within aisle 20 of the store and aisle 20 contains boxes of popcorn, the robot 12 downloads an AI counter model that can distinguish between different types of popcorn boxes and count them.

[0033] In step S26, the processing system 22 determines whether the context determined in step S20 has changed. If the context has not changed and the answer is no in step S26, the method proceeds to step S28.

[0034] In step S28, the processing system 22 counts the items using the downloaded count model.

[0035] In step S30, the processing system 22 uploads the item count to the central inventory management service 60. After the upload, the method returns to step S26 to check whether the context of the robot 12 has changed. If the context has changed and the answer is yes in step S26, it indicates that additional items should be counted, and the method returns to step S20.

[0036] In one embodiment, as shown in FIG. 8, the processing system 22 includes a control system 62, a context module 63, a transmission module 64, a receiver module 66, a tracking system 68, an obstacle avoidance system 70, a mapping system 72, a mobile platform 74, a power system 76, an edge AI computing module 78, and an attachment positioning system 80. The power system 76 includes a rechargeable battery that supplies power to the mobile inventory verification robot 12 and a docking port for charging the battery, and the battery can be recharged by docking the inventory verification robot 12 with a stationary power docking station (not shown).

[0037] In one embodiment, the context module 63 determines the context of the robot 12 based on the position of the robot within the store floor plan and which types of products are stored in each aisle. In one embodiment, the context module 63 determines the context based on user input. In one embodiment, the edge AI computing module 78 dynamically selects and uses an appropriate AI model from the AI model service 50 for detection and counting according to the context, using the position and other context information. In one embodiment, the edge AI computing module 78 downloads a context model 58 from the AI model service 50 that will be used to predict the context of the robot 12. In one embodiment, the transmission module 64 transmits the selection of the context-specific model to the AI model service 50. The download module 66 downloads the selected context-specific model from the AI model service 50. The control system 62 uses the feedback provided by the tracking system 68 to control the movement of the mobile inventory verification robot 12 along the inventory map stored in the mapping system 72. The tracking system 68 tracks the position of the mobile inventory verification robot 12 within a building such as a retail store. In one embodiment, the tracking system 68 uses feedback by tracking tags such as retroreflective tags on the ceiling of the building or infrared waypoints arranged on the floor of the building. The control system 62 further controls the movement of the mobile inventory verification robot 12 using an obstacle avoidance system 70 that detects obstacles in the path of the mobile inventory verification robot 12, including people, shopping carts, boxes of items to be stacked on the shelves, and floor displays. In one embodiment, an ultrasonic rangefinder on the robot 12 emits and receives ultrasonic signals, enabling the mobile inventory verification robot 12 to detect and avoid obstacles. The mobile platform 74 includes a motor, a mobile drive part 16, and wheels 18.

[0038] The inventory map stored within the mapping system 72 further includes an inventory layout diagram that defines the order in which the shelves and the items within the shelves are imaged and counted. When the mobile inventory verification robot 12 moves around the inventory map, images of the items on the shelves are captured in a predetermined order, and this predetermined order is the inventory layout diagram.

[0039] FIG. 9 is a flowchart of one embodiment of a method for automating an inventory verification procedure for items stored on shelves within a closed environment. The method includes step S10 of providing a mobile robotic device having a movable appendage with a sensory modality including a camera, step S12 of controlling the sensory modality to position the appendage to image items from a plurality of different viewpoints, step S14 of determining the context of the mobile robotic device, step S16 of retrieving an AI context-specific model based on the context, and step S18 of using the AI context-specific model to identify and count the items. In some embodiments, the method may also include one or more of step S20 of positioning the appendage on and along the sides of the item, step S22 of downloading the AI context-specific model from a cloud service, step S24 of determining the context of the mobile robotic device using the AI context model, step S26 of rotating the appendage around a vertical axis, step S28 of lowering and raising the appendage along the vertical dimension of the mobile robotic device, and step S30 of uploading the item count to an inventory management system.

[0040] In some embodiments, the AI model service 50 or the central inventory management service 60 or both are implemented using cloud computing. Although the present disclosure includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented with any other type of computing environment now known or later developed.

[0041] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0042] The characteristics are as follows.

[0043] On-demand self-service: Cloud consumers can provision computing capabilities, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider.

[0044] Broad network access: The capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (such as mobile phones, laptops, and PDAs).

[0045] Resource pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically assigned and reassigned according to demand. Consumers generally have no control or knowledge over the exact location of the provided resources, but have a degree of location independence in that they can specify the location at a higher level of abstraction (such as country, state, or data center).

[0046] Rapid scalability: Functions can be provisioned quickly and elastically, and in some cases automatically, to scale out rapidly and be released quickly to scale in rapidly. To the consumer, the functions available for provisioning often appear to be unlimited, and any amount can be purchased at any time.

[0047] Measured services: Cloud systems automatically control and optimize resource usage by leveraging some level of abstraction metering function appropriate for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, achieving transparency for both the provider and consumer of the services being utilized.

[0048] The service model is as follows.

[0049] Software as a Service (SaaS): The function provided to the consumer is to use the provider's application running on the cloud infrastructure. The application is accessible from various client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, and even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0050] Platform as a Service (PaaS): The function provided to consumers is to deploy the applications created or obtained by consumers, which are created using the programming languages and tools supported by the provider, onto the cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but have control over the deployed applications and, in some cases, the application hosting environment configuration.

[0051] Infrastructure as a Service (IaaS): The function provided to consumers is to provision processing, storage, networks, and other basic computing resources, and consumers can deploy and run any software that may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, the deployed applications, and, in some cases, limited control over the selected networking components (e.g., host firewalls).

[0052] The deployment models are as follows.

[0053] Private cloud: The cloud infrastructure is operated only for an organization. The cloud infrastructure can be managed by the organization or a third party and can exist on-premises or off-premises.

[0054] Community Cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (such as mission, security requirements, policies, and compliance considerations). The cloud infrastructure can be managed by an organization or a third party and can exist on-premises or off-premises.

[0055] Public Cloud: The cloud infrastructure is made available to the general public or a large industrial group and is owned by an organization that sells cloud services.

[0056] Hybrid Cloud: The cloud infrastructure remains a distinct entity but is a composition of two or more clouds (private, community, or public) that are linked together by standardized technologies or technologies that can assert ownership (such as cloud bursting for load balancing between clouds) that enable data and application portability.

[0057] The cloud computing environment is a service that aims to focus on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0058] Next, referring to FIG. 10, an exemplary cloud computing environment 150 is shown. As illustrated, the cloud computing environment 150 includes one or more cloud computing nodes 110 that can communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cellular phone 154A, a desktop computer 154B, a laptop computer 154C, or an automotive computer system 154N, or combinations thereof. The nodes 110 can communicate with each other. The nodes 110 can be physically or virtually grouped within one or more networks (not shown), such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described above. This enables the cloud computing environment 150 to provide infrastructure, platform, or software, or combinations thereof, as a service for which cloud consumers do not need to maintain resources on local computing devices. It should be understood that the types of computing devices 154A-N shown in FIG. 10 are merely exemplary, and that the computing nodes 110 and the cloud computing environment 150 can communicate with any type of computerized device via any type of network or network addressable connection, or both (e.g., using a web browser).

[0059] Next, referring to FIG. 11, a set of functional abstraction layers provided by the cloud computing environment 150 (FIG. 10) is shown. It should be understood that the components, layers, and functions shown in FIG. 11 are merely exemplary and that embodiments of the invention are not limited thereto. As illustrated, the following layers and corresponding functions are provided.

[0060] Hardware and software layer 160 includes hardware components and software components. Examples of hardware components include mainframe 161, RISC (Reduced Instruction Set Computer) architecture-based server 162, server 163, blade server 164, storage device 165, and network and networking components 166. In some embodiments, software components include network application server software 167 and database software 168.

[0061] Virtualization layer 170 provides an abstraction layer where the following examples of virtual entities can be provided: virtual server 171, virtual storage 172, virtual network 173 including virtual private network, virtual applications and operating systems 174, and virtual client 175.

[0062] As an example, the management layer 180 can provide the functions described below. Resource Provisioning 181 realizes the dynamic procurement of computing resources and other resources used to perform tasks within a cloud computing environment. Metering and Pricing 182 realizes cost tracking when resources are used within a cloud computing environment and the billing or invoicing for the consumption of such resources. As an example, such resources may include application software licenses. Security realizes identification and verification for cloud consumers and tasks, as well as protection for data and other resources. The User Portal 183 realizes access to the cloud computing environment for consumers and system administrators. Service Level Management 184 realizes the allocation and management of cloud computing resources so that the required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 185 realizes the pre-adjustment and procurement of cloud computing resources whose future requirements are expected, in accordance with the SLA.

[0063] The workload layer 190 provides examples of functions for which a cloud computing environment can be used. Examples of workloads and functions that can be provided from this layer include Mapping and Navigation 191, Software Development and Lifecycle Management 192, Virtual Classroom Education Delivery 193, Data Analytics Processing 194, Transaction Processing 195, and AI Model 196.

[0064] FIG. 12 shows a schematic diagram of an exemplary computer or processing system 22 that may implement a method for automating an inventory verification procedure for items stored on a shelf within a closed environment, according to one embodiment of the present disclosure. The computer system is merely an example of a suitable processing system and is not intended to imply any limitation as to the use or functionality of the embodiments of the methods described herein. The illustrated processing system may be operable in a number of other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with the processing system shown in FIG. 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices.

[0065] The computer system may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0066] The components of a computer system can include, without limitation, one or more processors or processing devices 100, a system memory 106, and a bus 104 that couples various system components including the system memory 106 to the processor 100. The processor 100 can include program modules 102 that implement the methods described herein. In one embodiment, a processing system 22 is implemented on the processor 100, and the program modules 102 include one or more of the modules shown in FIG. 8. The module 102 can be programmed within the integrated circuit of the processor 100 or can be loaded from the memory 106, the storage device 108, or the network 114, or a combination thereof.

[0067] The bus 104 can represent any one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus that uses any of a variety of bus architectures. By way of example and not limitation, such architectures can include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus.

[0068] The computer system can include various computer system readable media. Such media can be any available media that is accessible by the computer system and can include both volatile and nonvolatile media, removable and non-removable media.

[0069] System memory 106 may include a computer system readable medium in the form of volatile memory, such as random access memory (RAM) or cache memory or both. The computer system may further include other removable / non-removable volatile / non-volatile computer system storage media. By way of example only, a storage system 108 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (e.g., a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a removable non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media may be provided. In such cases, each may be connected to bus 104 by one or more data media interfaces.

[0070] The computer system may also communicate with one or more external devices 116, such as a keyboard, a pointing device, a display 118, one or more devices that enable a user to interact with the computer system, or any device that enables the computer system to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), or a combination thereof. Such communication may occur via an input / output (I / O) interface 110.

[0071] Furthermore, the computer system can communicate with one or more networks 114, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via the network adapter 112. As shown, the network adapter 112 communicates with other components of the computer system via the bus 104. Although not shown, it should be understood that other hardware components or software components or both can be used with the computer system. By way of example and not limitation, the examples include microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, and the like.

[0072] The present invention may be a system, method, or computer program product at any possible technical detailed integration level, or a combination thereof. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions to cause a processor to implement aspects of the present invention.

[0073] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction-executing device. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. In this specification, a computer-readable storage medium should not be construed to be an essentially transient signal such as, for example, a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0074] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each respective computing / processing device.

[0075] The computer-readable program instructions for carrying out the operations of the present invention may be source code or object code written in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or some combination of object-oriented programming languages such as Smalltalk, C++, and procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to customize the electronic circuit for implementing aspects of the present invention.

[0076] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0077] Such computer-readable program instructions cause the instructions executed via the processor of a computer or other programmable data processing apparatus to create means for implementing the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both, and thus a machine can be produced. Such computer-readable program instructions may also be stored in a computer-readable storage medium that includes instructions for implementing the manner of the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both, such that the computer-readable storage medium causes a computer, a programmable data processing apparatus, or other device, or a combination thereof, to function in a particular manner.

[0078] Such computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device such that the instructions executed thereon implement the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both, and a series of operational steps may be performed thereon to produce a computer-implemented process.

[0079] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described within the block may be performed in an order different than that depicted in the figures. For example, two blocks shown in succession may in fact be implemented as one step, executed simultaneously, substantially simultaneously, in a partially or wholly temporally overlapping manner, or the blocks may sometimes be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagram or flowchart diagram, or combinations of blocks of the block diagram or flowchart diagram or both, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0080] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of the stated feature, integer, step, operation, element, component, or combination thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0081] All means or steps and corresponding structures, materials, acts, and equivalents of the functional elements in the following claims are intended to include any structure, material, or act for performing the functions in combination with other claimed elements as specifically claimed. The description of the invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the invention. Embodiments were chosen and described in order to best explain the principles of the invention and its practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

[0082] Furthermore, although preferred embodiments of the invention have been described using specific terms, such description is for illustrative purposes only and it is to be understood that changes and variations may be made without departing from the following claims.

Claims

1. An apparatus for automating an inventory verification procedure for items stored on shelves within a closed environment, a mobile robotic device having at least one movable appendage, said at least one movable appendage including at least one camera and at least one additional sensory modality, said mobile robotic device, a positioning system configured to position said at least one movable appendage to take camera images of said items on said shelves within said closed environment from a number of different viewpoints using said at least one camera and said at least one additional sensory modality, a context module configured to determine the context of said mobile robotic device, dynamically searching for an AI context-specific model based on said context, identifying and counting said items on said shelves using said AI context-specific model and an edge AI computing module configured to perform An apparatus comprising.

2. The apparatus according to claim 1, wherein said positioning system uses camera vision and said at least one additional sensory modality to position said appendage on and along the sides of said items on said shelves.

3. The apparatus according to claim 1, wherein dynamically searching for said AI context-specific model includes downloading said AI context-specific model from a cloud hosting service.

4. The apparatus according to claim 1, further comprising determining the context of said mobile robotic device using an AI context model.

5. The apparatus according to claim 1, wherein said at least one movable appendage is configured to rotate about the longitudinal axis of said at least one appendage and descend and ascend along the vertical dimension of said mobile robotic device.

6. The apparatus according to claim 3, wherein said AI context-specific model includes an item type model and an item count model.

7. The apparatus according to claim 1, further comprising uploading the count of said items to an inventory management system.

8. A method for automating an inventory verification procedure for items stored on shelves within a closed environment, To provide a mobile robotic device having at least one movable appendage, wherein the at least one movable appendage includes at least one camera and at least one additional sensory modality, and the providing; Controlling the at least one camera and the at least one additional sensory modality to position the at least one movable appendage to capture camera images of the item on the shelf in the closed environment from a plurality of different viewpoints; Determining the context of the mobile robotic device; Dynamically searching for an AI context-specific model based on the context; Identifying and counting the items on the shelf using the AI context-specific model; Providing the mobile robotic device with an edge AI computing module configured to perform the above; A method including the above.

9. A program for causing a computer system to execute the method according to Claim 8.

10. A computer system for automating an inventory verification procedure for items stored on a shelf in a closed environment, A computer processor; A computer-readable storage medium; Program instructions stored on the computer-readable storage medium, which when executed by the processor cause the computer system to: Control at least one camera and at least one additional sensory modality of a mobile robotic device having at least one movable appendage, wherein the at least one movable appendage includes the at least one camera and the at least one additional sensory modality, and the control positions the at least one movable appendage to capture camera images of the item on the shelf in the closed environment from a plurality of different viewpoints; Determine the context of the mobile robotic device; Dynamically search for an AI context-specific model based on the context; Identify and count the items on the shelf using the AI context-specific model; Provide an edge AI computing module configured to perform the above; The program instructions for causing the above to be executed; A computer system comprising the above.

11. A computer system for automating the inventory verification procedure for items stored on a shelf within a closed environment, comprising a processing system and a mobile machine device having at least one movable attachment, the at least one movable attachment including at least one camera and at least one additional sensory modality, wherein the processing system is configured to position the at least one movable attachment to take camera images of the items on the shelf within the closed environment from a plurality of different viewpoints using the at least one camera and the at least one additional sensory modality; a context module configured to determine the context of the mobile machine device; download an AI context-specific model based on the context; use the AI context-specific model to identify and count the items on the shelf; a cloud service storing a plurality of AI context-specific models for download by an edge AI computing module; an inventory management system and the edge AI computing module configured to perform and comprising the edge AI computing module uploading the count of the items to the inventory management system.

Images