Real-time inventory and internal environment sensing method and system for object storage structures

Abstract

The present application relates to a real-time inventory system and process for use of a movable object storage structure, such as a refrigerator (11A, 11B). The system and process include a method for dynamically identifying objects put into or taken out of a refrigerator. The method comprises: detecting movement of an object in the refrigerator using one or more sensors coupled to the refrigerator, or sensing refrigerator opening conditions (11A, 11B); acquiring one or more images of at least a portion of the object while the object is put into or removed from the refrigerator (11A, 11B); and using the acquired image, tracking a motion of the object, determining a direction of the motion of the object, and identifying the object using a trained machine learning model, where the machine learning model is trained using, at least in part, a group-based training method that includes acquiring images from other refrigerators (36).

Description

[0001] Citation Statement

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 309,785, filed on February 14, 2022. Background Art

[0003] The present exemplary embodiment relates to object detection, tracking and identification, and incorporating gas sensing technology into storage compartments, such as but not limited to refrigerators, cooling units, cabinets, storage containers, etc. It is particularly applicable to refrigerators and will be described with particular reference to refrigerators. However, it should be understood that the present exemplary embodiment is also applicable to other similar applications. Summary of the invention

[0004] According to one embodiment of the present disclosure, a method for dynamically identifying objects placed in or taken out of a refrigerator, the method comprising: using one or more sensors coupled to the refrigerator to detect the movement of the object in the refrigerator, or sensing that the refrigerator has been opened; acquiring one or more images of at least a portion of the object when the object is placed in or taken out of the refrigerator; and using the acquired images to track the movement of the object, determine the direction of the movement of the object, and identify the object using a trained machine learning (ML) model, wherein the machine learning model is trained, at least in part, using a group-based training method that includes acquiring images from other refrigerators.

[0005] According to another embodiment of the present disclosure, a computer-readable storage medium includes executable computer code embodied in a tangible form, the computer-readable medium including: executable computer code operable to detect the movement of an object placed in or taken out of a refrigerator, using one or more sensors coupled to the refrigerator; executable computer code operable to acquire one or more images of at least a portion of the object; executable computer code operable to determine the direction of the movement of the object; executable computer code operable to identify the object based on the one or more images, wherein the computer code uses the acquired images to track the movement of the object, determine the direction of the movement of the object, and identify the object using a trained machine learning model, the machine learning model being trained, at least in part, using a group-based training method that includes acquiring images from other refrigerators.

[0006] According to another embodiment of the present disclosure, a storage system includes: a refrigerator; one or more cameras operably associated with the refrigerator; one or more sensors operably associated with the refrigerator; at least one processor operably associated with the refrigerator; and at least one storage circuit operably associated with the refrigerator, the at least one storage circuit including a computer-readable storage medium, the computer-readable storage medium storing computer code in a tangible form, wherein the computer code, when executed by the at least one processor, causes the storage system to perform the following operations: using one or more sensors coupled to the refrigerator to detect the movement of an object in the refrigerator, or sensing that the refrigerator has been opened; acquiring one or more images of at least a portion of the object when the object is placed in or taken out of the refrigerator; and using the acquired images, tracking the movement of the object, determining the direction of the movement of the object, and identifying the object using a trained machine learning model, wherein the machine learning model is trained, at least in part, using a group-based training method that includes acquiring images from other refrigerators. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a system block diagram of a cloud-based refrigerator real-time inventory system disclosed in an embodiment of the present application, and the refrigerator real-time inventory system includes: refrigerator No. 1 11A, refrigerators No. 2 to No. n 11B, a management machine 12, a cloud 13, smartphone / mobile device No. 1 14A, smartphone / mobile device No. 2 14B, smartphone / mobile device No. n 14C, and a database 36 (such as Spring House database).

[0008] Figure 2A , 2B2C is a flowchart of the overall process of a refrigerator real-time inventory system disclosed in an embodiment of the present application, and the refrigerator real-time inventory system process includes the following processes / sub-processes: the script starts from the command line interface (Command Line Interface, CL) 21, the application is initialized 22, the hardware is initialized 23, whether the sensor is triggered 24, start acquisition 25, the data lake uniform resource locator (Uniform Resource Locator, URL) is set based on the environment 26; the folder for the image is created 27, the camera acquires and saves the original image 28, whether the sensor is not triggered 29, the application programming interface (Application Programming Interface, API) calls the database 30, from the original image to the Joint Photographic Experts Group (Joint Photographic Experts Group) Group, JPEG) image conversion 31, process upload to data lake (or image upload to cloud) 32, queue of barcodes to be predicted 33, barcode detection function (or model for barcodes) 34, open source barcode reader 35 (such as zBar), database 36 (such as Spring House database), queue of images to be predicted 37, image prediction function 38, inference endpoint for object detection 39, trained object prediction model 40, object detection model training 41, image generation function 42, match image to sequence 43, is it the last prediction in the sequence 44, get image prediction and barcode detection 45, direction function 46, inference endpoint for direction 47, trained direction model 48, direction model training 49, annotation quality assurance (QA) service 50 (or image quality assurance service), create inventory changes 51, front-end application 52, inventory changes confirmed by administrator (such as Spring House administrator) 53, and image annotation service 54.

[0009] Figure 3 This is an overall process flow chart of another refrigerator real-time inventory system disclosed in an embodiment of the present application, which includes: in part, a local refrigerator 11A, a cloud-based image processing application associated with one or more image generation functions 42, functions of matching images to sequences 43 and accessing a database 36, a trained object prediction model 40, an image annotation service 54, an image annotation quality assurance service 50, a front-end application 52, a barcode detection function / model for barcodes 34, and a trained direction model 48.

[0010] Figure 4A , 4B4C is a schematic diagram of a refrigerator disclosed in an embodiment of the present application, and shows a light strip (on both sides of the opening and above each shelf) 60, an entrance plane sensor (on both sides of the opening, from top to bottom) 61, a camera (on both sides of each shelf) 62 and a proximity sensor (above each shelf) 63.

[0011] Figure 5 It is a high-level thread process flow chart of a refrigerator real-time inventory system disclosed in an embodiment of the present application. The refrigerator real-time inventory system includes an initial system process 100 for setting variables, creating threads and preparing an environment, a camera trigger thread 200 for checking sensors and updating variables, an image acquisition thread 300 for acquiring images and sending them to other threads when triggered, a data processing thread 400 for processing original images and uploading compressed photos, an image deletion thread 500 for deleting all images in a sequence in which an object has not entered a shelf, and a processed photo directory deletion thread 600 for waiting for a specified time and then deleting the processed photo directory.

[0012] FIG. 6A to FIG. 6G This is a method disclosed in the embodiments of the present application. Figure 5 The flowchart of the specific process of the refrigerator real-time inventory system shown in FIG. 1 includes a system initialization process 100 ( Figure 6B ) further process details executed, the image acquisition thread 300 ( Figure 6C ), camera triggers thread 200( Fig.6D ), data processing thread 400( Fig. 6E ), image deletion thread 500( Fig. 6F ), and, processed photo directory deletion thread 600 ( Figure 6G ).

[0013] Figure 7 This is an overall process flow chart of a cloud-based image processing operation for a refrigerator real-time inventory system disclosed in an embodiment of the present application. The cloud-based image processing operation includes: the refrigerator notifies the application programming interface that the image upload is completed 701, a process for image generation - creating an image in the database 702, communicating with the application programming interface / database 800, a process for object tracking / recognition 900, a process for barcode detection 1000, and a process for machine learning model training 1100.

[0014] FIG. 8A to FIG. 8F This is a method disclosed in the embodiments of the present application. Figure 7 A more specific flow chart of the cloud-based image processing operation for the refrigerator real-time inventory system is shown, and further operation details are provided: The refrigerator notifies the application programming interface that the image upload is completed 701 ( Fig. 8A ), communicate with the application programming interface / database 800 ( Fig. 8A), process 900 for object tracking / recognition ( Figure 8C ), process 1000 for barcode detection ( Figure 8F ), and, a process 1100 for machine learning model training ( Fig.8D ). Additionally, synthetic data processing 1200 for synthetically generating images for object detection / prediction training is also shown. DETAILED DESCRIPTION

[0015] Refrigerators are becoming increasingly complex and offer a range of functions. One application area, however, where progress has been relatively limited is inventory management. That is, users of refrigerators typically use the device to store fruit, vegetables, drinks, leftovers, and a variety of other foods. However, it is easy to forget what items are in the refrigerator, when they were placed there, and when certain items have expired or should be replaced.

[0016] According to one embodiment, the present application provides a method for detecting a moving object placed in a refrigerator, and providing a real-time inventory of the contents of the refrigerator based on the detected object placed in or removed from the refrigerator. One or more images of at least a portion of the object are captured. The direction of the movement of the object is tracked. The object is identified based on the one or more images. In various embodiments, the inventory for a storage structure, i.e., a refrigerator, is updated based on the object motion tracking and object identification results.

[0017] Other features include, but are not limited to, image recognition, digital inventory, item tagging and expiration tracking, temperature zones, adaptive exterior features, camera integration, drawers, modular shelving, user interface (UI) integration, computer vision, scanning, voice, crowdsourcing, sensing, and lighting features.

[0018] See also Figure 1 , shows a system block diagram of a cloud-based refrigerator real-time inventory system disclosed in an embodiment of the present application, the refrigerator real-time inventory system includes: refrigerator No. 1 11A, refrigerators No. 2 to No. n 11B, a management machine 12, a cloud 13, smartphone / mobile device No. 1 14A, smartphone / mobile device No. 2 14B, smartphone / mobile device No. n 14C, and a database 36 (such as the Spring House database).

[0019] As shown, the real-time inventory system and process include a system / method for dynamically identifying objects placed in or removed from a refrigerator 11A, the method comprising: using one or more sensors coupled to the refrigerator to detect movement of an object in the refrigerator 13 / 36, or sensing that the refrigerator has been opened; acquiring one or more images of at least a portion of the object when the object is placed in or removed from the refrigerator 13 / 36; and using the acquired images, tracking the movement of the object, determining the direction of the movement of the object, and identifying the object using a trained machine learning model 13 / 36, wherein the machine learning model is trained, at least in part, using a population-based training method that includes acquiring images from other refrigerators.

[0020] Figure 1 It is a high-level view of how the entire platform (such as the Spring House platform) works. Each refrigerator has its own users and inventory tracked by a database (such as the Spring House database), which is then operated by digital applications. In addition, each refrigerator has its own real-time inventory recognition model as a sub-model of the global model. This local model is more accurate and is based on previous purchases set by the user. When a specific user creates an inventory, the digital application uses this inventory to assist the user with recipe creation, food replenishment, food utilization, coaching, and statistics for food waste.

[0021] refer to Figure 2A , 2B 2C shows an overall process flow chart of a refrigerator real-time inventory system disclosed in an embodiment of the present application, and the refrigerator real-time inventory system process includes the following processes / sub-processes:

[0022] Initially, the script starts from the command line interface 21;

[0023] Then, application initialization 22 is performed, including environment (production vs. staging); user identification (ID); flags -n (upload), -I (local inference); data lake integration initialization (S3); camera configuration (gain, exposure, pixel format, reverse X axis, reverse Y axis), machine learning image data (width, height, confidence);

[0024] Then, hardware initialization 23, camera timing and synchronization signal generator; lighting system sensor and controller; camera trigger sensor array 1 to n; camera initialization including camera configuration (gain, exposure, pixel format, reverse X axis, reverse Y axis) and camera timing and synchronization signal generator;

[0025] The process then determines if the sensor is triggered 24, indicating that the refrigerator has been opened, if the refrigerator has been opened the process begins acquisition 25 and the process uploads to the data lake (or images are uploaded to the cloud) 32;

[0026] After starting to get 25, the process is executed in the following order:

[0027] Data lake URI settings based on environment 26;

[0028] A folder for images is created 27;

[0029] The camera acquires and saves the original image 28;

[0030] whether the sensor is not triggered 29, if no, return to camera acquisition 28, if yes, the process performs an application programming interface call to the database 30 and performs image conversion from raw image to Joint Photographic Experts Group 31;

[0031] The API call database 30 provides the following: Data Lake URI settings; Assembly / front end / refrigerator version; Expected image count; Camera configuration; and User ID. Then, after executing the API call, the process simultaneously creates a barcode queue to be predicted 33, an image queue to be predicted 37, and starts the image generation function 42 process.

[0032] Further processing of the images in the barcode queue 33 to be predicted includes a barcode detection function (or a model for barcodes) 34, and an open source barcode reader 35 (such as zBar) to read the barcodes in the processed images. The barcode results are stored in a database 36.

[0033] Further processing of images in the queue of images to be predicted 37 includes processing of image prediction functions 38, processing of inference endpoints for object detection 39, and processing of trained object prediction models 40. The process of object detection model training 41 provides supervised / unsupervised training of object prediction models, including: new annotations trained into the model; training manifests created using quality-assured annotations; and new annotations obtained from the serving subset.

[0034] The process of the image prediction function 38 outputs image data, including the predicted class of the object, i.e. the object type from the taxonomy, and data indicating the specific item associated with the image object. These data are stored in the database 36. In addition, the last prediction in the sequence determination 44 is made by the process, and if it is determined that the last prediction in the sequence has been processed, the process proceeds to obtain the image prediction and barcode detection 45 from the database 36.

[0035] After obtaining image prediction and barcode detection 45 from the database 36, the process performs the process of direction function 46, including: the process of inference endpoint 47 for direction, which uses the trained direction model 48 including the following sub-processes: train the new list into the current model using a linear learner algorithm; create a training list using direction, camera identification, image category, bounding box delta and frame position; calculate the change in bounding box from the first frame to the last frame; calculate the change in bounding box from frame to frame; separate annotations by image category and camera, and then sort frames by time; obtain new annotations from the service subset.

[0036] The image annotation quality assurance service 50 provides annotated training data to the direction model training process 49 including: evaluation of new annotations by Spring House members; and creation of a subset with trainable annotations.

[0037] Further processing of the image input to the image generation function 42 includes matching the image to a sequence 43 and then sorting the matched images in the database 36 .

[0038] Based on the process of obtaining image predictions and barcode detection 45, and the results of the execution of the orientation function 46, the process proceeds to create an inventory change 51, based on the following: the front-end application 52 includes, the item is placed in the inventory; and / or the user changes the inventory orientation and the status of the item. The inventory change 53 is confirmed by an administrator (such as a Spring House administrator), and the image annotation service 54 includes: obtaining images and bounding box inferences for inventory changes; confirming that the image contains a handheld object; and drawing a bounding box around the hand and the object on the hand. (In an exemplary embodiment, the annotations are saved in the Spring House database and sent to the core for training the object prediction model 41 and training the orientation model 49. However, other training methods are also within the scope of the present disclosure)

[0039] See also Figure 3 , showing an overall flow chart of another refrigerator real-time inventory system disclosed in an embodiment of the present application, the refrigerator real-time inventory system includes: in part, a local refrigerator 11A, a cloud-based image processing application associated with one or more image generation functions 42, functions of matching images to sequences 43 and accessing a database 36, a trained object prediction model 40, an image annotation service 54, an image annotation quality assurance service 50, a front-end application 52, a barcode detection function / model for barcodes 34, and a trained direction model 48.

[0040] The local refrigerator 11A includes processes related to image acquisition, including: sensors (motion and light sensing); lighting (lights recessed into the refrigerator and front LED lighting); image acquisition; data processing including: storing data (image data, metadata, sequence data); image conversion (raw to compressed format); and uploading images to the cloud.

[0041] The cloud-based real-time inventory image processing application 36 includes: images created in the database process; single frame model prediction, including object detection, text recognition; sound; and barcode processes; quantity prediction process (e.g., using OpenCV's infrared); item prediction process (based on the merging of single frame scores, we predict the items in the sequence); direction prediction process (bounding boxes and different sizes of hand patterns holding objects and hand patterns not holding objects); and the creation of inventory changes process (labeling items and directions).

[0042] The processes of the trained object prediction model 40 include image-related processes; bounding box prediction; and item prediction.

[0043] Image annotation service 54, and annotation quality assurance service 50, which includes: evaluation of new annotations by Spring House members; and creation of subsets with trainable annotations.

[0044] The front end application 52 includes: inventory change processes associated with items placed within inventory; and user changes to inventory directions and status of items.

[0045] Barcode detection function / model 34 for barcodes, and trained orientation model 48.

[0046] See also Figure 4A , 4B and 4C, which show a schematic diagram of a refrigerator disclosed in an embodiment of the present application, and show a light strip (on both sides of the opening and above each shelf) 60, an entrance plane sensor (on both sides of the opening, from top to bottom) 61, a camera (on both sides of each shelf) 62 and a proximity sensor (above each shelf) 63.

[0047] According to an exemplary embodiment of the present application, the light strips (on both sides of the opening and above each shelf) 60, the brightness and position of these lights are optimized to accurately acquire the original image from the camera (on both sides of each shelf) 62. The entrance plane sensor 61 and the proximity sensor 63 provide sensing functions to determine when to start the image acquisition thread, that is, to start acquiring images when the hand / object is placed in the refrigerator or taken out of the refrigerator.

[0048] According to an exemplary embodiment of the present application, there are two model sets (composed of smaller models) used to predict two elements in real-time inventory: tracking items and identifying items. Tracking runs on local hardware. When the user walks towards the refrigerator, the depth camera triggers the RGB camera to start acquiring. In parallel with acquisition, another thread is opened for processing. If it is determined that there is a handheld object in the sequence, the tracking model will sort the sequence to determine whether the object has passed through a specific plane. If so, the model then determines that the object has entered or left the refrigerator. If so, the local computer sends the relevant image to the cloud, on which another model set is run to identify the item. When the identification is completed, changes are made to the user database and inventory based on the above information.

[0049] See also Figure 5 ,and FIG. 6A to FIG. 6G , provides a high-level description of a refrigerator with a real-time inventory method and system disclosed in an embodiment of the present application. These figures, together with the following description, provide a thread execution process to provide a cloud-based real-time inventory associated with a refrigerator. The described thread and process provide an example of a real-time inventory and internal environment sensing system for an object storage structure (such as a refrigerator).

[0050] Figure 5 It is a high-level thread process flow chart of a refrigerator real-time inventory system disclosed in an embodiment of the present application, which includes an initial system process 100 for setting variables, creating threads and preparing the environment, a camera trigger thread 200 for checking sensors and updating variables, an image acquisition thread 300 for acquiring images and sending them to other threads when triggered, a data processing thread 400 for processing original images and uploading compressed photos, an image deletion thread 500 for deleting all images in a sequence where the object has not entered the shelf, and a processed photo directory deletion thread 600 for waiting for a specified time and then deleting the processed photo directory. The specified amount of time can be a user-specified amount of time or a specified amount of time programmed into the system by an administrator.

[0051] The user, developer, or system administrator starts the script, indicating whether to run it locally or with the upload to the cloud. In addition, they can specify whether they want uploaded photos to be deleted on the local drive, and how long they should be kept on the drive before being deleted. System initialization; setting the initial state of variables, preparing the environment, and creating helper threads that perform tasks throughout the system.

[0052] The camera trigger thread checks the object proximity sensor and the shelf entry plane sensor and updates variables that provide the following information to the rest of the system: object orientation, shelf number, whether to process and upload the raw image, and whether there is an object nearby.

[0053] When an object approaches, the image acquisition thread triggers the camera to acquire an image. When the object is no longer close, based on the logic of the camera trigger thread, the image acquisition thread stops acquiring images and returns to its normal waiting state. If the user pauses the system, this thread will be paused and will no longer acquire images. Since new images cannot be acquired, this effectively pauses the rest of the system without interrupting / disrupting any processing steps downstream.

[0054] If an object enters the rack, based on variables from the camera trigger thread, the image acquired by the image acquisition thread is fed to the data processing thread, which processes and uploads several compressed photos at a time. This thread is paused each time the camera acquires an image in order to allocate the computer's processing resources to camera acquisition. If delete after upload is specified at startup, the name of the directory containing the processed photos after upload is sent to the processed photo directory delete thread.

[0055] The processed photo directory deletion thread creates a new thread for each incoming sequence that pauses for a specified amount of time before deleting the directory given by the data processing thread. Each new sequence has its own thread, allowing waiting times to be independent of each other to ensure consistency and speed of the system.

[0056] If an object does not enter the rack after an image is acquired in the image acquisition thread, the raw image is sent to the image deletion thread, where each image is deleted in parallel in its own thread. When all raw images fed to it have been deleted, the image deletion thread loops back and waits for more input. Each of these five threads continues indefinitely until a user, developer, or system administrator pauses and shuts down the system.

[0057] FIG. 6A to FIG. 6G This is a method disclosed in the embodiments of the present application. Figure 5 The flowchart of the specific process of the refrigerator real-time inventory system shown in FIG. 1 includes a system initialization process 100 ( Figure 6B ) further process details executed, the image acquisition thread 300 ( Figure 6C ), camera triggers thread 200( Fig.6D ), data processing thread 400( Fig. 6E ), image deletion thread 500( Fig. 6F ), and, processed photo directory deletion thread 600 ( Figure 6G ).

[0058] System initialization 100 includes: obtaining user identification; setting up the environment (production and temporary storage); data lake integration initialization (S3); process flags -n (upload), -s (temporary storage), -d (delete processed files after uploading); creating auxiliary threads such as camera triggering, image acquisition, data processing, image deletion, directory deletion; enabling lighting; camera configuration (gain, exposure, pixel format, reverse X-axis, reverse Y-axis); establishing machine learning image data (width, height, confidence).

[0059] When the image acquisition thread 300 is triggered, it acquires images and sends them to other threads 400 / 500 / 600. This thread includes the following processes / sub-processes:

[0060] Initially, a signal 301 from the camera trigger thread is received, indicating that there is an image to be processed;

[0061] Then, the image acquisition thread performs the process of generating the directory 302;

[0062] Then, the process of acquiring an image 303 is performed, and preparation (S3) 304 is performed;

[0063] This thread then determines "has an object entered the rack" 305, and if so, alerts the data processing thread 307, and if not, alerts the image deletion thread 306.

[0064] The alert image deletion thread 306 obtains the image from the camera trigger thread 200 and initiates deletion of the image only if an indication is received from the "Has an object entered the shelf" process 305. The image deletion thread 500 will be described in further detail below.

[0065] The camera trigger thread performs the following process:

[0066] Initially, the thread performs the process of reading sensor 201, including 1) object detection thread: Is there an object nearby? 2) Is the object in a shelf? Which shelf?

[0067] The thread then proceeds to execute the image deletion determination algorithm 202; the shelf identification algorithm 203; the orientation algorithm 204; and the object near but not in the shelf 205. Data from the camera trigger thread 200 is passed to the image acquisition thread 300, including the process and yes / no upload, shelf number, object orientation, and affirmative determination of object near but not in the shelf.

[0068] The data processing thread 400 performs the following process:

[0069] Initially, the image acquisition thread 300 is initialized by being alerted that an image is available for processing;

[0070] The data processing thread 400 then determines "Is the camera activated?" 401, if not, then

[0071] Determine "Are there more raw photos to process?" 402, if yes, then

[0072] The original image subset algorithm is executed 403 and the image subset (number n in the image) is processed and uploaded 404;

[0073] The data processing thread 400 then determines "Have all photos in the sequence been processed?" 405; if so, a process is performed to alert the directory deletion thread 406, if not, it returns to the beginning of the thread.

[0074] The image deletion thread performs a process to delete all images in the sequence where the object did not enter shelf 500, which includes:

[0075] Initially a determination is made 501 as to whether their images are to be deleted; if so, a thread is spawned for each image in the array 502 .

[0076] The processed photo directory deletion thread executes the following process to delete the processed photo directory 600, the process includes:

[0077] Initially a process is performed to determine if there is a directory to be deleted 601, and if so, a thread is created for the directory 602;

[0078] A process is then executed to wait for a user-specified amount of time and then delete the directory of processed photos. Alternatively, the waiting period can be predetermined by a system administrator.

[0079] The following is provided as Figure 5 and FIG. 6A to FIG. 6G A further mid-level description of the refrigerator real-time inventory system shown.

[0080] Startup script:

[0081] The user navigates to the script location and runs the following command, which includes the various flags defined below:

[0082] To run the script locally (the script will not upload images to the cloud): "python app.py";

[0083] To run the script uploaded to the production environment: "python app.py -n";

[0084] To run the script and delete a directory of processed photos after uploading it to the cloud: "python app.py -dx" where "x" defines the number of seconds the script should wait before deleting the directory.

[0085] System Initialization ( Figure 6B )

[0086] When the script is initialized, it first processes the arguments parsed from the terminal launch. The user can add "-s" / "--s", "-n" / "--n", or "-dx" / "--delete x" (where x represents a number of the user's choice) as flags after the desired "python app.py". It will then prompt the user to enter a "user id" which will be used to classify the sequences in the front and back end of the fridge.

[0087] In one embodiment, the hardware configuration includes two "NVIDIA AGX Xavier Jetsons", one for controlling the lights, camera clocks, and the top three cameras and two sets of light curtains, and the other for controlling the bottom three cameras and two sets of light curtains. The script checks which computer it is running on, top or bottom, and sets the camera clocks and light levels accordingly.

[0088] The system then creates five threads that will run in parallel with the main image acquisition thread: a camera trigger thread, a data processing thread, a processed photo directory deletion thread, and an image deletion thread. Finally, the system configures the camera, sets up the environment for sending photos to a production or staging server, and enters the image acquisition thread.

[0089] Image acquisition thread ( Figure 6C )

[0090] This thread is constantly looping, waiting for a signal from the camera trigger thread 200. Once it receives this signal, it begins an uninterrupted sequence starting with generating a directory for the saved images and then proceeding to trigger the camera to acquire images. While the signal from the camera trigger thread is high, the camera will continue to acquire images, and when it is low, the camera will stop. Once completed, the S3 will be notified of the incoming image and alert the user.

[0091] This thread will then check the process and upload variables from the camera trigger thread. If the image should be kept, i.e. the process and upload variables are true, then the location of the raw image from the camera is provided to the data processing thread 400 for processing and possible upload. Otherwise, the location of the image is sent to the image deletion thread 500 for deletion. Once this is done, the loop resumes and waits for another signal from the camera trigger thread.

[0092] If at any time, while this thread is waiting for another camera trigger signal, the user interrupts the system, this thread will be paused. This effectively pauses the entire system, as no new images will be acquired and sent to the various processing threads. These processing threads can continue to run, and the sensor triggers will still update their variables, but the system will not respond to these inputs until the pause is revoked. This is very useful for maintaining refrigerator fronts during operation and other housekeeping tasks.

[0093] The obvious advantage of this design is that the team can run both sets of cameras simultaneously on different machines. If an object enters the front of the refrigerator without passing through the entry barrier and thus entering the shelf, the image will be discarded. This means that the system will ignore any images that are not part of the sequence despite the depth camera triggering, such as the user's legs when placing an item on the top shelf.

[0094] Camera trigger thread ( Fig.6D )

[0095] Once this thread is created, it continuously checks the Camera Trigger Thread 200 / Object Detection Thread 201 to see if an object is approaching and checks the status of the rack entry sensor. This information is then processed by the Image Delete Determination Algorithm 202, the Rack Identification Algorithm 203, and the Object Orientation Algorithm 204 to update the Process and Upload, Rack Number, and Object Orientation variables, respectively. The Process and Upload variables determine whether the raw image acquired in the Image Acquisition Thread 300 is sent for further processing or deleted. The Rack Number and Object Orientation variables provide key information about the object's location and orientation, which is sent to the back end for identification and further processing.

[0096] After the above algorithm is completed, the thread determines if there is an object nearby and if the object is in the shelf. If there is an object nearby and the object has not yet breached the entry plane of the shelf, the thread triggers the image acquisition thread to begin acquiring images by setting the trigger camera variable to true, otherwise it is set to false. The thread then loops back to the start and reads the sensor again. This provides real-time updates to variables that indicate if an object is close and is not affected by other threads.

[0097] Data processing thread ( Fig. 6E )

[0098] At the beginning of the loop 400 of the data processing thread, it checks if the camera is currently acquiring an image (by checking the same variable that triggers the camera and the image acquisition thread 300 checks whether it is acquiring / not acquiring an image). If the camera is not acquiring, it checks if there are more raw photos to process. If not, the loop repeats.

[0099] If the process and upload variables from the camera trigger thread 200 are true, the image acquisition thread 300 sends the location of the raw image to the data processing thread 400. This means that when the data processing thread 400 gets to the check for the raw photo, there is a photo to process and it continues instead of looping back.

[0100] The original image is broken down into subsets defined by the developer based on the original image subsetting algorithm 403. Each image in this new subset has its own thread and they are compressed and uploaded to the cloud in parallel 404. Once the subset is complete, the thread checks if there are more subsets and loops 405.

[0101] When there are no more raw images to process in a sequence, and if the user selected the processed photo deletion flag at startup, the thread passes the location of the processed photos to the directory deletion thread 406. Otherwise, they are left on the machine for developers to gain insights and research.

[0102] Image subsets and camera activation checks are critical to the speed of the system. Only a few images are processed at a time, so the system can respond more quickly to objects entering the refrigerator because all processing resources are not occupied by compression and upload. The thread will continue to process and upload images until exhausted, unless the camera is triggered, at which point it frees up the CPU cores to focus on image acquisition.

[0103] Image Deletion Thread( Fig. 6F )

[0104] When created, this thread continuously loops to check for images to delete 501. If the location of the original image is sent from the image acquisition thread 300, such as when an object approaches but does not enter the shelf, this thread spawns child threads for each image in the sequence 502. Each of these threads deletes the image and terminates. The maximum number of these threads can be adjusted by the developer to affect the speed of the system.

[0105] Processed photo directory deletion thread ( Figure 6G )

[0106] Once created, this thread continuously loops to check for directories to delete 601. Once there is a directory to delete from the data processing thread 400, this thread creates a child thread for that directory 602. This child thread waits for a specified time (specified when the script starts) and then deletes the directory containing the processed photos. Each incoming directory has its own child thread, so the waiting time does not increase by itself; no matter how many threads there are, each thread waits the same amount of time. After deleting the directory, the child thread terminates.

[0107] See also Figure 7 and FIG. 8A to FIG. 8F, showing an overall process flow chart of another cloud-based image processing operation for a refrigerator real-time inventory system disclosed in an embodiment of the present application.

[0108] Figure 7 This is an overall process flow chart of a cloud-based image processing operation for a refrigerator real-time inventory system disclosed in an embodiment of the present application. The cloud-based image processing operation includes: the refrigerator notifies the application programming interface that the image upload is completed 701, a process for image generation - creating an image in the database 702, communicating with the application programming interface / database 800, a process for object tracking / recognition 900, a process for barcode detection 1000, and a process for machine learning model training 1100.

[0109] Once the refrigerator has finished uploading the image to Amazon S3, it sends a callback to the API server to indicate that the upload is complete. (700 / 701 / 702) The API then triggers the Airflow inference pipeline, whose goal is to process the image, resulting in an inventory change of the items in the refrigerator. (700)

[0110] The inference pipeline starts by processing images from each camera separately. These images are downloaded and the image data is sent to one or more object detection models. These models can be different machine learning model architectures (e.g., YOLO vs FRCNN vs GapCNN and others) or different models of the same architecture. The results of these models can be used as described in this application, or, alternatively, the process can zoom in on all regions of interest and pass the cropped images into another object detection model or image classification model. This approach allows for more data on the object of interest to be used to "zoom in" on details, such as text on a product label.

[0111] From here, the results from each model are combined and all bounding boxes are passed into a non-maximum suppression algorithm to merge duplicate bounding boxes. This is especially important when using multiple object detection algorithms, as multiple algorithms will often find the same object of interest.

[0112] After eliminating duplicate bounding boxes, the result is sent to an object tracking algorithm (e.g., Deep SORT). Its goal is to associate bounding boxes across multiple frames of a video. The result is that each bounding box receives an object tracking identity, ensuring that the object (e.g., apple) in frame 1 is the same object in frame 100.

[0113] Finally, the process goes through each frame and determines if an object has entered or left the fridge. The previous predictions are used for the class of the object, and these predictions are combined to produce a food prediction and confidence score for changing the fridge inventory. This is the end of the part that runs on each camera.

[0114] FIG. 8A to FIG. 8F This is a method disclosed in the embodiments of the present application. Figure 7 A more specific flow chart of the cloud-based image processing operation for the refrigerator real-time inventory system is shown, and further operation details are provided: The refrigerator notifies the application programming interface that the image upload is completed 701 ( Fig. 8A ), communicate with the application programming interface / database 800 ( Fig. 8A ), process 900 for object tracking / recognition ( Figure 8C ), process 1000 for barcode detection ( Figure 8F ), and, a process 1100 for machine learning model training ( Fig.8D ). Additionally, synthetic data processing 1200 for synthetically generating images for object detection / prediction training is also shown.

[0115] Spring House Model Training Overview

[0116] The Spring House product, the exemplary real-time inventory system / process disclosed in this application, uses machine learning models as part of the inventory management system. An important part of creating any machine learning-based model is the process of training it. These models are trained to provide basic information that is used to locate and identify (classify) objects as they enter and exit the device. The inventory management process primarily uses computer vision models, which are trained based on image data and other relevant descriptive information. Once the model is trained, it can be used to predict possible locations and classifications from a video or a single image.

[0117] Training data collection

[0118] In order to train the desired model, sufficient training data must be collected or obtained. For this system, images of items are collected from videos of items entering and exiting the refrigeration device or from synthetically generated images; both of these processes are described in detail in other embodiments. Other information is also collected, such as product classification data (product name, brand, etc.), category information, and any other information used to generate predictions. This descriptive information is called "labels" because they represent reliable basic facts corresponding to each image. Images that are not useful for training are pruned from the dataset. The training data (images and labels) are stored in a way that allows the training process to map images to labels.

[0119] Model Type

[0120] Object localization and classification tasks can be performed by various model types; currently the most common are neural networks. For computer vision tasks, the following techniques are used:

[0121] Single-stage models: Both object localization and classification are performed using a single model.

[0122] Two-stage model: An initial model locates potential “regions of interest” (ROIs) but does not classify these items. Then, a second model uses the ROIs to classify each region. Then, unlikely ROIs are removed from the final result.

[0123] For training purposes, both one-stage and two-stage models can use the same images and labels.

[0124] Object Detection

[0125] In object detection, the desired output is a set of coordinates, in the image, which contain the object of interest. The training labels usually contain the ground truth values ​​of these coordinates.

[0126] Object Classification

[0127] The goal of any machine learning training process is to "learn from the data" so that it can later be used for inference using previously unseen data. More diversity in the training data generally results in a more robust model. As part of the training process, the original source images can be modified in various ways to produce more variation in the training data. These variations may include changes in brightness, contrast, blur, image translation, and human-part obstructions. The training process is iterative, cycling through training data examples in a loop. Typically, the model is periodically evaluated on a separate validation dataset using the training loop. From this validation set, metrics can be generated about the performance of the model. The trained model can be saved and used for inference later.

[0128] Further details of the threads / processes are described below:

[0129] The sub-process 701 / 702 associated with 700 (image generation) includes the following:

[0130] Initially, the refrigerator notifies the application programming interface that the image sequence upload is complete 701 ;

[0131] The process then performs image generation 702, including calling the Spring House API to create images from the path endpoints 711 and queueing the image for inference in the barcode 712.

[0132] The Spring House API / database 800 includes the following processes:

[0133] Create image from path 801;

[0134] Create image reasoning 802;

[0135] Create barcode inference 803;

[0136] Create Inventory Change 804;

[0137] Open source databases (such as "Postgres") 805;

[0138] "Did you find the barcode?" 811;

[0139] Use barcode products 812;

[0140] Use cross-refrigerator products 813; and

[0141] Create inventory change 814.

[0142] The object prediction process includes the following:

[0143] For each image, create a bounding box 901;

[0144] Passing the bounding box and image data to an object architecture algorithm (e.g., “Deep SORT”) 902 ;

[0145] evaluating whether an object has crossed a threshold into or out of the refrigerator 903;

[0146] Combined reasoning 904;

[0147] Create draft inventory changes for each camera based on enter / leave events. Associate pre-event reasoning with inventory changes 905;

[0148] Determine the maximum number of changes per product seen on a single product and save no more than those changes. Associate isolated reasoning with saved inventory changes 906;

[0149] Send images across multiple machine learning models for object detection 911;

[0150] Cropping the image to the bounding box and sending the image data to multiple image classification models 912;

[0151] Use the highest confidence result 913;

[0152] Merge bounding box 914;

[0153] Similar bounding boxes are filtered out using a non-maximum suppression algorithm 915 .

[0154] The machine learning model training 1100 process includes the following:

[0155] The image is recognized in the application 1101;

[0156] The images are pruned to only contain images with objects 1102;

[0157] Image data warehouse 1103;

[0158] Annotation service 1104;

[0159] Object prediction training 1105;

[0160] Image classification trainer 1106;

[0161] The image is sent to a bounding box 1111 drawn around the object on the hand;

[0162] Select a series of images for training 1112;

[0163] Enhance images to create more data by blurring, skewing, mirroring, adjusting color, and transforming images1113;

[0164] Preparing data for a specific model type 1114;

[0165] Model training 1115;

[0166] Evaluate the new model 1116 against the test suite;

[0167] Object detection model 1117;

[0168] Select a set of images for training 1121;

[0169] Preparing data for a specific model type 1122;

[0170] Model training 1123;

[0171] Evaluate the new model 1124 against the test suite; and

[0172] Image classification model 1125.

[0173] The barcode detection 1000 process includes the following:

[0174] barcode to be inferred Amazon SQS queue 1001;

[0175] Barcode detection 1002;

[0176] Download image 1011 from S3;

[0177] Send image farewell to open source barcode library (such as zBar) 1012;

[0178] Call the Spring House API to create a barcode inference endpoint 1013.

[0179] The synthetic data 1200 process includes the following:

[0180] capturing a series of images from multiple angles in a 360-degree light box 1201;

[0181] Using the segmentation model to remove background 1202;

[0182] Foreground image (food) 1203;

[0183] occlusion(hand)1204;

[0184] Adding the occlusion to the foreground image 1205;

[0185] The start and end keyframes are randomly determined 1206;

[0186] The foreground scales and moves appropriately throughout the sequence 1207;

[0187] Transformations such as blur are applied 1208;

[0188] Background added 1209;

[0189] Outputting a sequence of images and corresponding bounding boxes 1210; and

[0190] Background image 1211.

[0191] Certain parts of the detailed description of this application are presented in terms of operations on data bits expressed in algorithms and symbols, which are performed by conventional computer components, including a central processing unit (CPU), a memory device for the CPU, and a connected display device. These algorithmic descriptions and representations are the means used by technicians in the field of data processing technology to most effectively convey the content of their work to those skilled in the art. An algorithm is generally considered to be a self-consistent sequence of steps leading to a desired result. These steps are those steps that require physical operations on physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It turns out that it is sometimes convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc., mainly for common usage reasons.

[0192] It should be understood, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, it will be apparent from the discussion of this application that discussions using terms such as "processing" or "computing" or "arithmetic" or "determining" or "displaying" refer to the actions and processes of a computer system or similar electronic computing device that operate and transform data represented as physical (electronic) quantities in the computer system's registers and memories into other data similarly represented as physical quantities in the computer system's data, memories or registers, or other such information storage, transmission or display devices.

[0193] The exemplary embodiments also relate to an apparatus for performing the operations discussed herein. The apparatus may be specially constructed for the required purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as but not limited to any type of disk, including floppy disks, optical disks, CD-ROMs and magnetic optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any medium suitable for storing electronic instructions, and each coupled to a computer system bus.

[0194] The algorithms and displays presented in this application are essentially independent of any particular computer or other device. Various general-purpose systems can be used with programs that conform to the teachings of this application, or it may prove convenient to build more specialized equipment to perform the methods described in this application. It is apparent from the above description that various structures in these systems. In addition, the exemplary embodiments are not described with reference to any particular programming language. It is understood that a variety of programming languages ​​can be used to implement the teachings of the exemplary embodiments described in this application.

[0195] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media include read-only memory ("ROM"); random access memory ("RAM"); magnetic disk storage media; optical storage media; flash memory devices; and electrical, optical, acoustic or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), to name a few.

[0196] The methods illustrated in this specification may be implemented in a computer program product, which can be executed on a computer. The computer program product may include a non-transitory computer-readable recording medium on which the control program is recorded, such as a disk, a hard disk drive, etc. Common forms of non-transitory computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic storage medium, a CD-ROM, a DVD or any other optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM or other memory chip or a cassette, or any other tangible medium from which a computer can read and use.

[0197] Alternatively, the method may be implemented in a transient medium, such as in a transmissible carrier wave, where the control program is embodied as a data signal using a transmission medium, such as acoustic or light waves, such as those generated during radio wave and infrared data communications, and the like.

[0198] It is understood that the above disclosed variations and other features and functions, or alternatives thereof, may be combined into many other different systems or applications. Various currently unforeseeable or unanticipated substitutions, modifications, changes or improvements therein may be subsequently made by those skilled in the art, and these substitutions, modifications, changes or improvements are also intended to be included in the following claims.

[0199] The exemplary embodiments have been described with reference to the preferred embodiments. Obviously, modifications and alterations will occur to others upon reading and understanding the above detailed description. It is intended that the exemplary embodiments should be construed to include all such modifications and alterations as long as they come within the scope of the appended claims or their equivalents.

Claims

1. A method for dynamically identifying objects placed in or taken out of a refrigerator, characterized in that: The method comprises: Using one or more sensors coupled to the refrigerator, detecting movement of an object in the refrigerator, or sensing that the refrigerator has been opened; acquiring one or more images of at least a portion of the object when the object is placed in or removed from the refrigerator; and Using the acquired images, track the movement of the object, determine the direction of the movement of the object, and identify the object using a trained machine learning model, wherein the machine learning model is trained, at least in part, using a population-based training method that includes acquiring images from other refrigerators.

2. The method for dynamically identifying objects placed in or taken out of a refrigerator according to claim 1, characterized in that: The method further comprises: Based on the direction determination, it is determined whether the object is added to or removed from the refrigerator.

3. The method for dynamically identifying objects placed in or taken out of a refrigerator according to claim 1, characterized in that: The method further comprises: Based on the identified object and the determined direction, a real-time inventory record associated with the object within the refrigerator is updated.

4. The method for dynamically identifying an object placed in or taken out of a refrigerator according to claim 4, characterized in that: The method further comprises: On the smart device, information is displayed based on the real-time inventory record.

5. The method for dynamically identifying objects placed in or taken out of a refrigerator according to claim 1, characterized in that: The method further comprises: detecting a hand moving through an entry area of ​​the refrigerator; determining whether the hand is carrying the object; determining a direction of movement of the hand; and The real-time inventory record for the refrigerator is updated based on the hand direction determination and the determination as to whether the hand is carrying the object.

6. The method for dynamically identifying an object placed in or taken out of a refrigerator according to claim 1, characterized in that: The method further comprises: detecting the presence of the object within a scanning range of a scanner coupled to the refrigerator; scanning an object using the scanner; acquiring scan data from the scanner indicative of features of the object; and Feature information about the object is stored based on the scan data.

7. The method for dynamically identifying an object placed in or taken out of a refrigerator according to claim 6, characterized in that: The scanner is a near infrared NIR scanner.

8. The method for dynamically identifying an object placed in or taken out of a refrigerator according to claim 1, characterized in that: The trained machine learning models are two trained machine learning models and include: a first model for tracking objects and a second model for identifying objects.

9. The method for dynamically identifying an object placed in or taken out of a refrigerator according to claim 8, characterized in that: The tracking model is located on local hardware operably associated with the refrigerator, and the recognition model is located on a remote cloud-based system, the tracking model sorts the image sequence to determine whether the object enters or leaves the refrigerator, and then the local hardware sends one or more of the image sequence to the second model to identify the object.

10. The method for dynamically identifying an object placed in or taken out of a refrigerator according to claim 9, characterized in that: The method further comprises: When the user walks towards the refrigerator, the depth camera triggers the RGB camera to start acquiring the image sequence, and in parallel, a single thread is opened for processing to determine whether there is a handheld object in the image sequence, and if the object is entering or leaving the refrigerator, the local hardware sends one or more of the image sequences to the remote cloud-based system for identifying the object, and When identification is complete, a real-time inventory record associated with the object within the refrigerator is updated.

11. A computer-readable storage medium comprising executable computer code embodied in a tangible form, characterized in that: The computer readable medium comprises: executable computer code operable to detect movement of an object being placed into or removed from a refrigerator, utilizing one or more sensors coupled to said refrigerator; executable computer code operable to acquire one or more images of at least a portion of the object; executable computer code operable to determine a direction of said motion of said object; Executable computer code operable to identify the object based on the one or more images, wherein the computer code uses the acquired images to track the movement of the object, determine the direction of the movement of the object, and identify the object using a trained machine learning model, the machine learning model being trained, at least in part, using a population-based training method that includes acquiring images from other refrigerators.

12. The computer-readable storage medium according to claim 11, wherein: The computer readable storage medium further includes: Executable computer code is operable to determine whether the object is added to or removed from the refrigerator based on the direction determination.

13. The computer-readable storage medium according to claim 11, wherein: The computer readable storage medium further includes: Executable computer code is operable to update real-time inventory record data for the refrigerator based on the identified object and the determined direction.

14. The computer-readable storage medium according to claim 13, wherein: The computer readable storage medium further includes: executable computer code operable to detect a hand moving through an entry area of ​​said refrigerator; executable computer code operable to determine whether said hand is carrying said object; executable computer code operable to determine a direction of movement of said hand; and Executable computer code is operable to update real-time inventory record data for the refrigerator based on the direction determination of the hand and the determination as to whether the hand is carrying the object.

15. The computer-readable storage medium according to claim 11, wherein: The trained machine learning models are two trained machine learning models and include: a first model for tracking objects and a second model for identifying objects.

16. The computer-readable storage medium according to claim 15, wherein: The tracking model is located on local hardware operably associated with the refrigerator, and the recognition model is located on a remote cloud-based system, the tracking model sorts the image sequence to determine whether the object enters or leaves the refrigerator, and then the local hardware sends one or more of the image sequence to the second model to identify the object.

17. The computer-readable storage medium according to claim 16, wherein: The computer readable storage medium further includes: When the user walks towards the refrigerator, the depth camera triggers the RGB camera to start acquiring the image sequence, and in parallel, a single thread is opened for processing to determine whether there is a handheld object in the image sequence, and if the object is entering or leaving the refrigerator, the local hardware sends one or more of the image sequences to the remote cloud-based system for identifying the object, and When identification is complete, a real-time inventory record associated with the object within the refrigerator is updated.

18. A real-time inventory system for dynamically identifying objects placed in or taken out of a refrigerator, characterized in that: The system comprises: refrigerator; one or more cameras operably associated with the refrigerator; one or more sensors operably associated with the refrigerator; at least one processor operably associated with the refrigerator; and At least one storage circuit operably associated with the refrigerator, the at least one storage circuit comprising a computer-readable storage medium, the computer-readable storage medium storing computer code in a tangible form, wherein the computer code, when executed by the at least one processor, causes the storage system to perform the following operations: Using one or more sensors coupled to the refrigerator, detecting movement of an object in the refrigerator, or sensing that the refrigerator has been opened; acquiring one or more images of at least a portion of the object when the object is placed in or removed from the refrigerator; and Using the acquired images, track the movement of the object, determine the direction of the movement of the object, and identify the object using a trained machine learning model, wherein the machine learning model is trained, at least in part, using a population-based training method that includes acquiring images from other refrigerators.

19. The real-time inventory system according to claim 18, characterized in that: The operations also include: Based on the direction determination, it is determined whether the object is added to or removed from the refrigerator.

20. The real-time inventory system according to claim 18, characterized in that: The operations also include: Based on the identified object and the determined direction, a real-time inventory record associated with the object within the refrigerator is updated.

21. The real-time inventory system according to claim 18, characterized in that: The operations also include: On the smart device, information is displayed based on the real-time inventory record.

22. The real-time inventory system of claim 18, wherein: The operations also include: detecting a hand moving through an entry area of ​​the refrigerator; determining whether the hand is carrying the object; determining a direction of movement of the hand; and The real-time inventory record for the refrigerator is updated based on the hand direction determination and the determination as to whether the hand is carrying the object.

23. The real-time inventory system of claim 18, wherein: The operations also include: detecting the presence of the object within a scanning range of a scanner coupled to the refrigerator; scanning an object using the scanner; acquiring scan data from the scanner indicative of features of the object; and Feature information about the object is stored based on the scan data.

24. The real-time inventory system according to claim 23, characterized in that: The scanner is a near infrared NIR scanner.

25. The real-time inventory system of claim 18, wherein: The trained machine learning models are two trained machine learning models and include: a first model for tracking objects and a second model for identifying objects.

26. The real-time inventory system according to claim 25, characterized in that: The tracking model is located on local hardware operably associated with the refrigerator, and the recognition model is located on a remote cloud-based system, the tracking model sorts the image sequence to determine whether the object enters or leaves the refrigerator, and then the local hardware sends one or more of the image sequence to the second model to identify the object.

27. The real-time inventory system according to claim 26, characterized in that: The operations also include: When the user walks towards the refrigerator, the depth camera triggers the RGB camera to start acquiring the image sequence, and in parallel, a single thread is opened for processing to determine whether there is a handheld object in the image sequence, and if the object is entering or leaving the refrigerator, the local hardware sends one or more of the image sequences to the remote cloud-based system for identifying the object, and When identification is complete, a real-time inventory record associated with the object within the refrigerator is updated.

Images