System and method for tracking refrigerator inventory

Abstract

A refrigerator includes a plurality of storage compartments within an interior space of the refrigerator, a plurality of cameras mounted to a body or a door, and at least one computing device. The at least one computing device is configured to assign a direction for an item to determine whether the item is moved in, out, or within the refrigerator interior space.

Description

Background Technology

[0001] It is known that refrigerators can be equipped with cameras to photograph and record the items stored inside. In existing refrigerators with cameras for photographing and recording inventory, the cameras are arranged and designed so that each has a field of view facing inwards. When the location of stored items inside such refrigerators obstructs the camera's field of view of other items, the camera struggles to accurately record the obstructed items. Therefore, there is a need for a container equipped with an inventory tracking system capable of accurately photographing, tracking, and recording the inventory stored inside. Summary of the Invention

[0002] To address the aforementioned problems, a refrigerator includes a body and a door. The door is pivotally mounted on the body and movable relative to the body between an open position and a closed position. When the door is in the closed position, the body and the door define a plurality of storage compartments within the refrigerator's interior space. The refrigerator also includes at least one of a plurality of cameras mounted on the body and the door. Each camera is positioned to have a field of view including an entrance opening to at least one storage compartment when the door is in the open position. The plurality of cameras are configured to capture images of objects passing through the entrance openings when the objects are inserted into or removed from the at least one storage compartment. The refrigerator also includes at least one computing device operatively connected to the plurality of cameras.

[0003] The at least one computing device includes a session controller module configured to initiate a session and instruct each camera to begin capturing images based on a trigger signal received by the session controller. The at least one computing device also includes a synchronizer module configured to receive captured images from the plurality of cameras and group the captured images into frame sets, each frame set consisting of corresponding captured images captured simultaneously by different cameras among the plurality of cameras. The at least one computing device further includes an object recognition module configured to determine whether each captured image in a corresponding frame set includes the object and to assign a category identifier to the object.

[0004] The at least one computing device further includes an object tracking module configured to analyze consecutively captured images from each camera and assign tracking identifiers to associate the object in the analyzed image with the object in other analyzed images for each camera. The at least one computing device further includes an object merging module configured to compare tracking identifiers between analyzed images from different cameras and, when the object is detected by the object recognition module in captured images captured by multiple different cameras, assign a merging identifier to the object. The at least one computing device further includes a subsequence partitioning module configured to partition the session into subsequences using corresponding merging identifiers received from the object merging module. Each subsequence corresponds to one inventory change of the at least one storage compartment. The at least one computing device further includes a direction module configured to assign a direction to each subsequence based on the analyzed images to determine whether the object is moved in, out, or moving within the refrigerator's interior space. The at least one computing device further includes an inventory update module configured to communicate with a database to update the inventory status of the at least one storage compartment based on each inventory change and the assigned direction.

[0005] According to another aspect, a method for tracking inventory in a refrigerator includes initiating a session based on receiving a trigger signal, and capturing images using a plurality of cameras mounted on at least one of the body and the door, the door being pivotally mounted on the body and movable relative to the body between an open position and a closed position. When the door is in the closed position, the body and the door define a plurality of storage compartments within the refrigerator's interior space. Each camera is positioned to have a field of view through an entrance opening plane of the refrigerator's interior space when the door is in the open position, the entrance opening plane being accessible when the door is in the open position. The plurality of cameras are configured to capture images of objects passing through the entrance opening plane when the objects are loaded into or removed from the at least one storage compartment.

[0006] The method further includes determining, via a computing device communicating with the plurality of cameras, whether each captured image includes an object. The method also includes, via the computing device, analyzing consecutive captured images from each camera and assigning tracking identifiers to associate the object in the analyzed image with objects in other analyzed images for each camera. The method further includes, via the computing device, comparing tracking identifiers between analyzed images from different cameras, and, when the object is detected in captured images captured by multiple different cameras among the plurality of cameras, assigning a merge identifier to the object.

[0007] The method further includes, using the computing device, dividing the session into sub-sequences using corresponding merging identifiers, wherein each sub-sequence corresponds to one inventory change in the at least one storage compartment. The method also includes, using the computing device, assigning a direction to each sub-sequence based on the analyzed image to determine whether the object is moved in, out, or moving within the refrigerator's interior space. The method further includes, updating the inventory status in the database based on each inventory change and the assigned direction.

[0008] According to another aspect, a non-volatile computer-readable storage medium stores instructions that, when executed by a computer having a processor, cause the processor to perform the methods described above. Attached Figure Description

[0009] Figure 1 This is a block diagram of the operating environment of a refrigerator that includes an inventory tracking system.

[0010] Figure 2 This is a front view of the refrigerator with the door in the open position.

[0011] Figure 3 This is a perspective view of the refrigerator with the door in the open position.

[0012] Figure 4 This is a flowchart illustrating a method for tracking inventory inside a refrigerator.

[0013] Figure 5 It is a schematic diagram of a computer-readable medium or computer-readable device, containing processor-executable instructions configured to implement one or more of the described technical solutions. Detailed Implementation

[0014] the term The following contains definitions of some terms used in this application. These definitions cover various component examples and / or forms that fall within the scope of a particular term and can be used to implement this solution. These examples are not intended to be limiting. Furthermore, the components discussed in this application may be combined with other components, omitted, or laid out in other architectural forms.

[0015] As used in this application, "bus" refers to an interconnect architecture that operatively connects other computer components within or between computers. A bus can transmit data between computer components. Buses include, but are not limited to, memory buses, memory processor buses, peripheral buses, external buses, crossbar switches, and local buses. Buses can also interconnect internal components of devices through protocols such as Media Oriented Systems Transport (MOST), Controller Area Network (CAN), and Local Interconnect Network (LIN).

[0016] As used in this application, "component" refers to a computer-related entity (such as hardware, firmware, executing instructions, or a combination thereof). Computer components may include, for example, processes running on a processor, processors, objects, executable programs, execution threads, and the computer itself. One or more computer components may reside in a process and / or thread. Computer components may reside on a single computer and / or be distributed across multiple computers.

[0017] As used in this application, "computer communication" refers to communication between two or more communication devices (such as computers, personal digital assistants, cellular phones, network devices, vehicles, networked thermometers, infrastructure equipment, and roadside equipment). Communication methods include, for example, network transmission, data transmission, file transmission, app transmission, email, and Hypertext Transfer Protocol (HTTP) transmission. Computer communication can be achieved through any type of wired or wireless system and / or network configuration, such as local area networks (LANs), personal area networks (PANs), wireless personal area networks (WPANs), wireless networks (WANs), wide area networks (WANs), metropolitan area networks (MANs), virtual private networks (VPNs), cellular networks, token ring networks, peer-to-peer networks, ad hoc networks, mobile ad hoc networks, and vehicular ad hoc networks (VANETs).

[0018] Computer communication can employ any type of wired, wireless, or network communication protocol, including but not limited to Ethernet (such as IEEE 802.3), wireless local area networks (such as IEEE 802.11), communications access for land mobiles (CALM), global microwave interconnection access (WiMax), Bluetooth, Purple Bee protocol, ultra-wideband (UWAB), multiple-input multiple-output (MIMO), telecommunications and / or cellular network communication (such as Short Message Service (SMS), Multimedia Messaging Service (MMS), 3G, 4G, LTE, 5G, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Dedicated Short Range Enhanced (WAVE), CAT-M, LoRa), satellite communication, dedicated short range communication (DSRC), etc.

[0019] The term "communication interface" as used in this application may include input devices for receiving input and / or output devices for outputting data. The inputs and / or outputs can be used to control different functions, components, and systems. Specifically, "input devices" include, but are not limited to, keyboards, microphones, pointing devices, cameras, imaging devices, graphics cards, displays, buttons, knobs, etc. "Input devices" also include graphical input controls implemented within a user interface, which can be presented through various devices, such as hardware and software-based controls, interfaces, touchscreens, touchpads, or plug-and-play devices. "Output devices" include, but are not limited to, display devices and other devices used for outputting information and functions.

[0020] As used in this application, "computer-readable medium" refers to a non-transitory medium that stores instructions and / or data. Forms of computer-readable media include, but are not limited to, non-volatile and volatile media. Non-volatile media may include, for example, optical discs and magnetic disks. Volatile media may include, for example, semiconductor memory and dynamic memory. Common forms of computer-readable media include, but are not limited to, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, application-specific integrated circuits (ASICs), optical discs, other optical media, random access memory (RAM), read-only memory (ROM), memory chips or memory cards, memory sticks, and other media readable by computers, processors, or other electronic devices.

[0021] As used in this application, "database" may refer to a data table. In other examples, "database" may refer to a set of data tables. In still other examples, "database" may refer to a set of data storage devices and methods for accessing and / or operating these data storage devices. In one embodiment, the database may be stored on a disk, a data storage device, and / or a memory. The database may be stored locally or remotely and accessed via a network.

[0022] The term "data storage device" as used in this application may include, for example, a disk drive, a solid-state drive, a floppy disk drive, a magnetic tape drive, a compression drive, a flash memory card, and / or a memory stick. Furthermore, the disk may be a compact disk ROM (CD-ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and / or a digital video ROM drive (DVD ROM). The disk may store an operating system used to control or allocate resources of a computing device.

[0023] The term "display" as used in this application includes, but is not limited to, light-emitting diode (LED) display panels, liquid crystal display panels, cathode ray tube (CRT) displays, touchscreen displays, and the like, commonly used for displaying information. A display can receive user input (such as touch input, keyboard input, and input from various other input devices). A display can be accessed through various devices, such as remote systems. A display can also be physically integrated into portable or mobile devices.

[0024] The term "memory" as used in this application may include volatile memory and / or non-volatile memory. Non-volatile memory may include, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). Volatile memory may include, for example, random access memory (RAM), synchronous random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), and direct RAM bus RAM (DRRAM). The memory may store the operating system used to control or allocate resources of the computing device.

[0025] The term "module" as used in this application includes, but is not limited to, non-transitory computer-readable media storing instructions, instructions executable on a machine, hardware, firmware, software running on a machine, and / or combinations thereof, for performing one or more functions or operations, and / or triggering another module, method, and / or system to perform a corresponding function or operation. A module may also include logic circuits, software-controlled microprocessors, discrete logic circuits, analog circuits, digital circuits, programmable logic devices, storage devices storing executable instructions, logic gates, gate circuit combinations, and / or other circuit components. Multiple modules may be integrated into one module, or a single module may be distributed among multiple modules.

[0026] An "operable connection" or "operable connection" between entities refers to a connection that enables the transmission and / or reception of signals, physical communication, and / or logical communication. An operable connection may include wireless interfaces, firmware interfaces, physical interfaces, data interfaces, and / or electrical interfaces.

[0027] As used in this application, "portable device" refers to a computing device typically equipped with a display screen with user input functions (such as touch screen and keyboard) and a processor for computation. Portable devices include, but are not limited to, handheld devices, mobile devices, smartphones, laptops, tablets, e-readers, and smart speakers. In some embodiments, "portable device" may refer to a remote device that includes a processor for computation and / or a communication interface for remotely sending and receiving data.

[0028] The term "processor" as used in this application refers to a processor used to process signals and perform general-purpose calculations and arithmetic operations. The signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, bits, and bit streams, and the processor can receive, transmit, and / or detect these signals. Typically, a processor can be of various types, including multiple single-core and multi-core processors, coprocessors, and other single-core, multi-core, and coprocessor architectures. A processor may contain logic circuitry for performing operations and / or algorithms. A processor may also include any number of modules for executing instructions, tasks, or executable programs.

[0029] The term "user" as used in this application refers to a biological individual, such as a human being (e.g., an adult, a child, an infant).

[0030] System Overview The invention will now be described with reference to the accompanying drawings, which are for illustrative purposes only and are not intended to limit the invention. FIG1 is an exemplary component block diagram of the operating environment 100 of the refrigerator 102, which includes a trigger switch 104, a plurality of cameras 110, a lighting device 112, a refrigeration system 114, a door angle sensor 116, and a computing device 120.

[0031] The trigger switch 104, multiple cameras 110, lighting device 112, cooling system 114, and computing device 120 can all be interconnected via bus 124. The components of the operating environment 100, as well as the components of other systems, hardware architectures, and software architectures described in this application, can be combined, omitted, or constructed into different architectural forms according to different embodiments.

[0032] The computing device 120 may be integrated into the refrigerator 102 or implemented as another device such as a remote server 130 and connected via a network 132. The computing device 120 may employ various protocols to implement wired or wireless computer communication to receive and transmit signals between components within the operating environment 100. Furthermore, the computing device 120 may be operatively connected via a bus 124 (e.g., a bus based on Controller Area Network (CAN) or Local Interconnect Network (LIN) protocols) to enable data input and output between the computing device 120 and the components of the operating environment 100.

[0033] The refrigeration system 114 may employ a structure similar to that of existing known refrigerators, and therefore will not be described in detail. The refrigeration system 114 may include a temperature sensor for detecting the internal temperature of the refrigerator 102, and a fan for delivering air cooled by the refrigeration unit (not shown). The refrigeration unit may include a compressor (not shown) for compressing the refrigerant, a condenser (not shown) for condensing the compressed refrigerant, an expansion valve (not shown) for throttling and reducing the pressure of the condensed refrigerant, and an evaporator (not shown) for evaporating the depressurized refrigerant.

[0034] like Figure 2 As shown, refrigerator 102 includes a body 140 and a door 142, which are pivotally mounted and movable relative to the body 140 between an open position and a closed position, similar to a conventional refrigerator. Door 142 is connected to the body 140 via an upper hinge 144 and a lower hinge 154. When the door 140 is in the closed position, the body 140 and the door 142 define a plurality of storage compartments in the refrigerator's interior space 172. The plurality of storage compartments include a first storage compartment 174, a second storage compartment 180, a third storage compartment 182, a fourth storage compartment 184, and a fifth storage compartment 190, each of which is located within the body 140, in the illustrated embodiment. A sixth storage compartment 192, a seventh storage compartment 194, and an eighth storage compartment 200 are located on the door 142, in the illustrated embodiment. The number and location of storage compartments 174, 180, 182, 184, 190, 192, 194, and 200 may differ. Figure 2 and Figure 3 As shown.

[0035] like Figure 2 and Figure 3 As shown, the first storage compartment 174 has a first shelf 202 and a second shelf 204. The second storage compartment 180 has a third shelf 210. The third storage compartment 182 has a first drawer 212. The fourth storage compartment 184 has a second drawer 214. The fifth storage compartment 190 has a third drawer 220. The sixth storage compartment 192 has a first bottle rack 222. The seventh storage compartment 194 has a second bottle rack 224. The eighth storage compartment 200 has a third bottle rack 230.

[0036] Each storage compartment (174, 180, 182, 184, 190, 192, 194, 200) has a designated entrance opening, through which items can be placed into or retrieved from the corresponding storage compartment. See details. Figure 2The first storage compartment 174 defines a first entrance opening 232 leading to the first shelf 202 and the second shelf 204, and the second storage compartment 180 defines a second entrance opening 234 leading to the third shelf 210. The third storage compartment 182 defines a third entrance opening leading to the first drawer 212 (not visible in Figures 2 and 3), the fourth storage compartment 184 defines a fourth entrance opening leading to the second drawer 214 (not visible in Figures 2 and 3), and the fifth storage compartment 190 defines a fifth entrance opening leading to the third drawer 220 when each of the drawers 212, 214, and 220 is opened (not visible in Figures 2 and 3). The sixth storage compartment 192 defines a sixth entrance opening 250 leading to the first bottle rack 222, the seventh storage compartment 194 defines a seventh entrance opening 252 leading to the second bottle rack 224, and the eighth storage compartment 200 defines an eighth entrance opening 254 leading to the third bottle rack 230.

[0037] Multiple cameras 110 are mounted on the main body 140 and the door 142 to capture images of objects entering and exiting the various storage compartments 174, 180, 182, 184, 190, 192, 194, and 200. The multiple cameras 110 include: a first set of cameras 260a and 260b mounted on the main body 140 corresponding to the first storage compartment 174; a second set of cameras 262a and 262b mounted on the main body 140 corresponding to the second storage compartment 180; a third set of cameras 264a and 264b mounted on the main body 140 corresponding to the third storage compartment 182; a fourth set of cameras 270a and 270b mounted on the main body 140 corresponding to the fourth storage compartment 184; and a fifth set of cameras 272a and 272b mounted on the main body 140 corresponding to the fifth storage compartment 190. Figure 3 As can be seen more clearly, the multiple cameras 110 also include: a sixth set of cameras 274a and 274b installed on the door 142 and corresponding to the sixth storage compartment 192; a seventh set of cameras 280a and 280b installed on the door 142 and corresponding to the seventh storage compartment 194; and an eighth set of cameras 282a and 282b installed on the door 142 and corresponding to the eighth storage compartment 200.

[0038] like Figure 2 and Figure 3As shown, cameras with the suffix "a" are positioned on the left side of the corresponding storage compartment, and cameras with the suffix "b" are positioned on the right side of the corresponding storage compartment. The field of view of each of the multiple cameras 110 is not directed towards the interior of the refrigerator 102, but rather away from it. The first set of cameras 260a and 260b is positioned above the first storage compartment 174 and above the first shelf 202 and the second shelf 204, with its field of view covering the first entrance opening 232. The second set of cameras 262a and 262b is positioned above the second storage compartment 180 and above the third shelf 210, with its field of view covering the second entrance opening 234. The third set of cameras 264a and 264b is positioned above the first drawer 212, and when the first drawer 212 is opened, its field of view covers the third entrance opening leading to the first drawer 212 (not visible in Figures 2 and 3). The fourth set of cameras 270a and 270b are positioned above the second drawer 214. When the second drawer 214 is opened, their field of view covers the fourth entrance opening leading to the second drawer 214 (not visible in Figures 2 and 3). The fifth set of cameras 272a and 272b are positioned above the third drawer 220. When the third drawer 220 is opened, their field of view covers the fifth entrance opening leading to the third drawer 220 (not visible in Figures 2 and 3).

[0039] The sixth set of cameras 274a and 274b are mounted on door 142 and above the first bottle holder 222, their field of view covering the sixth entrance opening 250. The seventh set of cameras 280a and 280b are mounted on door 142 and above the second bottle holder 224, their field of view covering the seventh entrance opening 252. The eighth set of cameras 282a and 282b are mounted on door 142 and above the third bottle holder 230, their field of view covering the eighth entrance opening 254. With this arrangement, each of the multiple cameras 110 is positioned such that, when door 142 is in the open position, its field of view includes the corresponding entrance opening leading to the corresponding storage compartment. Using this configuration, the multiple cameras 110 are configured to capture images of objects passing through any of the entrance openings when said objects are being loaded into or removed from the corresponding storage compartment.

[0040] The computing device 120 controls the lighting device 112 during working hours (only when...). Figure 1 (Illustratively shown, the structure is similar to that of an existing refrigerator lighting device) is controlled to optimize the processing effect of the computer vision model described below. The computing device 120 also controls the configuration parameters of each of the multiple cameras 110, including frame rate and exposure parameters, so that the computer vision model can perform optimized processing.

[0041] Combination Figure 1 and Figure 2As shown, a trigger switch 104 is installed on the main body 140 of the refrigerator 102. The trigger switch 104 is operatively connected to a computing device 120 and configured to detect the position of the door 142 to determine whether the door 142 is in the open or closed position. Thus, the trigger switch 104 can send a trigger signal to the computing device 120 according to the open / closed state of the door 142. Specifically, when the door 142 moves from the closed position to the open position, the trigger switch 104 sends a start trigger signal to the computing device 120; when the door 142 moves from the open position to the closed position, the trigger switch 104 sends a stop trigger signal to the computing device 120. Within the scope of this disclosure, the trigger switch 104 can be at least one of a variety of position sensors, such as a mechanical switch, potentiometer, piezoelectric sensor, Hall effect sensor, and eddy current sensor. Furthermore, although the trigger switch 104 is illustrated as being mounted on the main body 140, it may also be additionally or alternatively mounted on the door 142.

[0042] The door angle sensor 116 is configured to detect the opening angle of the door 142 relative to the body 140. For example, the door angle sensor 116 may include a rotary encoder whose shaft rotates with the pivoting of the door 142 relative to the body 140 to output a signal and communicate with the computing device 120 to provide feedback on the angular position of the door 142 relative to the body 140.

[0043] Refer again Figure 1 The computing device 120 includes a processor 464, a memory 470, a data storage device 472, and a communication interface 474. All of these components are operably connected via a bus 124 to enable computer communication. The communication interface 474, through hardware and software cooperation, enables data input and output between the internal components of the computing device 120 and other components, networks, and data sources described in this application. The computing device 120 also includes a session controller module 480, a synchronizer module 482, a buffer module 484, an object recognition module 490, an object tracking module 492, an object merging module 494, a subsequence partitioning module 500, a direction module 502, and an inventory update module 504, used to receive image information from multiple cameras 110 and track the inventory of items inside the refrigerator 102.

[0044] The computing device 120 is operatively connected to multiple cameras 110. A session controller module 480 is configured to initiate a session and, based on a start trigger signal received from a trigger switch 104, instructs each of the multiple cameras 110 to begin capturing images. The multiple cameras 110 begin capturing images after the session controller module 480 receives the start trigger signal from the trigger switch 104, and cease capturing images after the session controller module 480 receives the stop trigger signal from the trigger switch 104. Each of the multiple cameras 110 is configured to compress the captured images, assign a session identification, a camera identification, and a frame identification to each captured image, and send the compressed image and its corresponding session identification, camera identification, and frame identification to a synchronizer module 482. This process can occur during or after a session, referring to the time period from when the door 142 is opened to when it is closed. A single session can be relatively long, potentially including the process of moving multiple items into and out of the refrigerator 102.

[0045] The synchronizer module 482 is configured to receive captured images from multiple cameras 110, along with a session identifier, camera identifier, and frame identifier corresponding to each captured image. Based on the frame identifier corresponding to each captured image, the synchronizer module 482 groups the captured images from the multiple cameras 110 into frame sets. Each frame set consists of captured images taken simultaneously by different cameras among the multiple cameras 110. In one embodiment, the synchronizer module 482 groups the captured images from the multiple cameras 110 according to the frame identifier corresponding to each image. In this case, different captured images have the same frame identifier, indicating that although these images were taken by different cameras, they were captured at the same time.

[0046] When the synchronizer module 482 receives each captured image from each of the multiple cameras 110, it receives, in addition to the frame identifier, the corresponding session identifier and camera identifier. The session identifier is associated with the session assigned by the session controller module 480, the camera identifier is associated with the specific camera among the multiple cameras 110 that captured the captured image, and as described above, the frame identifier is associated with the time the captured image was captured.

[0047] The buffer module 484 is configured to store the compressed captured image to an image buffer memory and, after the session ends, send the compressed captured image to a remote server (e.g., Figure 1The remote server 130 in the system performs post-processing. Through this structure, the system resources of the computing device 120 can be concentrated on image capture during the session and then used for post-processing after the session ends, thereby reducing the hardware capacity required for the computing device 120 to receive image information from multiple cameras 110 and track the inventory of the refrigerator 102.

[0048] The object recognition module 490 is configured to determine whether each captured image in each frame set contains an object, define a bounding box around the object in each captured image, and assign a category identifier to the object. The category identifier assigned by the object recognition module 490 indicates the inventory type of the object, which can be stored in the memory 470 and / or data memory 472 of the computing device 120, or remotely stored and accessed via the network 132. The inventory type can indicate the category of ingredients, food, meals, dishes, or other objects stored in a database.

[0049] The object recognition module 490 has a built-in computer vision model that can output bounding boxes and corresponding category labels for objects in each captured image. In one embodiment, the computer vision model is the YOLOv5 model. Without departing from the scope of protection of this application, the object recognition module 490 can also integrate other computer vision models to determine the existence of objects in the refrigerator 102 and assign category labels.

[0050] The computer vision model used in the object recognition module 490 can also be configured to remove images that do not contain objects from the frame set received from the synchronizer module 482 and send the updated image set to the object tracking module 492. The updated image set may contain only captured images of objects, and its number may be less than the number of frame set images received by the object recognition module 490 from the synchronizer module 482. The first captured image containing an object in the updated image set can indicate the start of a sequence corresponding to a hand reaching into or out of the refrigerator. The last captured image containing an object in the updated image set can indicate the end of the sequence. When the object recognition module 490 does not detect an object, such as detecting an empty hand and no items being placed or removed from the refrigerator, the last captured image containing an object can indicate the end of the sequence. Each sequence, or the detected action of reaching into or out of the refrigerator 102, can contain multiple objects. For example, a user may place or remove two items, such as an apple and a cup of yogurt, from the refrigerator in one sequence of actions. In this case, the object recognition module 490 assigns a first category label to the apple as an object in the captured image, and assigns a second category label to the yogurt as another object in the captured image.

[0051] The object tracking module 492 is configured to analyze consecutively captured images from each of the multiple cameras 110 and track objects in the consecutively captured images from each camera. In one embodiment, the object tracking module 492 compares the first n captured images in an updated image set within a sequence with the last n captured images in the same updated image set to track the movement of objects within the sequence. In the illustrated embodiment, n is 3, but it can also be a larger or smaller value. The time series of the captured images is determined based on the frame identifier associated with each image. For example, a captured image with a larger frame identifier value was captured later than a captured image with a smaller frame identifier. The camera that captured the images is determined based on the camera identifier corresponding to each image. In this way, the object tracking module 492 can associate the same object in consecutive images for each camera, thereby achieving the tracking of the object's movement over time.

[0052] In one embodiment, the object tracking module 492 is further configured to assign tracking identifiers to associate an object in a single analyzed image with the same object in other analyzed images for each of the plurality of cameras 172. The tracking identifier assigned by the object tracking module 492 is unique for each of the plurality of cameras 172, and also unique for each object captured in consecutive captured images by the plurality of cameras 172. Referring to the apple and yogurt example above, the tracking identifier assigned by the object tracking module 492 as an apple in a captured image is different from the tracking identifier assigned as yogurt in a captured image. Furthermore, in the first group of cameras 260a and 260b, the tracking identifier corresponding to the apple in the captured image taken by camera 260a is also different from the tracking identifier corresponding to the apple in the captured image taken by camera 260b. In this way, each tracking identifier assigned by the object tracking module 492 can uniquely identify a single object in consecutive captured images from a single camera among the plurality of cameras 172.

[0053] The object tracking module 492 can employ an observation-centric simple online and real-time tracking (OCSORT) algorithm to analyze continuously captured images from each of the multiple cameras 110. Based on this structure, the object tracking module 492 receives an updated image set with bounding boxes and category labels from the object recognition module 490. The OCSORT algorithm processes this image set and then outputs a further updated image set containing bounding boxes and category labels, and additionally assigns a corresponding tracking label to each object in each image.

[0054] The object merging module 494 is configured to compare tracking identifiers in the analyzed images from different cameras among the multiple cameras 110. When the object recognition module 490 detects the same object in the captured images captured by multiple different cameras among the multiple cameras 110, it assigns a merging identifier to that object. Using the apple and yogurt example above, the object merging module 494 is configured to compare the different tracking identifiers corresponding to the apple in the captured image captured by camera 260a with the different tracking identifiers corresponding to the apple in the captured image captured by camera 260b, and assign the same merging identifier to the object representing the apple in the captured images captured by both cameras (260a, 260b). The object merging module 494 determines the merging identifier based on each tracking identifier in the same sequence.

[0055] Each merge identifier assigned by the object merging module 494 is unique for a single object captured by multiple cameras 110 within a subsequence of a sequence. Here, a sequence refers to a single action of a hand reaching into or out of the refrigerator 102, and a subsequence refers to the process of a single item being moved into or out of the refrigerator 102. Therefore, each merge identifier assigned by the object merging module 494 can associate the same object in images captured by multiple different cameras within a sequence. The object merging module 494 implements its function through an algorithm that analyzes the objects within the bounding boxes of each captured image in a further updated image set and assigns the same merge identifier to the same object detected in images captured by different cameras within a sequence. Referring to the example of apples and yogurt, the merge identifier for apples is different from the merge identifier for yogurt.

[0056] The subsequence partitioning module 500 is configured to divide a session into multiple subsequences based on the corresponding merge identifier received from the object merging module 494. Each subsequence defined by the subsequence partitioning module 500 corresponds to a unique merge identifier, that is, to a single inventory change relative to a single item in the multiple storage compartments 172. Therefore, the subsequence partitioning module 500 is configured to partition the session based on a single inventory change in the multiple storage compartments 172.

[0057] The direction module 502 is configured to assign a motion direction to each subsequence based on the analyzed images to determine whether an object is moving into, out of, or within the refrigerator's interior space. In one embodiment, the direction module 502 employs a transformer-based classifier (e.g., a visual transformer such as CSWin Transformer) to compare the first n analyzed images with the last n analyzed images within a subsequence to determine the motion direction, where n can be 3. Since the analyzed images all contain objects, the first n analyzed images indicate that the object has entered the field of view of a specific camera, and the last n analyzed images indicate that the object has left the field of view of that specific camera. For example, the direction module 502 is configured to locate the centroid of the object's bounding box in the first n and last n analyzed images. The position of the centroid in each analyzed image, and the change in the centroid's position between different analyzed images, can indicate the movement and direction of the object within the bounding box. Since the cameras 274a, 274b, 280a, 280b, 282a, and 282b mounted on the door 142 can move relative to the main body 140, the door angle can be obtained through the door angle sensor 116, providing precise dimensional parameters for determining the movement of the object's bounding box centroid in the analyzed images. Furthermore, the installation positions of the cameras 260a, 260b, 262a, 262b, 264a, 264b, 272a, and 272b mounted on the main body 140 relative to the multiple storage compartments 174, 180, 182, 184, and 190 within the main body are known information, and can be determined based on the door angles relative to the multiple storage compartments 192, 194, and 200. Therefore, by determining the movement of the object's bounding box centroid in the analyzed images, the direction of motion of objects within the bounding boxes in 2n images currently being analyzed can be determined.

[0058] In another embodiment, the orientation module 502 can determine the direction of motion of the object by referring to the corresponding entrance openings 232, 234, 250, 252, 254 in the three-dimensional mesh space, based on the time between consecutive images, and also involving a plurality of cameras among the multiple cameras 110. In one embodiment, the orientation module 502 compares the first captured image and the last captured image of the multiple cameras 110 within a subsequence to determine the direction of motion of the object.

[0059] The direction module 502 implements its function through an algorithm. This algorithm determines the direction of motion of objects within a sub-sequence based on the known position of each camera in the multiple cameras 110 relative to the respective entrance openings 232, 234, 250, 252, and 254. Combined with... Figure 2Figure 3 illustrates that the positions of the first set of cameras 260a and 260b relative to the first entrance opening 232 are known information, based on the installation positions of the first set of cameras 260a and 260b on the main body 140 relative to the first entrance opening 232. Because the positions of the first set of cameras 260a and 260b relative to the first entrance opening 232 are known, the position of the first entrance opening 232 in the field of view of each camera 260a and 260b is also known. If the first captured image in a subsequence detects an object inside the plane defined by the first entrance opening 232, and the last captured image in the subsequence detects the object outside the plane defined by the first entrance opening 232, then the direction module 502 can determine that the direction of motion of the object is removal from the refrigerator 102. Conversely, if the first captured image in the subsequence detects an object outside the plane defined by the first entrance opening 232, and the last captured image in the subsequence detects the object inside the plane, then the direction module 502 can determine that the object's direction of motion is moving into the refrigerator 102.

[0060] The inventory update module 504 is configured to communicate with the memory 470 and / or data storage 472 via bus 124, or with the remote server 130 via network 132, to update the inventory status of at least one of the multiple storage compartments 172 based on each inventory change and the assigned orientation. The inventory update module 504 can also be configured to send all captured images of a single session to the remote server 130 via the data stream pipeline of network 132. The inventory update module 504 can also package the sub-sequence data corresponding to each merged identifier, including object identification information, tracking identifier information, and orientation information, and send it along with the captured images to the object storage service terminal.

[0061] When the orientation module 502 detects that an object has been removed from the refrigerator 102, the item that was originally located inside the refrigerator and corresponds to that specific object will be removed from the refrigerator's inventory. This inventory information can be stored in the memory 470, the data memory 472, and / or the remote server 130, and can be output to an application accessible to the portable device 510 for viewing the inventory information on the display screen of the portable device 510.

[0062] Methods for tracking inventory Combination Figure 4 Based on an exemplary embodiment, an inventory tracking method 600 for a refrigerator 102 will be described. Figure 4 The explanation will be combined with Figures 1 to 3 Proceed. For the sake of simplicity, Method 600 is presented as a step-by-step flowchart, but its components can be reorganized into different architectures, units, stages, and / or processes.

[0063] In step 602, method 600 includes sending a start trigger signal when door 142 moves from the closed position to the open position. Specifically, when trigger switch 104 detects that door 142 is in the open position, it sends a start trigger signal to session controller module 480.

[0064] In step 604, method 600 includes initiating a session based on a received start trigger signal. Specifically, the session controller module 480 initiates the session after receiving the start trigger signal from the trigger switch 104.

[0065] In step 606, method 600 includes the session controller module 480 receiving a start trigger signal and capturing images using a plurality of cameras 110 mounted on the body 140 and / or door 142. When the door 142 is in the closed position, the body 140 and the door 142 enclose an interior space of the refrigerator, and the field of view of each camera extends through an entrance opening plane of the interior space of the refrigerator, which is accessible when the door is open. Therefore, the plurality of cameras 110 are configured to capture images of objects passing through the entrance opening plane during a session, i.e., images of objects being placed into or removed from the interior space of the refrigerator.

[0066] In step 608, method 600 includes sending a stop trigger signal when door 142 switches from the open position to the closed position. Specifically, when position sensor 454 detects that door 142 has moved from the open position to the closed position, trigger switch 104 sends a stop trigger signal to session controller module 480. In step 608, method 600 further includes ending the session initiated in step 604 based on the received stop trigger signal. Specifically, after session controller module 480 receives the stop trigger signal from trigger switch 104, it ends the session initiated in step 602, and session controller 480 can be updated to assign a new session identifier for the next session. Multiple cameras 110 stop capturing images after session controller module 480 receives the stop trigger signal.

[0067] In step 610, method 600 includes grouping the captured images taken by the multiple cameras 110 in step 606. Specifically, synchronizer module 482 receives compressed images sent by the multiple cameras 110 and outputs frame sets to buffer module 484 and object recognition module 490, that is, grouping the captured images into frame sets, each frame set consisting of captured images taken by different cameras among the multiple cameras 110 at the same time. Continuing with the description in conjunction with step 610, method 600 also includes: receiving a session identifier, camera identifier, and frame identifier corresponding to each captured image taken by each camera among the multiple cameras 110. The session identifier is associated with the current session, the camera identifier is associated with the specific camera among the multiple cameras 110 that took the image, and the frame identifier is associated with the time the image was taken. Based on this, synchronizer module 482 groups the captured images according to the session identifier, camera identifier, and frame identifier for each captured image.

[0068] In step 620, method 600 includes storing the compressed captured image taken in step 606 into an image buffer memory. The compressed image can be sent to a post-processing end after the session ends. Specifically, buffer module 484 stores images captured by multiple cameras 110 during the session initiated in step 604, and sends the compressed image to synchronizer module 482 for post-processing after the session ends.

[0069] In step 622, method 600 includes determining whether each captured image taken in step 606 contains an object and assigning a category label to the object. Continuing with the explanation in conjunction with step 622, method 600 further includes: outputting a bounding box around the object and a category label for the object in each captured image using a computer vision model. Specifically, object recognition module 490 receives a set of frames output by synchronizer module 482 to determine whether each captured image in the corresponding frame set contains an object, assigns a category label to the object, and outputs a bounding box around the object.

[0070] In step 624, method 600 includes analyzing sequentially captured images from each of the multiple cameras 172 and assigning a corresponding tracking identifier to an object in each analyzed image from each camera. Specifically, object tracking module 492 receives sequentially captured images from object recognition module 490, assigns tracking identifiers to objects, and outputs the tracking identifiers to object merging module 494. In one embodiment, object tracking module 492 employs a simple, observation-centric online real-time tracking algorithm to analyze sequentially captured images and assign tracking identifiers.

[0071] In step 630, method 600 includes comparing tracking identifiers in the analyzed images of different cameras among the multiple cameras 172, and assigning a merge identifier to the object when the same object is detected in the captured images of multiple different cameras among the multiple cameras 172. Specifically, the object merging module 494 receives and compares the tracking identifiers from the object tracking module 492, and outputs the merge identifier to the subsequence partitioning module.

[0072] In step 632, method 600 includes dividing the session into multiple sub-sequences using corresponding merge identifiers, each sub-sequence corresponding to a single inventory change within the refrigerator's internal space. Specifically, the sub-sequence division module 500 receives a merge identifier from the object merging module 494 and outputs the sub-sequence to the direction module 502.

[0073] In step 634, method 600 includes assigning an orientation to each subsequence based on the analyzed image to determine whether the object is moving in, out, or moving within the interior space of the refrigerator. Specifically, orientation module 502 receives subsequences from subsequence segmentation module 500 and outputs the assigned motion orientation to inventory update module 504.

[0074] In step 640, method 600 includes updating the inventory status in the database based on each subsequence and its assigned direction. Specifically, the inventory update module 504 receives subsequences from the subsequence partitioning module 502 and the assigned movement direction from the direction module 502, and updates the inventory status stored in the database.

[0075] One aspect of this disclosure relates to a computer-readable medium having processor-executable instructions stored thereon for implementing at least one of the technical solutions described in this disclosure. Figure 5 An embodiment of a computer-readable medium or computer-readable device implemented in this manner is shown, wherein embodiment 700 includes a computer-readable medium 702 (such as a recordable optical disc CD-R, a digital video optical disc DVD-R, a USB flash drive, a hard disk drive, etc.) on which computer-readable data 704 is encoded. This encoded computer-readable data 704 (such as...) Figure 5 The binary data shown (consisting of multiple 0s and 1s) further includes a set of processor-executable computer instructions 710, configured to operate according to one or more principles described in this disclosure. In this embodiment 700, the processor-executable computer instructions 710 may be configured to execute method 712, such as... Figure 4 Method 600. In another aspect, the processor-executable computer instructions 710 can be configured to implement the system, such as... Figure 1 The operating environment 100 is described above. Those skilled in the art can design various such computer-readable media to implement the technical solutions described in this disclosure.

[0076] The terms “component,” “module,” “system,” “interface,” etc., used in this application generally refer to computer-related entities, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, processing unit, object, executable program, execution thread, program, or computer running on a processor. For instance, an application running on a controller and the controller itself can both be considered a component. One or more components may exist within a process or execution thread, and components may reside on a single computer or be distributed across two or more computers.

[0077] Furthermore, the technical solutions claimed in this application can be implemented as methods, apparatus, or articles of manufacture through standard programming or engineering techniques, controlling a computer through software, firmware, hardware, or any combination thereof to achieve the disclosed technical solutions. The term "article of manufacture" as used in this application is intended to cover computer programs accessible from any computer-readable device, carrier, or medium. Of course, various modifications can be made to this configuration without departing from the scope and spirit of the claims made in this application.

[0078] The term "computer-readable medium" as used in this application includes communication media, which typically carry computer-readable instructions or other data in the form of "modulated data signals" (such as carrier waves or other transmission mechanisms), including any information transmission medium. "Modulated data signals" refers to a signal in which one or more characteristics are set or altered to encode information.

[0079] Although the technical solutions of this application are described in language specific to structural features or method steps, it should be understood that the protected object defined by the appended claims is not limited to the specific features or steps described above. Rather, the specific features and steps described above are disclosed as exemplary embodiments. This application provides various modes of operation for each embodiment, and the order of one or more / all operations described should not be construed as implying a necessary sequential dependency between these operations. Those skilled in the art can understand alternative orders based on this description. Furthermore, not all operations are necessary in the embodiments provided in this application.

[0080] The word "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or," and an inclusive "or" may cover any combination thereof (such as A, B, or any combination thereof). Furthermore, the words "a" and "an" as used in this application are generally interpreted as "one or more" unless otherwise stated or the context clearly indicates the singular form. Additionally, "at least one A and B" and / or similar expressions generally refer to A or B, or both A and B. Moreover, the words "comprising," "having," "having," "accompanying," or variations thereof used in the Detailed Description or Claims have a similar meaning to "comprising."

[0081] Furthermore, unless otherwise stated, terms such as "first" and "second" are not intended to imply time, space, order, or other similar meanings, but are used solely as identifiers or names of features, units, items, etc. For example, the first channel and the second channel typically correspond to channel A and channel B, or two different / identical channels, or the same channel. Additionally, terms such as "comprising" and "including" generally mean including but not limited to.

[0082] Those skilled in the art will understand that various modifications can be made to the disclosed embodiments above, and their features and functions, or alternative or variant forms thereof, can be advantageously combined with a variety of other systems or applications. Furthermore, various unforeseen or unanticipated substitutions, modifications, variations, or improvements that may be subsequently made by those skilled in the art should also be covered by the appended claims.

Claims

1. A refrigerator, characterized in that, The refrigerator includes: A body and a door, wherein the door is pivotally mounted on the body and movable relative to the body between an open position and a closed position, and when the door is in the closed position, the body and the door define a plurality of storage compartments in the interior space of the refrigerator; A plurality of cameras are mounted on at least one of the main body and the door, each camera being positioned to have a field of view including an entrance opening to at least one storage compartment when the door is in the open position, the plurality of cameras being configured to capture images of objects passing through the entrance opening when the objects are loaded into or removed from the at least one storage compartment; and At least one computing device operably connected to the plurality of cameras, the at least one computing device comprising: The session controller module is configured to initiate a session and instruct each camera to start capturing images based on a trigger signal received by the session controller; The synchronizer module is configured to receive captured images from the plurality of cameras and group the captured images into frame sets, each frame set consisting of corresponding captured images captured simultaneously by different cameras among the plurality of cameras; An object recognition module is configured to determine whether each captured image in a corresponding frame set includes the object and to assign a category identifier to the object; The object tracking module is configured to analyze continuously captured images from each camera and assign tracking identifiers to associate the object in the analyzed image with the object in other analyzed images for each camera. The object merging module is configured to compare tracking identifiers between analyzed images from different cameras, and to assign a merging identifier to the object when the object is detected by the object recognition module in captured images captured by multiple different cameras among the plurality of cameras; The subsequence segmentation module is configured to segment the session into subsequences by receiving a corresponding merging identifier from the object merging module, wherein each subsequence corresponds to one inventory change of the at least one storage compartment; A direction module is configured to assign direction to each subsequence based on the analyzed image in order to determine whether the object is moving in, out, or within the interior space of the refrigerator; and The inventory update module is configured to communicate with the database to update the inventory status of the at least one storage compartment based on each inventory change and the assigned direction.

2. The refrigerator according to claim 1, characterized in that, Each of the plurality of cameras mounted on the main body has a field of view that is away from the interior space of the refrigerator.

3. The refrigerator according to claim 1 or 2, characterized in that, Each of the plurality of cameras has a field of view that covers at least a portion of at least one entrance opening leading to at least one of the plurality of storage compartments.

4. The refrigerator according to claim 1, characterized in that, Each of the plurality of storage compartments is associated with at least one of the plurality of cameras, such that the at least one camera has a field of view covering at least a portion of the corresponding entrance interface of a given storage compartment.

5. The refrigerator according to claim 4, characterized in that, Each of the plurality of storage compartments is associated with at least two cameras installed on opposite sides of each corresponding storage compartment.

6. The refrigerator according to claim 1, characterized in that, The direction module assigns the direction to each subsequence based on the known installation position of each camera on the main body.

7. The refrigerator according to claim 6, characterized in that, The refrigerator also includes a door angle sensor configured to determine the angle of the door relative to the body when the door is in the open position, and the direction module assigns the direction to each subsequence based on the angle of the door relative to the body.

8. The refrigerator according to claim 1, characterized in that, The object recognition module is configured to determine whether each captured image in the corresponding frame set includes multiple objects and to assign different category labels to each different object. The object tracking module is configured to assign different tracking identifiers to different objects in each analyzed image for each camera. The object merging module is configured to compare different tracking identifiers in analyzed images from different cameras, and to assign the same merging identifier to the same object when the same object is detected by the object recognition module in captured images captured by multiple different cameras among the plurality of cameras.

9. The refrigerator according to claim 8, characterized in that, The subsequence segmentation module is configured to divide the session into different subsequences by receiving a corresponding merging identifier from the object merging module, wherein each different subsequence corresponds to one inventory change of the at least one storage compartment.

10. A method for tracking inventory in a refrigerator, characterized in that, The method includes: Initiate a session based on the receipt of a trigger signal; Images are captured using a plurality of cameras mounted on at least one of the main body and the door, the door being pivotally mounted on the main body and movable relative to the main body between an open position and a closed position. When the door is in the closed position, the main body and the door define a plurality of storage compartments within the interior space of the refrigerator. Each camera is positioned to have a field of view through an entrance opening plane of the interior space of the refrigerator when the door is in the open position, the entrance opening plane being accessible. The plurality of cameras are configured to capture images of objects passing through the entrance opening plane when the objects are loaded into or removed from the at least one storage compartment. A computing device that communicates with the multiple cameras determines whether each captured image includes an object; The computing device analyzes continuously captured images from each camera and assigns tracking tags to associate the object in the analyzed image with the object in other analyzed images for each camera. The computing device compares tracking identifiers between analyzed images from different cameras, and assigns a merged identifier to the object when the object is detected in captured images captured by multiple different cameras among the plurality of cameras. Using the computing device, the session is divided into sub-sequences by the corresponding merge identifier, wherein each sub-sequence corresponds to one inventory change of at least one storage compartment; Using the computing device, orientations are assigned to each subsequence based on the analyzed images to determine whether the object is moving in, out, or within the interior space of the refrigerator; and Update the inventory status in the database based on each inventory change and the assigned direction.

11. The method according to claim 10, characterized in that, The method further includes sending a start trigger signal when the door moves from the closed position to the open position, and sending a stop trigger signal when the door moves from the open position to the closed position, wherein, upon receiving the start trigger signal, the plurality of cameras begin capturing images, and upon receiving the stop trigger signal, the plurality of cameras stop capturing images.

12. The method according to claim 10, characterized in that, The method further includes receiving a session identifier, a camera identifier, and a frame identifier for each captured image from each of the plurality of cameras, wherein the session identifier is associated with the session, the camera identifier is associated with a given camera among the plurality of cameras, and the frame identifier is associated with the capture time of the captured image.

13. The method according to claim 10 or 12, characterized in that, The method further includes grouping the captured images from the plurality of cameras into frame sets, wherein each frame set consists of corresponding captured images captured simultaneously by different cameras among the plurality of cameras.

14. The method according to claim 10 or 12, characterized in that, The method further includes assigning a category identifier to the object, and using a computer vision model to output a bounding box of the object and a category identifier for the object for each captured image.

15. The method according to claim 10 or 12, characterized in that, The analysis of continuously captured images and the assignment of tracking markers are performed using a simple, observation-centric online real-time tracking algorithm.

16. The method according to claim 10, characterized in that, The method further includes determining the angle of the door relative to the body when the door is in the open position, and assigning the direction to each subsequence based on the angle of the door relative to the body.

17. The method according to claim 10 or 16, characterized in that, Updating the inventory status includes determining inventory changes based on the corresponding item category information, merge identifier information, and direction information, and then recording the inventory changes in the database.

18. The method according to claim 10, characterized in that, The computing device, which communicates with the plurality of cameras, determines whether each captured image includes an object, including determining whether each captured image includes multiple objects; and The computing device analyzes continuously captured images from each camera and assigns tracking identifiers to associate the object in the analyzed image with the object in other analyzed images for each camera, including assigning different tracking identifiers to different objects.

19. The method according to claim 18, characterized in that, The tracking identifiers are compared between analyzed images from different cameras, and when the object is detected in captured images captured by multiple different cameras among the plurality of cameras, a merged identifier is assigned to the object, including assigning different tracking identifiers to different objects; and Specifically, the computing device divides the session into sub-sequences using corresponding merging identifiers, including dividing the session into different sub-sequences using corresponding merging identifiers.

20. A non-volatile computer-readable storage medium storing instructions, when executed by a computer having a processor, to cause the processor to perform a method comprising the following steps: Initiate a session based on the receipt of a trigger signal; Images are captured using a plurality of cameras mounted on at least one of the main body and the door, the door being pivotally mounted on the main body and movable relative to the main body between an open position and a closed position. When the door is in the closed position, the main body and the door define a plurality of storage compartments within the interior space of the refrigerator. Each camera is positioned to have a field of view through an entrance opening plane of the interior space of the refrigerator when the door is in the open position, the entrance opening plane being accessible. The plurality of cameras are configured to capture images of objects passing through the entrance opening plane when the objects are loaded into or removed from the at least one storage compartment. A computing device that communicates with the multiple cameras determines whether each captured image includes an object; The computing device analyzes continuously captured images from each camera and assigns tracking tags to associate the object in the analyzed image with the object in other analyzed images for each camera. The computing device compares tracking identifiers between analyzed images from different cameras, and assigns a merged identifier to the object when the object is detected in captured images captured by multiple different cameras among the plurality of cameras. Using the computing device, the session is divided into sub-sequences by the corresponding merge identifier, wherein each sub-sequence corresponds to one inventory change of at least one storage compartment; Using the computing device, orientations are assigned to each subsequence based on the analyzed images in order to determine whether the object is moving in, out, or moving within the interior space of the refrigerator; and Update the inventory status in the database based on each inventory change and the assigned direction.

Images