System and method for tracking inventory in a container

Abstract

A container system includes a main body, at least one camera, and at least one computing device in operable connection with the at least one camera. The at least one camera is positioned to have a field of view that includes an access opening to at least one storage compartment and is configured to capture images of items being loaded into or removed from the at least one storage compartment. The at least one computing device is configured to instruct the at least one camera to capture the images, analyze the captured images, generate a bounding box around an object of interest in a captured image, and assign a direction of movement to the object of interest as entering, exiting, or inside the housing. The direction of movement is assigned based on a change in size of the bounding box around the object of interest in the analyzed image.

Description

Technical Field

[0001] This application relates to the field of container inventory tracking technology, and more particularly to a system and method for tracking inventory in containers. Background Technology

[0002] Known refrigerators may include cameras for imaging and recording inventory arranged inside the refrigerator. Some known refrigerators include cameras for imaging and recording inventory, wherein said cameras are positioned and designed to each have a field of view pointing inwards from the refrigerator. When inventory items are placed in such refrigerators in a position that obstructs the camera's view of other items, the camera may have difficulty accurately recording the obstructed inventory.

[0003] US Patent Application Publication No. US2016 / 0088262A1 discloses a refrigerator comprising an imaging unit, a data storage unit, a control unit, and a display unit. The imaging unit generates an item loading / unloading video by recording video of stored items being loaded into or removed from the refrigerator. The data storage unit stores the item loading / unloading video, information about items that can be stored in the refrigerator, and information about the stored items. The control unit identifies the loading / unloading of items based on optical flow detected by visual recognition from the item loading / unloading video, and updates the information about the stored items in the refrigerator based on the identification information generated by visual recognition of the item loading / unloading and the item information stored in the data storage unit. The display unit displays the management status of the stored items to the user based on the updated information about the stored items.

[0004] US Patent Publication No. 11,692,769B2 discloses a refrigerator appliance comprising a cabinet defining a cooling compartment, a door rotatably hinged to the cabinet to provide selective access to the cooling compartment, and a camera assembly mounted to the cabinet for monitoring the cooling compartment. A controller is operatively coupled to the camera assembly and configured to use the camera assembly to acquire raw images, analyze the raw images to identify anchored objects, crop the raw images to generate reduced images around the anchored objects, and analyze the reduced images to identify food items being added to or removed from the cooling compartment.

[0005] Inventory tracking of refrigerators can be improved by taking into account aspects of the refrigerator and / or aspects of the images captured by the camera that were not considered in the aforementioned prior art solutions. Summary of the Invention

[0006] In view of the foregoing, a container system includes a body, at least one camera, and at least one computing device operatively connected to the camera. The body defines at least one storage compartment within a housing. The at least one camera is positioned to have a field of view including an entrance opening leading to the at least one storage compartment and is configured to capture images of items being loaded into or removed from the at least one storage compartment. The at least one computing device is configured to: instruct the at least one camera to capture the image based on a trigger signal received by the at least one computing device; analyze the image captured by the at least one camera and generate a bounding box around the object of interest in the captured image containing the object of interest; and assign a direction of movement to the object of interest, either entering, leaving, or within the housing. The direction of movement is assigned based on variations in the size of the bounding box around the object of interest in the analyzed image. Attached Figure Description

[0007] Figure 1 This is a 3D view of a refrigerator, with the door in the open position.

[0008] Figure 2 This is a front view of the refrigerator, with the door in the open position.

[0009] Figure 3 A block diagram illustrating the operating environment of a refrigerator, including an inventory tracking system.

[0010] Figure 4 This is a flowchart illustrating the direction in which items are distributed to enter, leave, or enter the refrigerator.

[0011] Figure 5 Here is an example of an analyzed image containing objects of interest.

[0012] Figure 6 A schematic diagram of a computer-readable medium or computer-readable device, including processor-executable instructions configured to implement at least one aspect. Detailed Implementation

[0013] Reference Figure 1 The refrigerator 102 is shown to include a body 104 and a door 106, the door 106 being pivotally mounted to the body 104 in a manner similar to that of a known refrigerator and being movable relative to the body 104. Figure 1 The refrigerator 102 shown is an example of a container that can store items such as food and beverages. Although it will be shown in more detail... Figure 1The refrigerator 102 shown may also include other containers (such as insulated boxes, pantry, closets, cabinets, cupboards, and similar storage containers) that can store items such as food and beverages, and may also include aspects for tracking inventory within the containers, as shown in more detail below. Therefore, the inventory tracking system shown in more detail below should not be construed as being limited to refrigerators unless explicitly shown so. Figure 1 and Figure 2 The refrigerator 102 shown is an example of a refrigerator that can adopt the aspects shown in more detail below. However, the refrigerator 102 may adopt other configurations, such as including two pivot doors, sliding doors, or having no doors. Figure 1 and Figure 2 The refrigerator 102 shown below is illustrated in more detail and should be understood to be adaptable to any of the alternative constructions described above. Figure 1 and Figure 2 In the illustrated embodiment, door 106 can be in multiple open positions ( Figure 1 The door 106 moves between one of its positions (shown in the figure) and a closed position. When the door 106 is in the closed position, the body 104 and the door 106 define at least one storage compartment within the refrigerated housing 108. For containers that do not include a refrigeration system (e.g., insulated boxes, food storage compartments, closets, cabinets, cupboards), when the door is in the closed position, the body and the door may simply define at least one storage compartment within the housing. As shown, when the door 106 is in the closed position, the body 104 and the door 106 define multiple storage compartments 112, 114, 116, 118, 120, 122, 124, and 126 within the refrigerated housing 108. The plurality of storage rooms includes a first storage room 112, a second storage room 114, a third storage room 116, a fourth storage room 118, a fifth storage room 120 (each located within the main body 104 in the illustrated embodiment), and a sixth storage room 122, a seventh storage room 124, and an eighth storage room 126 (located on the inside 128 of the door 106 in the illustrated embodiment). The number and location of the storage rooms 112, 114, 116, 118, 120, 122, 124, and 126 may vary. Figure 1 and Figure 2 As shown.

[0014] like Figure 1 As shown, the first shelf 132 and the second shelf 134 are arranged in the first storage compartment 112. The third shelf 136 is arranged in the second storage compartment 114. The first drawer 142 is arranged in the third storage compartment 116. The second drawer 144 is arranged in the fourth storage compartment 118. The third drawer 146 is arranged in the fifth storage compartment 120. The first storage box 152 is arranged in the sixth storage compartment 122. The second storage box 154 is arranged in the seventh storage compartment 124. The third storage box 156 is arranged in the eighth storage compartment 126.

[0015] In the illustrated embodiment, door 106 is connected to body 104 via upper hinge 162 and lower hinge 164. Door 106 is movable relative to body 104 between multiple open positions based on the angle of door 106 relative to body 104.

[0016] Each storage room 112, 114, 116, 118, 120, 122, 124, and 126 has a designated entrance opening, through which items can be placed into and retrieved from their respective storage rooms. More specifically, refer to... Figure 2 The first storage compartment 112 defines a first entrance opening 172 leading to the first shelf 132 and the second shelf 134, and the second storage compartment 114 defines a second entrance opening 174 leading to the third shelf 136. The third storage compartment 116 defines a third entrance opening leading to the first drawer 142 (in... Figure 1 and Figure 2 (Not visible in the middle), the fourth storage room 118 defines the fourth entrance opening leading to the second drawer 144 (in Figure 1 and Figure 2 (Not visible in the middle), the fifth storage room 120 is limited to the fifth entrance opening leading to the third drawer 146 (in Figure 1 and Figure 2 (Not visible in the middle) When the corresponding drawers 142, 144, and 146 are opened. The sixth storage compartment 122 defines a sixth entrance opening 180 leading to the first storage box 152, the seventh storage compartment 124 defines a seventh entrance opening 182 leading to the second storage box 154, and the eighth storage compartment 126 defines an eighth entrance opening 184 leading to the third storage box 156.

[0017] At least one camera is positioned to have a field of view including an entrance opening, such as at least one of entrance openings 172, 174, 180, 182, 184, and an opening leading to drawers 142, 144, 146. Figure 1 and Figure 2 As shown, multiple cameras 250 (see) Figure 3 The camera is mounted to the main body 104 and door 106 to capture images of objects moving into and out of the plurality of storage compartments 112, 114, 116, 118, 120, 122, 124, 126. Alternatively, however, at least one camera or the plurality of cameras 250 may be mounted in a location not attached to the refrigerator 102, while still having a field of view including the entrance opening. (Refer to back) Figure 1 and Figure 2In the illustrated embodiment, the plurality of cameras 250 includes: a first set of cameras 260a and 260b mounted on the main body 104 and associated with the first storage compartment 112; a second set of cameras 262a and 262b mounted on the main body 104 and associated with the second storage compartment 114; a third set of cameras 264a and 264b mounted on the main body 104 and associated with the third storage compartment 116; a fourth set of cameras 270a and 270b mounted on the main body 104 and associated with the fourth storage compartment 118; and a fifth set of cameras 272a and 272b mounted on the main body 104 and associated with the fifth storage compartment 120. Figure 1 As shown more clearly in the diagram, the plurality of cameras 250 also includes: a sixth set of cameras 274a, 274b mounted on door 106 and associated with the sixth storage compartment 122; a seventh set of cameras 280a, 280b mounted on door 106 and associated with the seventh storage compartment 124; and an eighth set of cameras 282a, 282b mounted on door 106 and associated with the eighth storage compartment 126. Figure 1 and Figure 2 As shown, cameras with the suffix "a" are located on the left side of the corresponding storage room, and cameras with the suffix "b" are located on the right side of the corresponding storage room.

[0018] exist Figure 1 and Figure 2 In the illustrated embodiment, each camera 260a, 260b, 262a, 262b, 264a, 264b, 266a, 266b, 270a, 270b, 272a, 272b on the body 104 has a field of view away from the interior of the refrigerator 102, rather than a field of view facing the interior of the refrigerator 102. When the door 106 is in the open position, each camera 274a, 274b, 280a, 280b, 282a, 282b on the door 106 also has a field of view away from the interior of the refrigerator 102. In an alternative configuration where at least one camera or the plurality of cameras 250 are mounted off-center from the refrigerator 102, the field of view of at least one or all cameras may face the interior of the refrigerator 102. Referring back to the illustrated embodiment, the first set of cameras 260a, 260b is positioned in the top of the first storage compartment 112, above the first shelf 132 and the second shelf 134, and its field of view covers the first entrance opening 172. The second set of cameras 262a and 262b are positioned at the top of the second storage compartment 114, above the third shelf 136, and their field of view covers the second entrance opening 174. The third set of cameras 264a and 264b are positioned above the first drawer 142, and their field of view covers the third entrance opening leading to the first drawer 142 when the first drawer 142 is opened (in... Figure 1 and Figure 2 (Not visible in the middle). The fourth set of cameras 270a and 272b are positioned above the second drawer 144, and their field of view covers the fourth entrance opening leading to the second drawer 144 when the second drawer 144 is opened (in the middle). Figure 1 and Figure 2 (Not visible in the middle). The fifth set of cameras 272a and 272b are positioned above the third drawer 146, and their field of view covers the fifth entrance opening leading to the third drawer 146 when the third drawer 146 is opened (in the middle). Figure 1 and Figure 2 (Not visible in the center). The sixth set of cameras 274a and 274b are mounted above the first storage box 152 on the door 106, and their field of view covers the sixth entrance opening 180. The seventh set of cameras 280a and 280b are mounted above the second storage box 154 on the door 106, and their field of view covers the seventh entrance opening 182. The eighth set of cameras 282a and 282b are mounted above the third storage box 156 on the door 106, and their field of view covers the eighth entrance opening 184.

[0019] In this way, each of the plurality of cameras 250, 260a, 260b, 262a, 262b, 264a, 264b, 272a, 272b, 274a, 274b, 280a, 280b, 282a, 282b, is positioned to have a field of view including corresponding entrance openings 172, 174, 180, 182, 184, which lead to corresponding storage rooms 112, 114, 116, 118, 120, 122, 124, 126 when the door 106 is in the open position. With this configuration, the plurality of cameras 250 are configured to capture images of an item passing through any of the entrance openings 172, 174, 180, 182, 184, when the item is being loaded into or removed from the corresponding storage compartment 112, 114, 116, 118, 120, 122, 124, 126. Figure 1 and Figure 2 As shown, specific cameras (e.g., cameras 260a, 262a, 264a, 270a, 272a, 274a, 274b, 280a, 280b, 282a, 282b) also have a field of view that may include the inner side 128 of door 106 when door 106 is in at least one of a plurality of open positions. The angle of door 106 relative to body 104 can be used to determine the orientation of an item, for example, when analyzing images captured by cameras 260a, 260b, 262a, 262b, 264a, 264b, 272a, 272b, 274a, 274b, 280a, 280b, 282a, 282b, whether the item is being loaded into or removed from the corresponding storage compartment 112, 114, 116, 118, 120, 122, 124, 126.

[0020] Reference Figure 3 The refrigerator 102 includes a computing device 400 operatively connected to each of the plurality of cameras 250. (Continuing to refer to...) Figure 3The refrigerator 102 may also include a trigger switch 404, a light 412, a cooling system 414, and a door angle sensor 416. The trigger switch 404, the plurality of cameras 250, the light 412, the cooling system 414, and the door angle sensor 416 may be connected to the computing device 400 via a bus 424 or a similar type of electrical connection.

[0021] Trigger switch 404 is operatively connected to computing device 400 and configured to detect the position of door 106 to indicate whether door 106 is in one of a plurality of open positions or a closed position. Trigger switch 404 is configured to send a trigger signal to computing device 400 based on whether door 106 is in one of a plurality of open positions or a closed position. More specifically, when door 106 moves from a closed position to one of a plurality of open positions, trigger switch 404 may be configured to send a start trigger signal to computing device 400, and when door 106 moves to a closed position, trigger switch 404 may be configured to send a stop trigger signal to computing device 400. Trigger switch 404 may include at least one of various types of position sensors, such as mechanical switches, potentiometers, piezoelectric sensors, Hall effect sensors, and eddy current sensors, without departing from the scope of this application. Furthermore, although trigger switch 404 is shown as being arranged on body 104, trigger switch 404 may be additionally or alternatively positioned on door 106.

[0022] Computer device 400 controls light 412 (only when...) Figure 3 (The lamp is depicted schematically, but is similar to a lamp in a known refrigerator). The operation of lamp 412 can be controlled by the state of trigger switch 404, similar to a lamp in a known refrigerator. The operation of lamp 412 can also be controlled by computing device 400 for optimization processing by the computer vision model shown below.

[0023] Cooling system 414 may be similar to the cooling system in a known refrigerator, and therefore will not be shown in detail. Cooling system 414 may include a temperature sensor for sensing the temperature inside refrigerator 102 and a blower fan for supplying air cooled by a cooling device (not shown), which may include a compressor (not shown) for compressing refrigerant, a condenser (not shown) for condensing the compressed refrigerant, an expansion valve (not shown) for depressurizing the condensed refrigerant, and an evaporator (not shown) for evaporating the depressurized refrigerant.

[0024] Door angle sensor 416 is configured to determine the angle of door 106 relative to body 104. Door angle sensor 416 may include, for example, a rotary encoder having a shaft that pivots with door 106 relative to body 104 to output and communicate with computing device 400, thereby conveying the relative angular position of door 106 relative to body 104.

[0025] Reference Figures 3 to 5The computing device 400 is typically configured to instruct at least one of the plurality of cameras 250 to capture an image based on a trigger signal received by the computing device 400. Figure 4 (At box 430 in the image). A trigger signal can be sent to computing device 400 by trigger switch 404. Computing device 400 is also configured to analyze images captured by the plurality of cameras 250 (in... Figure 4 (at box 432 in the middle), and generate bounding box 436 around object of interest 438 ( Figure 5 )(exist Figure 4 (See box 440 in the image). Object of interest 438 may correspond to an item being loaded into or removed from refrigerator 102. Figure 5 This is an example of an image captured by one of the plurality of cameras 250. More similar images from the same and different cameras are analyzed at box 432, and bounding boxes are generated at box 440 (similar to bounding box 436). The computing device 400 is also configured to assign a motion direction (at box 442) to the object of interest, said motion direction being entering, leaving, or entering the refrigerated housing 108 ( Figure 1 At least one of the following. The direction of motion is assigned based on the change in the size of the bounding box 436 surrounding the object of interest 438 in the analyzed image.

[0026] To accomplish the above tasks, computing device 400 may include a processor 464, a memory 470, a data storage device 472, and a communication interface 474, each operatively connected via a bus 424 for computer communication. The communication interface 474 provides software and hardware to facilitate data input and output between components of computing device 400 and other components, networks, and data sources, as illustrated in this application. Computing device 400 may also include a session controller module 480, a synchronizer module 482, a buffer module 484, an object detection module 490, an object tracking module 492, a direction module 494, an object merging module 502, and an inventory update module 504.

[0027] The session controller module 480 is configured to initiate a session and, based on a start trigger signal received by the session controller module 480 from the trigger switch 404, instructs each of the plurality of cameras 250 to begin capturing images. The plurality of cameras 250 begin capturing images after the session controller module 480 receives the start trigger signal from the trigger switch 404, and the plurality of cameras 250 stop capturing images after the session controller module 480 receives a stop trigger signal from the trigger switch 404. Each of the plurality of cameras 250 and / or computing device 400 can be configured to: compress the captured images; assign a session identifier (ID), a camera identifier (ID), and a frame identifier (ID) to each captured image; and send the compressed image along with its respective session ID, camera ID, and frame ID to the synchronizer module 482, which may occur during or after a session. A session refers to the time between the opening and closing of the door 106. A session can be relatively long and may include moving multiple items into and out of the refrigerator 102.

[0028] Synchronizer module 482 is configured to receive captured images from the plurality of cameras 250, along with a session ID, camera ID, and frame ID associated with each captured image, and to group the captured images into frame sets based on the frame ID associated with each captured image. Each frame set consists of corresponding captured images captured simultaneously from different cameras among the plurality of cameras 250. In one embodiment, synchronizer module 482 groups captured images from the plurality of cameras 250 based on the frame ID associated with each image. In this respect, the same frame ID between different captured images indicates that these images were captured at the same time, although from different cameras. In addition to the frame ID of each captured image received from each of the plurality of cameras 250, synchronizer module 482 also receives a session ID and a camera ID. The session ID is associated with a session assigned by session controller module 480, the camera ID is associated with a specific camera among the plurality of cameras 250 that captured the captured image, and the frame ID is associated with the time the captured image was captured, as described above.

[0029] The buffer module 484 is configured to store the compressed captured image as an image buffer memory, and is configured to send the compressed captured image to a remote server, such as remote server 506, via a network 508 (e.g., the Internet). Figure 3 This configuration allows the system resources of the computing device 400 to be used for image capture during a session and then for post-processing after the session ends, thereby reducing the capacity required for the computing device 400 to receive image information from the plurality of cameras 250 and track the inventory in the refrigerator 102.

[0030] Reference Figure 3 and Figure 5 The object detection module 490 employs a computer vision algorithm to identify objects in an image or video. More specifically, the object detection module 490 can be configured to: determine whether each captured image in a corresponding frame set contains an object of interest 438; define a bounding box 436 around each object of interest 438 in the corresponding captured image; and assign a category identifier to the object of interest 438. The category identifier assigned by the object detection module 490 indicates the inventory type of the object of interest 438. The inventory type can be stored in the memory 470 and / or data storage device 472 of the computing device 400, or stored remotely and accessed via a network 508. The inventory type can indicate the category of ingredients, food, meals, dishes, or other object types stored in a database.

[0031] Object detection module 490 includes a computer vision model that outputs around object 438 of a captured image containing an object of interest. Figure 5 ) bounding box 436 ( Figure 5 The object detection module 490 may include additional or alternative computer vision models to determine the presence and category identification of objects in the refrigerator 102 without departing from the scope of this application. The computer vision model used in the object detection module 490 may also be configured to remove images that do not contain objects of interest from the frame set received by the synchronizer module 482 and output an updated image set to the object tracking module 492. The updated image set (which may contain only captured images of the objects of interest) may include fewer images than the frame set received by the object detection module 490 from the synchronizer module 482. The first captured image in the updated image set containing the objects of interest may indicate the start of a sequence, showing a hand movement inside or outside the refrigerator. The last captured image in the updated image set containing the objects of interest may indicate the end of a sequence. In cases where the object detection module 490 does not identify an object of interest, such as when the object detection module 490 identifies an empty hand without any objects to be placed in or removed from the refrigerator, the last captured image containing the objects of interest may indicate the end of a sequence. Each sequence (or detected movement of refrigerator 102 into or out of the refrigerator) may contain multiple objects of interest.

[0032] The object tracking module 492 is configured to analyze sequentially captured images from each of the plurality of cameras 250, tracking objects of interest between the sequentially captured images. For example, the object tracking module 492 may compare the first n images captured in an updated image set during the sequence with the last n images captured in the updated image set during the sequence to track the movement of the object of interest during the sequence. In the illustrated embodiment, the number of images n is 3, but this number may be more or less. The temporal order of the captured images is determined based on the frame ID associated with each image. For example, a captured image with a higher frame ID may be captured at a later time compared to a captured image with a lower frame ID. The camera that captured the corresponding captured image is determined based on the camera ID associated with the corresponding captured image. In this respect, the object tracking module 492 associates the object of interest with itself between sequentially captured images, thereby tracking the movement of the object of interest over time.

[0033] The object tracking module 492 can also be configured to assign tracking identifiers to associate an object of interest (OP) in an analyzed image with other OOPs in other analyzed images from each of the plurality of cameras 250. The tracking identifier assigned by the object tracking module 492 is unique for each of the plurality of cameras 250, and also unique for each OOP captured sequentially by the plurality of cameras 250 in the captured images. For example, the object tracking module 492 is configured to assign a different tracking identifier to an OOP that appears as a lemon in a captured image than to an OOP that appears as a cup of yogurt in a captured image.

[0034] The object tracking module 492 may employ an observation-centric simple online and real-time tracking (OCSORT) algorithm to analyze sequentially captured images from each of the plurality of cameras 250. With this configuration, the object tracking module 492 receives an updated image set with bounding boxes and class IDs from the object detection module 490. The OCSORT algorithm processes this updated image set and then outputs a further updated image set with bounding boxes, class IDs, and also includes the associated tracking IDs of each object located in each captured image within the further updated image set.

[0035] The orientation module 494 employs computer algorithms to provide real-time orientation predictions for moving objects in the video stream. Using input parameters such as the object position (e.g., x, y coordinates) associated with the object of interest and gate angles (e.g., the angle of gate 106 relative to body 104), the orientation module 494 can track changes in the bounding box area or size and the x, y coordinates. Operating across the entire video, the orientation module 494 can generate predictions for each of the plurality of cameras 250.

[0036] The orientation module 494 can be configured to, for each of the plurality of cameras 250, output whether the item in the analyzed image associated with the object of interest was moved into the refrigerator, moved out of the refrigerator, or neither. The orientation module 494 may employ machine learning algorithms, such as a visual transformer (e.g., CSWin Transformer) or other machine learning algorithms capable of performing the functions described in this application. Alternatively, the orientation module 494 may employ a rule-based algorithm. Furthermore, the orientation module 494 may employ both rule-based and machine learning algorithms simultaneously.

[0037] When using rule-based algorithms and / or machine learning algorithms, the orientation module 494 can receive the sensed gate angle and object position information associated with the bounding box 436 surrounding the object of interest 438 as input parameters. For example, the orientation module 494 can receive the sensed gate angle and object position information from the object detection module 490 or the object tracking module 492 for the first n analyzed images and the last n analyzed images during the sequence. The number n can be 3 images; however, fewer or more can be used. Since the analyzed images are images containing the object of interest, the first n analyzed images indicate that the object has entered the field of view of a particular camera, and the last n images indicate that the object has left the field of view of a particular camera.

[0038] Changes in the angle of the gate 106 relative to the body 104 can cause changes in the object appearance, scale, orientation, and occlusion of at least some of the plurality of cameras 250. For example, since the cameras 274a, 274b, 280a, 280b, 282a, and 282b mounted to the gate 106 can move relative to the body 104, knowing the angle of the gate 106 relative to the body 104 when capturing a particular image can provide useful information when evaluating changes in the bounding box 436 and the object position around the object of interest 438 in the first n analyzed images and the next n analyzed images during the evaluation sequence. Furthermore, since the storage compartments 122, 124, and 126 on door 106 move relative to cameras 260a, 260b, 262a, 262b, 264a, 264b, 272a, and 272b mounted to the main body 104, the angle of door 106 relative to the main body 104 can provide useful information when evaluating changes in the bounding box 436 and object position associated with the object of interest in the first n analyzed images and the next n analyzed images during the evaluation sequence. Additionally, sensing changes in the door angle between the n analyzed images can be at least partially used to assign the direction of motion of the object of interest within the sequence. The sensing changes in the door angle between the n analyzed images can be changes in magnitude, i.e., differences in degrees or radians, and / or changes in direction, i.e., the opening or closing direction.

[0039] The computing device 400 (e.g., within the object detection module 490) can be configured to display the position and size of the bounding box 436 of the corresponding image in the analyzed images in pixel coordinates. Variations in the area or size of the bounding box 436 between analyzed images can indicate the orientation of the object of interest 438. For example, an increase in the area of ​​the bounding box 436 of the same camera from an earlier captured image to a later captured image can indicate that the object of interest 438 is moving toward the camera. Since the position of each camera relative to the plurality of storage chambers 112, 114, 116, 118, 120, 122, 124, 126 is known (this may require knowing the angle of the door 106 relative to the body 104 in a particular image), variations in the area or size of the bounding box 436 between analyzed images can indicate the orientation of the object of interest 438 as it enters, exits, or enters the refrigerated housing 108. Therefore, the orientation module 494 can be configured to output, for each of the plurality of cameras 250, whether the item associated with the object of interest 438 in the analyzed image is moved into the refrigerator, moved out of the refrigerator, or neither, based on changes in the area or size of the tracking bounding box.

[0040] The object location associated with the object of interest corresponds to a point within each bounding box of the n analyzed images. For example, the object location could be a similar predetermined location within each bounding box 436 of each of the n analyzed images. The object location within the bounding box 436 determines the object location of the object of interest 438 and can be represented as x, y coordinates within the analyzed images. In a more specific example, the computing device 400 (e.g., within the object detection module 490 or orientation module 494) is configured to locate the centroid or geometric center of the bounding box 436 surrounding the object of interest 438 in the first n analyzed images and the next n analyzed images. The location of the geometric center within each analyzed image, and the change in the centroid's location from one analyzed image to another, provides an indication of the movement and direction of motion of the object of interest 438 enclosed by the bounding box 436. Therefore, not only can changes in the area or size of the bounding box be tracked between analyzed images, but changes in the geometric center, which can be represented as x, y coordinates, can also be tracked between analyzed images and used to determine the direction of motion of the object of interest 438.

[0041] More specifically, when employing a rule-based algorithm, the orientation module 494 can, for each of the plurality of cameras 250, determine the orientation of the object of interest 438 over time between sequential images, referring to the corresponding entrance openings 172, 174, 180, 182, 184 in a three-dimensional grid space. In one embodiment, the orientation module 494 compares the first captured image from each camera during the sequence with the last captured image to determine the orientation. The rule-based algorithm is configured to determine the motion direction of the object of interest 438 within the sequence, for example, based on knowing the position of each of the plurality of cameras 250 relative to the entrance openings 172, 174, 180, 182, 184. (Refer to previous text) Figure 1 and Figure 2Because the mounting positions of the first set of cameras 260a and 260b on the main body 104 relative to the first entrance opening 172 are known, the positions of the first set of cameras 260a and 260b relative to the first entrance opening 172 are also known within the field of view of each camera 260a and 260b. If the first captured image of the sequence detects an object inside the plane defined by the first entrance opening 172, and the last captured image of the sequence detects an object outside the plane defined by the first entrance opening 172, then the direction module 494 can assign a direction of departure from the refrigerator 102 to the object of interest 438 relative to that particular camera. Conversely, if the first captured image of the sequence detects that the object of interest 438 is outside the plane defined by the first inlet opening 172, and the last captured image of the sequence detects that the object of interest 438 is inside the plane defined by the first inlet opening 172, then the direction module 494 may assign the direction of entering the refrigerator 102 to the object.

[0042] Reference Figure 3 The computing device 400 may also include a gate angle module 520, which is configured to determine a sensing gate angle based on features extracted from corresponding images of n images captured by corresponding cameras mounted to the body 104. For example, Figure 2 Any one of the cameras 260a, 262a, 264a, 270a, and 272a located to the left of the main body 104 can be used to capture images of the door 106 in one of several open positions. Features (e.g., the lens position of camera 280a, the lens position of camera 280b, and the lens position of camera 274a) can be extracted from the images to determine the angle of the door 106 relative to the main body 104, since the scale and orientation of the three positions are different for each door angle. Therefore, the door angle module 520 can be used in place of the door angle sensor 416 to measure and sense the door angle.

[0043] The object merging module 502 employs a computer algorithm that merges orientation predictions obtained from multiple cameras among the plurality of cameras 250. The algorithm of object merging module 502 processes the predictions from each camera, identifies unique inventory changes, and combines the results to produce a comprehensive set of merged predictions. The algorithm of object merging module 502 receives predictions from multiple cameras, which can be organized as a dictionary (cam_to_preds), where each camera is associated with a list of predictions. Each prediction can be represented as a tuple (track_id, class_id, change), where track_id is a unique identifier for the object of interest 438, class_id indicates the category of the object of interest 438, and change indicates a detected orientation change of the object of interest 438, such as entering, leaving the refrigerated housing 108, or no change. The algorithm within object merging module 502 can then iterate through the predictions of each camera, associating the track_ids of multiple cameras based on the movement of the object of interest 438, and creating a unique identifier (inventory_change_id) for each inventory change based on the category identifier of the object of interest 438 and the detected change. The algorithm of object merging module 502 then counts the occurrences of each unique inventory change in the predictions of each camera, and then combines the counts of each unique inventory change across all cameras. If a particular inventory change is observed in multiple cameras, the algorithm of object merging module 502 retains the maximum count, thereby increasing the likelihood of accurately representing the overall occurrence. The merged results are stored in a format that captures class_id, change, and occurrence count. The final result is a list of tables, each containing class_id (food), inventory change, and the number of inventory changes of that item observed across all cameras.

[0044] The inventory update module 504 is configured to communicate with the memory 470 and / or data storage device 472 via bus 424, and / or with the remote server 506 via network 508, to update the inventory status of at least one of the plurality of storage rooms 112, 114, 116, 118, 120, 122, 124, 126 based on each inventory change and its allocation direction. The inventory update module 504 can also be configured to send all captured images of a session to the remote server 506 via a data pipeline routed through network 508. The inventory update module 504 can also be configured to package corresponding sequence data, including object identification information, tracking identification information, and direction information for each merged identifier, and send the sequence data along with the captured images to the object storage service.

[0045] When an object of interest is detected by orientation module 494 as having left refrigerator 102, the object of interest located in the refrigerator and associated with that specific item is removed from the refrigerator's inventory. This inventory information can be stored in memory 470, data storage device 472, and / or remote server 506, and can be output to an application accessible by portable device 530 to view the inventory information on the display of portable device 530.

[0046] One aspect of this application relates to a computer-readable medium comprising processor-executable instructions configured to implement at least one aspect of the technical solutions proposed in this application. One aspect of a computer-readable medium or computer-readable device designed in this manner... Figure 6 As shown, implementation 700 includes a computer-readable medium 702 (e.g., CD-R, DVD-R, flash drive, hard disk platter, etc.) on which computer-readable data 704 is encoded. This encoded computer-readable data 704 (e.g., includes...) Figure 6 The binary data (multiple zeros and ones shown) further includes a set of processor-executable computer instructions 710 configured to operate according to at least one principle set forth in this application. In this implementation 700, the processor-executable computer instructions 710 may be configured to execute method 712, for example... Figure 4 The method. On the other hand, the processor-executable computer instructions 710 can be configured to implement the system, for example... Figure 3 The operating environment is 100. Many such computer-readable media can be designed by those skilled in the art to operate according to the technical solutions proposed in this application.

[0047] When used in this application, the terms "component," "module," "system," "interface," etc., are generally intended to refer to a computer-related entity, whether it is hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, processing unit, object, executable file, execution thread, program, or computer running on a processor. For instance, both an application running on a controller and the controller itself can be components. At least one component residing within a process or execution thread can reside on a single computer or be distributed across two or more computers.

[0048] Although the subject matter has been shown in language specific to structural features or method steps, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or steps described above. Rather, the specific features and steps described above are disclosed as exemplary aspects. This application provides various operations for each aspect. The order in which at least one or all operations are shown should not be construed as implying that these operations necessarily depend on the order. Alternative orders should be understood based on this application. Furthermore, not all operations are necessarily present in every aspect provided in this application.

[0049] When used in this application, "or" is intended to indicate an inclusive "or" rather than an exclusive "or". Furthermore, an inclusive "or" can include any combination thereof (e.g., A, B, or any combination thereof). Additionally, "a" as used in this application is generally interpreted as meaning "at least one", unless otherwise stated or clearly indicated from the context to be in the singular form. Furthermore, at least one of A and B and / or the like generally means A or B or both A and B. Moreover, within the scope of terms such as "comprising," "including," "having," "containing," or variations thereof used in detailed illustrations or claims, such terms are intended to be inclusive in a manner similar to the term "at least comprising".

[0050] Furthermore, unless otherwise stated, terms such as "first," "second," etc., are not intended to imply temporal, spatial, or sequential aspects. Rather, such terms are used only as identifiers, names, etc., for features, elements, items, etc. For example, the first channel and the second channel typically correspond to channel A and channel B, or two different channels, or two identical channels, or the same channel. Additionally, terms such as "including," "comprising," "having," and "containing" generally mean, but are not limited to, those included in the preceding text.

[0051] It should be understood that various aspects of the disclosed embodiments, as well as other features and functions, or alternatives or variations thereof, can be desirously combined with many other different systems or applications. Furthermore, the inventory tracking system specifically illustrated above with reference to a refrigerator can be used with other containers comprising a body defining at least one storage compartment, such as insulated boxes, food storage compartments, closets, cabinets, cupboards, and similar storage containers. Moreover, various alternatives, modifications, variations, or improvements that are not currently foreseen or anticipated can subsequently be made by those skilled in the art, and these are also intended to be covered by the appended claims.

Claims

1. A container system, characterized in that, The container system includes: A main body defining at least one storage chamber within the shell; At least one camera is positioned to have a field of view including an entrance opening leading to the at least one storage room, the at least one camera being configured to capture images of items being loaded into or removed from the at least one storage room; and At least one computing device, operatively connected to the at least one camera, the at least one computing device being configured to: Based on the trigger signal received by the at least one computing device, the at least one camera is instructed to capture the image; Analyze the images captured by the at least one camera, and generate a bounding box around the object of interest in the captured image containing the object of interest; and Assign a direction of movement to the object of interest, either entering, leaving, or within the housing, wherein the direction of movement is assigned based on the variation in the bounding box size around the object of interest in the analyzed image.

2. The container system according to claim 1, characterized in that, The at least one computing device is also configured to indicate the position and size of the bounding box of the corresponding image in the analyzed image in pixel coordinates.

3. The container system according to claim 2, characterized in that, The at least one computing device is also configured to assign object locations within each bounding box and associate x and y coordinates for the object locations in each corresponding image.

4. The container system according to claim 3, characterized in that, The object's position corresponds to the geometric center of the bounding box.

5. The container system according to claim 3, characterized in that, The at least one computing device is also configured to track changes in the bounding box size and x, y coordinates between the analyzed images.

6. The container system according to claim 5, characterized in that, The at least one computing device is also configured to assign a motion direction to the object of interest based on the changes in the x and y coordinates between the analyzed images.

7. The container system according to claim 6, characterized in that, The at least one camera includes a plurality of cameras, each of which is operatively connected to the at least one computing device. The at least one computing device is further configured to assign a predicted motion direction to the object of interest for each of the plurality of cameras that has captured an image containing the object of interest.

8. The container system according to claim 7, characterized in that, The at least one computing device is further configured to assign a predicted motion direction to the object of interest for each of the plurality of cameras; The predicted motion direction assigned to a particular camera among the plurality of cameras is based on changes in the bounding box size and x, y coordinates in the analyzed image captured by that particular camera.

9. The container system according to claim 7, characterized in that, The at least one computing device is further configured to: Analyze sequentially captured images from each of the plurality of cameras and assign a tracking identifier that associates the object of interest with itself in the analyzed sequentially captured images from different cameras among the plurality of cameras; as well as A motion direction is assigned to the object of interest based on the predicted motion direction from each of the plurality of cameras and the tracking identifier.

10. A non-transitory computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and the method by which the instructions, when executed by a computer having a processor, cause the processor to perform: Based on the trigger signal received by the processor, at least one camera is instructed to capture an image; Analyze the images captured by the at least one camera, and generate a bounding box around the object of interest in the captured image containing the object of interest; and Assign a direction of movement to the object of interest, either entering, leaving, or within the housing, wherein the direction of movement is assigned based on the variation in the bounding box size around the object of interest in the analyzed image.

Images