Refrigeration appliance with smart drawer

Abstract

Methods of operating a refrigeration appliance are provided. The refrigeration appliance includes a cabinet having a food storage compartment with a drawer slidably mounted within the food storage compartment. The refrigeration appliance also includes a sensor for detecting an atmospheric condition within the food storage compartment of the drawer and a camera assembly configured to monitor the drawer. The method generally includes acquiring an image using the camera assembly and analyzing the image to identify a first food and a second food in the food storage compartment of the drawer.

Description

Technical Field

[0001] The present invention relates generally to refrigeration appliances, and more specifically to systems and methods for managing the state of articles (such as agricultural products) stored in such refrigeration appliances. Background Technology

[0002] Refrigeration appliances typically include cabinets with freezer compartments. A wide variety of foods can be stored in the freezer compartment. The lower temperature of the freezer compartment relative to the ambient atmosphere can extend the shelf life of foods stored there.

[0003] Agricultural products (e.g., fruits and vegetables) stored in refrigerated appliances undergo various physical and chemical changes over time, such as ripening. Different agricultural products may be incompatible with each other, for example, they may have different storage requirements. For instance, the optimal temperature, humidity, and / or atmospheric composition for one agricultural product may differ from that of another. Different agricultural products may also be incompatible when stored together because each product undergoes different changes over time during storage; for example, one agricultural product may produce certain atmospheric chemicals that are harmful to other agricultural products as it ripens.

[0004] Therefore, refrigeration appliances with improved inventory management systems would be very useful. More specifically, refrigeration appliances with agricultural product inventory management systems capable of monitoring agricultural product inventory and tracking the status of such items during storage would be very useful. Summary of the Invention

[0005] Various aspects and advantages of the invention will be set forth in part in the description which follows, or may become apparent from the description, or may be learned by practice of the invention.

[0006] In an exemplary embodiment, a method of operating a refrigeration appliance is provided. The refrigeration appliance includes a cabinet having a food storage compartment, the food storage compartment having drawers slidably mounted within the food storage compartment. The drawers are slidable between a closed position and an open position. The drawers include walls defining a plurality of food storage compartments. The refrigeration appliance also includes sensors for detecting atmospheric conditions within the food storage compartments of the drawer and a camera assembly configured to monitor the drawer. The method includes acquiring images using the camera assembly and analyzing the images to identify a first food item and a second food item in the food storage compartments of the drawer. The method also includes detecting, using the sensors, that atmospheric conditions within the food storage compartments are above a predetermined threshold, and, based on the analysis of the images, identifying one of the first food item and the second food item as a source causing the atmospheric conditions to be above the predetermined threshold.

[0007] In another exemplary embodiment, a method of operating a refrigeration appliance is provided. The refrigeration appliance includes a cabinet defining a food storage compartment having drawers slidably mounted within the food storage compartment. The drawers are slidable between a closed position and an open position. The drawers include walls defining a plurality of food storage compartments. The refrigeration appliance also includes sensors operable to detect atmospheric conditions within the food storage compartments of the drawers, and a camera assembly positioned and configured to monitor the drawers. The method includes acquiring images using the camera assembly and analyzing the images to identify a first food item and a second food item in the food storage compartments of the drawers. The method also includes setting a first threshold for atmospheric conditions based on the identification of the first food item and setting a second threshold for atmospheric conditions based on the identification of the second food item. The method also includes monitoring atmospheric conditions by the sensors. The method also includes issuing a first user notification when atmospheric conditions reach the first threshold and issuing a second user notification when atmospheric conditions reach the second threshold.

[0008] In yet another exemplary embodiment, a method of operating a refrigeration appliance is provided. The refrigeration appliance includes a cabinet defining a food storage compartment having drawers slidably mounted within the food storage compartment. The drawers are slidable between a closed position and an open position. The drawers include walls defining a plurality of food storage compartments. The refrigeration appliance also includes sensors operable to detect atmospheric conditions within the food storage compartments of the drawers, and a camera assembly configured to monitor the drawers. The method includes acquiring images using the camera assembly and analyzing the images to identify a first food item and a second food item in the food storage compartments of the drawers. The method also includes determining that the first food item and the second food item are incompatible for co-storage and issuing a user notification including a suggestion to reposition one of the first food item and the second food item.

[0009] These and other features, aspects, and advantages of the invention will be better understood by referring to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. Attached Figure Description

[0010] The invention, including its preferred embodiments, is fully and readily disclosed in the description with reference to the accompanying drawings and is intended for use by those skilled in the art.

[0011] Figure 1 A front view of a refrigeration appliance according to an exemplary embodiment of the present invention is provided.

[0012] Figure 2 Provided Figure 1 A perspective view of a refrigeration appliance.

[0013] Figure 3 Provided Figure 1 A front view of a refrigeration appliance, with the door of the refrigeration appliance in the open position.

[0014] Figure 4 A front view of a portion of a refrigeration appliance according to one or more exemplary embodiments of the present invention is provided.

[0015] Figure 5 Provided Figure 4 A cross-sectional view of a part of a refrigeration appliance.

[0016] Figure 6 It shows refrigeration appliances (such as Figure 1 An exemplary image of a storage drawer and its contents in the open position (of an exemplary refrigeration appliance), which may be captured by a camera component in the refrigeration appliance.

[0017] Figure 7 A perspective view of a drawer in a refrigeration appliance that can be incorporated into one or more exemplary embodiments of the present invention is provided.

[0018] Figure 8 Provided Figure 7 A magnified view of a portion of the drawer.

[0019] Figure 9 A flowchart is provided for an exemplary method of operating a refrigeration appliance according to one or more exemplary embodiments of this method.

[0020] Figure 10 A flowchart is provided for another exemplary method for operating a refrigeration appliance according to one or more additional exemplary embodiments of the present invention.

[0021] Figure 11 A flowchart is provided for yet another exemplary method for operating a refrigeration appliance according to one or more additional exemplary embodiments of the present invention. Detailed Implementation

[0022] Reference will now be made in detail to embodiments of the invention, one or more examples of which are illustrated in the accompanying drawings. Each example is provided by way of explanation rather than limitation of the invention. Indeed, it will be apparent to those skilled in the art that various modifications and variations can be made to the invention without departing from its scope or spirit. For example, a feature shown or described as part of one embodiment may be used in combination with another embodiment to provide an additional embodiment. Therefore, the invention is intended to cover such modifications and variations that fall within the scope of the appended claims and their equivalents.

[0023] Figure 1This is a front view of an exemplary embodiment of the refrigeration appliance 100. Figure 2 This is a perspective view of refrigeration appliance 100. Figure 3 This is a front view of a refrigeration appliance 100, with its food preservation door 128 in the open position. The refrigeration appliance 100 extends along the vertical direction V between a top 101 and a bottom 102. The refrigeration appliance 100 also extends along the lateral direction L between a first side 105 and a second side 106. Figure 2 As shown, the lateral direction T can be perpendicular to the vertical direction V and the lateral direction L. The refrigeration appliance 100 extends along the lateral direction T between the front portion 108 and the rear portion 110.

[0024] Refrigeration appliance 100 includes an upper food preservation compartment 122 ( Figure 3 The refrigerator 100 comprises a cabinet or housing 120 arranged vertically below the food preservation compartment 122 and a lower freezer compartment or food frozen storage compartment 124. In some embodiments, an auxiliary food storage compartment (not shown) may be disposed (e.g., vertically V) between the food preservation compartment 122 and the food frozen storage compartment 124. Because the food frozen storage compartment 124 is disposed below the food preservation compartment 122, the refrigeration appliance 100 is often referred to as a bottom-mounted refrigerator. In an exemplary embodiment, the housing 120 also has a mechanical chamber (not shown) for accommodating a sealed cooling system (not shown). Using the teachings disclosed herein, those skilled in the art will understand that the invention can also be used with other types of refrigerators (e.g., door refrigerators). Therefore, the description set forth in this invention is for illustrative purposes only and is not intended to limit the invention in any way.

[0025] Each of the refrigerator doors 128 is rotatably hinged to the edge of the housing 120 for opening and closing the food preservation compartment 122. It should be noted that although two doors 128 in a "French door" configuration are illustrated, any suitable door arrangement utilizing one, two, or more doors is within the scope and spirit of the invention. A freezer door 130 is arranged below the refrigerator doors 128 for opening and closing the freezer compartment 124. In an exemplary embodiment, the freezer door 130 is coupled to a freezer drawer (not shown) slidably mounted within the freezer compartment 124. An auxiliary door 127 is coupled to an auxiliary drawer (not shown) slidably mounted within an auxiliary compartment (not shown).

[0026] The operation of the cooling appliance 100 can be regulated by a controller 134 operable to be connected to a user interface panel 136. The user interface panel 136 provides options for a user to manipulate the operation of the cooling appliance 100 to modify environmental conditions therein, such as temperature options. In some embodiments, the user interface panel 136 may be adjacent to the dispenser assembly. The panel 136 provides options for a user to manipulate the operation of the cooling appliance 100, such as temperature options, automatic or manual humidity control options (described in more detail below), etc. In response to user manipulation of the user interface panel 136, the controller 134 controls the operation of various components of the cooling appliance 100. The operation of the cooling appliance 100 can be regulated by the controller 134, for example, the controller 134 can regulate the operation of various components of the cooling appliance 100 in response to programming and / or user manipulation of the user interface panel 136.

[0027] Controller 134 may include memory and one or more microprocessors, CPUs, etc., such as general-purpose or special-purpose microprocessors capable of operating to execute programming instructions or microcontroller code associated with operating the refrigeration appliance 100. Memory may represent random access memory (such as DRAM) or read-only memory (such as ROM or FLASH). In one embodiment, the processor executes programming instructions stored in the memory. The memory may be a component separate from the processor or may be onboard within the processor. It should be noted that controller 134 as disclosed herein is capable of and can be operated to perform any of the methods and associated method steps disclosed herein.

[0028] The controller 134 can be located at various locations throughout the refrigeration appliance 100. In an illustrated embodiment, the controller 134 may be located within a door 128. In such embodiments, input / output (“I / O”) signals can be transmitted between the controller and various operating components of the refrigeration appliance 100. In one embodiment, the user interface panel 136 may represent a general-purpose I / O (“GPIO”) device or function block. In one embodiment, the user interface 136 may include input components, such as one or more of various electrical, mechanical, or electromechanical input devices including rotary dials, buttons, and touchpads. The user interface 136 may include display components, such as digital or analog display devices designed to provide operational feedback to the user. For example, the user interface 136 may include a touchscreen that provides both input and display functions. The user interface 136 may communicate with the controller via one or more signal lines or a shared communication bus.

[0029] Can Figure 3As shown, multiple food storage elements (such as housing 138, shelf 142, and drawer 140) are arranged within the food preservation storage compartment 122. As will be detailed below, drawer 140 may be configured for storing agricultural products (such as fruits and vegetables), and specifically, a refrigeration appliance may be operable to improve and be configured to improve the shelf life of the agricultural products stored therein. Therefore, drawer 140 may also be referred to as agricultural product drawer 140 or vegetable drawer 140 and fruit drawer 140. For example, in some embodiments, the refrigeration appliance may include two drawers 140, for example, such as... Figure 3 As shown.

[0030] Now, for general reference Figures 3 to 6 The refrigeration appliance 100 may also include an inventory management system, which is typically configured to monitor one or more compartments of the refrigeration appliance 100 to monitor the status of the inventory stored therein. More specifically, as detailed below, the inventory management system may include one or more sniffers or sensors 200 (see, for example, Figure 4 and Figure 5 ), camera 192 (see example, Figure 3 and Figure 5 ) or other detection devices used to monitor the food preservation compartment 122 and, in particular, drawer 140, to detect and monitor objects located in or removed from drawer 140 (e.g., typically by Figure 6 (Reference number 182 is used for identification). In this regard, the inventory management system can use data from each of these devices to obtain representations or knowledge of the identity, location, and / or other qualitative or quantitative characteristics of objects 182 (e.g., agricultural products such as fruits and / or vegetables) within drawer 140. Although the inventory management system is described herein as a monitoring drawer 140 for detecting objects 182, it should be understood that aspects of the invention can be used to monitor objects or articles in any other suitable appliance, room, etc.

[0031] like Figure 3 As schematically illustrated, the inventory management system may include a camera assembly 190, which is typically positioned and configured to acquire images of the refrigeration appliance 100 during operation. Specifically, according to an exemplary embodiment, the camera assembly 190 includes one or more cameras 192 mounted to a cabinet 120, a door 128, or otherwise positioned within the field of view of the food preservation compartment 122. Figure 3As shown, the camera 192 of the camera assembly 190 is mounted to the cabinet 120 at the front opening of the food preservation compartment 122 and is oriented to have a field of view 194 that covers the front opening and / or enters the food preservation compartment 122 and, in particular, enters the drawer 140, such as when the drawer 140 is in the open position, such as extending forward generally in the lateral direction T such that one or two drawers of the drawer 140 extend beyond the front edge (front) of one or more shelves 142 between the camera assembly 190 and the drawer 140, such as above the drawer 140.

[0032] Although Figure 3 A single camera 192 is shown, but it should be understood that the camera assembly 190 may include a plurality of cameras 192 disposed within the cabinet 120, wherein each of the plurality of cameras 192 has a specific monitoring area or range around the food preservation compartment 122. In this regard, for example, the field of view 194 of each camera 192 may be limited or focused on a specific area within the food preservation compartment 122, such as one camera 192 per drawer 140.

[0033] However, it may be necessary to position each camera 192 adjacent to the front opening of the food preservation compartment 122 and to orient each camera 190 such that the field of view 194 is directed into the food preservation compartment 112. This can mitigate or completely eliminate privacy concerns related to users obtaining images of the appliance 100. According to an exemplary embodiment, the camera assembly 190 can be used to improve the inventory management process of the refrigeration appliance 100. Therefore, each camera 192 may be positioned at the opening leading to the food preservation compartment 122 to monitor food (typically identified as object 182) being added to or removed from the food preservation compartment 122, particularly food being placed in or removed from drawer 140.

[0034] It should be understood that, according to alternative embodiments, camera assembly 190 may include any suitable number, type, size, and configuration of cameras 192 for acquiring images of the area within or around any suitable cooling appliance 100. Furthermore, it should be understood that each camera 192 may include features for adjusting the field of view and / or orientation.

[0035] It should be understood that the images acquired by the camera assembly 190 may vary in quantity, frequency, angle, resolution, detail, etc., in order to improve the clarity of specific areas around or within the refrigeration appliance 100. Furthermore, according to an exemplary embodiment, the controller 134 may be configured to illuminate the refrigeration compartment with one or more light sources before acquiring images. In particular, the controller 134 of the refrigeration appliance 100 (or any other suitable dedicated controller) is communicatively coupled to the camera assembly 190 and may be programmed or configured to analyze the images acquired by the camera assembly 190, for example, to identify items being added to or removed from the refrigeration appliance 100, as detailed below.

[0036] Generally, controller 134 may be operable to be coupled to camera assembly 190 for analyzing one or more images acquired by camera assembly 190 to extract useful information about object 182 located within drawer 140. In this regard, for example, the images acquired by camera assembly 190 may be used to extract barcodes, identify products, monitor product movement, or obtain other product information related to object 182. In particular, this analysis may be performed locally (e.g., on controller 134) or may be sent to a remote server for the analysis (e.g., in the "cloud," as those skilled in the art will recognize, refers to a remote server or database in a distributed computing environment comprising at least one remote server and local controller 134). Such analysis is intended to facilitate inventory management, for example, by identifying food items being added to or removed from food preservation compartment 122.

[0037] Specifically, according to an exemplary embodiment, the camera 192 (or the plurality of cameras 192 in the camera assembly 190 as a whole) is oriented downward from the top center of the cabinet 120 and has a field of view 194 covering the width of the food preservation compartment 122 (e.g., the total width of the two drawers 140). Figure 3 The diagram is schematically shown and corresponds to Figure 6 (Exemplary image). Thus, the field of view 194 of camera 192 and the resulting image acquired can capture any movement or motion of an object entering and / or leaving drawer 140. The images acquired by camera component 190 may include one or more still images, one or more video clips, or any other suitable type and number of images suitable for food identification (e.g., typically identified by reference numeral 182) or inventory analysis.

[0038] In particular, the camera assembly 190 can acquire images in response to any suitable trigger, such as an imaging schedule in which the camera assembly 190 periodically images and monitors the drawer 140. According to other embodiments, the camera assembly 190 can periodically capture low-resolution images until (e.g., by image differentiation of the low-resolution images) motion (such as opening, e.g., sliding one or two drawers 140 forward) is detected, at which point one or more high-resolution images can be acquired. According to other embodiments, the refrigeration appliance 100 may include one or more motion sensors (e.g., optical, acoustic, electromagnetic, etc.) that are triggered when an object 182 is being added to or removed from the drawer 140, and the camera assembly 190 may be operable to be coupled to such motion sensors to acquire images of the object 182 during such movement.

[0039] According to other embodiments, the refrigeration appliance 100 may include a door switch that detects when the refrigerator door 128 is opened, at which point the camera assembly 190 may begin acquiring one or more images. According to an exemplary embodiment, images 300 may be acquired continuously or periodically when the refrigerator door 128 is open and / or when one or both drawers 140 are in the open position. In this regard, acquiring images 300 may include determining that the door and / or drawer of the refrigeration appliance is open, and capturing images at a set frame rate when the door and / or drawer is open.

[0040] In particular, the movement of food between image frames can be used to determine whether food 182 is being removed from or added to the food preservation compartment 122. It should be understood that the images acquired by the camera assembly 190 may vary in quantity, frequency, angle, resolution, detail, etc., to improve the clarity of food 182. Furthermore, according to an exemplary embodiment, the controller 134 may be configured to illuminate a refrigerator light source (not shown) while acquiring image 300. Other suitable imaging triggers may also be employed and are within the scope of this invention.

[0041] like Figure 4 and Figure 5As shown, in various embodiments, the refrigeration appliance 100 may include an atmospheric condition sensor or sniffer 200 in fluid communication with the food preservation compartment 122. For example, the sensor 200 may be configured within a housing 120, such as within the food preservation compartment 122, or within a drawer 140 of the food preservation compartment 122, such that fluids (e.g., gases, such as air or other atmospheric gases within the food preservation compartment 122 (especially within the drawer 140, such as within a food storage compartment 144)) flow to and around and / or through the sensor 200, thereby enabling the sensor 200 to detect or monitor atmospheric conditions, such as the atmospheric composition, temperature, humidity, and other similar atmospheric conditions within the food preservation compartment 122 and the drawer 140.

[0042] In some implementations, multiple sensors 200 may be provided. For example, when sensor 200 is located in one drawer of drawer 140, another sensor 200 may be located in another drawer 140. Alternatively, multiple sensors 200 may be provided, each capable of operating and configured to measure different atmospheric conditions, such as temperature sensors and chemical substance sensors. For example, a chemical substance sensor may be a sniffer that detects or measures the concentration or type of a particular chemical substance (such as ethylene).

[0043] Now for reference Figure 7 and Figure 8 An exemplary drawer 140 may include a food storage compartment 144. The food storage compartment 144 may be defined by a plurality of walls of the drawer 140. For example, the plurality of walls may include a front wall 146, a rear wall 148, a left wall 150, and a right wall 152. Directional terms such as “left” and “right” are used herein with reference to the perspective of a user standing in front of the refrigeration appliance 100 approaching the items stored therein. One of the walls (e.g., as...) Figure 7 In an exemplary embodiment, the front wall 146 may include one or more ventilation holes 154 disposed in and passing through the wall (e.g., front wall 146). In such embodiments, the drawer 140 may include a humidity control knob, a slider or lever for adjusting the opening degree of the ventilation holes 154, such as slider 156, for example... Figure 8 As shown in the diagram, slider 156 is movable to selectively change the degree of opening or closing of one or more vents 154, such as in an open position where one or more vents are unobstructed to enhance air circulation into and through the food storage compartment 144, and in a closed position where one or more vents are closed by slider 156 to limit airflow into the food storage compartment 144.

[0044] For example, such as Figure 8 The slider 156 shown can travel along a single path within the track 158 (e.g., along such a path). Figure 8The direction 1000 shown in the figure moves in two generally opposite directions. Additionally, those skilled in the art will recognize that the slider 156 is movable through a plurality of intermediate positions between the open and closed positions, such as... Figure 8 An exemplary intermediate position is shown, in which one or more vents are partially blocked by slider 156. Therefore, slider 156 can move within track 158 in two opposing directions (e.g., back-and-forth movement) along direction 1000 between the open and closed positions and through multiple intermediate positions between them. Specifically, Figure 8 The exemplary embodiment shown includes a slider 156 and a plurality of vents 154 at an exemplary intermediate position, wherein one vent 154 is completely unobstructed or fully open, while another adjacent vent 154 is partially open, for example, partially blocked.

[0045] As described, changing the position of slider 156 allows for different humidity levels within the food storage compartment 144 of drawer 140. For example, when slider is in the closed or intermediate position, humidity (e.g., air with a relatively high moisture content compared to the ambient air outside the refrigeration appliance and / or the air in the rest of the food preservation compartment 122 outside drawer 140) may accumulate within the food storage compartment 144 of drawer 140. When slider 156 is in the open position, the humidity level within drawer 140 will reach equilibrium with the ambient humidity level, for example, in the rest of the food preservation compartment 122 outside drawer 140. Furthermore, the various intermediate positions provide different rates at which the humidity within the food storage compartment 144 of drawer 140 reaches equilibrium with the humidity in the rest of the food preservation compartment 122 outside drawer 140. For example, when the intermediate position of slider 156 is near the open position, equilibrium is reached faster (resulting in lower humidity within the food storage compartment 144 of drawer 140), while when the intermediate position of slider 156 is near the closed position, equilibrium is reached more slowly (resulting in higher humidity within the food storage compartment 144 of drawer 140). For example, Figure 8 The intermediate position shown (where one vent 154 is open and another vent is partially open and partially closed) allows some moisture to escape from the atmosphere within the food storage compartment 144, while also maintaining some humidity for produce, for example, products that prefer moderate storage humidity.

[0046] By using the teachings disclosed herein, those skilled in the art will understand that the present invention can be used in conjunction with other types of refrigerators, such as refrigerator / freezer combinations, door-to-door refrigeration appliances, bottom-mounted refrigeration appliances, compact refrigeration appliances, and any other type or style of refrigeration appliance. Therefore, other configurations of the refrigeration appliance 100 may be provided, and it should be understood that the configurations shown in the accompanying drawings and the descriptions set forth herein are merely examples for illustrative purposes.

[0047] The construction and configuration of a camera assembly 190 and a cooling appliance 100 according to an exemplary embodiment of the present invention have now been presented, and exemplary methods for operating a cooling appliance (such as cooling appliance 100) have been provided. Such methods can also be used to operate a camera assembly, such as camera assembly 190, or any other suitable camera assembly for monitoring appliance operation or inventory. In this regard, for example, controller 134 may be configured to implement one or more of the following exemplary methods. However, it should be understood that the exemplary methods discussed herein are merely illustrative of exemplary aspects of the invention and are not intended to be limiting.

[0048] Now go to Figure 9 Embodiments of the present invention may include a method 400 of operating a refrigeration appliance (such as the exemplary refrigeration appliance 100 described above). For example, the refrigeration appliance may include a controller and a cabinet having a food storage compartment with drawers slidably mounted within it. The drawers are slidable between a closed position and an open position. The drawers may include a plurality of walls defining the food storage compartment. Again, as an example, the refrigerator may also include sensors for detecting atmospheric conditions within the food storage compartment of the drawer, and a camera assembly positioned and configured to monitor the drawer, as described above.

[0049] like Figure 9 As shown, method 400 includes (at step 410) using a camera assembly to acquire an image of the cooling chamber of the refrigeration appliance. For example, camera assembly 190 of the refrigeration appliance 100 may acquire an image 300 of the food storage compartment 144 of the food preservation compartment 122 and / or drawer 140 (e.g., as shown in the image). Figure 6 As shown in the diagram, drawer 140 may include multiple objects 182 in its field of view. In this respect, camera assembly 190 of refrigeration appliance 100 may acquire one or more images (e.g., such as image 300) of food preservation compartment 122, freezer compartment 124, or any other area or region within or around refrigeration appliance 100.

[0050] In some embodiments, the method may also include, and / or the refrigeration appliance may be configured to identify one or more food items, such as identifying a first food item and a second food item based on one or more images. In some embodiments, the identification of food items may be performed using a camera assembly 190. For example, the refrigeration appliance may include a camera, and the step of identifying food items may include identifying food items based on images captured by the camera. In some embodiments, the operation of the camera may be associated with door opening; for example, the camera may be operable and configured to capture images whenever a door is opened and / or whenever a door is closed after an opening is detected. Those skilled in the art will understand the structure and operation of cameras, and therefore, for the sake of brevity, cameras are not shown or described in further detail herein. In such embodiments, the controller 134 of the refrigeration appliance 100 may be configured to perform image-based processing, such as identifying food items based on images of the food (e.g., photographs of the food taken using camera 192 of the camera assembly 190). For example, the controller 134 may be configured to identify food items by comparing the image with stored images of known or previously identified food items.

[0051] exist Figure 9 In the exemplary embodiment shown, method 400 may include step 420 of analyzing the image acquired at step 410 to identify a first food item and a second food item in the food storage compartment of the drawer. For example, the controller 134 of the refrigeration appliance 100 (or any other suitable dedicated controller) is communicatively coupled to the camera assembly 190 and may be programmed or configured to analyze the images acquired by the camera assembly 190, for example, to identify items stored in the refrigeration appliance 100, as detailed above.

[0052] Step 420 includes analyzing the image to identify objects, such as at least a first food item and a second food item, disposed in the food storage compartment 144 of drawer 140. It should be understood that this analysis can utilize any suitable image analysis technique, image decomposition, image segmentation, image processing, etc. This analysis can be performed entirely by controller 134, transferred to a remote server, performed with user assistance (e.g., via user interface panel 136), or in any other suitable manner. According to an exemplary embodiment of the invention, the analysis performed at step 420 may include a machine learning image recognition process.

[0053] According to an exemplary embodiment, the image analysis may use any suitable image processing technique, image recognition process, etc. As used herein, the term "image analysis" and the like can generally be used to refer to any suitable method for observing, analyzing, decomposing, extracting features, classifying, etc., one or more images, videos, or other visual representations of objects. As explained in more detail below, the image analysis may include implementing image processing techniques, image recognition techniques, or any suitable combination thereof. In this regard, the image analysis may use any suitable image analysis software or algorithm to continuously or periodically monitor objects within the food preservation compartment 122, such as those within its drawer 140. It should be understood that the image analysis or processing may be performed locally (e.g., by controller 134) or remotely (e.g., by transferring image data to a remote server or network, e.g., in the cloud).

[0054] Specifically, analyzing one or more images may include implementing image processing algorithms. As used herein, the term "image processing," etc., is generally intended to refer to any suitable method or algorithm for analyzing images that does not rely on artificial intelligence or machine learning techniques (e.g., compared to machine learning image recognition processes described below). For example, image processing algorithms may rely on image differentiation, such as performing a pixel-by-pixel comparison of two sequentially acquired images. This comparison can help identify substantial differences between sequentially acquired images, such as identifying movement, the presence of a specific object, the existence of a specific condition, etc. For example, when a specific condition exists, one or more reference images may be acquired, and these reference images may be stored for future comparison with images acquired during the operation of the appliance. The similarity and / or differences between the reference images and the acquired images can be used to extract useful information for improving the performance of the appliance. For example, image differentiation can be used to determine when pixel-level motion metrics exceed a predetermined motion threshold.

[0055] The processing algorithm may also include measures for separating or eliminating noise in image comparison, for example, noise caused by image resolution, data transmission errors, inconsistent lighting, or other imaging errors. By eliminating such noise, the image processing algorithm can improve the accuracy of object detection, avoid false object detection, and separate important objects, regions, or patterns within an image. Furthermore, or alternatively, the image processing algorithm may use other suitable techniques for identifying or recognizing specific items or objects, such as edge matching, divide-and-conquer search, grayscale matching, receptive field histograms, or other suitable routines (e.g., executed at controller 134 based on one or more captured images from one or more cameras). Other image processing techniques may also be employed and are within the scope of this invention.

[0056] In addition to the image processing techniques described above, image analysis may include the use of artificial intelligence (“AI”), such as machine learning image recognition processes, neural network classification modules, any other suitable artificial intelligence (AI) techniques and / or any other suitable image analysis techniques, examples of which will be described in more detail below. Furthermore, the exemplary image analysis or evaluation processes described below can each be used alone, in combination, or interchangeably to extract detailed information about the image being analyzed to improve the performance of one or more methods described in this invention, or otherwise improve the operation of the appliance. According to exemplary embodiments, any suitable number of image processing, image recognition, or other image analysis techniques, or combinations thereof, can be used to perform accurate analysis of the acquired images.

[0057] In this regard, the image recognition process can utilize any suitable artificial intelligence technique, such as any suitable machine learning technique, or, for example, any suitable deep learning technique. According to an exemplary implementation, the image recognition process may include an image recognition method implementing a region-based convolutional neural network (“R-CNN”). Generally, R-CNN may include obtaining an input image and extracting candidate regions comprising potential objects or regions of the image. In this regard, a “candidate region” may be one or more regions in the image that may belong to a particular object, and may also include neighboring regions sharing common pixel characteristics. Features of the candidate regions are then computed using the convolutional neural network, and the extracted features are then used to determine the classification of each particular region.

[0058] According to other implementations, the image segmentation process can be combined with R-CNN image recognition. Generally, image segmentation creates a pixel-based mask for each object in an image and enables a more detailed or refined understanding of the various objects in a given image. In this respect, image segmentation may involve dividing an image into segments (e.g., into groups of pixels containing similar attributes), which can be analyzed individually or in parallel to obtain a more detailed representation of one or more objects in the image, rather than processing the entire image (e.g., a large number of pixels, many of which may not contain useful information). This may be referred to herein as “mask R-CNN” or similar terms, rather than the conventional R-CNN architecture. For example, mask R-CNN may be based on fast R-CNN, which is slightly different from R-CNN. For example, R-CNN first applies a convolutional neural network (“CNN”) and then assigns it to a recommendation region on the covn5 attribute map, instead of assigning it to the initially split recommendation region. Furthermore, according to exemplary implementations, standard CNNs can be used to acquire, identify, or detect any other qualitative or quantitative data related to one or more objects or regions within one or more images. Additionally, the K-means algorithm can be used.

[0059] According to other embodiments, the image recognition process may use any other suitable neural network process while still remaining within the scope of the invention. For example, the step of analyzing one or more images may include using a deep belief network (“DBN”) image recognition process. A DBN image recognition process typically includes stacking many separate unsupervised networks, each using the hidden layer of one network as input to the next layer. According to yet another embodiment, the step of analyzing one or more images may include implementing a deep neural network (“DNN”) image recognition process, which typically includes using a neural network (a computational system inspired by biological neural networks) with multiple layers between input and output. Other suitable image recognition processes, neural network processes, artificial intelligence analysis techniques, and combinations of the methods described above or other known methods may be used while still remaining within the scope of the invention.

[0060] Furthermore, it should be understood that various transfer techniques can be used, but this particular technique is not necessary. If transfer techniques are used for learning, the neural network architecture can be pre-trained on public datasets (such as VGG16 / VGG19 / ResNet50) and then the final layer can be retrained on an appliance-specific dataset. Additionally, or alternatively, the image recognition process may include detecting specific conditions based on comparisons of initial conditions, relying on image subtraction, image stacking, image concatenation, etc. For example, subtracted images can be used to train multi-class neural networks for future comparisons and image classification.

[0061] It should be understood that the machine learning image recognition model can be actively trained by the appliance using new images, the training data can be provided by the manufacturer or another remote source, or it can be trained in any other suitable manner. For example, according to an exemplary embodiment, the image recognition process relies at least in part on a neural network trained using multiple images of appliances with different configurations, experiencing different conditions, or interacting in different ways. The training data can be stored locally or remotely and can be transferred to a remote server for training other appliances and the model.

[0062] It should be understood that image processing and machine learning image recognition processes can be used in combination to improve image analysis, object detection, or extract other useful qualitative or quantitative data or information from one or more images, which can be used to improve the operation or performance of appliances. In fact, the methods described in this invention can be used interchangeably with any or all of these techniques to improve image analysis processes and promote improved appliance performance and consumer satisfaction. The image processing algorithms and machine learning image recognition processes described in this invention are merely exemplary and are not intended to limit the scope of the invention in any way.

[0063] Please refer to the following for more details. Figure 9 The exemplary method 400 may also include a step 430 of detecting that the atmospheric conditions within a food storage compartment (e.g., food storage compartment 144) are above a predetermined threshold. For example, the predetermined threshold may be a default value stored in the memory of a controller. One or more sniffers or sensors 200 may be used to monitor and / or detect atmospheric conditions, as described above. The predetermined threshold may be, for example, an ethylene level. Additional exemplary atmospheric conditions and their corresponding predetermined thresholds include temperature, humidity levels, and / or the levels or concentrations of any other chemical substances or components in the atmosphere within the drawer (e.g., drawer 140).

[0064] Method 400 may also include step 440, identifying one of the first and second food items as a source of atmospheric conditions exceeding a predetermined threshold based on image analysis. For example, steps 420 and 440 may use multiple images from the same set of images or the same image, wherein the set of images includes multiple images of the same region or location taken over time. For example, identifying one of the first and second food items as a source of atmospheric conditions may include image analysis, thereby identifying color changes in the food, such as darkening or browning of fruits, vegetables, or other similar agricultural products, based on time-series images of the same objects in the drawer.

[0065] Furthermore, it should be understood that these steps are not necessarily performed in a given order. For example, detection step 430 may occur before identifying the first and second food products, such as food identification occurring in response to detecting atmospheric conditions above a predetermined threshold. As one example among many possible examples, atmospheric conditions may be ethylene levels, and the predetermined threshold may be excessive ethylene levels, for example, because ethylene levels may be harmful to the storage of at least one agricultural product, such ethylene levels may be excessive. The method could therefore include detecting excessive ethylene levels and, in response to detecting ethylene levels, acquiring and analyzing images to locate the source of the ethylene levels.

[0066] In some implementations, method 400 may further include issuing a user notification. The user notification may include an indication or identification that one of the first food and the second food has been identified as a source causing atmospheric conditions to exceed a predetermined threshold.

[0067] In some implementations, sensors can be used to detect atmospheric conditions within the food storage compartment of the drawer when the drawer is in the closed position, such as when step 430 can be performed. For example, when the drawer is in the closed position, the controller can send or query the sensors, wherein the closed position can be detected by the controller based on a position switch or position sensor (e.g., a Hall effect sensor) and / or based on images from a camera component, wherein the controller can analyze such images to identify and detect when the drawer is in the closed position. Advantageously, measuring or detecting atmospheric conditions when the drawer is in the closed position allows for a more accurate reading of the atmosphere within the drawer itself (such as in the food storage compartment therein), rather than the environmental conditions outside the drawer, such as in the rest of the food storage compartment and / or outside the refrigeration appliance.

[0068] In some implementations, the camera assembly can be positioned and configured to monitor the food storage compartments of the drawer when it is in the open position. For example, it is advantageous that when images are acquired while the drawer is in the open position, the images of the contents inside the drawer may be clearer (e.g., less obstructed), such as when the drawer is in the open position, the drawer can be extended outward away from other contents and structures (e.g., shelves) in the refrigerator, thereby allowing for a clearer and more complete view of the interior of the drawer and its contents when the drawer is in the open position.

[0069] In some embodiments, the drawer may also include a vent disposed on and passing through one of the plurality of walls. In such embodiments, a camera assembly may be configured to monitor the vent of the drawer, for example, the vent may be positioned within the field of view of the camera. Such embodiments may also include determining an optimal humidity level for at least one of a first food item and a second food item and determining an optimal position of a slider at the vent corresponding to the determined optimal humidity. Exemplary embodiments in which the camera assembly is positioned and configured to monitor the vent of the drawer may also include analyzing images to determine whether the slider at the vent is in the optimal position, and issuing a user notification when the slider at the vent is not in the optimal position, wherein the user notification may be, for example, an audible user notification and / or a visible user notification as detailed below, and may also be provided locally and / or remotely as detailed below.

[0070] Now go to Figure 10 Embodiments of this disclosure may include a method 500 for operating a refrigeration appliance (such as the exemplary refrigeration appliance 100 described above). For example, the refrigeration appliance may include a controller and multiple food storage drawers, as described above.

[0071] Method 500 also includes an image acquisition step 510 and an analysis and recognition step 520 similar to steps 410 and 420 described above, and for the sake of brevity, such descriptions will not be repeated.

[0072] Compared to the predetermined thresholds described above with respect to exemplary method 400, method 500 may include defining new or additional thresholds for one or more atmospheric conditions in a refrigerating appliance (such as drawer 140). For example, one or more thresholds may be based on and / or in response to the anticipated or expected ethylene level of an identified food, such as a particular type of agricultural product, where the anticipated or expected ethylene level corresponds to the identified food becoming ripe (or overripe, etc.). Therefore, in some embodiments, exemplary method 500 may include a step 530 of setting a first threshold for the atmospheric condition based on the identification of a first food and a step 540 of setting a second threshold for the atmospheric condition based on the identification of a second food. In at least some embodiments, the first food may be different from the second food, and therefore the first threshold may also be different from the second threshold, although different foods may not necessarily have different thresholds.

[0073] like Figure 10 As shown, method 500 may further include step 550 of monitoring atmospheric conditions, wherein a first threshold and a second threshold for atmospheric conditions are set at steps 530 and 540. Such monitoring may be performed at least in part by sensors, such as by a controller of a refrigeration appliance capable of operating to communicate with sensors. For example, exemplary methods may include using and / or monitoring atmospheric conditions by sensors.

[0074] Still referencing Figure 10 Method 500 may also include the step of issuing one or more user notifications. Such notifications may be issued locally (e.g., on the user interface panel 136 of the refrigeration appliance 100) and / or remotely (such as on a remote device not directly and physically attached to or connected to the refrigeration appliance, such as a smartphone, smart home system, or other similar device). User notifications may include one or more of visual notifications (e.g., illuminating an indicator light or providing a text notification) and / or audible notifications (such as a ringtone or alarm tone). For example, method 500 may include step 560 of issuing a first user notification when a first atmospheric condition threshold based on the identification of a first food item is reached, and step 570 of issuing a second user notification when a second atmospheric condition threshold based on the identification of a second food item is reached. Thus, for example, customized and responsive monitoring and inventory management may be provided in method 500, wherein each food item is tracked individually and specifically based on atmospheric conditions that are more apparent or sensitive to a particular identified food item.

[0075] In some implementations, the first atmospheric condition threshold in step 530 may include a first ethylene level and the second atmospheric condition threshold in step 540 may include a second ethylene level.

[0076] In some implementations, the sensor may be operable to detect atmospheric conditions within the food storage compartment of the drawer when the drawer is in the closed position. For example, when the drawer is in the closed position, the controller may send or query the sensor, wherein the closed position may be detected by the controller based on a position switch or position sensor (e.g., a Hall effect sensor) and / or based on images from a camera component, wherein the controller may analyze such images to identify and detect when the drawer is in the closed position. Advantageously, measuring or detecting atmospheric conditions when the drawer is in the closed position allows for a more accurate reading of the atmosphere within the drawer itself (such as in the food storage compartment therein), rather than the environmental conditions outside the drawer, such as in the rest of the food storage compartment and / or outside the refrigeration appliance.

[0077] In some implementations, the camera assembly can be configured to monitor the food storage compartments of the drawer when the drawer is in the open position. For example, it is advantageous that when images are acquired while the drawer is in the open position, the images of the contents inside the drawer may be clearer (e.g., less obstructed), such as when the drawer is in the open position, the drawer can be extended outward away from other contents and structures (e.g., shelves) in the refrigerator, thereby allowing for a clearer and more complete view of the drawer's interior when the drawer is in the open position.

[0078] In some embodiments, the drawer may also include a vent through one of a plurality of walls. In such embodiments, a camera assembly may be configured to monitor the drawer's vent, for example, the vent may be positioned within the camera's field of view. Such embodiments may also include determining an optimal humidity level for at least one of a first food item and a second food item and determining an optimal position of a slider at the vent corresponding to the determined optimal humidity. Furthermore, such embodiments may also include, or alternatively include, a humidity level as a first threshold and a second threshold as atmospheric conditions, for example, atmospheric conditions may be humidity, and the first threshold and the second threshold may each be a humidity level. Exemplary embodiments in which the camera assembly is positioned and configured to monitor the drawer's vent may also include analyzing images to determine whether the slider at the vent is in an optimal position, and issuing a user notification when the slider at the vent is not in an optimal position, wherein the user notification may be, for example, an audible user notification and / or a visible user notification as described above, and may also be provided locally and / or remotely as described above.

[0079] Now go to Figure 11Embodiments of the present invention may also include a method 600 for operating a refrigeration appliance (such as the exemplary refrigeration appliance 100 described above). For example, the refrigeration appliance may include a controller and multiple food storage drawers, as described above.

[0080] Method 600 also includes an image acquisition step 610 and an analysis and recognition step 620 similar to steps 410 / 510 and 420 / 520 described above, and for the sake of brevity, such descriptions will not be repeated.

[0081] Method 600 may also include step 630 of determining incompatibility between the first food and the second food for co-storage. For example, such incompatibility may include different optimal humidity levels and / or temperatures. Alternatively, such incompatibility may include, or alternatively, that one of the first and second foods produces ethylene, for example, when the food matures or ages, it generates or releases a large amount of ethylene (compared to other agricultural products), and that the first food and the other of the first foods are sensitive to ethylene, for example, in cases where exposure to the level of ethylene released by one food may accelerate the aging rate of the other food. In such embodiments, the determination of incompatibility may be based at least in part on the rate at which one of the first and second foods produces ethylene.

[0082] After determining that the first food item and the second food item are incompatible for co-storage, and in response to such determination, method 600 may then include a step 640 of providing a user notification that includes a suggestion to reposition one of the first and second food items. For example, when the refrigeration appliance includes more than one drawer, the suggestion may include a suggestion to move one food item to another drawer. Alternatively, the suggestion may include a suggestion to move one food item to another part of the food preservation compartment, such as the exterior of one or more drawers, or may include a suggestion to store one food item at room temperature, such as outside the refrigeration appliance. Furthermore, such examples are not required; for example, the user notification may simply provide a suggestion to remove one food item from a drawer or reposition it without specifying where that food item should be moved.

[0083] This written description uses examples to disclose the invention, including the best mode of implementation, and also enables those skilled in the art to practice the invention, including making and using any device or system and performing any incorporated methods. The patentable scope of the invention is defined by the claims and may include other examples that may occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they include structural elements that are not indistinguishable from the literal language of the claims, or if they include equivalent structural elements that differ only slightly from the literal language of the claims.

Claims

1. A method for operating a refrigeration appliance, characterized in that, The refrigeration appliance includes a cabinet with a food storage compartment, the food storage compartment having a drawer slidably mounted therein, the drawer sliding between a closed position and an open position, the drawer including walls defining multiple food storage compartments, sensors for detecting atmospheric conditions within the food storage compartments of the drawer, and a camera assembly configured to monitor the drawer. The method includes: Use the camera component to acquire images; Analyze the image to identify the first and second food items in the food storage compartment of the drawer; The sensor detected that the atmospheric conditions inside the food storage compartment were higher than a predetermined threshold; and Based on the analysis of the image, one of the first food item and the second food item is identified as the source causing the atmospheric conditions to exceed the predetermined threshold.

2. The method according to claim 1, characterized in that, The atmospheric conditions include ethylene levels.

3. The method according to claim 1, characterized in that, The sensor is operable to detect the atmospheric conditions within the food storage compartment of the drawer when the drawer is in the closed position.

4. The method according to claim 1, characterized in that, The camera assembly is configured to monitor the food storage compartment of the drawer when the drawer is in the open position.

5. The method according to claim 1, characterized in that, The drawer also includes a vent through one of the plurality of walls, and the camera assembly is configured to monitor the vent of the drawer.

6. The method according to claim 5, characterized in that, The method further includes determining an optimal humidity level for at least one of the first food and the second food, determining an optimal position of a slider at the vent corresponding to the determined optimal humidity, analyzing the image to determine whether the slider at the vent is in the optimal position, and issuing a user notification when the slider at the vent is not in the optimal position.

7. A method for operating a refrigeration appliance, characterized in that, The refrigeration appliance includes a cabinet with a food storage compartment having a drawer slidably mounted within the food storage compartment, the drawer sliding between a closed position and an open position, the drawer including walls defining multiple food storage compartments, sensors for detecting atmospheric conditions within the food storage compartments of the drawer, and a camera assembly configured to monitor the drawer. The method includes: Use the camera component to acquire images; Analyze the image to identify the first and second food items in the food storage compartment of the drawer; Based on the identification of the first food item, a first threshold for atmospheric conditions is set; Based on the identification of the second food item, a second threshold for the atmospheric conditions is set; The atmospheric conditions are monitored by the sensor; When the atmospheric conditions reach the first threshold, a first user notification is issued; and When the atmospheric conditions reach the second threshold, a second user notification is issued.

8. The method according to claim 7, characterized in that, The first threshold includes a first ethylene level and the second threshold includes a second ethylene level.

9. The method according to claim 7, characterized in that, The sensor is configured to detect the atmospheric conditions within the food storage compartment of the drawer when the drawer is in the closed position.

10. The method according to claim 7, characterized in that, The camera assembly is configured to monitor the food storage compartment of the drawer when the drawer is in the open position.

11. The method according to claim 7, characterized in that, The drawer also includes a vent through one of the plurality of walls, and wherein the camera assembly is configured to monitor the vent of the drawer.

12. The method according to claim 11, characterized in that, The method further includes: determining an optimal humidity level for at least one of the first food and the second food; determining an optimal position for a slider at the vent corresponding to the determined optimal humidity; analyzing the image to determine whether the slider at the vent is in the optimal position; and issuing a user notification when the slider at the vent is not in the optimal position.

13. A method for operating a refrigeration appliance, characterized in that, The refrigeration appliance includes a cabinet with a food storage compartment having a drawer slidably mounted within the food storage compartment, wherein the drawer can slide between a closed position and an open position. The drawer includes walls defining multiple food storage compartments, sensors for detecting atmospheric conditions within the food storage compartments of the drawer, and a camera assembly configured to monitor the drawer. The method includes: Use the camera component to acquire images; Analyze the image to identify the first and second food items in the food storage compartment of the drawer; It was determined that the first food and the second food are incompatible for co-storage; and Send a user notification that includes a suggestion to reposition one of the first food item and the second food item.

14. The method according to claim 13, characterized in that, "Determining that the first food and the second food are incompatible for co-storage" is based at least in part on the rate at which one of the first food and the second food produces ethylene.

15. The method according to claim 13, characterized in that, The drawer also includes a vent through one of the plurality of walls, and the camera assembly is configured to monitor the vent of the drawer.

16. The method according to claim 15, characterized in that, The method further includes determining an optimal humidity level for the other of the first food and the second food, determining an optimal position for a slider at the vent corresponding to the determined optimal humidity, analyzing the image to determine whether the slider at the vent is in the optimal position, and issuing a user notification when the slider at the vent is not in the optimal position.

17. The method according to claim 13, characterized in that, The sensor is configured to detect the atmospheric conditions within the food storage compartment of the drawer when the drawer is in the closed position.

18. The method according to claim 13, characterized in that, The camera assembly is configured to monitor the food storage compartment of the drawer when the drawer is in the open position.

Images