Method and device for recognizing positions of food materials in refrigerator, refrigerator and computer readable storage medium
By acquiring images at multiple rotation angles during the refrigerator door closing process, statistically counting the values, and combining them with image segmentation technology, the problem of misjudging the position of food caused by the signal delay of the angle sensor was solved, achieving higher position recognition accuracy.
Patent Information
- Application Number
- CN202410564345.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-08
- Publication Date
- 2025-11-11
AI Technical Summary
In existing technologies, the delay in angle sensor signals makes it difficult for refrigerators to accurately identify the specific location of food during the door closing process, leading to misjudgments in food area identification.
By capturing images at each rotation angle during the door closing process, counting the number of ingredients located in the door area and the box area at each rotation angle, and combining image segmentation technology with a set spacing threshold, the region affiliation of the ingredients is determined.
It improves the accuracy of food location recognition during the closing stage, overcomes the error in area judgment caused by the delay of angle sensor signals, and ensures the accuracy of food location recognition.
Smart Images
Figure CN120926677A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of refrigeration equipment technology, such as a method and apparatus for identifying the location of food in a refrigerator, a refrigerator, and a computer-readable storage medium. Background Technology
[0002] Currently, with the rapid development of science and technology, smart home appliances are becoming increasingly diverse in function. In the field of refrigeration equipment, refrigerators, in addition to traditional freezing and refrigeration functions, also have food management capabilities. To achieve food management, refrigerators are equipped with cameras on the door or cabinet. These cameras identify food items by capturing images and then display their storage locations or recommend recipes based on the identified items. However, when a large number of items are stored on the refrigerator door and the door is closed, the refrigerator struggles to determine the exact location of the food items from the camera images, hindering food management.
[0003] To determine the specific location of food items from images captured by a camera, a method for identifying the area where food items are located has been disclosed. The method includes: during the door closing phase, obtaining the angle value between the door and the plane containing the cabinet opening using an angle sensor, and obtaining the food item image captured by the camera in real time; if the food item image is located in the cabinet area, determining that the food item corresponding to the food item image is located in the cabinet area; if the food item image is located in the door area, determining that the food item corresponding to the food item image is located in the door area.
[0004] In the process of implementing the embodiments of this disclosure, at least the following problems were found in the related art:
[0005] Because the door moves in real time during the closing phase, while the angle value between the door and the plane where the cabinet opening is located is obtained by the angle sensor and the food area is identified, the door is continuously in the closing phase. Due to the signal delay of the angle value detected by the angle sensor, the area identification result determined based on the angle value detected by the angle sensor may be inconsistent with the food area corresponding to the current angle value of the door, resulting in misjudgment of the food area.
[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0007] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.
[0008] This disclosure provides a method and apparatus for identifying the location of food items in a refrigerator, a refrigerator, and a computer-readable storage medium, to overcome the problem of area judgment errors caused by the delay of angle sensor signals and improve the accuracy of food item location identification in images captured during the door closing phase.
[0009] In some embodiments, the refrigerator includes a cabinet with an opening, a connecting shaft on one side of the opening in a vertical direction, and a refrigerator compartment; a door rotatably disposed at the opening via the connecting shaft, the door including a door body and a plurality of bottle holders mounted on the door body; and a camera mounted on the top of the refrigerator compartment. The method includes: when the door is closed, acquiring images at each associated rotation angle of the door, the images including bottle holder food in the plurality of bottle holders and / or drawer food in the drawers; obtaining a door area and a cabinet area corresponding to each rotation angle based on the images at each rotation angle; obtaining the current area of the food at each rotation angle, the current area including either the door area or the cabinet area; and determining the area affiliation of the food based on the current area of the food at each rotation angle.
[0010] Optionally, the door area refers to the area consisting of the main body of the door and multiple bottle holders when the door is closed, and the cabinet area refers to the area where the door is closed, coplanar with the bottom surface of the bottom drawer of the refrigerator compartment, and located in front of the cabinet opening. The width of the cabinet area along the width direction of the cabinet is equal to the width of the cabinet, and the length of the cabinet area along the thickness direction of the cabinet gradually decreases as the rotation angle decreases.
[0011] In some embodiments, determining the region affiliation of an ingredient based on its current region at each rotation angle includes: obtaining a first count value, which represents the sum of counts of ingredients located in the door region at all rotation angles; obtaining a second count value, which represents the sum of counts of ingredients located in the box region at all rotation angles; and determining the region affiliation of the ingredient based on the first count value and the second count value.
[0012] In some embodiments, determining the region affiliation of an ingredient based on a first count value and a second count value includes: if the first count value is greater than the second count value, determining that the ingredient belongs to the door area; or, if the first count value is less than the second count value, determining that the ingredient belongs to the box area.
[0013] In some embodiments, obtaining a first count value and a second count value includes: when food is included in the door area, determining that the food is located in the door at a certain rotation angle, and updating the first count at the rotation angle; determining the first count at all rotation angles as the first count value; when food is included in the box area, determining that the food associated with the food image is located in the box at a certain rotation angle, and updating the second count at the rotation angle; determining the second count at all rotation angles as the second count value.
[0014] Optionally, updating the first count under the rotation angle includes: updating the first count under the rotation angle to the sum of the current first count and the value 1.
[0015] Optionally, the door area includes food ingredients, including: obtaining the total area of a first food ingredient image and the area of a first partial food ingredient image within the door area; if the ratio of the area of the first partial food ingredient image to the total area of the first food ingredient image is greater than or equal to the first area ratio, it is determined that the door area includes food ingredients.
[0016] Optionally, updating the first count under the rotation angle includes: updating the first count under the rotation angle to the sum of the current first count and the value 1.
[0017] Optionally, the container area includes food ingredients, which includes: obtaining the total area of a second food ingredient image and the area of a second partial food ingredient image within the container area; if the ratio of the area of the second partial food ingredient image to the total area of the second food ingredient image is greater than or equal to the second area ratio, it is determined that the container area includes food ingredients.
[0018] In some embodiments, obtaining the door region corresponding to each rotation angle based on the captured images at each rotation angle includes: segmenting the images corresponding to the door in the captured images at each rotation angle into multiple continuous line segments based on image segmentation technology; and determining the region enclosed by all the continuous line segments as the door region corresponding to the rotation angle. Wherein, the images corresponding to the door in the captured images represent captured images with the door outline.
[0019] In some embodiments, obtaining the box area corresponding to each rotation angle based on the captured image at each rotation angle includes: determining the spacing threshold at the rotation angle according to the correspondence between the set rotation angle and the set spacing threshold; determining the spacing threshold as the length value of the box area along the thickness direction of the box at the rotation angle; and determining the box width value as the width value of the box area along the width direction of the box.
[0020] In some embodiments, obtaining images at each rotation angle includes: periodically obtaining images at the closing angle via the camera when the rotation angle is greater than or equal to an angle threshold but less than the maximum angle; or controlling the camera to stop capturing images when the rotation angle is less than the angle threshold.
[0021] Optionally, the angle threshold is greater than or equal to 35° and less than or equal to 45°, and the maximum angle is 90°.
[0022] In some embodiments, the apparatus includes a processor and a memory storing program instructions, the processor being configured to, when executing the program instructions, perform the method described above for identifying the location of food items in a refrigerator.
[0023] In some embodiments, a refrigerator includes: a cabinet having an opening, a connecting shaft being provided on one side of the opening in a vertical direction, the cabinet including a refrigeration compartment; a door rotatably disposed at the opening via the connecting shaft, the door including a door body and a plurality of bottle holders mounted on the door body; a camera mounted on the top of the refrigeration compartment; and a device for identifying the location of food items in the refrigerator, as described above, mounted on the cabinet.
[0024] In some embodiments, the computer-readable storage medium stores program instructions that, when executed, cause a computer to perform the method described above for identifying the location of food items in a refrigerator.
[0025] The method and apparatus for identifying the location of food in a refrigerator, the refrigerator, and the computer-readable storage medium provided in this disclosure can achieve the following technical effects:
[0026] In this embodiment, when the door is in the closing phase, images are captured at each rotation angle associated with the door. Then, based on these images, the door area and cabinet area corresponding to each rotation angle are obtained, and the current area of the food item at each rotation angle is also determined. Finally, based on the current area of the food item at each rotation angle, the area affiliation of each food item is determined. Thus, by comprehensively analyzing and determining the area affiliation of the food item at each rotation angle, this embodiment overcomes the problem of area judgment errors caused by angle sensor signal delay, improving the accuracy of food item position recognition in the images captured during the closing phase.
[0027] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description
[0028] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:
[0029] Figure 1 This is a schematic diagram of the regional structure of the box area and the door area provided in the embodiments of this disclosure;
[0030] Figure 2 This is a schematic diagram of a method for identifying the location of food items in a refrigerator, provided in an embodiment of this disclosure;
[0031] Figure 3 This is a schematic diagram of another method for identifying the location of food items in a refrigerator, provided in an embodiment of this disclosure;
[0032] Figure 4 This is a schematic diagram of another method for identifying the location of food items in a refrigerator, provided in an embodiment of this disclosure;
[0033] Figure 5 This is a schematic diagram of another method for identifying the location of food items in a refrigerator, provided in an embodiment of this disclosure;
[0034] Figure 6 This is a schematic diagram of a device for identifying the location of food items in a refrigerator, provided in an embodiment of this disclosure;
[0035] Figure 7 This is a schematic diagram of a refrigerator provided in an embodiment of this disclosure. Detailed Implementation
[0036] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.
[0037] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.
[0038] Unless otherwise stated, the term "multiple" means two or more.
[0039] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.
[0040] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.
[0041] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.
[0042] In this embodiment of the disclosure, smart home appliances refer to home appliances formed by introducing microprocessors, sensor technology and network communication technology into home appliances. They have the characteristics of intelligent control, intelligent sensing and intelligent application. The operation of smart home appliances often relies on the application and processing of modern technologies such as the Internet of Things, the Internet and electronic chips. For example, smart home appliances can be connected to electronic devices to enable users to remotely control and manage smart home appliances.
[0043] In this embodiment of the disclosure, the refrigerator includes a cabinet and a door, and a camera.
[0044] The cabinet has an opening, and a connecting shaft is provided vertically on one side of the opening. The cabinet includes a refrigeration compartment. Optionally, the refrigeration compartment also has an opening. The refrigeration compartment includes a shelf area and a drawer area, with the drawer area located below the shelf area. One or more drawers are arranged vertically in the drawer area.
[0045] The door is rotatably mounted at the opening of the box via a connecting shaft. The door includes a main body and multiple bottle holders installed on the main body, with food items stored in the multiple bottle holders.
[0046] A camera is mounted on top of the refrigerated compartment. The camera is configured to capture images of the door when it is closed or of the front of the compartment opening. Understandably, the camera is also configured to capture images of the food inside the refrigerated compartment. There may be one or multiple cameras.
[0047] Optionally, multiple bottle holders are arranged vertically along the inner side of the main body of the door.
[0048] Optionally, the cabinet also includes a freezer compartment, with the refrigerator compartment located above the freezer compartment.
[0049] Based on the above refrigerator structural configuration, combined with Figure 2 As shown in the embodiments of this disclosure, a method for identifying the location of food items in a refrigerator is provided, including:
[0050] S01, with the refrigerator door in the closed position, acquires images at each associated rotation angle of the door, including images of food items in multiple bottle holders and / or drawer contents. Bottle holder contents include one or more types of food, and drawer contents include one or more types of food.
[0051] In this step, the refrigerator determines that the door is in the closed stage as follows: the refrigerator obtains the current rotation angle of the door; if the current rotation angle is greater than a rotation angle threshold and the current rotation angle shows a decreasing trend, the refrigerator determines that the door is in the closed stage. Here, the rotation angle represents the angle between the plane of the door and the plane of the refrigerator opening. The rotation angle threshold is greater than or equal to 45° and less than or equal to the maximum rotation angle, which is determined by the refrigerator model.
[0052] S02, the refrigerator obtains the door area and cabinet area corresponding to each rotation angle based on the images captured at each rotation angle.
[0053] In this step, combined Figure 1 As shown, the door area represents the area consisting of the main body of the door and multiple bottle holders when the door is closed. The cabinet area represents the area where the door is closed, coplanar with the bottom surface of the bottom drawer of the refrigerator compartment, and located in front of the cabinet opening. The width of the cabinet area along the width direction of the cabinet is equal to the width of the cabinet, and the length of the cabinet area along the thickness direction of the cabinet gradually decreases as the rotation angle decreases.
[0054] It should be noted that when the door is in the closing stage, the door area is different at different rotation angles, and the width of the box area along the width direction of the box remains unchanged, while the length of the box area along the thickness direction of the box gradually decreases as the rotation angle decreases.
[0055] S03, the refrigerator obtains the current area of the food at each rotation angle, the current area including the door area or the cabinet area.
[0056] In this step, when the door is closed, the food items captured in the image may be located in the door area or the container area at different rotation angles. Therefore, this embodiment obtains the current area of the food items at each rotation angle, rather than relying on the confirmation of the area at a single rotation angle, to achieve accurate identification of the food item's location.
[0057] S04, the refrigerator determines the area belonging of the food based on the current area of the food at each rotation angle.
[0058] The method for identifying the location of food items in a refrigerator, as provided in this disclosure, involves acquiring images of the door at each associated rotation angle when the door is closed. Based on these images, the method then obtains the door area and refrigerator area corresponding to each rotation angle, and identifies the current area of the food item at each rotation angle. Finally, based on the current area of the food item at each rotation angle, the area affiliation of each food item is determined. Thus, by comprehensively analyzing and determining the area affiliation of the food item at each rotation angle, this method overcomes the problem of area judgment errors caused by angle sensor signal delay, improving the accuracy of food item location identification within images captured during the door-closing phase.
[0059] Optionally, combined Figure 3 As shown, the refrigerator determines the food's region affiliation based on the food's current region at each rotation angle, including:
[0060] S11, the refrigerator obtains a first count value, which represents the sum of the counts of food items located in the door area under all rotation angles. Here, "food items" refers to any type of food item in the bottle holder area, or any type of food item in the drawer area.
[0061] S12, the refrigerator obtains a second count value, which represents the sum of the counts of food items located in the refrigerator area under all rotation angles.
[0062] S13, the refrigerator determines the area of the food based on the first count value and the second count value.
[0063] In this embodiment, after obtaining the food items included in the images captured at each rotation angle, the present disclosure calculates the sum of the counts of food items located in the door area at all rotation angles to obtain a first count value, and calculates the sum of the counts of food items located in the cabinet area at all rotation angles to obtain a second count value. Then, based on the first and second count values, the region to which the food items belong is determined. Thus, this embodiment can statistically analyze and determine the region to which the food items belong by obtaining the sum of the counts of food items located in the door area and the sum of the counts of food items located in the cabinet area at all rotation angles. This helps overcome the problem of region judgment errors caused by angle sensor signal delay and improves the accuracy of food item position recognition in images captured during the door closing phase.
[0064] Optionally, the refrigerator determines the zone affiliation of food items based on the first count value and the second count value, including:
[0065] If the first count value is greater than the second count value, the refrigerator determines that the area containing the food belongs to the door area. Alternatively,
[0066] If the first count value is less than the second count value, the refrigerator determines that the area containing the food is classified as the refrigerator compartment area.
[0067] Thus, if the first count value is greater than the second count value, it indicates that the number of times the food appears in the door area across all rotation angles is higher than the number of times it appears in the cabinet area across all rotation angles. Therefore, the food is determined to belong to the door area. Conversely, if the first count value is less than the second count value, it indicates that the number of times the food appears in the door area across all rotation angles is lower than the number of times it appears in the cabinet area across all rotation angles. Therefore, the food is determined to belong to the cabinet area. Thus, even with signal delays at individual or multiple rotation angles, the location of the food can be accurately identified through counting statistics. This successfully overcomes the problem of area judgment errors caused by angle sensor signal delays and improves the accuracy of food position identification in images captured during the door closing phase.
[0068] Optionally, combined Figure 4 As shown, the refrigerator statistically obtains a first count value and a second count value, including:
[0069] S21, if the food is included in the door area, the refrigerator determines that the food is located in the door at the rotation angle and updates the first count at the rotation angle. The initial value of the first count is zero.
[0070] In this step, the first number of times the refrigerator updates its rotation angle includes: the first number of times the refrigerator updates its rotation angle is the sum of the current first number and the value 1.
[0071] The door area includes food items, comprising: the total area of a first food item image obtained by the refrigerator and the area of a first partial food item image within the door area; the refrigerator determines that the door area includes food items if the ratio of the first partial food item image area to the total area of the first food item image is greater than or equal to a first area ratio. Understandably, if the ratio of the first partial food item image area to the total area of the first food item image is less than the first area ratio, the door area does not include food items. The first partial food item image area within the door area represents the area value of a partial food item image located within the door area. This partial food item image is located within the door area, while other parts of the food item image are located outside the door area. The first area ratio is greater than or equal to 50% and less than 100%.
[0072] S22, the refrigerator determines the first count value for all rotation angles.
[0073] S23, if the refrigerator includes food within the cabinet area, it determines that the food associated with the food image at the rotation angle is located within the cabinet, and updates the second count at the rotation angle. The initial value of the second count is zero.
[0074] In this step, the second number of times the refrigerator updates its rotation angle includes: the second number of times the refrigerator updates its rotation angle is the sum of the current second number and the value 1.
[0075] The refrigerator determines that the area within the cabinet includes food, comprising: the total area of a second food image obtained by the refrigerator and the area of a second partial food image within the cabinet area; if the ratio of the second partial food image area to the total area of the second food image is greater than or equal to a second area ratio, the refrigerator determines that the area within the cabinet includes food. Understandably, if the ratio of the second partial food image area to the total area of the second food image is less than the second area ratio, the refrigerator determines that the area within the cabinet does not include food. The second partial food image area within the cabinet area represents the area value of a partial food image located within the cabinet area. Wherein, a portion of the food image is located within the cabinet area, and the other portion of the food image is located outside the cabinet area. Wherein, the second area ratio is greater than or equal to 50% and less than 100%. It should be noted that the second area ratio may be equal to or unequal to the first area ratio. Preferably, the second area ratio may be equal to the first area ratio.
[0076] S24, the refrigerator determines the second count value for all rotation angles.
[0077] In this embodiment, when food is included within the door area, the system determines that the food is located within the door at each rotation angle and updates the first count at each rotation angle. Then, it determines the first count across all rotation angles as a first count value. When food is included within the cabinet area, the system determines that the food image associated with the food is located within the cabinet at each rotation angle and updates the second count at each rotation angle. Thus, this embodiment can statistically analyze and determine the regional affiliation of food by obtaining the sum of counts of food located within the door area and the sum of counts of food located within the cabinet area at all rotation angles. This helps overcome the problem of regional judgment errors caused by angle sensor signal delay and improves the accuracy of food position recognition in images captured during the door closing phase.
[0078] It should be noted that as the rotation angle gradually decreases, there is some overlap between the door area and the box area corresponding to the rotation angle. As an example, at a certain rotation angle, the part of the door area near the door rotation axis and the part of the box area near the door rotation axis overlap. If there is food in the overlapping area, the food will be located in both the door area and the box area at the same time. In this case, the first count and the second count at the rotation angle are updated synchronously.
[0079] Optionally, the refrigerator obtains the door area corresponding to each rotation angle based on the images captured at each rotation angle, including:
[0080] The refrigerator uses image segmentation technology to divide the image corresponding to the door in the captured image at a rotation angle into multiple continuous line segments. The image corresponding to the door represents the captured image containing the door's outline.
[0081] The refrigerator defines the area enclosed by all continuous line segments as the door area corresponding to the rotation angle.
[0082] Thus, this embodiment of the present disclosure uses image segmentation technology to segment the image corresponding to the door in the image captured at the rotation angle into multiple continuous line segments, and determines the area enclosed by all continuous line segments as the door area corresponding to the rotation angle. In this way, this embodiment of the present disclosure can identify the door area from the image captured at the rotation angle using image segmentation technology, providing a strong basis for subsequent determination of the food area's location, and further improving the accuracy of food position identification in the image captured during the door closing phase.
[0083] Optionally, combined Figure 5 As shown, the refrigerator obtains the cabinet area corresponding to each rotation angle based on the images captured at each rotation angle, including:
[0084] S31, the refrigerator determines the spacing threshold at the set rotation angle based on the correspondence between the set rotation angle and the set spacing threshold. In this step, the set spacing threshold gradually decreases as the set rotation angle decreases. Simultaneously, the correspondence between the set rotation angle and the set spacing threshold can be obtained through fitting multiple experiments.
[0085] S32, the refrigerator determines the spacing threshold as the length of the cabinet area along the thickness direction of the cabinet at the rotation angle.
[0086] S33, the refrigerator determines the cabinet width value as the width value of the cabinet area along the cabinet width direction.
[0087] In this embodiment, based on the correspondence between a set rotation angle and a set spacing threshold, the spacing threshold is determined as the length of the box area along the thickness direction of the box at the set rotation angle. Then, the width of the box is determined as the width of the box area along the width direction of the box. Thus, this embodiment can use the spacing threshold determined based on the correspondence between the set rotation angle and the set spacing threshold as the length of the box area along the thickness direction of the box at the set rotation angle. This allows for real-time adjustment of the length of the box area along the thickness direction, avoiding misidentification of food location due to an excessively large box area and avoiding the possibility of food location being identified as the door area due to an excessively small box area. This reduces misjudgment of food location caused by unreasonable box area configuration, thereby further improving the accuracy and reliability of food location identification.
[0088] Understandably, the width of the box area along the width direction is the same at each rotation angle. The width of the box area along the width direction at each rotation angle is the same as the box width.
[0089] Optionally, the refrigerator acquires images at each rotation angle, including:
[0090] When the rotation angle is greater than or equal to a critical angle value but less than the maximum angle value, the refrigerator periodically captures images at the rotation angle using a camera. Alternatively,
[0091] When the rotation angle is less than the critical angle value, the refrigerator controls the camera to stop capturing images.
[0092] The angle threshold is greater than or equal to 35° and less than or equal to 45°, with a maximum angle of 90°. Preferably, the angle threshold is 38°.
[0093] Thus, in this embodiment, when the rotation angle is greater than or equal to the angle threshold but less than the maximum angle, the camera periodically acquires images at different rotation angles to obtain images at different rotation angles in a time-series manner, ensuring the amount of data collected and improving the reliability of food location identification. However, when the rotation angle is less than the angle threshold, the spacing threshold at that angle is smaller, and correspondingly, the length of the container area along the thickness direction of the container is also smaller. In this case, the food in the captured image is most likely located in the door area. If the aforementioned steps are continued to determine the food area, it will affect the magnitude of the first or second count value, leading to misjudgments in food location identification. Therefore, when the rotation angle is less than the angle threshold, this embodiment controls the camera to stop capturing images to ensure the accuracy of food location identification.
[0094] In practical applications, the method for identifying the location of food items in a refrigerator specifically performs the following steps:
[0095] First, with the refrigerator door closed, it captures images at each associated rotation angle of the door. These images include food items in multiple bottle holders and / or drawer contents. Specifically, the bottle holder foods include milk and cucumber, while the drawer contents include purple cabbage.
[0096] Secondly, the refrigerator obtains the current area of the food at each rotation angle, including the door area or the cabinet area.
[0097] Then, the refrigerator calculates a first count value, representing the sum of the counts of food items located in the door area under all rotation angles, and a second count value, representing the sum of the counts of food items located in the refrigerator body under all rotation angles. Specifically, the sum of the counts of milk in the door area under all rotation angles is 11, and the sum of the counts of cucumber in the door area under all rotation angles is 9. The sum of the counts of cucumber in the door area under all rotation angles is 12, and the sum of the counts of cucumber in the door area under all rotation angles is 8. The sum of the counts of purple cabbage in the door area under all rotation angles is 7, and the sum of the counts of purple cabbage in the door area under all rotation angles is 13.
[0098] Finally, based on the results, the first count value of the milk was greater than the second count value, and the first count value of the cucumber was greater than the second count value. Therefore, the refrigerator determined that the milk and cucumber belonged to the door area. However, the first count value of the purple cabbage was less than the second count value, so the refrigerator determined that the purple cabbage belonged to the cabinet area.
[0099] Combination Figure 6 As shown, this embodiment of the disclosure provides a device 70 for identifying the location of food items in a refrigerator, including a processor 700 and a memory 701. Optionally, the device 70 may further include a communication interface 702 and a bus 703. The processor 700, communication interface 702, and memory 701 can communicate with each other via the bus 703. The communication interface 702 can be used for information transmission. The processor 700 can call logical instructions in the memory 701 to execute the method for identifying the location of food items in a refrigerator as described in the above embodiment.
[0100] Furthermore, the logic instructions in the aforementioned memory 701 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.
[0101] The memory 701, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 100 executes functional applications and data processing by running the program instructions / modules stored in the memory 701, that is, it implements the method for identifying the location of food items in the refrigerator in the above embodiments.
[0102] The memory 701 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 701 may include high-speed random access memory and may also include non-volatile memory.
[0103] Combination Figure 7 As shown, this embodiment of the present disclosure provides a refrigerator 100, including: a refrigerator body and a door, a camera, and the aforementioned device 70 for identifying the location of food items in the refrigerator. The refrigerator body has an opening, and a connecting shaft is provided on one side of the opening in a vertical direction. The refrigerator body includes a refrigeration compartment. Optionally, the refrigeration compartment has an opening. The refrigeration compartment includes a shelf area and a drawer area, with the drawer area located below the shelf area. One or more drawers are arranged sequentially in the vertical direction in the drawer area. The door is rotatably mounted at the opening of the refrigerator body via the connecting shaft. The door includes a door body and multiple bottle holders installed on the door body, and the multiple bottle holders store food items. The camera is installed on the top of the refrigeration compartment. The camera is configured to capture an image of the door when it is closed or to capture an image of the front side of the opening.
[0104] The device 70 for identifying the location of food items in the refrigerator is installed in the refrigerator body. The installation relationship described herein is not limited to placement inside the refrigerator body, but also includes installation and connection with other components of the refrigerator 100, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the device 70 for identifying the location of food items in the refrigerator can be adapted to any feasible refrigerator body, thereby realizing other feasible embodiments.
[0105] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described method for identifying the location of food items in a refrigerator.
[0106] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.
[0107] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.
[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0109] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
Claims
1. A method for identifying the location of food items in a refrigerator, characterized in that, The refrigerator includes a cabinet with an opening, and a connecting shaft along a vertical direction on one side of the opening. The cabinet includes a refrigeration compartment. The door is rotatably mounted at the opening via the connecting shaft. The door includes a door body and multiple bottle holders mounted on the door body. The camera is installed on the top of the refrigerated area, using methods including: With the door closed, capture images at each rotation angle associated with the door, including bottle ingredients in multiple bottle holders and / or drawer ingredients in multiple drawers; Based on the images captured at each rotation angle, the door area and box area corresponding to each rotation angle are obtained; Obtain the current area of the food at each rotation angle; the current area includes the door area or the box area. Based on the current area of the food at each rotation angle, determine the area to which the food belongs.
2. The method according to claim 1, characterized in that, Based on the current region of the food at each rotation angle, determine the region to which the food belongs, including: The first count value is obtained by statistics. The first count value represents the sum of the counts and values of the food items located in the door area under all rotation states. The second count value is obtained by statistics. The second count value represents the sum of the counts and values of the food in the box area under all rotation states. The regional affiliation of the ingredients is determined based on the first and second count values.
3. The method according to claim 2, characterized in that, Based on the first and second count values, the regional affiliation of the ingredients is determined, including: If the first count value is greater than the second count value, the area containing the food is determined to belong to the door area; or, If the first count value is less than the second count value, the area containing the food is determined to belong to the container area.
4. The method according to claim 2, characterized in that, The statistics yielded a first count value and a second count value, including: If the door area includes food, determine that the food is located within the door during rotation, and update the first count during rotation. Obtain the first count for all rotation states and obtain the first count value; If the food is included within the box area, determine that the food image associated with the rotating state is located in the box, and update the second rotation state. Obtain the second number of rotations under all rotation states, and obtain the second count value.
5. The method according to any one of claims 1 to 4, characterized in that, Based on the images captured at each rotation state, the corresponding door area for each rotation state is obtained, including: Based on image segmentation technology, the image corresponding to the door in the captured image under rotation is segmented into multiple continuous line segments; The area enclosed by all continuous line segments is defined as the gate area corresponding to the rotation state. The width of the box is determined to be the width of the box area along the width direction of the box.
6. The method according to any one of claims 1 to 4, characterized in that, Based on the images captured at each rotation angle, the box area corresponding to each rotation state is obtained, including: Based on the correspondence between the set rotation angle and the set spacing threshold, the spacing threshold under the rotation angle is determined; The spacing threshold is determined as the length of the box area along the thickness direction of the box at the rotation angle.
7. The method according to any one of claims 1 to 4, characterized in that, Obtain images at each rotation angle, including: When the rotation angle is greater than or equal to the critical angle value but less than the maximum angle value, images are periodically captured by the camera at the rotation angle; or, When the door closing angle is less than the critical angle value, the camera will stop capturing images.
8. A device for identifying the location of food items in a refrigerator, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when running the program instructions, perform the method for identifying the location of food items in a refrigerator as described in any one of claims 1 to 7.
9. A refrigerator, characterized in that, include: The cabinet has an opening, and a connecting shaft is provided on one side of the opening in a vertical direction. The cabinet includes a refrigeration area. The door is rotatably mounted at the opening of the box via a connecting shaft. The door includes a main body and multiple bottle holders mounted on the main body. The camera is installed on the top of the cold storage area; The device for identifying the location of food items in a refrigerator as described in claim 8 is installed in the refrigerator body.
10. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed, they cause the computer to perform the method for identifying the location of food items in the refrigerator as described in any one of claims 1 to 7.