Refrigerator and food material occupation amount determining method thereof

Abstract

The invention relates to a refrigerator and a food material occupation amount determining method thereof. The refrigerator comprises a refrigerator body which is provided with at least one chamber; the compartment comprises at least one storage compartment; at least one image acquirer configured to acquire compartment images of the corresponding storage compartments; the at least one controller is connected with each image collector and is configured to obtain the compartment image of each storage compartment; for each compartment image, performing scale scaling processing on the compartment image to obtain reference compartment images under different scales; determining a target detection frame of each food material target in the compartment images according to the reference compartment images under different scales; and for each storage compartment, determining the food material occupation amount of the storage compartment according to the area proportion of the target detection frame in the compartment image corresponding to the storage compartment in the corresponding compartment image. The method can reduce the determination cost of the food material occupation amount.

Description

A refrigerator and a method for determining the amount of food it occupies. Technical Field

[0001] This application relates to the field of refrigerators, and in particular to a refrigerator and a method for determining the amount of food occupied by the refrigerator. Background Technology

[0002] With the rapid development of smart home appliances and IoT technology, refrigerators are no longer just refrigeration devices for storing food, but have gradually evolved into home food management centers integrating intelligent sensing, data analysis, and energy-saving control. To improve the intelligence level and user experience of refrigerators, various sensors (such as temperature and humidity sensors) and image acquisition devices can be integrated inside the refrigerator to realize the perception and analysis of information such as the status of food and storage capacity.

[0003] In traditional technologies, refrigerators often rely on simple weight sensors or infrared sensors to estimate the amount of food inside. For example, radar can be used to acquire point cloud data of the refrigerator's interior to determine the food occupancy, or multiple infrared transceivers can be placed on the refrigerator walls to determine the amount of infrared laser light received by the receivers.

[0004] However, radar or multi-channel infrared transceivers are mostly integrated from hardware, and the calibration and maintenance costs for each piece of hardware are relatively high, and they also impose certain limitations on the internal structure design of the refrigerator. Therefore, there is an urgent need for a method that can determine the amount of food to be consumed at a lower cost. Summary of the Invention

[0005] This application provides a refrigerator and a method for determining the amount of food occupied, so as to reduce the cost of determining the amount of food occupied.

[0006] In a first aspect, some embodiments provide a refrigerator, including:

[0007] The enclosure has at least one compartment; the compartment includes at least one storage compartment.

[0008] At least one image acquisition device is configured to acquire compartment images of the corresponding storage compartment;

[0009] At least one controller is connected to each image acquisition unit and is configured as follows:

[0010] Obtain the compartment images of each storage compartment;

[0011] For each compartment image, the compartment image is scaled to obtain reference compartment images at different scales;

[0012] Based on reference compartment images at different scales, determine the target detection bounding boxes for each food item in the compartment images;

[0013] For each storage compartment, the amount of food occupied in the storage compartment is determined based on the area ratio of the target detection box in the corresponding compartment image.

[0014] The refrigerator provided in the above embodiments ensures the targeted and completeness of image acquisition by acquiring independent images of each storage compartment, avoiding overlapping or occlusion issues between different compartments. Multi-scale scaling processing is performed on each image to generate reference images at different scales. This allows the model to simultaneously capture the overall outline of large-volume food items and the local details of small-volume food items, significantly improving the detection coverage of food items of different sizes. Target detection based on multi-scale reference images effectively fuses feature information from different scales, enhancing the model's robustness in complex scenes, thereby more accurately locating and identifying various food targets in the image. The food occupancy of the storage compartment is quantified by the area ratio of the target detection box in the image, effectively eliminating interference from non-food backgrounds and objectively reflecting the actual space occupied by the food. Furthermore, the determination of food occupancy can be completed using only an image acquisition device. Compared to using composite sensing systems such as radar point clouds or infrared arrays, this significantly reduces hardware costs and structural complexity, further achieving efficient and reliable visual perception of the refrigerator's internal space at a limited cost.

[0015] Secondly, some embodiments also provide a method for determining the amount of food occupied in a refrigerator, including:

[0016] Obtain the compartment images of each storage compartment;

[0017] For each compartment image, the compartment image is scaled to obtain reference compartment images at different scales;

[0018] Based on reference compartment images at different scales, determine the target detection bounding boxes for each food item in the compartment images;

[0019] For each storage compartment, the amount of food occupied in the storage compartment is determined based on the area ratio of the target detection box in the corresponding compartment image.

[0020] The refrigerator food occupancy determination method provided in the above embodiments ensures the targetedness and completeness of image acquisition by acquiring independent compartment images of each storage compartment, avoiding overlapping or occlusion issues between different compartments. Multi-scale scaling processing is performed on each image to generate reference images at different scales, enabling the model to simultaneously capture the overall outline of large-volume food items and the local details of small-volume food items, significantly improving the detection coverage of food items of different sizes. Target detection based on multi-scale reference images effectively fuses feature information at different scales, enhancing the model's robustness in complex scenes, thereby more accurately locating and identifying various food targets in the image. The food occupancy of the storage compartment is quantified by the area ratio of the target detection box in the image, effectively eliminating interference from non-food backgrounds and objectively reflecting the actual space occupied by the food.

[0021] Thirdly, some embodiments also provide a device for determining the amount of food ingredients consumed, including:

[0022] The acquisition module is used to acquire images of each storage compartment.

[0023] The scaling module is used to scale the images of each compartment to obtain reference compartment images at different scales.

[0024] The detection module is used to determine the target detection box of each food item in the compartment image based on the reference compartment images at different scales;

[0025] The determination module is used to determine the amount of food occupied in each storage compartment based on the area ratio of the target detection box in the corresponding compartment image.

[0026] The food occupancy determination device provided in the above embodiments ensures the targetedness and completeness of image acquisition by acquiring independent images of each storage compartment, avoiding overlapping or occlusion of views between different compartments. Multi-scale scaling processing is performed on each image to generate reference images at different scales. This allows the model to simultaneously capture the overall outline of large-volume food items and the local details of small-volume food items, significantly improving the detection coverage of food items of different sizes. Target detection based on multi-scale reference images effectively fuses feature information at different scales, enhancing the model's robustness in complex scenes, thereby more accurately locating and identifying various food targets in the image. The food occupancy of the storage compartment is quantified by the area ratio of the target detection box in the image, effectively eliminating interference from non-food backgrounds and objectively reflecting the actual space occupied by the food.

[0027] Fourthly, some embodiments also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0028] Obtain the compartment images of each storage compartment;

[0029] For each compartment image, the compartment image is scaled to obtain reference compartment images at different scales;

[0030] Based on reference compartment images at different scales, determine the target detection bounding boxes for each food item in the compartment images;

[0031] For each storage compartment, the amount of food occupied in the storage compartment is determined based on the area ratio of the target detection box in the corresponding compartment image.

[0032] The readable storage medium provided in the above embodiments stores a computer program that, when executed by a processor, acquires independent images of each storage compartment, ensuring the targeted and complete nature of image acquisition and avoiding overlapping or occlusion of views between different compartments. Multi-scale scaling processing is performed on each image to generate reference images at different scales, enabling the model to simultaneously capture the overall outline of large-volume ingredients and the local details of small-volume ingredients, significantly improving the detection coverage of ingredients of different sizes. Target detection based on multi-scale reference images effectively fuses feature information at different scales, enhancing the model's robustness in complex scenes, thereby more accurately locating and identifying various food targets in the image. The area ratio of the target detection box in the image is used to quantify the food occupancy of the storage compartment, effectively eliminating interference from non-food backgrounds and objectively reflecting the actual space occupied by the food.

[0033] Fifthly, some embodiments also provide a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0034] Obtain the compartment images of each storage compartment;

[0035] For each compartment image, the compartment image is scaled to obtain reference compartment images at different scales;

[0036] Based on reference compartment images at different scales, determine the target detection bounding boxes for each food item in the compartment images;

[0037] For each storage compartment, the amount of food occupied in the storage compartment is determined based on the area ratio of the target detection box in the corresponding compartment image.

[0038] The computer program provided in the above embodiments, when executed by the processor, acquires independent images of each storage compartment, ensuring the targetedness and completeness of image acquisition and avoiding overlapping or occlusion of views between different compartments. Multi-scale scaling processing is performed on each image to generate reference images at different scales. This allows the model to simultaneously capture the overall outline of large-volume ingredients and the local details of small-volume ingredients, significantly improving the detection coverage of ingredients of different sizes. Target detection based on multi-scale reference images effectively fuses feature information at different scales, enhancing the model's robustness in complex scenes, thereby more accurately locating and identifying various food targets in the image. The area ratio of the target detection box in the image is used to quantify the food occupancy of the storage compartment, effectively eliminating interference from non-food backgrounds and objectively reflecting the actual space occupied by the food. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 is a schematic block diagram of a first type of refrigerator structure provided in some embodiments of this application;

[0041] Figure 2 is a schematic block diagram of a second refrigerator structure provided in some embodiments of this application;

[0042] Figure 3 is a schematic block diagram of the structure of the processing device in the refrigerator provided in some embodiments of this application;

[0043] Figure 4 is a schematic block diagram of a third refrigerator structure provided in some embodiments of this application;

[0044] Figure 5 is a schematic block diagram of a fourth refrigerator structure provided in some embodiments of this application;

[0045] Figure 6 is a flowchart illustrating a first method for determining the amount of food stored in a refrigerator, provided in some embodiments of this application.

[0046] Figure 7A is a flowchart illustrating a target detection box determination step provided in some embodiments of this application;

[0047] Figure 7B is a schematic diagram of a target detection box provided in some embodiments of this application;

[0048] Figure 8 is a flowchart illustrating a step for obtaining a target feature map according to some embodiments of this application;

[0049] Figure 9A is a flowchart illustrating a step for obtaining a target feature map according to some embodiments of this application;

[0050] Figure 9B is a schematic diagram of a process for determining the amount of food ingredients provided in some embodiments of this application;

[0051] Figure 10 is a schematic flowchart of a compartment temperature control step provided in some embodiments of this application;

[0052] Figure 11 is a flowchart illustrating another compartment temperature control step provided in some embodiments of this application;

[0053] Figure 12 is a flowchart illustrating the steps for outputting a food occupancy reminder according to some embodiments of this application;

[0054] Figure 13 is a flowchart illustrating the steps for determining the overall machine power-on rate reward score according to some embodiments of this application;

[0055] Figure 14 is a flowchart illustrating a device for determining the amount of food ingredients provided in some embodiments of this application;

[0056] Figure 15 is an internal structural diagram of a computer device provided in some embodiments of this application. Detailed Implementation

[0057] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0058] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0059] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0060] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0061] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0062] The refrigerator 1 provided in this application will now be described with reference to the accompanying drawings, the overall structure of which is shown in Figure 1. The refrigerator 1 includes a cabinet 10 and a processing device 20.

[0063] As shown in Figure 2, the housing 10 has at least one storage compartment.

[0064] Storage rooms are typically divided into freezer rooms and refrigerator rooms (referred to as refrigerator rooms). They can also be further divided into chambers with special functions, such as chambers for storing fruits and vegetables. Refrigerator rooms can maintain a temperature range of approximately 4°C to store food, medicine, or biological agents in a refrigerated state. Freezer rooms can maintain a temperature range of approximately -18°C to store food, medicine, or biological agents in a frozen state.

[0065] The storage compartment has an opening that can be opened and closed via a door 11 hinged to the outer casing, or via a drawer 12. When a refrigerator compartment and a freezer compartment are provided, one opening can be opened and closed via a door (e.g., the refrigerator compartment), and the other opening can be opened and closed via a drawer 12 (e.g., the freezer compartment).

[0066] The housing 10 employs a vapor compression refrigeration cycle to generate energy for maintaining the target temperature. The refrigeration cycle consists of a compressor 161, a condenser, a throttling device, and an evaporator. The refrigeration cycle involves a series of processes, including compression, condensation, expansion, and evaporation, to cool the storage compartment and maintain an ideal low-temperature storage environment inside.

[0067] In a vapor compression refrigeration cycle, a low-temperature, low-pressure refrigerant enters the compressor 161, which compresses it into a high-temperature, high-pressure refrigerant gas and discharges the compressed refrigerant gas. The discharged refrigerant gas flows into the condenser, where the condenser condenses the compressed refrigerant into a liquid phase, and the heat is released to the surrounding environment through the condensation process.

[0068] The throttling device causes the high-temperature, high-pressure liquid refrigerant formed in the condenser to expand into a low-pressure liquid refrigerant. The evaporator evaporates the refrigerant that has expanded in the throttling device and returns the low-temperature, low-pressure refrigerant gas to the compressor 161. The evaporator can achieve a cooling effect by exchanging heat with the material to be cooled through the latent heat of refrigerant evaporation. In this application, the evaporator exchanges heat with air to form air for cooling the storage compartment, thereby cooling the storage compartment. The throttling device can be a capillary tube.

[0069] A filter is also installed downstream of the condenser. The filter is used to filter impurities in the refrigerant, improve the heat exchange efficiency of the refrigeration unit, and reduce the risk of pipe blockage.

[0070] A liquid receiver can also be installed on the suction side of the compressor 161. The liquid receiver is used to separate the refrigerant into gas and liquid phases. The liquid receiver is a shell-shaped component. The gas-liquid mixed refrigerant fluid enters the liquid receiver for basic phase separation. The gas enters the gas passage and undergoes gravity settling to separate droplets, while the liquid enters the liquid space and separates into bubbles. The gas flows out from the gas outlet and is then drawn into the compressor 161, preventing the compressor 161 from carrying liquid in the suction and reducing the service life of the compressor 161.

[0071] The compressor 161 and condenser can be located at the lower rear of the housing, while the evaporator can be located at the rear of the housing corresponding to the storage compartment. The evaporator and condenser can also be arranged in other locations according to the industrial design of the housing 10, which will not be listed here. The location where the evaporator is located has sufficient space to allow air to flow. The air is driven by the fan 162 to deliver the air generated by the evaporator for cooling the storage compartment to the target location and to draw in air from the storage compartment, forming an air circulation. In one or more embodiments of this application, the fan 162 includes a refrigeration fan and a freezing fan. In one or more embodiments of this application, the fan 162 can also be configured in conjunction with the condenser.

[0072] In one or more embodiments of this application, the evaporator may also be divided into two parts for the refrigerator compartment and the freezer compartment, referred to as the refrigerator compartment cooler and the freezer compartment cooler.

[0073] A defrosting element is provided in the housing 10. The defrosting element is configured to generate heat for defrosting the evaporator, thereby putting the evaporator in a defrosting state. In one or more embodiments of this application, the defrosting element includes a defrosting heater 163, which may be an electric heating tape or an electric heater. In one or more embodiments of this application, the defrosting element may also be a combination of an electric heating tape or an electric heater, and a heat exchanger or heat exchange piping. When defrosting conditions are met, the heat exchange piping is opened, and the high-temperature, high-pressure refrigerant discharged from the compressor 161 enters the heat exchange piping, exchanges heat with the surrounding air, raises the air temperature, and further provides heat to melt the frost layer on the evaporator surface, thereby putting the evaporator in a defrosting state. The heat exchange piping may be located below the evaporator, utilizing the principle that hot air has a lower density and rises to guide the air to remove the ice or frost layer on the evaporator. The defrosting element composed of an electric heating tape or an electric heater may also be located around the evaporator in other positions, such as above or to one side of the evaporator.

[0074] A display 164 is installed on the cabinet 10.

[0075] The cabinet 10 is equipped with a refrigeration system, which is configured to transfer heat from the inside of the refrigerator to the outside through the circulation of refrigerant.

[0076] As shown in Figure 3, the hardware configuration of the processing device 20 is illustrated. The processing device 20 includes components such as a processor 201, volatile memory 203, non-volatile memory 202, a display device 204, an operation device 205, a communication interface 206, and a drive device 207, which are interconnected via a bus 208. The processor 201 can be a dedicated processor 201, a central processing unit, etc. The processor 201 can access the storage unit to execute instructions or application programs stored in the storage unit to achieve related functions. The display device 204 is a display device 204 for displaying various information, the operation device 205 is an operation device for receiving various operations, and the drive device 207 is a hardware terminal that interacts with the storage medium. In one or more embodiments of this application, the storage medium includes media such as CD-ROM, floppy disk, and optical-magnetic-optical disk that record information in an optical, electrical, or magnetic manner. The storage medium can also be a semiconductor memory such as ROM or flash memory that records information in an electrical manner.

[0077] In one or more embodiments of this application, the processing device 20 may be a controller 13. The controller 13 is disposed in the housing 10.

[0078] In one or more embodiments of this application, the processing device 20 may be communicatively connected to the controller 13, for example, by a terminal device 15 and / or a cloud server 14.

[0079] In one or more embodiments of this application, some functions of the processing device 20 may be implemented by the controller 13, and some functions may be implemented by the terminal device 15 and / or the cloud server 14.

[0080] Controller 13 can communicate with terminal device 15 and / or server 14. The network between controller 13 and terminal device 15, or between controller 13 and server 14, can be the Internet, cellular network, Wi-Fi network, low power wide area network (LPWAN), WAN, LAN, etc., based on standards and protocols such as LoRa, Sigfox, and NB-IoT.

[0081] The cabinet 10 can be used in home environments to store daily necessities such as food and cold drinks; it can also be used in commercial places such as restaurants, hotels, supermarkets, and convenience stores to store ingredients, food, and drinks to meet customer needs; and it can also be used in places such as hospitals and laboratories to store medicines and biological samples to meet medical and scientific research needs.

[0082] Server 14 can provide various network services, such as resource and data access for refrigerator 1 controller 13 and terminal device 15. Server 14 has higher performance and reliability. Server 14 can connect to multiple refrigerator 1 controllers 13, multiple terminal devices 15, and other smart home appliance terminals.

[0083] Terminal device 15 is an electronic device with intelligent functions. It can connect to the aforementioned networks to achieve functions such as remote control, data exchange, and human-computer interaction. Terminal device 15 includes smartphones, tablets, smart speakers, wearable devices, smart home appliances (such as smart TVs), and smart in-vehicle devices, etc. The interaction methods between terminal device 15 and users include, but are not limited to: operating on the screen with a finger or stylus, performing various operations through buttons, voice control, gesture control, iris recognition, and facial recognition, etc.

[0084] In one or more embodiments of this application, the housing 10 is communicatively connected to the sensor assembly 30. At least a portion of the sensors in the sensor assembly 30 are disposed within the housing 10.

[0085] As shown in Figures 4 and 5, in one or more embodiments of this application, the sensor assembly 30 includes at least one temperature sensor; the temperature sensor may include at least one of a compartment temperature sensor 31, an evaporator temperature sensor 32, and an ambient temperature sensor 33.

[0086] In one or more embodiments of this application, at least one controller is connected to a temperature sensor. The controller is configured to: determine initial control parameters corresponding to the current ambient temperature based on a parameter initialization model; wherein the parameter initialization model is obtained through reinforcement learning based on refrigerator operating data under a preset sample ambient temperature; for each iteration, control the refrigerator to operate in a single cycle according to the refrigerator control parameters corresponding to the current iteration, and obtain the performance evaluation data corresponding to the refrigerator's operation in this single cycle; wherein the refrigerator control parameters in the first iteration are the initial control parameters; if the performance evaluation data corresponding to the current single cycle meets a preset update condition, update the refrigerator control parameters corresponding to the current iteration based on the performance evaluation data corresponding to the current single cycle to obtain the refrigerator control parameters corresponding to the next iteration; continue executing the next iteration until the performance evaluation data in the latest iteration no longer meets the preset update condition.

[0087] For example, the compartment temperature sensor 31 includes a refrigerator compartment temperature sensor 311 and a freezer compartment temperature sensor 312. The refrigerator compartment temperature sensor 311 is installed in the refrigerator compartment of the cabinet 10 to detect the temperature of the refrigerator compartment; the freezer compartment temperature sensor 312 is installed in the freezer compartment of the cabinet 10 to detect the temperature of the freezer compartment.

[0088] In one or more embodiments of this application, the compartment temperature sensor 31 further includes a fruit and vegetable compartment temperature sensor 313.

[0089] In one or more embodiments of this application, the compartment temperature sensor 31 further includes a variable temperature compartment temperature sensor 314.

[0090] For example, an evaporator temperature sensor 32 is disposed on the evaporator for detecting the temperature of the evaporator.

[0091] In one or more embodiments of this application, the evaporator temperature sensor includes a refrigerator compartment cooler temperature sensor 321 and a freezer compartment cooler temperature sensor 322.

[0092] In one or more embodiments of this application, the ambient temperature sensor 33 includes an indoor temperature sensor 331.

[0093] In one or more embodiments of this application, the ambient temperature sensor 33 includes an indoor temperature sensor 331 and an outdoor temperature sensor (not shown). The outdoor temperature can also be obtained by querying a server.

[0094] In one or more embodiments of this application, the sensor assembly 30 further includes a humidity sensor.

[0095] In one or more embodiments of this application, the sensor assembly 30 further includes a door switch sensor to detect the opening and closing of the door 11.

[0096] In one or more embodiments of this application, the sensor assembly 30 may also include other sensors, such as vibration sensors, weight sensors, etc.

[0097] In one or more embodiments of this application, the sensor assembly 30 further includes an electrical parameter sensor 34. The number of electrical parameter sensors 34 is not limited, and the electrical parameter sensors 34 can be used to detect one or more of the following: electrical charge, peak electrical charge, valley electrical charge, current, voltage, and energy efficiency.

[0098] In one or more embodiments of this application, the electrically driven actuator 16 in the housing 10 includes a compressor 161, a fan 162, and a defrost heater 163.

[0099] In one or more embodiments of this application, the electrically driven actuators 16 in the housing 10 include a compressor 161, a fan 162, a defrost heater 163, a display 164, and may also include, for example, a water pump in an ice-making module and a motor in an ice-crushing module.

[0100] Currently, traditional refrigerators rely heavily on technologies such as weight sensors, infrared beam detectors, or radar point cloud data to monitor the amount of food inside. However, these methods face significant limitations in practical application and widespread adoption. Weight sensors, typically placed below shelves, can only detect overall weight changes and cannot distinguish between individual food items and containers, let alone determine spatial distribution. They are also susceptible to errors due to vibration or movement. Infrared technology, for example, can estimate space occupancy by arranging numerous infrared transmitters and receivers in an array along the refrigerator's inner wall and assessing beam obstruction. However, this approach is costly, complex to install, and inaccurate for detecting transparent packaging or small items. While radar-based methods for acquiring 3D point cloud data can provide a more comprehensive depiction of the internal contours, their core drawback lies in the complexity of data processing algorithms and the high computational resource requirements, leading to high system costs. Furthermore, their effectiveness in identifying tightly packed or low-reflectivity food items is inconsistent. Clearly, relying on traditional sensors to determine the amount of food inside a refrigerator has numerous shortcomings.

[0101] To overcome the above problems, in some optional embodiments, referring to Figure 6, a method for determining the amount of food occupied in a refrigerator is provided, which is applied to the controller in the refrigerator and may include the following steps:

[0102] S601, Obtain the compartment images of each storage compartment.

[0103] Storage compartments refer to independent spaces or areas within the refrigerator's interior designed for storing food. Each storage compartment may contain one or more shelves or drawers for categorizing and storing different types of food. Compartment images refer to image data of a specific storage compartment within the refrigerator, captured by an image acquisition device (such as a camera), used for subsequent analysis of food occupancy within that compartment.

[0104] It should be noted that in this embodiment, each image acquisition device corresponding to the same storage compartment is positioned within the projection area of ​​that storage compartment on the corresponding refrigerator door. The projection area refers to the visual coverage or region of the image acquisition device corresponding to the same storage compartment on the refrigerator door, ensuring that the camera can accurately capture images within that compartment without interference from other compartments. The advantage of this arrangement is that by placing the image acquisition device for the same storage compartment within the projection area of ​​that compartment on the door, the field of view of each camera precisely corresponds to the space of the compartment it is responsible for, effectively avoiding image distortion, obstruction, or information redundancy caused by shooting angle deviations or cross-area shooting. Simultaneously, this layout allows the camera to acquire images at a near-vertical overhead angle, minimizing perspective distortion and improving the geometric realism of the image, providing a structurally sound and clearly defined visual input foundation for subsequent accurate calculation of food space occupancy.

[0105] In one optional embodiment, the images of each storage compartment are acquired in two ways: either directly from the image acquisition device, or by acquiring the images of each storage compartment when the image acquisition device is activated; wherein, the activation conditions of the image acquisition device include at least one of responding to a door closing operation and the image acquisition device reaching a preset activation time. The advantage of this configuration is that by setting the activation conditions of the image acquisition device to responding to a door closing operation or reaching a preset timed acquisition time, the latest storage status can be obtained promptly after the user completes food storage and retrieval, ensuring the real-time nature and accuracy of the data. Furthermore, it allows for continuous monitoring of changes in the refrigerator's internal environment through periodic acquisition when no one is operating the refrigerator. This achieves dynamic, efficient, and energy-saving continuous tracking and updating of the amount of food inside the refrigerator without affecting normal user operation or causing redundant power consumption.

[0106] For example, the way to directly obtain the compartment images of each storage compartment from the image acquisition device can be as follows: when the conditions for the image acquisition device to be turned on are met, the image acquisition device is started; after the image acquisition device acquires the compartment image of the storage compartment, it sends the compartment image to the controller; the controller directly obtains the compartment image of the storage compartment.

[0107] For example, in this embodiment, the image acquisition device can be turned on periodically to obtain images of each storage compartment; alternatively, it can be turned on after the refrigerator door has been closed for a preset period of time to obtain images of each storage compartment. This embodiment does not limit this. It should be noted that since the opening and closing of the refrigerator door usually indicates that the user may change or retrieve the food stored in the refrigerator, turning on the image acquisition device only after the refrigerator door has been closed for a preset period of time to obtain images of each storage compartment can ensure that the image acquisition device significantly reduces the collection and processing of invalid image data, further avoiding the occupation and waste of invalid resources.

[0108] S602, for each compartment image, scale the compartment image to obtain reference compartment images at different scales.

[0109] The reference compartment image is obtained by scaling the original compartment image, providing different scale versions of the compartment image for multi-scale target detection, improving the accuracy and robustness of detecting ingredients of different sizes. The target detection bounding box is a rectangular box used to identify the location of a specific target (such as food) in the image. Its coordinates and size indicate the specific location and range of the target in the image, facilitating subsequent calculations and analysis.

[0110] In some embodiments, the method of scaling the cubicle image to obtain reference cubicle images at different scales can be as follows: The cubicle image can be scaled up based on at least one preset magnification scale to obtain a reference cubicle image at the corresponding magnification scale; alternatively, the cubicle image can be scaled down based on at least one preset reduction scale to obtain a reference cubicle image at the corresponding reduction scale. It should be noted that in this embodiment, scaling the cubicle image can be performed by only magnification, only reduction, or both simultaneously; this embodiment does not limit the scope of the scaling. The reduction scale can be understood as a scale smaller than the current scale of the cubicle image; the magnification scale can be understood as a scale larger than the current scale of the cubicle image.

[0111] It should be noted that reference compartment images at different scales contain different levels of information. Small-scale images focus on detailed features, meaning they contain more small-sized ingredients or local details of ingredients; large-scale images, on the other hand, contain broader contextual information, meaning they contain more large-volume ingredients or spatial relationships between ingredients. Scale-scaling the compartment images helps capture the diverse features of ingredients at different scales.

[0112] S603, based on reference compartment images at different scales, determines the target detection bounding boxes for each food item in the compartment image.

[0113] The target detection box can be understood as a detection box used to visually annotate ingredients. This detection box is used to represent the coordinates and category confidence of the ingredients in the compartment image.

[0114] In some embodiments, the method for determining the target detection boxes of each food item in the compartment image based on reference compartment images at different scales can be as follows: It can be based on a target detection model, where the reference compartment images at different scales are input into the target detection model to obtain the target detection boxes of each food item in the compartment image. Alternatively, it can be based on image fusion processing of the reference compartment images at different scales to obtain a fused compartment image; then, based on a target detection model, the fused compartment image is input into the target detection model to obtain the target detection boxes of each food item in the compartment image.

[0115] S604, for each storage compartment, determine the amount of food occupied in the storage compartment based on the area ratio of the target detection box in the corresponding compartment image.

[0116] Among them, the food occupancy refers to the proportion of space or area occupied by food in a specific storage compartment inside the refrigerator. It is estimated by calculating the area proportion of the food target detection box in the compartment image, and is used to reflect the actual storage amount of food in that compartment.

[0117] In one alternative embodiment, for each target detection box, the proportion of food ingredients in the corresponding compartment image of the storage compartment is determined; the sum of the proportions of food ingredients in all target detection boxes is taken as the food ingredient occupancy of the storage compartment.

[0118] In one optional embodiment, the overlap between each target detection box is determined; for a first target detection box that does not overlap, a first proportion of the target detection box in the image of the corresponding storage compartment is determined; for a second target detection box that overlaps, detection box fusion processing is performed on the second target detection box that overlaps to obtain a fused detection box; for each fused detection box, a second proportion of the fused detection box in the image of the corresponding storage compartment is determined; and the sum of each first proportion and each second proportion is used as the food occupancy of the storage compartment.

[0119] In the above embodiments, by acquiring independent images of each storage compartment, the targeted and complete nature of image acquisition is ensured, avoiding overlapping or occlusion of views between different compartments. Multi-scale scaling processing is performed on each image to generate reference images at different scales. This allows the model to simultaneously capture the overall outline of large-volume ingredients and the local details of small-volume ingredients, significantly improving the detection coverage of ingredients of different sizes. Target detection based on multi-scale reference images effectively integrates feature information at different scales, enhancing the robustness of the model in complex scenes, thereby more accurately locating and identifying various food targets in the image. The area ratio of the target detection box in the image is used to quantify the food occupancy of the storage compartment, effectively eliminating interference from non-food backgrounds and objectively reflecting the actual space ratio occupied by the food.

[0120] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the step of determining the target detection box of each food item in the compartment image according to the reference compartment images at different scales in the above embodiments in S603 is refined.

[0121] Referring to the target detection box determination steps shown in Figure 7A, the steps include:

[0122] S701, for each scale of the reference compartment image, extract the food features in the reference compartment image to obtain the reference feature map of the reference compartment image at the corresponding scale.

[0123] Among them, food features refer to the unique attributes or information extracted from food images that can represent or distinguish different foods, such as shape, texture, and color, which are used for subsequent target detection and recognition. The reference feature map reflects the feature information of food and background at different scales and is used for subsequent feature fusion and target detection.

[0124] In some embodiments, for each scale of the reference compartment image, the food features in the reference compartment image are extracted based on a feature extraction network to obtain a reference feature map of the reference compartment image at the corresponding scale.

[0125] For example, given a compartment image with dimensions (H, W), and defining a set of scale factors {S1, S2, ..., S...} i(e.g., S1=0.5, S2=1, S3=1.5). The reference compartment image size input to the feature extraction network is represented as [batch, C, H, W], where batch represents the batch size (the number of images sequentially input into the network); C represents the number of channels; H represents the length of the reference compartment image; and W represents the width of the reference compartment image. Taking a compartment image size of 600×200, batch=3, and number of channels=3 as an example, the reference compartment image size under the original size input to the feature extraction network is represented as [4, 3, 600, 200]. For each scale factor, the corresponding reference compartment image S is generated through scaling and dilation operations. i Its dimensions are (H) Si W Si In practice, bilinear interpolation or other suitable image scaling and dilation algorithms can be used to ensure image quality. Taking the aforementioned scale factors S1=0.5, S2=1, S3=1.5 as an example, the original-size compartment image is scaled to obtain the reference compartment image I. 0.5 Reference compartment image I1 and reference compartment image I 1.5 , among which, I 0.5 I1 can be characterized as [3,3,300,100], and I1 can be characterized as [3,3,600,200]. 1.5 It can be represented as [3,3,900,700]; in this embodiment, the reference compartment images of the above three scales can be input into the feature extraction network.

[0126] For example, the generated reference compartment images at different scales are input into the feature extraction network for feature extraction. Since images at different scales contain different levels of information—small-scale images focus on detailed features, while large-scale images contain broader contextual information—by extracting features separately, the model can capture the diverse features of the food at different scales. For instance, small-scale images may be more helpful in detecting small-sized food items or local details of food items, while large-scale images are more helpful in identifying large-volume food items or spatial relationships between food items. Here, we assume the total downsampling factor of the feature extraction network is set to 32 (i.e., after 5 downsampling steps with stride=2), then the reference compartment image I... 0.5 Reference compartment image I1 and reference compartment image I 1.5 The feature map sizes obtained after feature extraction are as follows: Reference compartment image I 0.5 The corresponding feature map is: [3, 256, 9, 3]; the feature map corresponding to the reference compartment image I1 is: [3, 256, 18, 6]; the reference compartment image I1... 1.5The corresponding feature maps are: [3, 256, 2, 9]; where 4 represents the batch size; 256 represents the number of channels C, which is usually determined by the network structure of the feature extraction network. The number of channels in the feature map output by the backbone network of YOLOv8 in the last layer is usually 256, 512, 1024, etc. Here, 256 is used as an example for explanation; 9 and 3 both represent the spatial size of the extracted feature map.

[0127] S702, fuses reference feature maps at different scales to obtain the target feature map of the compartment image.

[0128] The target feature map is a comprehensive feature representation at a unified scale, obtained by scaling and fusing reference feature maps at different scales. This target feature map incorporates details and contextual information from different scales, enhancing the model's ability to recognize targets (such as food ingredients), and is used for the final generation of target detection boxes and the calculation of food ingredient occupancy.

[0129] In some embodiments, a feature fusion network is used to fuse reference feature maps at different scales to obtain the target feature map of the compartment image.

[0130] S703, perform target detection on the target feature map to obtain the target detection boxes of each food item in the compartment image.

[0131] In some embodiments, target detection is performed on the target feature map based on the target detection network to obtain the target detection box of each food target in the compartment image.

[0132] For example, as shown in the target detection box diagram in Figure 7B, in this embodiment, the original size of the compartment image can be input into the feature extraction network; the feature extraction network performs scale scaling and feature extraction processing on the compartment image to obtain the target feature map; the target feature map is input into the target detection network to perform target detection on the target feature map to obtain the target detection box of each food target in the compartment image; and the food targets are marked in the compartment image in the form of detection boxes.

[0133] In the above embodiments, by extracting food features at each scale and constructing reference feature maps, the system can capture local details and global contextual information of food at different sizes. Furthermore, by fusing the multi-scale reference feature maps, the system effectively integrates cross-level visual expressions from fine textures to macroscopic structures, significantly enhancing the model's adaptability to scale changes and complex scenes. Finally, detection is performed based on the fused target feature map, which not only improves the accuracy of recognizing food of different sizes and occlusions, but also significantly reduces the common problems of missed detections and false detections in single-scale detection. This enables the system to stably and accurately locate each food target even under diverse storage conditions.

[0134] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the step of fusing reference feature maps at different scales to obtain the target feature map of the compartment image in S702 of the above embodiments is refined.

[0135] Referring to Figure 8, the steps for obtaining the target feature map include:

[0136] S801 performs scale restoration processing on reference feature maps at different scales to obtain reference feature maps at the same scale.

[0137] In some embodiments, this embodiment can perform scale restoration processing on reference feature maps at different scales, restoring the reference feature maps at different scales to their original scale or unifying them into reference feature maps at a uniform scale. For example, for reference feature maps smaller than the original scale, upsampling processing is performed on the reference feature image to restore the reference feature image at the original scale. For reference feature maps larger than the original scale, downsampling processing is performed on the reference feature image to restore the reference feature image at the original scale. Optionally, the upsampling processing can be bilinear interpolation, and the downsampling processing can be pooling.

[0138] For example, the reference feature maps at all scales are restored to [3,256,18,6], i.e., the original scale; for the reference compartment image I 0.5 The corresponding feature map [3, 256, 9, 3] is upsampled to 18×6; the feature map [3, 256, 18, 6] corresponding to the reference compartment image I1 is not processed; the reference compartment image I 1.5 The corresponding feature map [3, 256, 2, 9] is downsampled to 18×6, and all three feature maps are unified to the original scale of 18×6.

[0139] S802, fuses reference feature maps at the same scale to obtain the target feature map of the compartment image.

[0140] In some embodiments, the reference feature maps at the same scale can be weighted and summed to achieve fusion processing of the reference feature maps at the same scale and obtain the target feature map of the compartment image. For example, taking a weight of 1:1:1 as an example, the reference feature maps at the same scale can be fused to obtain the target feature map of the compartment image; the size of the target feature map is [3, 256, 18, 6].

[0141] In the above embodiments, by scaling reference feature maps at different scales, they are unified into the same scale space, eliminating the difficulty of feature alignment caused by scale differences. On this basis, deep fusion of feature maps at the same scale effectively aggregates detailed features and semantic information extracted at multiple scales, enhancing the integrity and consistency of feature expression, thereby significantly improving the accuracy and robustness of the target detection model in recognizing food morphology, boundaries and contextual relationships.

[0142] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the step of determining the amount of food occupied in the storage compartment according to the area ratio of the target detection box in the corresponding compartment image in the above embodiments S604 is refined.

[0143] Referring to Figure 9A, the steps for obtaining the target feature map include:

[0144] S901 constructs the target mask image corresponding to the storage compartment image.

[0145] The target mask image is a binary image corresponding to the compartment image, where each pixel indicates whether the location belongs to the food target area. In the target mask image, pixels in the food target area are set to a first pixel value (e.g., 1), while pixels in the non-food background area are set to a second pixel value (e.g., 0). Each pixel in the target mask image has the first pixel value.

[0146] In some embodiments, an empty binary mask image can be initialized according to the scale of the compartment image, serving as the target mask image. The scale of the target mask image is the same as that of the compartment image; all pixel values ​​in the target mask image are 0.

[0147] S902 sets the pixel value of the area where the food target is located in the target mask image corresponding to the compartment image to the second pixel value based on the position coordinates of each target detection box in the compartment image.

[0148] In some embodiments, for each target detection box, the second position coordinates of the target detection box in the target mask image are determined based on the first position coordinates of the target detection box in the compartment image; and the second pixel value is set based on the second position coordinates and the pixel points of the area where the food target is located.

[0149] For example, for each target detection box, the coordinate range x in the target mask image is traversed. i1 ≤x≤x i2 and y i1 ≤y≤y i2Let M(x,y) be the second pixel value. Here, x and y represent the horizontal and vertical coordinates of the target mask image, respectively; x i1 and y i1 The x and y coordinates represent the top-left corner of the target detection bounding box, respectively; x i2 and y i2 These represent the x-coordinate and y-coordinate of the upper right corner of the target detection box, respectively.

[0150] S903 determines the amount of food occupied in the storage compartment based on the percentage of pixels with the second pixel value in the mask image corresponding to each target compartment.

[0151] In some implementations, the number of first pixels corresponding to the second pixel value is determined from the target mask image; the number of second pixels corresponding to all pixels in the storage compartment image is determined; and the ratio between the number of first pixels and the number of second pixels is used to determine the amount of food occupied in the storage compartment.

[0152] It should be noted that, as shown in Figure 9B, the process of determining the amount of food consumed is illustrated. Since each pixel in the target mask image represents an actual area of ​​1 unit (in the image coordinate system), it can be understood as the area S of the union of all the pixels (i.e., gray squares) mapped by the detection boxes in the target mask image. union =N, which is the area occupied by the food target in the target mask image.

[0153] For example, taking a cubicle image with a length of 500 pixels and a width of 300 pixels as an example, the pixel area S of the cubicle image is... shelf =500×300=150000 pixels². When the number of pixels detecting the second pixel value in the target mask image is 6500, the pixel area of ​​the food target in the target mask image is S. union =6500 pixels². Based on this, the amount of food ingredients used... .

[0154] In the above embodiments, by constructing an initial mask image and marking the food area with specific pixel values ​​according to the target detection box coordinates, the food and background areas can be clearly separated, and the spatial distribution of the food can be intuitively represented. Then, by statistically analyzing the proportion of marked pixels, the food occupancy is calculated. This method effectively eliminates overlapping areas between detection boxes, avoids repeated calculations, and makes the space occupancy assessment more accurate.

[0155] Based on the technical solutions of the above embodiments, some optional embodiments are also provided, in which the steps of the above embodiments are refined.

[0156] Referring to the compartment temperature control steps shown in Figure 10, the steps include:

[0157] S1001 controls the amount of cold air delivered by the fan to each storage compartment based on the amount of food occupied in each compartment, in order to adjust the temperature in each compartment.

[0158] In some embodiments, where each storage compartment corresponds to a fan, and the fan can supply air to the corresponding storage compartment through an air outlet, the control parameters of the fan can be determined according to the food occupancy range of the storage compartment; the fan in the corresponding storage compartment is controlled based on the control parameters to adjust the airflow of cold air supplied to each storage compartment, and further adjust the compartment temperature in each storage compartment.

[0159] For example, when the food occupancy rate of a storage compartment exceeds a first preset food occupancy threshold (e.g., 80%), the fan can be controlled to increase the airflow of cold air into each storage compartment to achieve strong cooling and preservation. When the food occupancy rate of a storage compartment is less than a second preset food occupancy threshold (e.g., 20%), the fan can be controlled to reduce the airflow of cold air into each storage compartment to achieve weak cooling and energy saving. When the food occupancy rate of a storage compartment is neither greater than the first preset food occupancy threshold (e.g., 80%) nor less than the second preset food occupancy threshold, there is no need to adjust the fan's airflow. It should be noted that in this embodiment, the first preset food occupancy threshold is greater than the second preset food occupancy threshold.

[0160] In the above embodiments, by dynamically adjusting the fan output volume according to the real-time food occupancy of each storage compartment, the precise on-demand allocation of cooling resources is achieved: the air supply is increased in compartments with high food occupancy to quickly cool down and ensure the food preservation effect; the air supply is reduced in compartments with low occupancy or no occupancy to avoid wasting cooling capacity. Thus, while maintaining a suitable storage temperature in each area, the overall energy efficiency of the refrigerator is significantly improved, achieving an optimal balance between preservation performance and energy-saving operation.

[0161] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the fan is configured with multiple air outlets; each air outlet corresponds to a storage compartment. Based on this, the steps in the above embodiments are further refined.

[0162] Referring to the compartment temperature control steps shown in Figure 11, the steps include:

[0163] S1101 controls the airflow of the fan to deliver cold air to at least two storage compartments based on the average food occupancy value between the food occupancy of at least two storage compartments corresponding to the fan, so as to adjust the compartment temperature in at least two storage compartments.

[0164] In some embodiments, each fan may have multiple air outlets, with different outlets supplying air to different storage compartments. Therefore, it is not possible to determine the airflow to a storage compartment solely based on the amount of food occupied. Thus, in this embodiment, the target compartment corresponding to the fan is determined; the average food occupancy value between the target compartments is determined; the control parameters of the fan are determined based on the food occupancy range to which the average food occupancy value belongs; and the corresponding fan is controlled based on the control parameters to adjust the airflow of cold air supplied to each storage compartment, further adjusting the compartment temperature within each storage compartment.

[0165] For example, when the average occupancy rate of food ingredients is greater than a first preset food ingredient occupancy threshold (e.g., 80%), the fan can be controlled to increase the airflow of cold air to each storage compartment to achieve strong cooling and preservation. When the average occupancy rate of food ingredients is less than a second preset food ingredient occupancy threshold (e.g., 20%), the fan can be controlled to reduce the airflow of cold air to each storage compartment to achieve weak cooling and energy saving. When the average occupancy rate of food ingredients is neither greater than the first preset food ingredient occupancy threshold (e.g., 80%) nor less than the second preset food ingredient occupancy threshold, there is no need to adjust the fan's airflow. It should be noted that in this embodiment, the first preset food ingredient occupancy threshold is greater than the second preset food ingredient occupancy threshold.

[0166] In the above embodiments, frequent start-stop of fans or drastic changes in air volume caused by fluctuations in the occupancy of a single compartment are avoided, thereby improving operational stability and energy efficiency; sufficient cooling is ensured when the overall storage load of multiple compartments is high, and energy is saved together when the overall load is low, thereby achieving a balance between temperature equilibrium in multiple compartments and overall energy consumption optimization while simplifying control logic and reducing hardware costs.

[0167] Based on the technical solutions of the above embodiments, some optional embodiments are also provided, in which the steps of the above embodiments are refined.

[0168] The steps for outputting the food occupancy reminder, as shown in Figure 12, include:

[0169] For each storage compartment, S1201 determines the food occupancy reminder to be output to the user based on the occupancy range of the food occupancy in that storage compartment.

[0170] The food occupancy reminder is used to inform users of the current storage status of food (e.g., if the food quantity is too low and needs to be replenished, or if the food quantity is too high and it is recommended to use it first and then buy more), in order to improve user experience and convenience. This reminder can be displayed through mobile devices, refrigerator displays, or other terminal devices.

[0171] In some embodiments, for each storage compartment, based on the occupancy range to which the food occupancy of the storage compartment belongs, a food occupancy reminder that matches the occupancy range to be output to the user is determined.

[0172] For example, if the food occupancy level exceeds a first occupancy threshold (e.g., 90%), a food occupancy reminder can be sent to the user's device: "Your refrigerator compartment is almost full. We suggest you use up the food you have before purchasing more." If the food occupancy level is below a second occupancy threshold (e.g., 30%), it is determined that the food in that compartment may need to be replenished, and a reminder message is proactively pushed to the user via a mobile device or the refrigerator's own display screen. After a confirmation button is triggered, other normal operations can continue.

[0173] In the above embodiments, by mapping the amount of food used to a preset occupancy range, the refrigerator can intelligently identify whether the storage compartment is full, moderately full, or empty, and generate differentiated user reminders accordingly: prompting replenishment when the stock is too low to avoid food shortages; suggesting reasonable storage when the compartment is close to full to optimize storage space; thereby achieving a leap from passive recording to proactive management, significantly improving the refrigerator's interactive intelligence and ease of use, helping users to plan food purchases and storage more scientifically, and reducing food waste.

[0174] Based on the technical solutions of the above embodiments, some optional embodiments are also provided, in which the control process of the refrigerator is described in detail.

[0175] Refer to Figure 13 for the method of determining the amount of food needed in the refrigerator, which includes:

[0176] S1301 Obtains the compartment images of each storage compartment;

[0177] S1302 performs scale scaling on each compartment image to obtain reference compartment images at different scales;

[0178] S1303 extracts the food features from the reference compartment image for each scale to obtain the reference feature map of the reference compartment image at the corresponding scale.

[0179] S1304 performs scale restoration processing on reference feature maps at different scales to obtain reference feature maps at the same scale.

[0180] S1305 performs fusion processing on reference feature maps at the same scale to obtain the target feature map of the compartment image;

[0181] S1306 performs target detection on the target feature map to obtain the target detection box of each food item in the compartment image;

[0182] S1307 constructs a target mask image for each storage compartment corresponding to the compartment image;

[0183] S1308 sets the pixel value of the area where the food target is located in the target mask image corresponding to the compartment image to the second pixel value based on the position coordinates of each target detection box in the compartment image;

[0184] S1309 determines the amount of food occupied in the storage compartment based on the percentage of pixels with the second pixel value in each target mask image corresponding to the storage compartment;

[0185] S1310 controls the amount of cold air delivered by the fan to each storage compartment based on the amount of food occupied in each compartment, so as to adjust the temperature in each storage compartment.

[0186] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0187] Based on the same inventive concept, this application also provides a food quantity determination device for implementing the above-described method for determining the food quantity in a refrigerator. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the food quantity determination device provided below can be found in the limitations of the refrigerator food quantity determination method described above, and will not be repeated here.

[0188] In an exemplary embodiment, as shown in FIG14, a device for determining the amount of food ingredients is provided, including: an acquisition module 1401, a scaling module 1402, a detection module 1403, and a determination module 1404.

[0189] The acquisition module 1401 is used to acquire the compartment images of each storage compartment;

[0190] The scaling module 1402 is used to scale the compartment image for each compartment image to obtain reference compartment images at different scales.

[0191] The detection module 1403 is used to determine the target detection box of each food item in the compartment image based on the reference compartment images at different scales;

[0192] The determination module 1404 is used to determine the amount of food occupied in each storage compartment based on the area ratio of the target detection box in the corresponding compartment image.

[0193] In some embodiments, the detection module 1403 is further configured to extract food features from the reference compartment image for each scale of the reference compartment image to obtain a reference feature map of the reference compartment image at the corresponding scale; perform fusion processing on the reference feature maps at different scales to obtain a target feature map of the compartment image; and perform target detection on the target feature map to obtain a target detection box for each food target in the compartment image.

[0194] In some embodiments, the detection module 1403 is further configured to perform scale restoration processing on reference feature maps at different scales to obtain reference feature maps at the same scale; and to perform fusion processing on reference feature maps at the same scale to obtain target feature maps of the compartment image.

[0195] In some embodiments, the determining module 1404 is further configured to construct a target mask image corresponding to the storage compartment image; each pixel in the target mask image is a first pixel value; according to the position coordinates of each target detection box in the compartment image, the pixel in the area where the food target is located in the target mask image corresponding to the compartment image is set as a second pixel value; and the amount of food occupied in the storage compartment is determined according to the proportion of the number of pixels with the second pixel value in each target mask image corresponding to the storage compartment.

[0196] In some embodiments, the food occupancy determination device further includes: an adjustment module, used to control the air volume of the fan delivering cold air to each storage compartment according to the food occupancy of different storage compartments, so as to adjust the compartment temperature in each storage compartment.

[0197] In some embodiments, the adjustment module is further configured to, for each fan, control the airflow of cold air delivered by the fan to at least two storage compartments based on the average food occupancy value between the food occupancy amounts of the at least two storage compartments corresponding to the fan, so as to adjust the compartment temperature within the at least two storage compartments.

[0198] In some embodiments, the acquisition module 1401 is further configured to acquire compartment images of each storage compartment when the image acquisition device is turned on; wherein the image acquisition device turning on condition includes at least one of responding to a door closing operation and the image acquisition device operating time reaching a preset opening time.

[0199] In some embodiments, the food occupancy determination device further includes: a reminder module, used to determine, for each storage compartment, a food occupancy reminder to be output to the user terminal based on the occupancy range to which the food occupancy of the storage compartment belongs.

[0200] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 15. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for determining the amount of food occupied in a vacuum drawer. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0201] Those skilled in the art will understand that the structure shown in FIG15 is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the computer device to which the embodiments of this application are applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0202] In an alternative embodiment, the computer device shown in FIG15 may be the aforementioned refrigerator. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0203] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0204] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0205] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0206] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0207] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0208] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A refrigerator, characterized in that, include: The enclosure is constructed with at least one compartment; the compartment includes at least one storage compartment. At least one image acquisition device is configured to acquire compartment images of a corresponding storage compartment; at least one controller is connected to each of the image acquisition devices and is configured to acquire compartment images of each storage compartment. For each compartment image, the compartment image is scaled to obtain reference compartment images at different scales; Based on reference compartment images at different scales, determine the target detection bounding boxes for each food item in the compartment images; For each storage compartment, the amount of food occupied in the storage compartment is determined based on the area ratio of the target detection box in the corresponding compartment image.

2. The refrigerator according to claim 1, characterized in that, When the controller determines the target detection boxes of each food item in the compartment image based on reference compartment images at different scales, it is configured to: extract food item features from the reference compartment image at each scale to obtain a reference feature map of the reference compartment image at the corresponding scale; perform fusion processing on the reference feature maps at different scales to obtain a target feature map of the compartment image; and perform target detection on the target feature map to obtain the target detection boxes of each food item in the compartment image.

3. The refrigerator according to claim 1, characterized in that, When the controller performs fusion processing on reference feature maps at different scales to obtain the target feature map of the compartment image, it is configured to: perform scale restoration processing on reference feature maps at different scales to obtain reference feature maps at the same scale. The reference feature maps at the same scale are fused to obtain the target feature map of the compartment image.

4. The refrigerator according to claim 1, characterized in that, When the controller determines the amount of food occupied in the storage compartment based on the area ratio of the target detection box in the corresponding compartment image, it is further configured to: construct a target mask image of the storage compartment image; each pixel in the target mask image is a first pixel value; and set the pixel of the area where the food target is located in the target mask image of the storage compartment image to a second pixel value according to the position coordinates of each target detection box in the compartment image. The amount of food occupied in each storage compartment is determined based on the percentage of pixels with the second pixel value in each target mask image corresponding to the storage compartment.

5. The refrigerator according to claim 1, characterized in that, Each image acquisition device corresponding to the same storage compartment is set in the projection area of ​​the corresponding storage compartment on the corresponding door of the refrigerator.

6. The refrigerator according to any one of claims 1-5, characterized in that, The refrigerator further includes: a refrigeration system disposed within the cabinet and configured to generate cold air through the circulation of refrigerant; a fan disposed within the cabinet and configured to deliver cold air to the compartment through an air outlet disposed on the inner wall of the compartment; the controller is further configured to: control the airflow of the fan to each storage compartment according to the amount of food occupied in each storage compartment, so as to adjust the compartment temperature in each storage compartment.

7. The refrigerator according to claim 6, characterized in that, The fan is equipped with multiple air outlets; each air outlet corresponds to a storage compartment; correspondingly, when the controller performs the operation of controlling the airflow of the fan to each storage compartment based on the amount of food occupied in different storage compartments, so as to adjust the compartment temperature in each storage compartment, it is configured to: for each fan, control the airflow of the fan to the at least two storage compartments based on the average amount of food occupied between the at least two storage compartments corresponding to the fan, so as to adjust the compartment temperature in the at least two storage compartments.

8. The refrigerator according to any one of claims 1-5, characterized in that, When the controller acquires images of each storage compartment, it is configured to acquire images of each storage compartment when the image acquisition device is turned on. The image acquisition device turning on conditions include at least one of responding to a door closing operation and the image acquisition device operating time reaching a preset opening time.

9. The refrigerator according to any one of claims 1-5, characterized in that, The controller is also configured to: for each storage compartment, determine the food occupancy reminder to be output to the user terminal based on the occupancy range to which the food occupancy of the storage compartment belongs.

10. A method for determining the amount of food ingredients used, characterized in that, The method includes: acquiring compartment images of each storage compartment; scaling each compartment image to obtain reference compartment images at different scales; determining target detection boxes for each food item in the compartment images based on the reference compartment images at different scales; and determining the food item occupancy of each storage compartment based on the area ratio of the target detection boxes in the corresponding compartment images of the storage compartment.

Images