Refrigerator and food material recognition method thereof

Abstract

The application relates to a refrigerator and a food material recognition method thereof. The refrigerator comprises a cabinet, an image collector and a controller configured to: acquire at least one frame of target detection image collected by the image collector; determine whether a hand region and different food materials in each frame of target detection image overlap; in the case that the hand region and the different food materials do not overlap in all frames of target detection image, select a target food material of the current access action according to the variation degree of the relative distance between the hand region and the different food materials in each frame of target detection image; and in the case that the hand region and any food material overlap in at least one frame of target detection image, determine the overlap scores of the different food materials and the hand region in at least one overlap index in each frame of target detection image, and select at least one food material as the target food material of the current access action based on the weighted sum value between the overlap scores. The method can improve the food material recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of refrigerators, and in particular to a refrigerator and a method for identifying food items thereon. Background Technology

[0002] With the development of smart home appliance control technology, smart refrigerators with visual recognition capabilities are gradually becoming the mainstream in the market. These refrigerators, by installing cameras inside the compartments and combining them with image target detection algorithms, can automatically identify the types of food that users store or retrieve, thereby achieving intelligent food management and providing users with convenient services such as expiration reminders and recipe recommendations.

[0003] In traditional technologies, after a user completes a storage or retrieval action, the system selects a keyframe and calculates the center distance between the food detection frame and the hand detection frame to identify the food closest to the hand as the target food; alternatively, it directly determines the operated food based on whether the food frame and the hand frame overlap. Additionally, some solutions determine the storage or retrieval operation by tracking the direction of the hand's center of mass movement.

[0004] However, food items inside a refrigerator are often placed close together. In a single frame image, the candidate food item closest to the hand may not be the target food held by the user, but rather the background food item next to it, leading to misidentification and further affecting the accuracy of food identification. Summary of the Invention

[0005] This application provides a refrigerator and a method for identifying food ingredients thereon to improve the accuracy of food ingredient detection.

[0006] In a first aspect, some embodiments provide a refrigerator, including:

[0007] The enclosure includes at least one compartment;

[0008] The image acquisition device is configured to acquire motion detection images within the room;

[0009] The controller, connected to the image acquisition unit, is configured as follows:

[0010] Acquire at least one frame of target detection image captured by the image acquisition device;

[0011] Determine whether the hand region overlaps with different food ingredients in each frame of the target detection image;

[0012] If the hand region does not overlap with different food ingredients in all frames of target detection images, the target food ingredient for this storage action is selected based on the degree of change in the relative distance between the hand region and different food ingredients in each frame of target detection images.

[0013] If, in at least one frame of the target detection image, the hand region overlaps with any food ingredient, the overlap score between different food ingredients and the hand region in each frame of the target detection image is determined under at least one overlap index. Based on the weighted sum of the overlap scores, at least one food ingredient is selected as the target food ingredient for this access action.

[0014] The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of target detection images; the overlap area between the hand-held food item and other food items in each frame of target detection images; and the degree of change in relative distance. The overlap area index is used to characterize the cumulative degree of overlap in area between the hand region and any food item in all target detection images. The degree of change in relative distance index is used to characterize the total fluctuation of the relative position between the hand region and any food item in all target detection images as the frames change.

[0015] The refrigerator provided in the above embodiments, by acquiring at least one frame of target detection image, provides a sufficient data foundation for subsequent multi-frame temporal analysis, effectively overcoming random errors caused by insufficient information in a single frame image; by determining whether the hand region overlaps with different foods in each frame, it can initially screen candidate foods that have a direct interaction relationship with the hand, laying a reliable premise for subsequent layered determination; when the hand region does not overlap with any food in all frames, the target food is further selected based on the degree of change in the relative distance between the hand and the food in each frame, which can accurately identify the food that is continuously held by the hand by utilizing cross-frame displacement stability features, effectively avoiding misjudging food that accidentally approaches in the background as the target; and when there is at least one frame where the hand region overlaps with the food, the frequency of occurrence and overlap are considered together. The overlap index is weighted and scored using three overlap metrics: overlap area, relative distance change, and overlap ratio. The overlap area metric is not the cross-joint ratio of a single frame, but the cumulative overlap across all target detection images. This comprehensively reflects the sufficiency of contact between the food and the hand throughout the entire action. The relative distance change metric is not the spatial distance of a single frame, but the total fluctuation of the relative position across all target detection images. This accurately characterizes the displacement stability of the food relative to the hand. Thus, the interaction intensity between the food and the hand is quantified from multiple dimensions, including contact frequency, contact sufficiency, and motion stability. This approach takes into account the complementarity of features across different dimensions and achieves synergistic enhancement among multiple metrics through weighted fusion. As a result, even in complex placement scenarios, it can accurately distinguish between the handheld target food and adjacent or stacked interfering food.

[0016] Secondly, some embodiments also provide a refrigerator food identification method, including:

[0017] Acquire at least one frame of target detection image captured by the image acquisition device;

[0018] Determine whether the hand region overlaps with different food ingredients in each frame of the target detection image;

[0019] If the hand region does not overlap with different food ingredients in all frames of target detection images, the target food ingredient for this storage action is selected based on the degree of change in the relative distance between the hand region and different food ingredients in each frame of target detection images.

[0020] If, in at least one frame of the target detection image, the hand region overlaps with any food ingredient, the overlap score between different food ingredients and the hand region in each frame of the target detection image is determined under at least one overlap index. Based on the weighted sum of the overlap scores, at least one food ingredient is selected as the target food ingredient for this access action.

[0021] The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of target detection images; the overlap area between the hand-held food item and other food items in each frame of target detection images; and the degree of change in relative distance. The overlap area index is used to characterize the cumulative degree of overlap in area between the hand region and any food item in all target detection images. The degree of change in relative distance index is used to characterize the total fluctuation of the relative position between the hand region and any food item in all target detection images as the frames change.

[0022] The refrigerator food identification method provided in the above embodiments, by acquiring at least one frame of target detection image, provides a sufficient data foundation for subsequent multi-frame time-series analysis, effectively overcoming random errors caused by insufficient information in a single frame image; by determining whether the hand region overlaps with different foods in each frame, it can initially screen candidate foods that have a direct interaction relationship with the hand, laying a reliable premise for subsequent layered determination; when the hand region does not overlap with any food in all frames, the target food is further selected based on the degree of change in the relative distance between the hand and the food in each frame, which can accurately identify the food that is continuously held by the hand by utilizing cross-frame displacement stability features, effectively avoiding misjudging food that accidentally approaches in the background as the target; and when there is at least one frame where the hand region overlaps with the food, the occurrence of multiple overlaps is considered. The system uses a weighted scoring of three overlap metrics: number of overlapping objects, overlapping area, and relative distance change. The overlapping area metric is not the cross-joint ratio of a single frame, but rather the cumulative degree of overlap across all target detection images. This comprehensively reflects the sufficiency of contact between the food and the hand throughout the entire action. Similarly, the relative distance change metric is not the spatial distance of a single frame, but rather the total fluctuation of the relative position across all target detection images as the frames change. This accurately characterizes the displacement stability of the food relative to the hand. Thus, the system quantifies the interaction intensity between the food and the hand from multiple dimensions, including contact frequency, contact sufficiency, and motion stability. It takes into account the complementarity of features across different dimensions and achieves synergistic enhancement among multiple metrics through weighted fusion. This allows the system to accurately distinguish between the handheld target food and adjacent or stacked interfering food even in complex placement scenarios.

[0023] Thirdly, some embodiments also provide a refrigerator food identification device, including:

[0024] The acquisition module is used to acquire at least one frame of target detection image captured by the image acquisition device;

[0025] The determination module is used to determine whether the hand region overlaps with different food ingredients in each frame of the target detection image;

[0026] The selection module is used to select the target food for the current access action based on the degree of change in the relative distance between the hand region and different foods in each frame of the target detection image when the hand region does not overlap with different foods in all frames of the target detection image; when the hand region overlaps with any food in at least one frame of the target detection image, the module determines the overlap score between different foods and the hand region in each frame of the target detection image under at least one overlap index, and selects at least one food as the target food for the current access action based on the weighted sum of the overlap scores.

[0027] The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of target detection images; the overlap area between the hand-held food item and other food items in each frame of target detection images; and the degree of change in relative distance. The overlap area index is used to characterize the cumulative degree of overlap in area between the hand region and any food item in all target detection images. The degree of change in relative distance index is used to characterize the total fluctuation of the relative position between the hand region and any food item in all target detection images as the frames change.

[0028] The refrigerator food identification device provided in the above embodiments, by acquiring at least one frame of target detection image, provides a sufficient data foundation for subsequent multi-frame time-series analysis, effectively overcoming random errors caused by insufficient information in a single frame image; by determining whether the hand region overlaps with different foods in each frame, it can initially screen candidate foods that have a direct interaction relationship with the hand, laying a reliable premise for subsequent layered determination; when the hand region does not overlap with any food in all frames, the target food is further selected based on the degree of change in the relative distance between the hand and the food in each frame, which can accurately identify the food that is continuously held by the hand by utilizing cross-frame displacement stability features, effectively avoiding misjudging food that accidentally approaches in the background as the target; and when there is at least one frame where the hand region overlaps with the food, the occurrence of multiple overlaps is considered. The system uses a weighted scoring of three overlap metrics: number of overlapping objects, overlapping area, and relative distance change. The overlapping area metric is not the cross-joint ratio of a single frame, but rather the cumulative degree of overlap across all target detection images. This comprehensively reflects the sufficiency of contact between the food and the hand throughout the entire action. Similarly, the relative distance change metric is not the spatial distance of a single frame, but rather the total fluctuation of the relative position across all target detection images as the frames change. This accurately characterizes the displacement stability of the food relative to the hand. Thus, the system quantifies the interaction intensity between the food and the hand from multiple dimensions, including contact frequency, contact sufficiency, and motion stability. It takes into account the complementarity of features across different dimensions and achieves synergistic enhancement among multiple metrics through weighted fusion. This allows the system to accurately distinguish between the handheld target food and adjacent or stacked interfering food even in complex placement scenarios.

[0029] Fourthly, some embodiments also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0030] Acquire at least one frame of target detection image captured by the image acquisition device;

[0031] Determine whether the hand region overlaps with different food ingredients in each frame of the target detection image;

[0032] If the hand region does not overlap with different food ingredients in all frames of target detection images, the target food ingredient for this storage action is selected based on the degree of change in the relative distance between the hand region and different food ingredients in each frame of target detection images.

[0033] If, in at least one frame of the target detection image, the hand region overlaps with any food ingredient, the overlap score between different food ingredients and the hand region in each frame of the target detection image is determined under at least one overlap index. Based on the weighted sum of the overlap scores, at least one food ingredient is selected as the target food ingredient for this access action.

[0034] The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of target detection images; the overlap area between the hand-held food item and other food items in each frame of target detection images; and the degree of change in relative distance. The overlap area index is used to characterize the cumulative degree of overlap in area between the hand region and any food item in all target detection images. The degree of change in relative distance index is used to characterize the total fluctuation of the relative position between the hand region and any food item in all target detection images as the frames change.

[0035] The readable storage medium provided in the above embodiments, on which the computer program is stored, when executed by a processor, acquires at least one frame of target detection image, providing a sufficient data foundation for subsequent multi-frame temporal analysis, effectively overcoming random errors caused by insufficient information in a single frame image; by determining whether the hand region overlaps with different foods in each frame, candidate foods that have a direct interaction relationship with the hand can be initially screened, laying a reliable foundation for subsequent layered determination; when the hand region does not overlap with any food in all frames, the target food is further selected based on the degree of change in the relative distance between the hand and the food in each frame, which can accurately identify the food that is continuously held by the hand by utilizing cross-frame displacement stability features, effectively avoiding misjudging food that accidentally approaches in the background as the target; and when at least one frame has a hand region overlapping with food... When the overlap ratio is not a single frame intersection-union ratio, but the cumulative degree of overlap across all target detection images, it can comprehensively reflect the sufficiency of contact between the food and the hand throughout the entire action. The degree of change in relative distance is not a single frame spatial distance, but the total fluctuation of the relative position across all target detection images, which can accurately characterize the displacement stability of the food relative to the hand. Thus, the interaction intensity between the food and the hand is quantified from multiple dimensions such as contact frequency, contact sufficiency, and motion stability. It takes into account the complementarity of features in each dimension and achieves synergistic enhancement among multiple indicators through weighted fusion, so that it can accurately distinguish the handheld target food from adjacent or stacked interfering food even in complex placement scenarios.

[0036] Fifthly, some embodiments also provide a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0037] Acquire at least one frame of target detection image captured by the image acquisition device;

[0038] Determine whether the hand region overlaps with different food ingredients in each frame of the target detection image;

[0039] If the hand region does not overlap with different food ingredients in all frames of target detection images, the target food ingredient for this storage action is selected based on the degree of change in the relative distance between the hand region and different food ingredients in each frame of target detection images.

[0040] If, in at least one frame of the target detection image, the hand region overlaps with any food ingredient, the overlap score between different food ingredients and the hand region in each frame of the target detection image is determined under at least one overlap index. Based on the weighted sum of the overlap scores, at least one food ingredient is selected as the target food ingredient for this access action.

[0041] The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of target detection images; the overlap area between the hand-held food item and other food items in each frame of target detection images; and the degree of change in relative distance. The overlap area index is used to characterize the cumulative degree of overlap in area between the hand region and any food item in all target detection images. The degree of change in relative distance index is used to characterize the total fluctuation of the relative position between the hand region and any food item in all target detection images as the frames change.

[0042] The computer program provided in the above embodiments, when executed by the processor, acquires at least one frame of target detection image, providing a sufficient data foundation for subsequent multi-frame temporal analysis and effectively overcoming random errors caused by insufficient information in a single frame image. By determining whether the hand region overlaps with different foods in each frame, it can initially screen candidate foods that have a direct interaction relationship with the hand, laying a reliable foundation for subsequent layered determination. When the hand region does not overlap with any food in all frames, the target food is further selected based on the degree of change in the relative distance between the hand and the food in each frame. It can accurately identify the food that is continuously held by the hand using cross-frame displacement stability features, effectively avoiding misjudging food that accidentally approaches in the background as the target. When there is at least one frame where the hand region overlaps with the food... When overlapping occurs, a weighted score is calculated based on three overlap indicators: frequency of occurrence, overlap area, and degree of change in relative distance. The overlap area indicator is not the cross-joint ratio of a single frame, but the cumulative degree of overlap across all target detection images, which can comprehensively reflect the sufficiency of contact between the food and the hand throughout the entire action. The degree of change in relative distance is not the spatial distance of a single frame, but the total fluctuation of the relative position across all target detection images as it changes between frames, which can accurately characterize the displacement stability of the food relative to the hand. Thus, the interaction intensity between the food and the hand is quantified from multiple dimensions such as contact frequency, contact sufficiency, and motion stability. This approach takes into account the complementarity of features in each dimension and achieves synergistic enhancement among multiple indicators through weighted fusion, thereby accurately distinguishing the handheld target food from adjacent or stacked interfering food even in complex placement scenarios. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A schematic block diagram of a first refrigerator structure provided for some embodiments of this application;

[0045] Figure 2 A schematic block diagram of a second refrigerator structure provided for some embodiments of this application;

[0046] Figure 3 Schematic block diagram of the structure of the processing device in the refrigerator provided in some embodiments of this application;

[0047] Figure 4 A schematic block diagram of a third refrigerator structure provided for some embodiments of this application;

[0048] Figure 5 A schematic block diagram of a fourth refrigerator structure provided in some embodiments of this application;

[0049] Figure 6 A flowchart illustrating a first refrigerator food identification method provided in some embodiments of this application;

[0050] Figure 7 A flowchart illustrating a first step for determining overlap scoring, provided for some embodiments of this application;

[0051] Figure 8A A flowchart illustrating a step for determining the degree of change in relative distance, provided for some embodiments of this application;

[0052] Figure 8B A relative distance diagram provided for some embodiments of this application;

[0053] Figure 9 A flowchart illustrating a second step for determining overlap scoring, provided for some embodiments of this application;

[0054] Figure 10 A flowchart illustrating a third step for determining overlap scoring, provided for some embodiments of this application;

[0055] Figure 11 A flowchart illustrating a step for determining overlap in some embodiments of this application;

[0056] Figure 12 A flowchart illustrating a step for determining a target ingredient, provided for some embodiments of this application;

[0057] Figure 13 A flowchart illustrating a second refrigerator food identification method provided in some embodiments of this application;

[0058] Figure 14 A schematic flowchart of a refrigerator food identification device provided in some embodiments of this application;

[0059] Figure 15 This is an internal structural diagram of a computer device provided for some embodiments of this application. Detailed Implementation

[0060] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0061] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0062] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0063] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0064] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0065] The refrigerator 1 provided in this application will now be described with reference to the accompanying drawings. The overall structure of the refrigerator 1 is as follows: Figure 1 As shown. Refrigerator 1 includes a cabinet 10 and a processing device 20.

[0066] like Figure 2 As shown, the housing 10 has at least one storage compartment.

[0067] Storage rooms are typically divided into freezer rooms and refrigerator rooms (referred to as refrigerator rooms). They can also be further divided into chambers with special functions, such as chambers for storing fruits and vegetables. Refrigerator rooms can maintain a temperature range of approximately 4°C to store food, medicine, or biological agents in a refrigerated state. Freezer rooms can maintain a temperature range of approximately -18°C to store food, medicine, or biological agents in a frozen state.

[0068] The storage compartment has an opening that can be opened and closed via a door 11 hinged to the outer casing, or via a drawer 12. When a refrigerator compartment and a freezer compartment are provided, one opening can be opened and closed via a door (e.g., the refrigerator compartment), and the other opening can be opened and closed via a drawer 12 (e.g., the freezer compartment).

[0069] The housing 10 employs a vapor compression refrigeration cycle to generate energy for maintaining the target temperature. The refrigeration cycle consists of a compressor 161, a condenser, a throttling device, and an evaporator. The refrigeration cycle involves a series of processes, including compression, condensation, expansion, and evaporation, to cool the storage compartment and maintain an ideal low-temperature storage environment inside.

[0070] In a vapor compression refrigeration cycle, a low-temperature, low-pressure refrigerant enters the compressor 161, which compresses it into a high-temperature, high-pressure refrigerant gas and discharges the compressed refrigerant gas. The discharged refrigerant gas flows into the condenser, where the condenser condenses the compressed refrigerant into a liquid phase, and the heat is released to the surrounding environment through the condensation process.

[0071] The throttling device causes the high-temperature, high-pressure liquid refrigerant formed in the condenser to expand into a low-pressure liquid refrigerant. The evaporator evaporates the refrigerant that has expanded in the throttling device and returns the low-temperature, low-pressure refrigerant gas to the compressor 161. The evaporator can achieve a cooling effect by exchanging heat with the material to be cooled through the latent heat of refrigerant evaporation. In this application, the evaporator exchanges heat with air to form air for cooling the storage compartment, thereby cooling the storage compartment. The throttling device can be a capillary tube.

[0072] A filter is also installed downstream of the condenser. The filter is used to filter impurities in the refrigerant, improve the heat exchange efficiency of the refrigeration unit, and reduce the risk of pipe blockage.

[0073] A liquid receiver can also be installed on the suction side of the compressor 161. The liquid receiver is used to separate the refrigerant into gas and liquid phases. The liquid receiver is a shell-shaped component. The gas-liquid mixed refrigerant fluid enters the liquid receiver for basic phase separation. The gas enters the gas passage and undergoes gravity settling to separate droplets, while the liquid enters the liquid space and separates into bubbles. The gas flows out from the gas outlet and is then drawn into the compressor 161, preventing the compressor 161 from carrying liquid in the suction and reducing the service life of the compressor 161.

[0074] The compressor 161 and condenser can be located at the lower rear of the housing, while the evaporator can be located at the rear of the housing corresponding to the storage compartment. The evaporator and condenser can also be arranged in other locations according to the industrial design of the housing 10, which will not be listed here. The location where the evaporator is located has sufficient space to allow air to flow. The air is driven by the fan 162 to deliver the air generated by the evaporator for cooling the storage compartment to the target location and to draw in air from the storage compartment, forming an air circulation. In one or more embodiments of this application, the fan 162 includes a refrigeration fan and a freezing fan. In one or more embodiments of this application, the fan 162 can also be configured in conjunction with the condenser.

[0075] In one or more embodiments of this application, the evaporator may also be divided into two parts for the refrigerator compartment and the freezer compartment, referred to as the refrigerator compartment cooler and the freezer compartment cooler.

[0076] A defrosting element is provided in the housing 10. The defrosting element is configured to generate heat for defrosting the evaporator, thereby putting the evaporator in a defrosting state. In one or more embodiments of this application, the defrosting element includes a defrosting heater 163, which may be an electric heating tape or an electric heater. In one or more embodiments of this application, the defrosting element may also be a combination of an electric heating tape or an electric heater, and a heat exchanger or heat exchange piping. When defrosting conditions are met, the heat exchange piping is opened, and the high-temperature, high-pressure refrigerant discharged from the compressor 161 enters the heat exchange piping, exchanges heat with the surrounding air, raises the air temperature, and further provides heat to melt the frost layer on the evaporator surface, thereby putting the evaporator in a defrosting state. The heat exchange piping may be located below the evaporator, utilizing the principle that hot air has a lower density and rises to guide the air to remove the ice or frost layer on the evaporator. The defrosting element composed of an electric heating tape or an electric heater may also be located around the evaporator in other positions, such as above or to one side of the evaporator.

[0077] A display 164 is installed on the cabinet 10.

[0078] The cabinet 10 is equipped with a refrigeration system, which is configured to transfer heat from the inside of the refrigerator to the outside through the circulation of refrigerant.

[0079] like Figure 3 As shown in the figure, the hardware configuration of the processing device 20 is as follows. The processing device 20 includes components such as a processor 201, volatile memory 203, non-volatile memory 202, display device 204, operation device 205, communication interface 206, and drive device 207, which are interconnected via a bus 208. The processor 201 can be a dedicated processor 201, a central processing unit, etc. The processor 201 can access the storage unit to execute instructions or application programs stored in the storage unit to achieve related functions. The display device 204 is a display device 204 for displaying various information, the operation device 205 is an operation device for receiving various operations, and the drive device 207 is a hardware terminal that interacts with the storage medium. In one or more embodiments of this application, the storage medium includes media such as CD-ROM, floppy disk, and optical-magnetic-optical disk that record information in an optical, electrical, or magnetic manner. The storage medium can also be a semiconductor memory such as ROM or flash memory that records information in an electrical manner.

[0080] In one or more embodiments of this application, the processing device 20 may be a controller 13. The controller 13 is disposed in the housing 10.

[0081] In one or more embodiments of this application, the processing device 20 may be communicatively connected to the controller 13, for example, by a terminal device 15 and / or a cloud server 14.

[0082] In one or more embodiments of this application, some functions of the processing device 20 may be implemented by the controller 13, and some functions may be implemented by the terminal device 15 and / or the cloud server 14.

[0083] Controller 13 can communicate with terminal device 15 and / or server 14. The network between controller 13 and terminal device 15, or between controller 13 and server 14, can be the Internet, cellular network, Wi-Fi network, low power wide area network (LPWAN), WAN, LAN, etc., based on standards and protocols such as LoRa, Sigfox, and NB-IoT.

[0084] The cabinet 10 can be used in home environments to store daily necessities such as food and cold drinks; it can also be used in commercial places such as restaurants, hotels, supermarkets, and convenience stores to store ingredients, food and beverages to meet customer needs; and it can also be used in places such as hospitals and laboratories to store medicines and biological samples to meet medical and scientific research needs.

[0085] Server 14 can provide various network services, such as resource and data access for refrigerator 1 controller 13 and terminal device 15. Server 14 has higher performance and reliability. Server 14 can connect to multiple refrigerator 1 controllers 13, multiple terminal devices 15, and other smart home appliance terminals.

[0086] Terminal device 15 is an electronic device with intelligent functions. It can connect to the aforementioned networks to achieve functions such as remote control, data exchange, and human-computer interaction. Terminal device 15 includes smartphones, tablets, smart speakers, wearable devices, smart home appliances (such as smart TVs), and smart in-vehicle devices, etc. The interaction methods between terminal device 15 and users include, but are not limited to: operating on the screen with a finger or stylus, performing various operations through buttons, voice control, gesture control, iris recognition, and facial recognition, etc.

[0087] In one or more embodiments of this application, the housing 10 is communicatively connected to the sensor assembly 30. At least a portion of the sensors in the sensor assembly 30 are disposed within the housing 10.

[0088] like Figure 4 and Figure 5As shown, in one or more embodiments of this application, the sensor assembly 30 includes at least one temperature sensor; the temperature sensor may include at least one of a compartment temperature sensor 31, an evaporator temperature sensor 32, and an ambient temperature sensor 33.

[0089] For example, the compartment temperature sensor 31 includes a refrigerator compartment temperature sensor 311 and a freezer compartment temperature sensor 312. The refrigerator compartment temperature sensor 311 is installed in the refrigerator compartment of the cabinet 10 to detect the temperature of the refrigerator compartment; the freezer compartment temperature sensor 312 is installed in the freezer compartment of the cabinet 10 to detect the temperature of the freezer compartment.

[0090] In one or more embodiments of this application, the compartment temperature sensor 31 further includes a fruit and vegetable compartment temperature sensor 313.

[0091] In one or more embodiments of this application, the compartment temperature sensor 31 further includes a variable temperature compartment temperature sensor 314.

[0092] For example, an evaporator temperature sensor 32 is disposed on the evaporator for detecting the temperature of the evaporator.

[0093] In one or more embodiments of this application, the evaporator temperature sensor includes a refrigerator compartment cooler temperature sensor 321 and a freezer compartment cooler temperature sensor 322.

[0094] In one or more embodiments of this application, the ambient temperature sensor 33 includes an indoor temperature sensor 331.

[0095] In one or more embodiments of this application, the ambient temperature sensor 33 includes an indoor temperature sensor 331 and an outdoor temperature sensor (not shown). The outdoor temperature can also be obtained by querying a server.

[0096] In one or more embodiments of this application, the sensor assembly 30 further includes a humidity sensor.

[0097] In one or more embodiments of this application, the sensor assembly 30 further includes a door switch sensor to detect the opening and closing of the door 11.

[0098] In one or more embodiments of this application, the sensor assembly 30 may also include other sensors, such as vibration sensors, weight sensors, etc.

[0099] In one or more embodiments of this application, the sensor assembly 30 further includes an electrical parameter sensor 34. The number of electrical parameter sensors 34 is not limited, and the electrical parameter sensors 34 can be used to detect one or more of the following: electrical charge, peak electrical charge, valley electrical charge, current, voltage, and energy efficiency.

[0100] In one or more embodiments of this application, the electrically driven actuator 16 in the housing 10 includes a compressor 161, a fan 162, and a defrost heater 163.

[0101] In one or more embodiments of this application, the electrically driven actuators 16 in the housing 10 include a compressor 161, a fan 162, a defrost heater 163, a display 164, and may also include, for example, a water pump in an ice-making module and a motor in an ice-crushing module.

[0102] Currently, to accurately identify the categories of food items stored or retrieved by users and achieve automatic updates to the food list, traditional technologies mainly employ two approaches: The first is a single-frame distance-based method, where a keyframe is selected after the user completes the storage or retrieval action, and the food item closest to the hand is identified as the target food item by calculating the center distance between the food item detection box and the hand detection box; the second is a simpler method based on hand contact relationships, which directly determines the operated food item based on whether there is an overlap between the food item box and the hand box, or supplements this with the direction of the hand's centroid movement trajectory to determine whether the storage or retrieval operation is performed. However, both of these methods have inherent drawbacks. While the single-frame distance-based method is simple to implement, because food items inside a refrigerator are often placed closely side by side, the candidate food item closest to the hand in a single-frame image is often not the target food item held by the user, but rather the background food item placed adjacent to the target, making it difficult to effectively distinguish between the target and the background; at the same time, this method relies entirely on single-frame information and lacks modeling of the continuous interaction between the hand and the food items during the action. When the target food item and the background food items are extremely close in space, the single-frame distance information cannot provide sufficient discrimination criteria. On the other hand, while the simple method based on hand contact introduces overlap information, it only focuses on the binary feature of whether overlap exists, failing to quantify the degree of overlap and contact stability. Furthermore, when food items are stacked vertically, the bottom item typically has a larger area and a more stable position. Relying solely on the overlap area or contact relationship can easily lead to misidentification of the untouched bottom stacked item as the target item. Therefore, traditional technologies suffer from the inability to effectively distinguish between the handheld target item and adjacent or stacked background food items using multi-frame temporal interaction information, a problem that urgently needs improvement.

[0103] To overcome the above problems, in some alternative embodiments, see Figure 6 A method for identifying food items in a refrigerator is provided, which can be applied to the controller in the refrigerator and may include the following steps:

[0104] S601 acquires at least one frame of target detection image captured by the image acquisition device.

[0105] The target detection image refers to the specific frame image selected from the motion detection images acquired by the image acquisition device for food identification and analysis. It should be noted that the target detection image can be a motion detection image directly acquired by the image acquisition device, or a close-up image obtained by cropping the hand area in the motion detection image.

[0106] In some optional embodiments, at least one frame of motion detection image is acquired from the image acquisition device based on the current access action; for each motion detection image, the hand region in the motion detection image is cropped to obtain a hand detection image; both the hand detection image and the motion detection image are used as target detection images.

[0107] In some embodiments, when the current access action representation is stored, a preset number of action detection images are sequentially selected from the memory as target detection images; when the current access action representation is retrieved, a preset number of action detection images are selected from the memory in reverse order as target detection images.

[0108] The preset quantity refers to a pre-defined threshold for the number of motion detection images selected from the memory. This quantity is used to determine the total number of target detection images involved in the food identification calculation, ensuring that the algorithm has sufficient temporal information for multi-frame statistical analysis while also considering computational efficiency.

[0109] For example, when the current access action is characterized as being stored, the method of sequentially selecting a preset number of motion detection images from the memory as target detection images is as follows: when the current access action is characterized as being stored, the controller determines the frames of motion detection images stored in the memory in a time sequence, and reads the preset number of frames of motion detection images from the memory in the order from the earliest stored to the latest stored, as target detection images; wherein, the preset number of frames is a sequence of frames arranged continuously in time, the reading start position is the first frame of the motion detection image sequence corresponding to the current access action in the memory, and the reading proceeds forward in time until the number of frames read reaches the preset number.

[0110] It's important to note that during the storage process, as the user moves the food from the outside of the refrigerator towards the inside, in the initial phase (when the hand just enters the camera's field of view), the food is not yet obscured by other items inside the refrigerator, and the hand's grip on the food is most stable. At this point, the food is fully exposed, its outline is clear, and its relative position to the hand is well-defined. As the hand continues to move further in, the food gradually approaches its storage location, and existing background food inside the refrigerator begins to enter the camera's field of view, creating spatial proximity or obstruction with the hand-held food. At this point, the features of the hand-held food are easily interfered with by the background food. Therefore, selecting images sequentially from the start of the action prioritizes retaining the frames where the food features are clearest and interference is least prevalent, specifically when the hand first enters the field of view.

[0111] For example, when the current access action represents retrieval, the method of selecting a preset number of motion detection images from the memory in reverse order as target detection images is as follows: when the current access action represents retrieval, the controller determines the frames of motion detection images stored in the memory in time sequence, and reads a preset number of frames of motion detection images from the memory in reverse order from the latest stored to the earliest stored, as target detection images; wherein, the preset number of frames is a sequence of frames arranged continuously in time sequence, the reading start position is the last frame of the motion detection image sequence corresponding to the current access action in the memory, and the reading proceeds in reverse along the time sequence until the number of frames read reaches the preset number.

[0112] For example, when the current access action is represented as "storing," the user moves the food from inside the refrigerator to the outside. In the initial stage of the action (when the hand just picks up the food from its storage location), the food is initially inside the refrigerator, closely packed with surrounding background food. At this point, there is significant occlusion and proximity interference between the food and the background food, making it difficult to clearly distinguish. As the hand continues to move outward and gradually moves away from its original storage location, the food gradually moves away from the background food group. In the later stage of the action (when the hand is about to leave the camera's field of view), the food is completely free from the interference area of ​​the background food and exposed to an open field of view. Its features are most complete and clear, and its relative positional relationship with the hand is most stable. Therefore, selecting images in reverse chronological order, starting from the end of the action, can preferentially retain the frames where the food features are clearest after it has been freed from background interference.

[0113] For example, for a complete food storage and retrieval action, a keyframe sequence and a close-up image sequence are obtained from the original video. For instance, the keyframes include keyframe 0, keyframe 1, ..., keyframe 5 ([f0][f1][f2][f3][f4][f5]); the close-up images include close-up image 0, close-up image 1, and close-up image 2 ([crop0][crop1][crop2]). Therefore, the original frame sequence is: [f0][f1][f2][f3][f4][f5] + [crop0][crop1][crop2]. By employing different keyframe extraction strategies for "storage" and "retrieval," the recognition focus is on the time period when the relationship between the target food and the hand is most stable, thereby reducing interference from irrelevant background before and after the action. When the action is "storage," the earlier keyframes are selected because the target hand is more clearly visible in the first few frames before and after storage. The frames involved in the calculation are: [f0][f1][f2][f3] + [crop0][crop1][crop2], a total of 7 frames participating in the recognition calculation. When the action is "removal", the later keyframes are selected because the later frames better reflect the state of being continuously held by the hand after the food is removed from its original position. The frames involved in the calculation are: [f2][f3][f4][f5] + [crop0][crop1][crop2], a total of 7 frames involved in the recognition calculation.

[0114] In the above embodiments, the direction of selecting action detection images from the memory is adaptively determined according to the type of storage and retrieval action (storage or retrieval). That is, sequential selection is performed when storing and reverse selection is performed when retrieval. This ensures that the acquired target detection images can always focus on the period when the interaction between the hand and the food is most stable and the characteristics of the target food are clearest. In the storage action, the food is continuously held by the hand in the early stage of the action, and its state is clear. Sequential selection can prioritize the retention of high-quality images in this stage. In the retrieval action, the food is taken away from its original position by the hand in the later stage of the action and exposed to the camera's field of view. Reverse selection can prioritize the capture of the image with the most complete features in this separation stage. This effectively avoids interference factors such as occlusion, motion blur, and sudden changes in illumination in the beginning and end stages of the action, and significantly improves the overall quality of the input image sequence.

[0115] S602 determines whether the hand region overlaps with different food ingredients in each frame of the target detection image.

[0116] The hand region refers to the image region corresponding to the user's hand identified by the target detection algorithm in the target detection image. It is usually represented in the form of a bounding box, which has specific position coordinates and size information to characterize the spatial position and range of the hand in the image.

[0117] In some embodiments, the controller acquires the identified hand region and each food region in each frame of the target detection image, where both the hand region and the food region are represented by detection boxes. For each frame of the target detection image, the controller calculates the intersection area between the detection boxes of the hand region and each food region in that frame, and determines whether the intersection area is greater than zero. If the intersection area is greater than zero, the controller determines that there is an overlap between the food region and the hand region in that frame; if the intersection area is equal to zero, the controller determines that there is no overlap between the food region and the hand region in that frame. After traversing all frames, the controller records the overlap determination results between each food region and the hand region in each frame.

[0118] S603 If the hand region does not overlap with different ingredients in all frames of target detection images, select the target ingredient for this access action based on the degree of change in the relative distance between the hand region and different ingredients in each frame of target detection images; if the hand region overlaps with any ingredient in at least one frame of target detection images, determine the overlap score between different ingredients and the hand region in each frame of target detection images under at least one overlap index, and select at least one ingredient as the target ingredient for this access action based on the weighted sum of the overlap scores.

[0119] The overlap index refers to a multi-dimensional evaluation metric used to quantify the interaction between different food items and the hand region in each frame of the target detection image. The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of the target detection image, the overlap area between the handheld food item and other food items in each frame of the target detection image, and the degree of change in relative distance.

[0120] The frequency of occurrence index refers to the frequency of a particular food ingredient being detected in all target detection images. This index reflects the degree of continuous visibility of the food ingredient throughout the entire action process; that is, the more frequently the food ingredient appears, the more continuous and stable it is in time, and the more it matches the characteristics of a target food ingredient that is continuously held and moved by the hand.

[0121] The overlap area index is used to characterize the degree of cumulative overlap between the hand region and any food item in all target detection images. This index reflects the continuity and sufficiency of contact between the food item and the hand; that is, the larger the total overlap area, the longer and more fully the food item maintains contact with the hand during the hand movement, and the more it matches the characteristics of the target food item actually held by the hand.

[0122] The relative distance change index is used to characterize the total fluctuation of the relative position between the hand region and any food item across all target detection images. This index reflects the displacement stability of the food item relative to the hand; that is, the smaller the change, the more stable the relative position of the food item with the hand is throughout the entire movement, and the more it matches the motion characteristics of a target food item stably held by the hand.

[0123] In some embodiments, when the hand region does not overlap with different ingredients in all frames of the target detection image, the method for selecting the target ingredient for this access action based on the degree of change in the relative distance between the hand region and different ingredients in each frame of the target detection image is as follows: when the hand region does not overlap with any ingredients in all frames of the target detection image, the controller calculates the relative distance between the center point of the ingredient region and the center point of the hand region for each ingredient in each frame of the target detection image, and determines the inter-frame distance difference of the ingredient based on the relative distance difference between two adjacent frames. The degree of change in the relative distance between the ingredient and the hand region is obtained by summing all the inter-frame distance differences. After traversing all ingredients, the controller compares the degree of change values ​​of each ingredient and selects the ingredient with the smallest degree of change as the target ingredient for this access action.

[0124] In some embodiments, when there is an overlap between the hand region and any food in at least one frame of the target detection image, the overlap score between different food and the hand region in each frame of the target detection image is determined under at least one overlap index, and at least one food is selected as the target food for this access action based on the weighted sum of the overlap scores. The method is as follows: when there is an overlap between the hand region and any food in at least one frame of the target detection image, the controller calculates the overlap score of each food under the occurrence frequency index, the overlap area index, and the relative distance change index; the controller multiplies the overlap scores of each food under the three indices by their respective preset weight coefficients and adds them together to obtain the weighted sum of each food; after traversing all food, the controller selects the food with the highest weighted sum as the target food for this access action.

[0125] For example, based on the weighted sum of the overlapping scores, at least one ingredient is selected as the target ingredient for this access action, according to the following formula.

[0126] total_score=a×overlap_score+b×delta_score+c×frame_score

[0127] Wherein, total_score is the weighted sum of the overlapping scores, overlap_score is the overlapping score under the frequency index, delta_score is the overlapping score under the overlapping area index, frame_score is the overlapping score under the relative distance change index, a is the overlapping weight of the frequency index (e.g., 0.35), b is the overlapping weight of the overlapping score under the overlapping area index (e.g., 0.4), and c is the overlapping weight of the relative distance change index (e.g., 0.25).

[0128] For example, taking ingredients including tomatoes and eggs as an example, the scores are: Tomato: overlap_score=2.02 / 2.02=1.0; delta_score=34 / 34=1.0; frame_score=7 / 7=1.0; total=0.35×1.0+0.40×1.0+0.25×1.0=1.0; Egg: overlap_score=0.30 / 2.02=0.149; delta_score=34 / 128=0.266; frame_score=5 / 7=0.714; total=0.35×0.149+0.40×0.266+0.25×0.714=0.337. Obviously, the total score of tomatoes (1.0) is greater than that of eggs (0.337). Therefore, the ingredient operated on in this access operation is tomatoes.

[0129] In the above embodiments, by acquiring at least one frame of target detection image, a sufficient data foundation is provided for subsequent multi-frame temporal analysis, effectively overcoming the random errors caused by insufficient information in a single frame image; by determining whether the hand region overlaps with different foods in each frame, candidate foods that have a direct interaction relationship with the hand can be initially screened, laying a reliable premise for subsequent layered determination; when the hand region does not overlap with any food in all frames, the target food is further selected based on the degree of change in the relative distance between the hand and the food in each frame, which can accurately identify the food that is continuously held by the hand by utilizing cross-frame displacement stability features, effectively avoiding misjudging food that accidentally approaches in the background as the target; and when there is at least one frame where the hand region overlaps with the food, the frequency of occurrence and the overlap area are considered together. The system uses a weighted scoring of three overlapping indicators: overlap area, relative distance change, and total overlap. The overlap area indicator is not the cross-joint ratio of a single frame, but the cumulative degree of overlap across all target detection images. This comprehensively reflects the sufficiency of contact between the food and the hand throughout the entire action. The relative distance change indicator is not the spatial distance of a single frame, but the total fluctuation of the relative position across all target detection images. This accurately characterizes the displacement stability of the food relative to the hand. Thus, the system quantifies the interaction intensity between the food and the hand from multiple dimensions, including contact frequency, contact sufficiency, and motion stability. It takes into account the complementarity of features in each dimension and achieves synergistic enhancement among multiple indicators through weighted fusion. This allows the system to accurately distinguish between the handheld target food and adjacent or stacked interfering food even in complex placement scenarios.

[0130] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the steps for determining the overlap score between different food ingredients and hand regions in each frame of target detection image under the degree of change index in the above embodiments are refined.

[0131] See Figure 7 The steps for determining overlap scoring, as shown, include:

[0132] S701 determines the degree of change in the relative distance between the food and the hand region in each frame of the target detection image for each food item.

[0133] In some embodiments, each food item in each frame of the target detection image is traversed, and for the center point of the food item region and the center point of the hand region in each frame, the first distance difference on the horizontal axis and the second distance difference on the vertical axis are calculated respectively. The sum of the first distance difference and the second distance difference is used as the relative distance between the food item and the hand region in the current frame. The controller obtains the relative distance between the food item in two adjacent frames and calculates the difference between the two as the inter-frame distance difference. After traversing all adjacent frame pairs, all inter-frame distance differences are summed, and the summed value is used as the degree of change of the relative distance between the food item and the hand region in each frame of the target detection image.

[0134] S702 uses the minimum degree of change in all ingredients and the ratio between them as the degree of change in the ingredients as the overlap score under the degree of change index.

[0135] In some embodiments, the change value of the relative distance corresponding to each of all ingredients is obtained, and the minimum change value is determined from all change values. For each ingredient, the controller calculates the ratio between the minimum change value and the change value of the ingredient itself, and uses the calculated ratio as the overlap score of the ingredient under the change index. In the edge case where the change value is equal to zero, the controller directly sets the overlap score of the ingredient under the change index to the preset maximum value.

[0136] For example, taking tomatoes and eggs as ingredients, the relative distance variation of tomatoes is 34, and that of eggs is 128. Obviously, tomatoes have the smallest variation among all ingredients. Therefore, based on the formula: overlap score delta_score under the variation index = min_delta (i.e., the smallest variation among all ingredients) / vec_delta (i.e., the variation of the ingredients), the overlap score of tomatoes under the variation index is determined to be 1 (34 / 34=1), and the overlap score of eggs under the variation index is 0.266 (34 / 128=0.266).

[0137] In the above embodiments, by determining the degree of change in the relative distance between each food item and the hand area frame by frame, and using the ratio between the minimum degree of change among all food items and the degree of change of each food item itself as the overlap score under the degree of change index, the normalization processing of this index is achieved, making the scores between different food items comparable and avoiding dimensional interference caused by differences in the movement amplitude of different food items or differences in the size of the detection box. The smaller the degree of change in relative distance of the food item, the closer its ratio is to 1, and the higher the score. This characteristic is exactly in line with the physical law that the displacement change of the target food item held by the hand should be minimized relative to the hand, so as to effectively quantify the stability of the interaction between the food item and the hand. In addition, the score in the form of ratio does not rely on external thresholds or prior parameters, but is adaptively calculated entirely based on the relative relationship of the characteristics of each food item in the current frame sequence. This not only improves the objectivity and robustness of the scoring results, but also enhances the generalization ability of the algorithm under different refrigerator internal layouts, different types of food items, and different placement postures.

[0138] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the steps for determining the degree of change in the relative distance between the food and hand regions in each frame of the target detection image in the above embodiments are refined.

[0139] See Figure 8A The steps shown for determining the degree of change in relative distance include:

[0140] For each frame of target detection image, S801 determines the distance difference between the center point of the food region and the center point of the hand region.

[0141] In some embodiments, the food region detection box of each food item in the target detection image of the current frame is obtained, and the geometric center point of the food region detection box is determined. At the same time, the geometric center point of the hand region detection box is obtained. The controller calculates the difference between the abscissa value of the food region center point and the abscissa value of the hand region center point, and uses the absolute value of the difference as the first distance difference. The controller also calculates the difference between the ordinate value of the food region center point and the ordinate value of the hand region center point, and uses the absolute value of the difference as the second distance difference.

[0142] For each food ingredient, S802 uses the sum of the distance differences between the food ingredient and the hand region in each target detection image as the degree of change in the relative distance between the food ingredient and the hand region in each frame of the target detection image.

[0143] In some embodiments, for each food ingredient, a first sum between the absolute values ​​of the first distance differences corresponding to the food ingredients in each target detection image is determined, and a second sum between the absolute values ​​of the second distance differences corresponding to the food ingredients in each target detection image is determined. The sum between the first sum and the second sum is used as the degree of change in the relative distance between the food ingredients and the hand region in each frame of the target detection image.

[0144] For example, such as Figure 8B The diagram illustrates relative distances, including eggs and tomatoes. For each frame of the target detection image, the relative distance 1 between the center point 1 of the egg detection box and the center point 2 of the hand detection box, and the relative distance 2 between the center point 3 of the tomato detection box and the center point 2 of the hand detection box, are determined sequentially. The sum of the distance differences between the egg and hand regions in each target detection image is used as the degree of change in the relative distance between the egg and hand regions in each frame of the target detection image, and the sum of the distance differences between the tomato and hand regions in each target detection image is used as the degree of change in the relative distance between the tomato and hand regions in each frame of the target detection image. Here, taking eggs as an example, the first distance difference on the horizontal axis between center point 1 (x1, y1) and center point 2 (x2, y2) is determined (first distance difference dx = x1 – x2); the first sum of the absolute values ​​of the first distance differences between the corresponding ingredients in each target detection image is determined (first sum ddx = ∑dx). n(where n is the number of frames); and, determine the second distance difference on the ordinate between center point 1 (x1, y1) and center point 2 (x2, y2) (second distance difference dy = y1 – y2); determine the second sum of the absolute values ​​of the second distance differences corresponding to the food items in each target detection image (second sum ddy = ∑dy). n (where n is the number of frames), the sum between the first sum and the second sum is used as the degree of change in the relative distance between the food and hand regions in each frame of the target detection image (the degree of change in relative distance vec_delta=|ddx|+|ddy|).

[0145] In the above embodiments, by determining the first distance difference and the second distance difference between the center point of the food region and the center point of the hand region on the horizontal and vertical coordinates for each frame of the target detection image, and using the sum of the two as the relative distance between the food and the hand in that frame, the spatial positional relationship between the hand and the food on the two-dimensional plane can be completely described with extremely low computational cost, avoiding the square root operation caused by using Euclidean distance, and effectively reducing the overhead of frame-by-frame calculation; by summing the relative distances between each frame, the degree of change of the relative distance between the food and the hand region in each frame can be used as the degree of change of the relative distance between the food and the hand region in each frame. This summation can accurately quantify the total displacement fluctuation of the food relative to the hand in consecutive frames. That is, the smaller the degree of change, the more stable the relative position of the food and the hand is during the movement, and the more consistent it is with the motion characteristics of the target food that is continuously held by the hand.

[0146] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the steps for determining the overlap score between different food ingredients and hand regions in each frame of target detection image under the occurrence frequency index in the above embodiments are refined.

[0147] See Figure 9 The steps for determining overlap scoring, as shown, include:

[0148] S901 determines the number of target detection images containing each food ingredient in each frame of target detection images.

[0149] In some embodiments, each food ingredient in each frame of the target detection image is traversed, and the presence of a food ingredient region detection result in the current frame is determined frame by frame. If it exists, the count value of the food ingredient is incremented by one. After the controller traverses all target detection images, the count value corresponding to each food ingredient is determined as the number of images in which the food ingredient appears in the target detection image.

[0150] S902 uses the ratio between the number of images and the total number of images in the target detection image as the overlap score under the occurrence index.

[0151] In some embodiments, the number of images in each frame of target detection images of the food ingredient is obtained, and the total number of images in the target detection images involved in the calculation is obtained; the controller calculates the ratio between the number of images and the total number of images, and uses the calculated ratio as the overlap score of the food ingredient under the occurrence frequency index.

[0152] For example, taking tomatoes and eggs as food ingredients, since the number of target detection images containing food ingredients is the number of times the food ingredients appear in each target detection image, for example, tomatoes appear 7 times, eggs appear 5 times, and the total number of target detection images is 7. Based on this, using the formula: overlap score under the occurrence frequency index frame_score = frame_count (i.e., the number of occurrences, i.e., the number of images) / frame_all (i.e., the total number of target detection images), the overlap score of tomatoes under the occurrence frequency index is determined to be 1 (7 / 7=1), and the overlap score of eggs under the degree of change index is 0.714 (5 / 7=0.714).

[0153] In the above embodiments, by counting the number of images of each food ingredient appearing in the target detection image and using the ratio between the number of images and the total number of images in the target detection image as the overlap score under the occurrence index, a quantitative assessment of the continuous visibility of the food ingredient during the movement is achieved. The higher the ratio, the more stably the food ingredient is detected in more frames, which is more in line with the physical law that the target food ingredient that is continuously held by the hand and moves with the hand should have continuous appearance characteristics in time sequence. This effectively distinguishes it from background interference food ingredients that only appear briefly in individual frames due to accidental hand sweep or detection jitter.

[0154] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the steps for determining the overlap score between different food ingredients and hand regions in each frame of target detection image under the overlap area index in the above embodiments are refined.

[0155] See Figure 10 The steps for determining overlap scoring, as shown, include:

[0156] S1001 determines the total overlap area between the food and the hand region in each frame of the target detection image for each food item.

[0157] In some embodiments, the food region and hand region of each food item in each frame of the target detection image are obtained frame by frame. For the current food item in the current frame, the intersection area between the food region and the hand region is calculated as the overlapping area in the frame. After the controller traverses all target detection images, the overlapping area corresponding to the food item in each frame is accumulated, and the accumulated sum is used as the total overlapping area between the food item and the hand region in each frame of the target detection image.

[0158] S1002 uses the ratio between the total overlapping area of ​​ingredients and the largest total overlapping area among all ingredients as the overlap score under the overlap area index.

[0159] In some embodiments, the total overlap area corresponding to each of all ingredients is obtained, and the maximum total overlap area value is determined from all total overlap areas. For each ingredient, the controller calculates the ratio between the total overlap area of ​​the ingredient and the maximum total overlap area value, and uses the calculated ratio as the overlap score of the ingredient under the overlap area index. In the edge case where the maximum total overlap area value is zero, the controller directly sets the overlap score of the ingredient under the overlap area index to zero.

[0160] For example, taking tomatoes and eggs as ingredients, the total overlap area of ​​tomatoes is 2.02 and the total overlap area of ​​eggs is 0.30. Obviously, tomatoes have the largest total overlap area among all ingredients. Therefore, based on the formula: overlap_score = overlap_sum (i.e., the total overlap area of ​​ingredients) / max_ov (i.e., the largest total overlap area among all ingredients), the overlap score of tomatoes is determined to be 1 (2.02 / 2.02=1) and the overlap score of eggs is 0.149 (0.30 / 2.02=0.149).

[0161] In the above embodiments, the total overlap area is obtained by accumulating the overlap area between each food ingredient and the hand region in each frame of the target detection image. The ratio between the total overlap area of ​​the food ingredient and the largest total overlap area among all food ingredients is used as the overlap score under the overlap area index. This achieves normalization processing of the index, making the overlap degree between different food ingredients comparable and avoiding dimensional interference caused by differences in the size of the food ingredients or the size of the detection box. The larger the total overlap area of ​​the food ingredient, the closer its ratio is to 1, and the higher the score. This characteristic is consistent with the target food ingredient that is continuously held by the hand and should maintain contact with the hand during the movement. This method aligns with the physical laws of sustained and sufficient contact and overlap, thus accurately reflecting the continuity and sufficiency of contact between the food and the hand. Furthermore, this accumulation method integrates the overlap information of each frame throughout the entire action, avoiding misjudgments caused by accidental contact or detection errors when relying solely on single-frame overlap determination. It also effectively distinguishes the handheld target food from background food that is only briefly obscured due to spatial proximity in a few frames. Moreover, this ratio-based scoring does not rely on external thresholds and is entirely based on the relative relationships of the overlap features of each food item in the current frame sequence for adaptive calculation, exhibiting strong generalization ability and scene adaptability.

[0162] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the steps for determining whether there is overlap between the hand region in each frame of the target detection image and different food items in the target detection image are refined.

[0163] See Figure 11 The steps for determining overlap, as shown, include:

[0164] S1101 If there are at least two identical food items in the target detection image of the current frame, then select the food item that is closest to the hand region from the at least two identical food items, and determine whether there is an overlap between the food item and the hand region in the target detection image.

[0165] In some embodiments, all food regions identified in the target detection image of the current frame are acquired, and the food regions are grouped according to food category, so that food regions of the same food category are grouped together. For each group, the controller determines whether the number of food regions in the group is greater than or equal to two. If so, the controller calculates the distance between each food region in the group and the hand region, and selects the food region with the smallest distance as the representative region of the category in the frame. Based on the representative region, the controller calculates the intersection area between the representative region and the hand region, and determines whether there is an overlap between the food category and the hand region in the frame based on whether the intersection area is greater than zero.

[0166] For example, if there are two tomatoes in the target detection image, the distance difference between tomato 1 and the hand area is 1, and the distance difference between tomato 2 and the hand area is 2. Obviously, tomato 1 is closer to the hand area. Therefore, tomato 1 is selected to determine whether there is any overlap between the tomato and the hand area.

[0167] In some embodiments, when at least two identical ingredients exist in the target detection image of the current frame, the ingredient closest to the hand region is selected from these identical ingredients, and the overlap between the ingredient and the hand region is determined based on this ingredient. This effectively solves the problem of redundant candidate interference caused by the segmentation and detection of the same ingredient into multiple instances or the batch placement of ingredients of the same type in the image. It avoids overlap judgment conflicts caused by the competition between multiple candidate boxes of the same ingredient and significantly reduces the data processing volume of subsequent cross-frame feature statistics and scoring calculation. At the same time, this distance-optimized strategy ensures that no matter how many times the same category appears in the image, only one most representative relationship record between the hand and the ingredient is retained for each ingredient category in each frame, ensuring the clarity and consistency of the feature sequence. This provides a clean and reliable data foundation for subsequent ingredient recognition based on multi-frame statistics, thereby improving the recognition accuracy and processing efficiency of the algorithm in scenarios of batch storage or dense placement of the same type of ingredients.

[0168] Based on the technical solutions of the above embodiments, some optional embodiments are also provided. In these optional embodiments, the steps in the above embodiments are refined when the hand region does not overlap with different ingredients in all frames of target detection images.

[0169] See Figure 12 The steps for identifying the target ingredient shown include:

[0170] S1201 If the hand region does not overlap with different food ingredients in all frames of target detection images, and the degree of change in the relative distance between each food ingredient and the hand region is the same, select the food ingredient that appears most frequently, or the food ingredient with the smallest average relative distance to the hand region in all frames of target detection images, as the target food ingredient for this storage action.

[0171] In some embodiments, when the hand region and each food item do not overlap in all frames of the target detection image, and the cumulative value of the change in the relative distance between each food item and the corresponding food item is the same, the controller counts the number of times each food item appears in each frame of the target detection image and selects the food item with the most occurrences as the target food item for this access action; or, the controller calculates the average relative distance between each food item and the hand region in each frame of the target detection image and selects the food item with the smallest average relative distance as the target food item for this access action. Specifically, when the controller uses "most occurrences" for selection, if there are multiple food items with the most occurrences, the controller further selects the food item with the smallest average relative distance as the target food item; when the controller uses "smallest average relative distance" for selection, if there are multiple food items with the smallest average relative distance, the controller further selects the food item with the most occurrences as the target food item.

[0172] In some embodiments, the average relative distance in this embodiment is determined by the following formula:

[0173]

[0174] Where mean_dist_i is the average relative distance of the i-th ingredient, dist_i is the relative distance of the i-th ingredient, and frame_count is the number of times the i-th ingredient appears.

[0175] It should be noted that if the hand region does not overlap with any of the different food ingredients in all frames of the target detection image, and the degree of variation in the relative distance between each food ingredient and the hand region is the same, selecting the food ingredient that appears most frequently or has the smallest average relative distance to the hand region in all frames of the target detection image as the target food ingredient for this access action still cannot yield the final target food ingredient. In this case, the food ingredient with the smallest reference distance to the hand region from all categories of food ingredients will be selected as the target food ingredient to ensure that there is still a recognition result even in extreme cases. The reference distance determined in this case can be the degree of variation in relative distance, the relative distance between each food ingredient and the hand region in any frame, or the average relative distance between each food ingredient and the hand region in all frames of the target detection image. This embodiment does not limit this to any particular type.

[0176] In the above embodiments, in the extreme case where the hand region and each food item do not overlap in all frames and the relative distance of each food item changes to the same degree, a fallback selection strategy is further adopted, using the most frequently occurring or the smallest average relative distance. This effectively ensures the reliability and robustness of the algorithm's output when conventional discrimination dimensions fail. That is, when no overlapping information is available and the stability index cannot distinguish each candidate food item, by counting the number of times the food item appears in multiple frames, the food item that is continuously visible and more stable in time can be selected first. Alternatively, by calculating the average relative distance in each frame, the food item that is spatially closest to the hand can be selected first. Both can provide reasonable physical judgment criteria under the boundary condition of converging discrimination criteria. At the same time, this fallback mechanism ensures that the recognition process can output results normally without interruption or error in any extreme scenario. This not only improves the integrity of the system and the consistency of the user experience, but also avoids the problem of missing food item list updates due to no output, further consolidating the algorithm's engineering implementation capability in complex and ever-changing home use scenarios.

[0177] Based on the technical solutions of the above embodiments, some optional embodiments are also provided, in which the control process of the refrigerator is described in detail.

[0178] See Figure 13 The refrigerator food identification method shown includes:

[0179] When the action representation of the current access is stored, S1301 sequentially selects a preset number of action detection images from the memory as target detection images; when the action representation of the current access is retrieved, it selects a preset number of action detection images from the memory in reverse order as target detection images.

[0180] S1302 If there are at least two identical food items in the target detection image of the current frame, then select the food item that is closest to the hand region from the at least two identical food items, and determine whether there is any overlap between the food items and the hand region in the target detection image;

[0181] S1303, in the case that the hand region does not overlap with different food ingredients in all frames of target detection images, selects the target food ingredient for the current storage action based on the degree of change in the relative distance between the hand region and different food ingredients in each frame of target detection images;

[0182] S1304 If, in the case that the hand region overlaps with any food ingredient in at least one frame of the target detection image, determine the overlap score between different food ingredients and the hand region in each frame of the target detection image under at least one overlap index.

[0183] For each frame of target detection image, S1305 determines the distance difference between the center point of the food region and the center point of the hand region.

[0184] S1306 For each food ingredient, the sum of the distance differences between the food ingredient and the hand region in each target detection image is used as the degree of change in the relative distance between the food ingredient and the hand region in each frame of target detection image;

[0185] S1307 uses the minimum degree of change in all ingredients and the ratio between them and the degree of change of the ingredients as the overlap score under the degree of change index.

[0186] S1308 determines the number of target detection images containing each food ingredient in each frame of target detection images;

[0187] S1309 uses the ratio between the number of images and the total number of images in the target detection images as the overlap score under the occurrence index;

[0188] S1310 For each food item in each frame of target detection image, determine the total overlap area between the food item and the hand region in each frame of target detection image;

[0189] S1311 uses the ratio of the total overlapping area of ​​ingredients to the largest total overlapping area among all ingredients as the overlap score under the overlap area index.

[0190] S1312 selects at least one ingredient as the target ingredient for this storage / retrieval action based on the weighted sum of the overlap scores under the degree of change index, the frequency of occurrence index, and the overlap area index.

[0191] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0192] Based on the same inventive concept, this application also provides a refrigerator food identification device for implementing the refrigerator food identification method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more refrigerator food identification device embodiments provided below can be found in the limitations of the refrigerator food identification method described above, and will not be repeated here.

[0193] In one exemplary embodiment, such as Figure 14 As shown, a refrigerator food identification device is provided, including: an acquisition module 1401, a determination module 1402, and a selection module 1403. Wherein,

[0194] The acquisition module 1401 is used to acquire at least one frame of target detection image acquired by the image acquisition device;

[0195] The determination module 1402 is used to determine whether the hand region in each frame of the target detection image overlaps with different food ingredients;

[0196] The selection module 1403 is used to select the target food for the current access action based on the degree of change in the relative distance between the hand region and different food in each frame of the target detection image when the hand region does not overlap with different food in all frames of the target detection image; when the hand region overlaps with any food in at least one frame of the target detection image, the module determines the overlap score between different food and the hand region in each frame of the target detection image under at least one overlap index, and selects at least one food as the target food for the current access action based on the weighted sum of the overlap scores.

[0197] The overlap index includes the frequency of different ingredients in each frame of the target detection image, the overlap area between the handheld ingredient and other ingredients in each frame of the target detection image, and the degree of change in relative distance.

[0198] In some embodiments, the selection module 1403 is further configured to determine the degree of change in the relative distance between the food and the hand region in each frame of target detection images for each food item; and to use the minimum degree of change among all food items and the ratio between the minimum degree of change and the degree of change of the food items as the overlap score under the degree of change index.

[0199] In some embodiments, the selection module 1403 is further configured to determine, for each frame of target detection image, the distance difference between the center point corresponding to the food region and the center point corresponding to the hand region; and for each food, the sum of the distance differences between the food and the hand region in each target detection image is used as the degree of change in the relative distance between the food and the hand region in each frame of target detection image.

[0200] In some embodiments, the selection module 1403 is further configured to determine the number of target detection images in which each food ingredient appears for each food ingredient in each frame of target detection images; and to use the ratio between the number of images and the total number of target detection images as the overlap score under the occurrence index.

[0201] In some embodiments, the selection module 1403 is further configured to determine the total overlap area between the food and the hand region in each frame of target detection image for each food item in each frame of target detection image; and to use the ratio between the total overlap area of ​​the food items and the largest total overlap area among all food items as the overlap score under the overlap area index.

[0202] In some embodiments, the determining module 1402 is further configured to, if there are at least two identical food ingredients in the target detection image of the current frame, select the food ingredient that is closest to the hand region from among the at least two identical food ingredients, and determine whether there is an overlap between the food ingredients and the hand region in the target detection image.

[0203] In some embodiments, the acquisition module 1401 is further configured to sequentially select a preset number of motion detection images from the memory as target detection images when the current access action representation is stored; and to select a preset number of motion detection images from the memory in reverse order as target detection images when the current access action representation is retrieved.

[0204] In some embodiments, the selection module 1403 is further configured to select the food that appears most frequently or has the smallest average relative distance to the hand in each frame of the target detection image when there is no overlap between the hand region and different food ingredients in all frames of the target detection image and the degree of change of the relative distance between each food ingredient and the hand region is the same, as the target food ingredient for this access action.

[0205] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for identifying food items in a vacuum drawer. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0206] Those skilled in the art will understand that Figure 15 The structures shown are merely block diagrams of some structures related to the embodiments of this application and do not constitute a limitation on the computer devices on which the embodiments of this application are applied. Specific computer devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0207] In one alternative embodiment, Figure 15 The computer device shown may be the aforementioned refrigerator. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0208] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0209] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0211] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0212] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0213] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A refrigerator, characterized in that, The refrigerator includes: The enclosure includes at least one compartment; An image acquisition device is configured to acquire motion detection images within the room; The controller, connected to the image acquisition unit, is configured as follows: Acquire at least one frame of target detection image captured by the image acquisition device; Determine whether the hand region overlaps with different food ingredients in each frame of the target detection image; If the hand region does not overlap with different food ingredients in all frames of target detection images, the target food ingredient for this storage action is selected based on the degree of change in the relative distance between the hand region and different food ingredients in each frame of target detection images. If, in at least one frame of the target detection image, the hand region overlaps with any food ingredient, the overlap score between different food ingredients and the hand region in each frame of the target detection image is determined under at least one overlap index. Based on the weighted sum of the overlap scores, at least one food ingredient is selected as the target food ingredient for this access action. The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of target detection image; the overlap area between the hand-held food item and other food items in each frame of target detection image; and the degree of change in relative distance. The overlap area index is used to characterize the cumulative degree of overlap in area between the hand region and any food item in all target detection images. The degree of change in relative distance index is used to characterize the total fluctuation of the relative position between the hand region and any food item in all target detection images as the frames change.

2. The refrigerator according to claim 1, characterized in that, When the controller performs the task of determining the overlap score between different food items and the hand region in each frame of target detection images under the metric of degree of change, it is configured to: For each food item in each frame of the target detection image, determine the degree of change in the relative distance between the food item and the hand region in each frame of the target detection image; The ratio between the smallest degree of change among all ingredients and the degree of change of the ingredient itself is used as the overlap score under the degree of change index.

3. The refrigerator according to claim 2, characterized in that, When determining the degree of change in the relative distance between the food and the hand region in each frame of target detection image, the controller is configured to: For each frame of target detection image, determine the distance difference between the center point of the food region and the center point of the hand region; For each food ingredient, the sum of the distance differences between the food ingredient and the hand region in each target detection image is used as the degree of change in the relative distance between the food ingredient and the hand region in each frame of the target detection image.

4. The refrigerator according to claim 1, characterized in that, When the controller performs the task of determining the overlap score between different food items and the hand region in each frame of target detection images based on the frequency of occurrence, it is configured to: For each food ingredient in each frame of target detection images, determine the number of target detection images in which the food ingredient appears; The ratio between the number of images and the total number of images in the target detection image is used as the overlap score under the occurrence index.

5. The refrigerator according to claim 1, characterized in that, When the controller performs the task of determining the overlap score between different food items and the hand region in each frame of target detection images based on the overlap area index, it is configured to: For each food item in each frame of the target detection image, determine the total overlap area between the food item and the hand region in each frame of the target detection image; The ratio between the total overlapping area of ​​the ingredients and the largest total overlapping area among all ingredients is used as the overlap score under the overlapping area index.

6. The refrigerator according to claim 1, characterized in that, When determining whether there is overlap between the hand region and different food items in the target detection image in each frame, the controller is configured to: If there are at least two identical food items in the target detection image of the current frame, then among the at least two identical food items, the food item that is closest to the hand region is selected to determine whether there is any overlap between the food item and the hand region in the target detection image.

7. The refrigerator according to claim 1, characterized in that, The refrigerator also includes: The memory is configured to store at least one motion detection image in a sequential, time-order manner. Accordingly, when the controller acquires at least one frame of target detection image captured by the image acquisition device, it is configured to: When the action representation is stored in this access action, a preset number of action detection images are sequentially selected from the memory as target detection images; When the action representation is retrieved in this access operation, a preset number of action detection images are selected from the memory in reverse order as target detection images.

8. The refrigerator according to claim 1, characterized in that, In the case that the hand region does not overlap with different food items in all frames of target detection images, the controller is further configured to: If the hand region does not overlap with any of the different food ingredients in all frames of the target detection image, and the degree of change in the relative distance between each food ingredient and the hand region is the same, then the food ingredient that appears most frequently or has the smallest average relative distance to the hand region in all frames of the target detection image is selected as the target food ingredient for this access action.

9. A method for identifying food items in a refrigerator, characterized in that, The method includes: Acquire at least one frame of target detection image captured by the image acquisition device; Determine whether the hand region overlaps with different food ingredients in each frame of the target detection image; If the hand region does not overlap with different food ingredients in all frames of target detection images, the target food ingredient for this storage action is selected based on the degree of change in the relative distance between the hand region and different food ingredients in each frame of target detection images. If, in at least one frame of the target detection image, the hand region overlaps with any food ingredient, the overlap score between different food ingredients and the hand region in each frame of the target detection image is determined under at least one overlap index. Based on the weighted sum of the overlap scores, at least one food ingredient is selected as the target food ingredient for this access action. The overlap index includes at least one of the following: the frequency of occurrence of different food items in each frame of target detection image; the overlap area between the hand-held food item and other food items in each frame of target detection image; and the degree of change in relative distance. The overlap area index is used to characterize the cumulative degree of overlap in area between the hand region and any food item in all target detection images. The degree of change in relative distance index is used to characterize the total fluctuation of the relative position between the hand region and any food item in all target detection images as the frames change.

10. The method according to claim 9, characterized in that, Determine the overlap score between different food items and the hand region in each frame of the target detection image under a change index, including: For each food item in each frame of the target detection image, determine the degree of change in the relative distance between the food item and the hand region in each frame of the target detection image; The ratio between the smallest degree of change among all ingredients and the degree of change of the ingredient itself is used as the overlap score under the degree of change index.

Images