Method for controlling refrigeration device, refrigeration device and computer program product
By combining a multimodal pre-trained large model with a convolutional neural network, refrigerator image data is directly processed, solving the robustness and efficiency problems of refrigerator item status detection, and realizing fast and accurate item status recognition and personalized preservation strategies.
Patent Information
- Application Number
- CN202411172872.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2026-03-03
AI Technical Summary
Existing methods for detecting the state of items in refrigerators are not robust enough due to factors such as lighting, shooting angle, and occlusion. Furthermore, deep learning models require extensive manual annotation and retraining, which limits their application in detecting the freshness of items.
By utilizing a multimodal pre-trained large model to directly process image data, the feature and location information of objects can be obtained. The status of objects can be determined by analyzing the feature and location information without the need for model adjustment and manual annotation. Combined with convolutional neural networks to extract object features, the category, quantity and location of objects can be detected quickly and accurately.
It improves the efficiency and accuracy of item state detection, reduces the technical overhead of data collection and model training, and achieves highly robust detection for different item categories.
Smart Images
Figure CN121594630A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of household appliances, and more particularly to a method for controlling a refrigeration device, a refrigeration device, and a computer program product for at least assisting in implementing the steps of the method according to this application. Background Technology
[0002] In current refrigerators, machine vision methods can be used to process images of items captured by cameras installed inside the refrigerator to determine the status of the items. However, on the one hand, traditional image processing algorithms are not robust enough to detect the status of items under different refrigerator conditions because images of items inside the refrigerator are easily affected by factors such as lighting, shooting angle, and occlusion. On the other hand, the deep learning models used in machine vision methods mostly employ fully supervised training methods to learn the detection of status information for different categories of items. This requires collecting a large number of image samples for each item category, manually labeling these samples, and retraining the model for each new item category. These factors greatly limit the application of deep learning models in detecting the freshness of items.
[0003] Therefore, current image processing methods for detecting the state of items inside a refrigerator have room for improvement. Summary of the Invention
[0004] The purpose of this application is to provide a method for controlling a refrigeration device, a refrigeration device, and a computer program product to at least partially solve the problems in the prior art.
[0005] According to a first aspect of this application, a method for controlling a refrigeration device is provided, the method including a step of acquiring the state of an item in the refrigeration device, the acquisition step including:
[0006] - Step S1: Based on the pre-given item category information and image data of the items to be detected in the refrigeration equipment at various times, obtain the feature information and / or location information of each item belonging to each item category at each time; and
[0007] - Step S2: Determine the state information of each item to be detected based on the feature information and / or location information of each item belonging to the same item category at each time.
[0008] The core concept of this application also includes: utilizing a multimodal pre-trained large model to directly process image data of each frame of an item without any model adjustments, in order to obtain time-related feature information and / or location information of each item. By analyzing the feature information and / or location information, the state information of each item to be detected can be determined. This scheme fully utilizes the generalization ability of the multimodal pre-trained large model for image target detection, not only possessing the advantages of high robustness of deep learning, but also eliminating the technically costly steps of data collection, manual annotation, and model training required by commonly used fully supervised training methods for deep neural networks, thereby effectively improving the detection efficiency of state information of different categories of items in refrigeration equipment.
[0009] According to an optional embodiment of this application, the status information of the items includes at least one of the following: the category of each item, the total quantity of items of the same category, the storage location of each item, the storage time of each item, and the freshness of each item. The total quantity of the items includes, for example, the total number, total volume, and / or total weight of the items. Considering that the status information of the items is correlated with the user's expected behavior towards the items—such as where the item is retrieved, when it is consumed, how it is consumed, and / or how much of the item is consumed, when the item is purchased, and / or how much of the item is purchased—determining this status information provides a reliable basis for the user's decision regarding the items.
[0010] According to another optional embodiment of this application, the item category information and image data of the items to be detected at various times are input into a multimodal pre-trained large model, thereby determining the category of each item to be detected at each time and the total number of items of the same category. Optionally, the location information and / or image information of each item belonging to each item category at each time can be obtained, and the feature information of each item at each time can be obtained based on the image information of each item at each time. Here, the generalization ability of the multimodal pre-trained large model for image object detection is fully utilized to quickly and accurately obtain the category and quantity of each item, as well as the location information and / or image information at each time. In particular, the item category information input into the multimodal pre-trained large model can be changed according to actual needs without any training or fine-tuning of the model.
[0011] According to another optional embodiment of this application, feature information of each item at each time step can be obtained using a convolutional neural network based on the image information of each item at each time step. Here, the efficient processing capability of the convolutional neural network for image classification tasks is fully utilized to quickly and accurately extract the feature information of each item from the image information.
[0012] According to another optional embodiment of this application, step S2 may include:
[0013] - Step S21: Calculate the similarity between the first feature information of each item belonging to the same item category at the first time and other feature information at one or more other times, wherein the other times are different from the first time;
[0014] - Step S22: Determine whether the similarity exceeds a preset similarity threshold;
[0015] - Step S23: If the similarity exceeds a preset similarity threshold, then the item belonging to the first feature information obtained based on the image data at the first time point and the item belonging to the other feature information obtained based on the image data at other times points are marked as the same item, and the storage time of the item is determined based on the first time point and the other times points; and
[0016] - Step S24: If the similarity does not exceed the preset similarity threshold, then the item associated with the first feature information obtained based on the image data at the first moment is marked as an item that has been taken out.
[0017] Here, steps S21 to S24 can be executed iteratively until all items belonging to the same item category, obtained based on the image data from the first moment, have been marked. In this way, the storage time of items can be accurately and efficiently determined by utilizing the similarity of the feature information of items at different times, and the retrieved items can be quickly filtered out.
[0018] According to another optional embodiment of this application, step S2 may further include:
[0019] - Step S25: After all items belonging to the same item category obtained based on the image data at the first moment have been marked, determine whether there are any unmarked items among the items belonging to the same item category obtained based on image data at other moments; and
[0020] - Step S26: If there is an unmarked item, mark the item as a newly placed item and store the other time associated with the newly placed item.
[0021] Here, newly added items can be quickly and accurately identified, and their characteristic information and / or location information, along with other associated times, can be stored. These other times are marked as the initial times of the newly added items. Based on subsequently acquired image data about the items, the storage time of the items can be updated, and the freshness of the items can be assessed by utilizing the negative correlation between the storage time of the item category and its freshness.
[0022] According to another optional embodiment of this application, the feature information of the item may include a first feature value vector of the item at a first time moment and other feature value vectors of the item at one or more other times moment moment, wherein the other times moment moment moment moment is different from the first time moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment momentary ...
[0023] According to another optional embodiment of this application, step S2 may further include:
[0024] - Step S27: After calculating the storage time of the item, determine whether the positional deviation between the first position information of the item at a first moment and other position information at one or more other moments exceeds a preset positional deviation threshold; and
[0025] - Step S28: If the deviation exceeds a preset position deviation threshold, the item is marked as a moved item, and the storage location of the item is updated with other position information of the item.
[0026] This method allows for the quick and timely marking of items whose positions have changed, avoiding interference with item identification due to changes in item position, and laying the foundation for the execution of various functions of refrigeration equipment (such as food preservation function, food position indication, etc.).
[0027] According to another optional embodiment of this application, the method may further include a notification step of the status of items in the refrigeration equipment, the notification step including:
[0028] - Step S3: Output notification information about the items based at least on the determined status information of each item.
[0029] Optionally, the notification information sent may include, for example, the status information of each item. This is because the status of an item is related to the user's expected behavior toward that item, so that the user can take appropriate measures in a timely manner based on the current status information of the item, such as where to get the item, when to consume the item and / or how much of the item to consume, when to purchase the item and / or how much of the item to purchase, etc.
[0030] Optionally, the notification information sent may also include, for example, recommended available recipes for each item category based at least on the item's status information, saving users the tedious process of thinking about or searching for recipes and further improving the user's experience in handling items.
[0031] Optionally, the notification message may also include reminders about items that the user needs to purchase, thereby promptly reminding the user to purchase the corresponding items to replenish their supplies and avoid shortages.
[0032] According to another optional embodiment of this application, the method may further include a preservation treatment step for the articles in the refrigeration equipment, the preservation treatment step including:
[0033] - Step S4: Adjust the preservation parameters of the refrigeration equipment for the storage area where the corresponding item is located, based at least on the determined status information of each item.
[0034] Optionally, the preservation parameters of the refrigeration equipment for the storage area of the respective item can be adjusted at least based on the determined category, storage time, and storage location of each item. These preservation parameters include, for example, humidity parameters, and / or temperature parameters, and / or oxygen content parameters of the storage area. In this way, personalized preservation strategies adapted to the state information of the items can be developed, effectively slowing down the rate of freshness decline of the items in the refrigeration equipment.
[0035] According to a second aspect of this application, a refrigeration device is provided, which may include the following components:
[0036] - An image acquisition unit, configured to acquire image data of the item to be detected in the refrigeration device at various times; and
[0037] - A control unit configured to implement the method according to this application.
[0038] According to another optional embodiment of this application, the refrigeration device may further include a notification unit configured to output notification information about the items based at least on the determined status information of each item.
[0039] According to another optional embodiment of this application, the refrigeration equipment may further include a preservation treatment unit configured to preserve items in various storage areas of the refrigeration equipment.
[0040] According to another optional embodiment of this application, the refrigeration device is, for example, a refrigerator.
[0041] According to a third aspect of this application, a computer program product, such as a computer-readable program carrier, is provided, comprising computer program instructions that, when executed by a processor, at least assist in implementing the steps of the method according to this application. Attached Figure Description
[0042] The principles, features, and advantages of this application will be better understood below with reference to the accompanying drawings. The drawings include:
[0043] Figure 1 A flowchart illustrating a method for controlling a refrigeration device according to an exemplary embodiment of this application is shown.
[0044] Figure 2 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown;
[0045] Figure 3 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown;
[0046] Figure 4 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown;
[0047] Figure 5 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown;
[0048] Figure 6 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown; and
[0049] Figure 7 A schematic block diagram of a refrigeration device according to an exemplary embodiment of this application is shown.
[0050] List of reference numerals
[0051] 1 Refrigeration equipment
[0052] 11 Image Acquisition Unit
[0053] 12 Control Units
[0054] 13 Notification Unit
[0055] 14 Preservation Processing Units Detailed Implementation
[0056] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit the scope of protection of this application.
[0057] Figure 1 A flowchart illustrating a method for controlling a refrigeration device according to an exemplary embodiment of this application is shown. The following exemplary embodiments describe the method according to this application in more detail.
[0058] The method may include a step of obtaining the state of the items in the refrigeration device 1. For example... Figure 1 As shown, the acquisition steps may include steps S1 and S2. In step S1, feature information and / or location information of each item belonging to each item category at each time is acquired based on pre-given item category information and image data of the items to be detected in the refrigeration device 1 at each time. In the current embodiment of this application, the category information of the items to be detected in the refrigeration device 1 is pre-given. For example, a user can pre-given N item categories that they use daily, wherein the number of item categories N is unlimited, and the number of items in each item category is also unlimited. The image acquisition unit 11 of the refrigeration device 1 can acquire image data of the items to be detected in the refrigeration device 1 at each time. The image acquisition unit 11 is configured, for example, as cameras arranged at multiple locations in the storage space of the refrigeration device 1, which can acquire image data of each item in the storage space from various angles without blind spots.
[0059] Here, pre-given item category information and image data of the items at various times can be input into a loaded multimodal pre-trained large model, thereby obtaining image information of each item belonging to each item category at each time. The multimodal pre-trained large model specifically refers to a deep neural network model based on a transformer structure. It has learned the semantic correspondence between different modalities of data—such as image data and text data—using self-supervised training based on a large amount of unlabeled sample data, and can use text data to guide image processing algorithms to perform image processing tasks such as item classification, item recognition, and item detection. For example, the multimodal pre-trained large model includes, but is not limited to, the SAM model (Segment Anything Model), the DINO (Distillation with no labels) model, and the ClipSeg (Contrastive Language-Image Pre-training Segmentation) model. These multimodal pre-trained large models can directly process the image data of each frame of the item without any model adjustments.
[0060] Specifically, on the one hand, the information of each item category can be sequentially input into the multimodal pre-trained large model. The prompt word encoder of the multimodal pre-trained large model can obtain text features about the item categories based on the input item category information, that is, the names of items in each category, such as vegetables like potatoes, carrots, and green vegetables, and fruits like apples, cherries, and pears. On the other hand, the image data of each frame of items can be input into the multimodal pre-trained large model. The image encoder of the multimodal pre-trained large model can convert the image data of each frame into image features. Since the multimodal pre-trained large model has learned the semantic correspondence between image data and text data, the decoder of the multimodal pre-trained large model can use text features and image features to extract image information about each item in each item category from each frame of image data. At the same time, the image information is associated with the time information of the corresponding frame of image data.
[0061] Furthermore, by inputting pre-given item category information and image data of the items at various times into the loaded multimodal pre-trained large model, the position information of each item belonging to each item category at each time can also be obtained. Specifically, the item category information and image data of the items at various times can be input into the loaded multimodal pre-trained large model to obtain the position information of each item belonging to each item category at each time. Specifically, the prompt word encoder of the multimodal pre-trained large model can obtain text features about the item category based on the input item category information; the image encoder of the multimodal pre-trained large model can convert each frame of image data about the item into image features; and the decoder of the multimodal pre-trained large model can use the text features and image features to extract the position information of each item belonging to each item category at each time from each frame of image data, that is, the information about the position of each item in the storage space of the refrigeration device 1, while the position information is associated with the time information of the corresponding frame of image data.
[0062] Next, feature information of each item at each time step can be obtained based on image information of each item at each time step, for example, using a convolutional neural network. In this application, "feature information" refers to information related to the shape features of an item—such as its size, geometry, and color—that can be extracted from image data about the item. Based on this feature information, two different items can be distinguished. Here, convolutional neural networks specifically refer to image classification models based on convolutional neural networks, such as the VGG (Visual Geometry Group) model, the GoogleLeNet model, the RestNet model, and the EfficientNet model, which have been trained on specific datasets. Here, the efficient processing capability of convolutional neural networks for image classification tasks is fully utilized to quickly and accurately obtain feature information of each item based on image information. The feature information of an item obtained using a convolutional neural network can include a first feature value vector of the item at a first time step (assumed to be the starting time) and other feature value vectors of the item at one or more other times, wherein the other times are different from the first time step. It should be noted that since each image information is associated with the time information of the corresponding frame of image data, each feature value vector is also associated with corresponding time information.
[0063] In step S2, the state information of each item to be detected can be determined based on the feature information and / or location information of each item belonging to the same item category at each time. In this application, "item state information" is used to characterize the physical state of the item in the refrigeration equipment as follows: the physical state is correlated with the user's expected behavior towards the item, such as where the item is taken, when the item is consumed, how the item is consumed and / or how much of the item is consumed, when the item is purchased and / or how much of the item is purchased, etc. The item state information may include, for example, at least one of the following: item category, total quantity of items of the same item category, storage location of each item, storage time of each item, and freshness of each item, wherein the total quantity of the items includes, for example, the total number, total volume, and / or total weight of the items.
[0064] For example, the item category information and image data of the items to be detected at various times are input into a multimodal pre-trained large model, thereby determining the category of each item to be detected at each time, as well as the quantity of each item (including number, volume and / or weight, etc.), and then counting the total number of items of the same item category (including total number, total volume and / or total weight, etc.).
[0065] The following combination Figure 2 The flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application details step S2. The following only describes steps S2... Figure 1 The differences between the embodiments shown are omitted for brevity, and the same steps will not be repeated.
[0066] like Figure 2 As shown, step S2 may include steps S21 to S24. In step S21, the similarity between the first feature information of each item belonging to the same item category at a first time point and other feature information at one or more other times points is calculated, wherein the other times points are different from the first time point. Here, the feature information is represented in the form of feature value vectors. Therefore, when analyzing the similarity between two feature value vectors—for example, a first feature value vector and other feature value vectors—the similarity between the first feature information of the item and other feature information can be characterized by the cosine distance between the first feature value vector of the item and other feature value vectors.
[0067] In step S22, it is determined whether the similarity exceeds a preset similarity threshold. The embodiments of this application are based on the following assumption: there are certain differences between items belonging to the same item category, that is, no two items are completely identical. Therefore, the similarity of the feature information of each item can be used to distinguish items of the same item category.
[0068] If the similarity exceeds a preset similarity threshold, that is, the cosine distance between the first feature vector of the item and other feature vectors is less than a preset distance threshold, then in step S23, the item belonging to the first feature information obtained based on the image data at the first time and the item belonging to the other feature information obtained based on the image data at other times are marked as the same item, and the storage time of the item is determined based on the first time and the other times.
[0069] If the similarity does not exceed the preset similarity threshold, that is, the cosine distance between the first feature value vector of the item and other feature value vectors is greater than or equal to the preset distance threshold, it means that at other times it is impossible to match a feature value vector with a high similarity to the first feature value vector. Therefore, the item is no longer in the storage space of the refrigeration device 1 at other times. In step S24, the item that belongs to the first feature information obtained based on the image data at the first time is marked as an item that has been taken out.
[0070] Next, steps S21 to S24 are executed repeatedly until all items belonging to the same item category obtained based on the image data at the first moment have been marked, thereby completing the item marking process for that item category and obtaining the storage time of the items still existing in the storage space of the refrigeration equipment 1.
[0071] According to the configuration scheme of this application, a multimodal pre-trained large model can be used to directly process the image data of each frame of an item to obtain time-related feature information and / or location information of each item. By analyzing the feature information and / or location information, the state information of each item to be detected can be determined. In particular, the item category information input into the multimodal pre-trained large model can be changed according to actual needs, such as adding new item categories, deleting existing item categories, or replacing existing item categories with new ones, without requiring any training or fine-tuning of the model. This scheme fully utilizes the generalization ability of the multimodal pre-trained large model for image object detection, possessing not only the advantages of high robustness of deep learning, but also eliminating the high technical overhead steps of data collection, manual annotation, and model training required by commonly used fully supervised training methods for deep neural networks, thereby effectively improving the detection efficiency of state information of different item categories in refrigeration equipment.
[0072] Figure 3 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown. The following only describes the process... Figure 2 The differences between the embodiments shown are omitted for brevity, and the same steps will not be repeated.
[0073] Step S2 may further include steps S25 and S26. After all items belonging to the same item category acquired based on the image data at the first moment have been marked, i.e., after exiting the loop from S21 to S24, step S25 can determine whether there are any unmarked items among the items belonging to the same item category acquired based on image data at other moments. If there are unmarked items, i.e., the item did not exist at the first moment, then in step S26, the item is marked as a newly added item, and the other moments associated with the item are stored. Here, the feature information and / or location information of the newly added item, as well as the associated other moments, can be stored, and the other moments are regarded as the initial moments of the newly added item. Based on the subsequently acquired image data about the item, the storage time of the item can be determined, thereby identifying the freshness of the item. In this way, newly added items can be quickly and accurately identified, and the storage time of the item can be updated in a timely manner.
[0074] Figure 4 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown. The following only describes the process... Figure 2 The differences between the embodiments shown are omitted for brevity, and the same steps will not be repeated.
[0075] like Figure 4 As shown, step S2 may further include steps S27 and S28. After calculating the storage time of the item, in step S27, it can be determined whether the positional deviation between the first position information of the item at a first moment and other position information at one or more other moments exceeds a preset positional deviation threshold. If the deviation exceeds the preset positional deviation threshold, in step S28, the item is marked as a moved item—that is, the item's position in the storage space of the refrigeration device 1 has moved—and the storage position of the item is updated with other position information of the item. If the deviation does not exceed the preset positional deviation threshold, step S28 is skipped directly. In this way, items whose positions have changed can be marked quickly and timely, and the position information of the corresponding items can be updated, avoiding interference from item position changes on item identification, thereby laying the foundation for the execution of various functions of the refrigeration device 1, such as adjusting the operating parameters of the food preservation function according to the position changes of the items, indicating the current position of the items according to user needs, etc.
[0076] Figure 5 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown. The following only describes the process... Figure 1 The differences between the embodiments shown are omitted for brevity, and the same steps will not be repeated.
[0077] After obtaining the status of the items in the refrigeration device 1, the method may further include a notification step regarding the status of the items in the refrigeration device 1. For example... Figure 5 As shown, the notification step may include step S3. In step S3, notification information about the items may be output based at least on the determined status information of each item. Here, optical and / or acoustic notification information may be output to the user through the notification unit 13 (e.g., the display screen and / or voice system of the refrigeration device 1). The notification information particularly includes the status information of each item, namely, at least one of the following: the category of each item, the total quantity of items of the same category, the storage location of each item, the storage time of each item, and the freshness of each item. This is because the status of an item is correlated with the user's expected behavior towards that item, so that the user can take appropriate measures in a timely manner based on the current status information of the item, such as where to take the item, when to eat the item and / or how much of the item to eat, when to purchase the item and / or how much of the item to purchase, etc.
[0078] In addition, recommended available recipes can be determined and output based at least on the categories of each item contained in the status information of the item. Optionally, the user's eating habits or preferences can also be introduced as influencing factors of the recommended available recipes, so that the user is spared the tedious process of thinking or looking up recipes, and the user's experience in handling items is further improved.
[0079] Optionally, based on the status information of the items—especially the total quantity of items of the same category, the storage time of each item, and / or the freshness of each item—the system can determine and output prompts about the items that the user needs to purchase. For example, if the total quantity of a certain category of items is lower than a preset total quantity threshold, or if the freshness of a certain category of items is lower than a preset freshness threshold, the system can promptly remind the user to purchase the corresponding items to replenish the stock and avoid a shortage of items.
[0080] Figure 6 A flowchart illustrating a method for controlling a refrigeration device according to another exemplary embodiment of this application is shown. The following only describes the process... Figure 1 The differences between the embodiments shown are omitted for brevity, and the same steps will not be repeated.
[0081] The method may also include a preservation process for the items in the refrigeration equipment 1. For example... Figure 6As shown, the preservation process may include step S4. In step S4, the preservation parameters of the refrigeration equipment 1 for the storage area where the corresponding item is located are adjusted based at least on the determined state information of each item. Specifically, the preservation parameters of the refrigeration equipment 1 for the storage area where the corresponding item is located can be adjusted based at least on the determined category, storage time, and storage location of each item. These preservation parameters include, for example, humidity parameters, and / or temperature parameters, and / or oxygen content parameters of the storage area. This allows for the development of personalized preservation strategies adapted to the state information of the items, effectively slowing down the rate of freshness decline of the items in the refrigeration equipment 1.
[0082] In addition, it should be noted that the step numbers described herein do not necessarily represent the order of steps, but are merely a reference numeral. The order may be changed depending on the specific circumstances, as long as the technical objective of this application can be achieved.
[0083] Figure 7 A schematic block diagram of a refrigeration device according to an exemplary embodiment of this application is shown.
[0084] like Figure 7 As shown, the refrigeration device 1 may include the following components:
[0085] - An image acquisition unit 11, configured to acquire image data of an item to be detected in the refrigeration device 1 at various times, the image acquisition unit 11 being configured, for example, as cameras arranged at multiple locations within the storage space of the refrigeration device 1; and
[0086] - Control unit 12, which is configured to implement the method according to this application.
[0087] Optionally, the cooling device 1 may further include a notification unit 13 configured to output notification information about the items based at least on the determined status information of each item. The notification unit 13 may be configured, for example, as a display screen of the cooling device 1 for outputting optical notification information and / or a voice system for outputting acoustic notification information.
[0088] Optionally, the refrigeration equipment 1 may further include a preservation treatment unit 14, which is configured to preserve items in various storage areas of the refrigeration equipment 1, such as adjusting preservation treatment parameters such as humidity parameters, and / or temperature parameters, and / or oxygen content parameters of the storage areas.
[0089] Optionally, the refrigeration device 1 is particularly a refrigerator.
[0090] If an embodiment includes an "and / or" association between a first feature and a second feature, it should be interpreted as follows: according to one implementation, the embodiment has not only the first feature but also the second feature; according to another implementation, the embodiment has either only the first feature or only the second feature.
[0091] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of this application, even when only a single embodiment is described with respect to a particular feature. The feature examples provided in this application are intended for illustrative purposes and not for limitation, unless otherwise stated. In practice, multiple features may be combined with each other as needed and where technically feasible. Various substitutions, modifications, and alterations are also conceived without departing from the spirit and scope of this application.
Claims
1. A method for controlling a refrigeration device (1), the method comprising a step of acquiring the state of an item in the refrigeration device (1), the acquisition step comprising: Step S1: Based on the pre-given item category information and image data of the items to be detected in the refrigeration equipment (1) at various times, obtain the feature information and / or location information of each item belonging to each item category at each time; and Step S2: Determine the state information of each item to be detected based on the feature information and / or location information of each item belonging to the same item category at each time.
2. The method according to claim 1, wherein, The status information of the items includes at least one of the following: category of each item, total quantity of items of the same category, storage location of each item, storage time of each item, and freshness of each item. The total quantity of the items includes the total number, total volume, and / or total weight of the items.
3. The method according to claim 2, wherein, The item category information and image data of the items to be detected at each time step are input into a multimodal pre-trained large model, thereby determining the category of each item to be detected at each time step and the total number of items of the same category, and / or obtaining the location information and / or image information of each item belonging to each item category at each time step, and obtaining the feature information of each item at each time step based on the image information of each item at each time step.
4. The method according to claim 3, wherein, Based on the image information of each item at each time, the feature information of each item at each time is obtained by using a convolutional neural network.
5. The method according to any one of claims 1 to 4, wherein, Step S2 includes: Step S21: Calculate the similarity between the first feature information of each item belonging to the same item category at the first time and other feature information at one or more other times, wherein the other times are different from the first time; Step S22: Determine whether the similarity exceeds a preset similarity threshold; Step S23: If the similarity exceeds a preset similarity threshold, then the item associated with the first feature information obtained based on image data at the first time point and the item associated with the other feature information obtained based on image data at other times points are marked as the same item, and the storage time of the item is determined based on the first time point and the other times points; and Step S24: If the similarity does not exceed the preset similarity threshold, then the item belonging to the first feature information obtained based on the image data at the first moment is marked as an item that has been taken out; In this process, steps S21 to S24 are executed repeatedly until all items belonging to the same item category obtained based on the image data at the first moment have been marked.
6. The method according to claim 5, wherein, Step S2 further includes: Step S25: After all items belonging to the same item category acquired based on the image data at the first moment have been marked, determine whether there are any unmarked items among the items belonging to the same item category acquired based on image data at other moments; and Step S26: If there is an unmarked item, mark the item as a newly placed item and store the other time associated with the newly placed item.
7. The method according to any one of claims 1 to 4, wherein, The feature information of the item includes a first feature value vector of the item at a first time moment and other feature value vectors of the item at one or more other times moment moment, wherein the other times moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment moment 1 feature value vector of the item at a first time moment ...
8. The method according to claim 7, wherein, The cosine distance between the first feature vector of the item and other feature vectors represents the similarity between the first feature information of the item and other feature information. If the cosine distance is less than a preset distance threshold, the similarity exceeds the preset similarity threshold; if the cosine distance is greater than or equal to the preset distance threshold, the similarity does not exceed the preset similarity threshold.
9. The method according to claim 5, wherein, Step S2 further includes: Step S27: After calculating the storage time of the item, determine whether the positional deviation between the first position information of the item at a first moment and other position information at one or more other moments exceeds a preset positional deviation threshold; and Step S28: If the deviation exceeds a preset position deviation threshold, the item is marked as a moved item, and the storage location of the item is updated with other position information of the item.
10. The method according to any one of claims 1 to 4, wherein, The method further includes a notification step of the status of the items in the refrigeration equipment (1), the notification step including: Step S3: Output notification information about the items based at least on the determined status information of each item.
11. The method according to claim 10, wherein, The notification information sent includes status information for each item, and / or recommended available recipes for each item category based at least on the status information of the items, and / or prompts about items that the user needs to purchase.
12. The method according to any one of claims 1 to 4, wherein, The method further includes a preservation process for the items in the refrigeration equipment (1), the preservation process including: Step S4: Adjust the preservation parameters of the refrigeration equipment (1) for the storage area where the corresponding item is located, based at least on the determined status information of each item.
13. The method according to claim 12, wherein, The refrigeration equipment (1) adjusts the preservation parameters of the storage area where the corresponding item is located based at least on the determined category, storage time and storage location of each item, wherein the preservation parameters include the humidity parameters, and / or temperature parameters, and / or oxygen content parameters of the storage area.
14. A refrigeration device (1), the refrigeration device (1) comprising the following components: An image acquisition unit (11) is configured to acquire image data of the item to be detected in the refrigeration device (1) at various times; and Control unit (12) configured to implement the method according to any one of claims 1 to 13.
15. The refrigeration equipment (1) according to claim 14, wherein, The refrigeration equipment (1) also includes: Notification unit (13), configured to output notification information about the items based at least on the determined status information of each item; and / or A preservation unit (14) is configured to preserve items in the various storage areas of the refrigeration equipment (1).
16. The refrigeration equipment (1) according to claim 14 or 15, wherein, The refrigeration equipment (1) is a refrigerator.
17. A computer program product, such as a computer-readable program carrier, comprising or storing computer program instructions that, when executed by a processor, at least partially implement the steps of the method according to any one of claims 1 to 13.