Food material management method and intelligent refrigerator

Abstract

Embodiments of the present application provide a food material management method and a smart refrigerator. The method comprises: after determining identification information of a plurality of identification items of a target food material in the refrigerator, notifying the identification information of the plurality of identification items through a preset notification mode; determining target information of the target food material based on identification information of a second identification item and / or user modification information of a first identification item; wherein the first identification item is an identification item corresponding to user modification information, and the second identification item is an identification item other than the first identification item among the plurality of identification items. In the above method, the modified user modification information can accurately represent the identification result of the corresponding first identification item, and the confirmed identification information can also accurately represent the identification result of the corresponding second identification item. Therefore, based on these two parts of information, the food material can be accurately managed.

Description

Technical Field

[0001] This application relates to the field of smart home appliance technology, and in particular to a food management method and a smart refrigerator. Background Technology

[0002] With the rapid development of technology and the improvement of people's living standards, the application of household appliances is becoming increasingly widespread. Refrigerators, as one of the most frequently used household appliances, can keep food fresh and cool. By identifying relevant information about food placed in or taken out of the refrigerator, food can be intelligently managed.

[0003] Due to factors such as the strong obstruction of food packaging or the inconsistent storage and retrieval methods, the identification results of food-related information may not be accurate enough. Inaccurate food identification results may lead to the inability to accurately manage food in the future. Summary of the Invention

[0004] This application provides a food management method and a smart refrigerator for precise food management.

[0005] In a first aspect, embodiments of this application provide a method for managing food ingredients, the method comprising:

[0006] After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method;

[0007] Based on the identification information of the second identification item and / or the user modification information of the first identification item, the target information of the target ingredient is determined; wherein, the first identification item is the identification item corresponding to the user modification information, and the second identification item is the other identification items among the plurality of identification items besides the first identification item.

[0008] The above solution notifies users of the identification information of multiple identification items through a preset notification method, so that users can promptly discover identification errors or unidentified identification information in the identification items and modify them. The modified information can accurately represent the identification result of the corresponding first identification item, and the confirmed identification information can also accurately represent the identification result of the corresponding second identification item. Therefore, based on these two parts of information, food ingredients can be managed accurately.

[0009] In some optional implementations, if the user modification information is received, the target information of the target ingredient is determined based on the identification information of the second identification item and / or the user modification information of the first identification item, including:

[0010] The user modification information of each first identification item and the identification information of each second identification item are determined as the target information.

[0011] In the above scheme, if user modification information is received, that is, if the first identification item is among multiple identification items, it means that the user has modified the identification information of the first identification item, while the identification information of the second identification item is considered not to be modified. Therefore, the user modification information of each first identification item and the identification information of each second identification item are used as the target information of the target ingredient.

[0012] In some optional implementations, before determining the user modification information of each first identification item and the identification information of each second identification item as the target information, the method further includes:

[0013] Determine whether the user-modified information is among the preset correct information or preset incorrect information in the first identification item;

[0014] Determining the target information as the user modification information of each first identification item and the identification information of each second identification item includes:

[0015] If the user modification information is among the preset correct information of the first identification item, then the user modification information of the first identification item and the identification information of the second identification item are determined as the target information; or

[0016] If the user modification information is in the preset error information of the first identification item, then the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information are determined as the target information; wherein, the target error type is determined based on the correspondence between the preset error information and the error type.

[0017] Before determining the user-modified information of the first identification item as the target information, the above scheme needs to first determine whether the user-modified information is valid. Based on this, by setting preset correct information and preset error information, if the user-modified information is in the preset correct information, it means that the user-modified information is correct and valid; if the user-modified information is in the preset error information, it means that the user-modified information is incorrect but valid. Both of these cases are considered valid modifications, and the user-modified information of the first identification item can be determined as the target information. In addition, by also determining the target error type of the user-modified information as the target information, it is convenient for the user to adjust the user-modified information based on the target information.

[0018] In some optional implementations, if the user modification information is in the preset error information of the first identification item, then before determining the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information as the target information, the method further includes:

[0019] Based on at least one of the following: the similarity between the identification information of the first identification item and the candidate information of the first identification item, the preset partition of the refrigerator, the preset user information, and the layout of existing food in the refrigerator, the target candidate information of the first identification item is determined.

[0020] The target information is determined by the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information, including:

[0021] The user modification information of the first identification item, the identification information of the second identification item, the target error type of the user modification information, and the target candidate information of the first identification item are determined as the target information.

[0022] The above solution addresses the scenario where user modifications to information are based on preset error messages in the first identification item. Although the user modifications are effective, they are not accurate. By predicting the target candidate information for the first identification item and using the target candidate information as the target information, it becomes easier for users to adjust their modified information later.

[0023] In some optional implementations, if the user modification information is not received, the target information of the target ingredient is determined based on the identification information of the second identification item and / or the user modification information of the first identification item, including:

[0024] The identification information of the plurality of identification items is determined as the target information.

[0025] If no user modification information is received in the above scheme, that is, if the first identification item is not among the multiple identification items, it means that the user has not modified the identification information. The identification information of the second identification item is considered to be unchanged. Therefore, the identification information of the multiple identification items is used as the target information of the target ingredient.

[0026] In some optional implementations, the method further includes:

[0027] If the identification information of the multiple identification items contains identification information indicating that it is not identified, and the cumulative number exceeds a preset number, a reminder will be issued through a preset reminder method; wherein, the cumulative number is the number of times that the user modification information has not been received for a continuous period of time, and the identification information of the multiple identification items contains identification information indicating that it is not identified.

[0028] In the above scheme, the identification information of the second identification item is considered to be unmodifiable, which may be because the user thinks it does not need to be modified or the user does not cooperate in making the modification. If there is identification information indicating that the identification information is not identified among multiple identification items, it means that the identification information needs to be modified. It can be determined that the user did not cooperate in making the modification because the modification information was not received. Based on this, it is necessary to count the cumulative number of times the user did not cooperate in making the modification. If the cumulative number is too high, the user needs to be reminded so as to better manage the ingredients in the future.

[0029] Secondly, embodiments of this application provide a smart refrigerator, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor performs the following:

[0030] After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method;

[0031] Based on the identification information of the second identification item and / or the user modification information of the first identification item, the target information of the target ingredient is determined; wherein, the first identification item is the identification item corresponding to the user modification information, and the second identification item is the other identification items among the plurality of identification items besides the first identification item.

[0032] In some optional implementations, if the user modification information is received, the processor specifically executes:

[0033] The user modification information of each first identification item and the identification information of each second identification item are determined as the target information.

[0034] In some optional implementations, before determining the user modification information of each first identification item and the identification information of each second identification item as the target information, the processor further performs:

[0035] Determine whether the user-modified information is among the preset correct information or preset incorrect information in the first identification item;

[0036] The processor specifically executes:

[0037] If the user modification information is among the preset correct information of the first identification item, then the user modification information of the first identification item and the identification information of the second identification item are determined as the target information; or

[0038] If the user modification information is in the preset error information of the first identification item, then the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information are determined as the target information; wherein, the target error type is determined based on the correspondence between the preset error information and the error type.

[0039] In some optional implementations, if the user modification information is in the preset error information of the first identification item, then before determining the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information as the target information, the processor further executes:

[0040] Based on at least one of the following: the similarity between the identification information of the first identification item and the candidate information of the first identification item, the preset partition of the refrigerator, the preset user information, and the layout of existing food in the refrigerator, the target candidate information of the first identification item is determined.

[0041] The processor specifically executes:

[0042] The user modification information of the first identification item, the identification information of the second identification item, the target error type of the user modification information, and the target candidate information of the first identification item are determined as the target information.

[0043] In some optional implementations, if the user modification information is not received, the processor specifically executes:

[0044] The identification information of the plurality of identification items is determined as the target information.

[0045] In some alternative implementations, the processor also performs:

[0046] If the identification information of the multiple identification items contains identification information indicating that it is not identified, and the cumulative number exceeds a preset number, a reminder will be issued through a preset reminder method; wherein, the cumulative number is the number of times that the user modification information has not been received for a continuous period of time, and the identification information of the multiple identification items contains identification information indicating that it is not identified.

[0047] Thirdly, embodiments of this application also provide a food ingredient management device, including:

[0048] The notification module is used to notify the recipient of the identification information of multiple identification items of the target food in the refrigerator through a preset notification method after determining the identification information of the multiple identification items.

[0049] The information determination module is used to determine the target information of the target ingredient based on the identification information of the second identification item and / or the user modification information of the first identification item; wherein, the first identification item is the identification item corresponding to the user modification information, and the second identification item is the other identification items among the plurality of identification items besides the first identification item.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a smart refrigerator. When the program is run on the smart refrigerator, it causes the smart refrigerator to perform any of the food management methods described in the first aspect above.

[0051] Furthermore, the technical effects of any of the implementation methods in aspects two to four can be found in the technical effects of different implementation methods in aspect one, and will not be repeated here. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0054] Figure 2 A refrigerator architecture diagram provided for an embodiment of this application;

[0055] Figure 3 A flowchart illustrating the first food ingredient management method provided in this application embodiment;

[0056] Figure 4 A flowchart illustrating the second food ingredient management method provided in this application embodiment;

[0057] Figure 5 A flowchart illustrating the third food ingredient management method provided in this application embodiment;

[0058] Figure 6 A flowchart illustrating the fourth food ingredient management method provided in this application embodiment;

[0059] Figure 7 This is a schematic diagram of the structure of the food ingredient management device provided in the embodiments of this application;

[0060] Figure 8 This is a schematic diagram of the structure of the smart refrigerator provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0063] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0064] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, it can refer to a direct connection, an indirect connection through an intermediate medium, or a connection within two devices. Those skilled in the art can understand the specific meaning of the above term in this application based on the specific circumstances.

[0065] Due to factors such as the strong obstruction of food packaging or the inconsistent storage and retrieval methods, the identification results of food-related information may not be accurate enough. Inaccurate food identification results may lead to the inability to accurately manage food in the future.

[0066] This application provides a food management method and a smart refrigerator for precise food management.

[0067] See Figure 1 The diagram shown is an application scenario illustration provided by some embodiments of this application, which includes a refrigerator 100 and a cloud 200.

[0068] The refrigerator 100 has functions such as food preservation, refrigeration, and food entry. This embodiment does not limit the specific type, size, or power of the refrigerator.

[0069] Refrigerator 100 also communicates with cloud 200 via various communication methods. This allows refrigerator 100 to connect via Local Area Network (LAN), Wireless Local Area Network (WLAN), and other networks. Cloud 200 can provide refrigerator 100 with various content and interactive features. For example, refrigerator 100 can send and receive information, such as receiving food detection data and sending image data. Cloud 200 includes one or more servers, which can be of one or more types. Functions such as image recognition are implemented through the servers.

[0070] In some embodiments, such as Figure 2 As shown, the refrigerator 100 includes a housing 110, a door 120, a camera 130, and a processor 140, wherein:

[0071] The casing 110 includes one or more storage units, each storage unit having an opening; a weight sensor is provided at the bottom of each storage unit to collect the weight of the food stored in the corresponding storage unit.

[0072] The door 120 is movably connected to the housing 110 and is used to cover the storage unit;

[0073] The shooting device 130 is connected to the housing 110 and is used to capture images.

[0074] In practical applications, when a user opens the refrigerator, they will store or take out food. Therefore, in order to facilitate the management of food, the processor 140 needs to control the shooting device 130 to start shooting images after the door 120 is opened.

[0075] The processor acquires images captured by the imaging device 130 and identifies the target image (an image containing food ingredients, or an image containing the barcode corresponding to the food ingredients). Taking an image containing food ingredients as an example:

[0076] Based on the 3D point cloud of the hand region in the depth image, the position of the hand's center of mass can be identified through a 3D algorithm. By tracking the hand's center of mass, the motion trajectory of the hand's center of mass and the position information of the hand's end can be determined. After determining that the hand region has entered the preset area, the presence of food in the hand can be determined based on the depth information of the hand's end region. If so, the corresponding image is used as a candidate image, and the region of interest (ROI) is selected from the candidate images as the target image.

[0077] The processor sends multiple consecutive frames of target images to the cloud 200, which then detects the food ingredients based on the target images.

[0078] It is understandable that some refrigerators have independent identification functions. For these refrigerators, the target image does not need to be sent to the cloud; instead, the food type can be identified through the built-in identification module. This embodiment does not limit the specific implementation method of food type identification, or it may obtain the food type through methods such as scanning a code.

[0079] In addition, the processor can determine whether the user wants to store or retrieve food based on the direction of the hand's center of gravity movement trajectory. Based on this, combined with the weight sensor and the hand's center of gravity movement trajectory, the processor identifies the location information of the food, i.e., which storage unit the food is stored in or retrieved from.

[0080] For example, when a weight sensor in a storage unit detects an increase in weight, it determines to store the food in the storage unit; when a weight sensor in a storage unit detects a decrease in weight, it determines to remove the food from the storage unit.

[0081] The shooting device 130 can be a color camera, such as a three-primary-color (Red Green Blue, RGB) mode camera; or it can be a monochrome camera.

[0082] The above Figure 2 The positions of the processor 140 and the camera device 130 are only illustrative examples, and the specific positions of the processor 140 and the camera device 130 can be set according to the refrigerator structure.

[0083] In some embodiments, in order to enable the shooting device to obtain a better shooting field of view while reducing the obstruction of the shooting field of view by objects such as refrigerators and human bodies, the shooting device 130 is disposed on the top of the housing 110, for example, the shooting device 130 is disposed at the top center of the housing 110.

[0084] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with reference to the accompanying drawings and specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0085] This application provides a first method for managing food ingredients, such as... Figure 3 As shown, it includes the following steps:

[0086] Step S301: After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method.

[0087] This embodiment does not specifically limit the multiple identification items of the target ingredient. Referring to the above embodiments, in practice, the type, location, production date, quantity, and weight of the ingredient can be identified. Based on this, the multiple identification items may include some or all of the type, location, and weight. In addition, besides the directly identifiable identification items mentioned above, the multiple identification items may also include indirect identification items (such as shelf life) determined based on the directly identifiable items.

[0088] Among the above-mentioned identification items, there may be identification errors or identification information that indicates that the identification items are not identified. Users need to assist in confirming or modifying the identification information of the above-mentioned identification items. Based on this, after determining the identification information of the multiple identification items of the target food in the refrigerator, this embodiment will notify the user through a preset notification method.

[0089] This embodiment does not specifically limit the preset notification method, for example:

[0090] 1) The refrigerator includes a voice playback device (such as a speaker) that plays voice information containing multiple recognition items.

[0091] 2) The refrigerator includes a display screen that displays identification information containing multiple identification items;

[0092] 3) The refrigerator includes a voice playback device and a display screen, wherein the voice playback device plays voice containing recognition information of multiple recognition items, and / or the display screen displays an interface containing recognition information of multiple recognition items.

[0093] The above-mentioned preset notification methods are merely illustrative examples. In actual applications, refrigerators may be equipped with other interactive devices, and corresponding notification methods can be used, which will not be elaborated here.

[0094] Step S302: Based on the identification information of the second identification item and / or the user modification information of the first identification item, determine the target information of the target ingredient.

[0095] Wherein, the first identification item is the identification item corresponding to user modification information, and the second identification item is the other identification item among the plurality of identification items besides the first identification item.

[0096] In this embodiment, after the identification information of multiple identification items is notified through a preset notification method, the user can promptly discover whether there are identification errors or unidentified identification information in the identification items, and then modify this information.

[0097] This embodiment does not limit the specific implementation method for determining the above-mentioned user-modified information, for example:

[0098] 1) The refrigerator includes a voice acquisition device (such as a microphone), through which the user records voice containing the user's modified information, including the first recognition item;

[0099] 2) The refrigerator includes a display screen. The user performs corresponding operations on the interface containing multiple identification items displayed on the display screen, triggering an instruction to modify the user information containing the first identification item.

[0100] 3) The refrigerator includes a voice acquisition device and a display screen. The user enters voice information containing the first identification item through the voice acquisition device, and / or the user performs corresponding operations on the interface containing multiple identification items displayed on the display screen to trigger the instruction containing the first identification item of the user modification information.

[0101] The above method for determining user-modified information is merely an example. In actual applications, refrigerators may be equipped with other interactive devices, and corresponding methods can be used to determine user-modified information, which will not be elaborated here.

[0102] In this embodiment, after determining the target information of the target ingredients, the target information can be stored in the ingredient database, supporting users to perform non-real-time interactions through the client on the terminal, such as querying the location of ingredients, recipe recommendations, and purchase reminders.

[0103] The above solution notifies users of the identification information of multiple identification items through a preset notification method, so that users can promptly discover identification errors or unidentified identification information in the identification items and modify them. The modified information can accurately represent the identification result of the corresponding first identification item, and the confirmed identification information can also accurately represent the identification result of the corresponding second identification item. Therefore, based on these two parts of information, food ingredients can be managed accurately.

[0104] During implementation, if user modification information is received, it means that the first identification item is among the multiple identification items; if user modification information is not received, it means that the first identification item is not among the multiple identification items.

[0105] Optionally, for scenarios where user-modified information is received, this application embodiment provides a second method for managing food ingredients, such as... Figure 4 As shown, it includes the following steps:

[0106] Step S401: After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method.

[0107] Step S401 can be referred to in the above embodiment, and will not be repeated here.

[0108] Step S402: Determine the user modification information of each first identification item and the identification information of each second identification item as the target information.

[0109] In the above scheme, if user modification information is received, that is, if the first identification item is among multiple identification items, it means that the user has modified the identification information of the first identification item, while the identification information of the second identification item is considered not to be modified. Therefore, the user modification information of each first identification item and the identification information of each second identification item are used as the target information of the target ingredient.

[0110] During implementation, users may trigger some invalid user modification information. Therefore, before determining the user modification information of the first identification item as the target information, it is necessary to first determine whether the above user modification information is a valid modification.

[0111] Optionally, for scenarios involving receiving user-modified information, this application provides a third method for managing food ingredients, such as... Figure 5 As shown, it includes the following steps:

[0112] Step S501: After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method.

[0113] Step S501 can be referred to in the above embodiment, and will not be repeated here.

[0114] Step S502: Determine whether the user-modified information is in the preset correct information or preset incorrect information of the first identification item.

[0115] Because users may trigger some invalid modifications, especially in voice interaction, users are prone to triggering some invalid modification commands when they are not guided.

[0116] Based on this, each identification item in this embodiment corresponds to preset correct information and preset incorrect information; the preset correct information includes multiple valid information indicating that the user has made correct modifications; the preset incorrect information includes multiple valid information indicating that the user has made incorrect modifications (such as unclear or unreasonable).

[0117] If the user modifies the information in the preset correct information, it means that the user's modified information is correct and valid, and the following step S503 is executed; if the user modifies the information in the preset incorrect information, it means that the user's modified information is incorrect and valid, and the following step S504 is executed.

[0118] Both of the above situations are considered valid modifications, and the user modification of the first identification item can be identified as the target information.

[0119] For example, if the first identification item is a category, the preset correct information includes all common food categories, such as radish, apple, tomato, etc.; the preset incorrect information includes all common unclear descriptions, such as vegetables, green vegetables, etc.

[0120] If the first identification item is location, the preset correct information includes all accurate descriptions of the refrigerator's location, such as the first shelf; the preset incorrect information includes all common inaccurate descriptions of location, such as above, left, etc.

[0121] If the first identification item is weight, the default correct information includes all reasonable weight ranges, such as 0.1 to 10 kg; the default incorrect information includes all unreasonable weight ranges, such as greater than 100 kg.

[0122] If the first identification item is the shelf life, the default correct information includes all reasonable durations, such as within 365 days; the default incorrect information includes all unreasonable durations, such as 5 years.

[0123] The preset correct information and preset incorrect information of the first identification item mentioned above are merely illustrative examples, and this application is not limited thereto.

[0124] Step S503: If the user modification information is in the preset correct information of the first identification item, then the user modification information of the first identification item and the identification information of the second identification item are determined as the target information.

[0125] As described above, if the user's modified information falls within the preset correct information, it indicates that the user's modified information is correct and valid. Therefore, the user's modified information in the first identification item and the identification information in the second identification item can be directly identified as the target information. In other words, the user's modification is successful, and the correct information corresponding to multiple identification items is obtained.

[0126] Step S504: If the user modification information is in the preset error information of the first identification item, then the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information are determined as the target information.

[0127] The target error type is determined based on the correspondence between the preset error information and the error type.

[0128] As mentioned above, if the user's modified information appears in the preset error messages, it means that the modified information is both erroneous and valid. That is, whether the user's modification failed or there is an error corresponding to a specific item, it is necessary to determine the target error type of the user's modified information. This will allow the user to adjust their modified information based on the target error type when viewing the target information later, resulting in more accurate target information.

[0129] In this embodiment, there is a pre-defined correspondence between each identification item. The correspondence includes the association between the pre-defined error information of the identification item and the error type. Based on the correspondence of the identification item, the target error type corresponding to the user's modified information can be accurately determined.

[0130] Before determining the user-modified information of the first identification item as the target information, the above scheme needs to first determine whether the user-modified information is valid. Based on this, by setting preset correct information and preset error information, if the user-modified information is in the preset correct information, it means that the user-modified information is correct and valid; if the user-modified information is in the preset error information, it means that the user-modified information is incorrect but valid. Both of these cases are considered valid modifications, and the user-modified information of the first identification item can be determined as the target information. In addition, by also determining the target error type of the user-modified information as the target information, it is convenient for the user to adjust the user-modified information based on the target information.

[0131] In some optional implementations, for the scenario where the user modifies information in the preset error information of the first identification item, the following steps are further included before step S504:

[0132] Based on at least one of the following: the similarity between the identification information of the first identification item and the candidate information of the first identification item, the preset partition of the refrigerator, the preset user information, and the layout of existing food in the refrigerator, the target candidate information of the first identification item is determined.

[0133] As mentioned above, if the user's modified information is among the preset error messages, it means that the user's modified information is both incorrect and valid. That is, even if the user's modification fails, there are still information corresponding to certain identification items that are incorrect. Based on this, the correct information that these identification items might correspond to can be predicted, i.e., the target candidate information for the first identification item can be predicted.

[0134] This embodiment can determine the target candidate information of the first identification item in the following ways, but not limited to:

[0135] 1) Based on the similarity between the identification information of the first identification item and all candidate information, determine the N target candidate information that is most similar to the identification information;

[0136] For example, if the first identification item is a type and the identification information is tomato, other types of food (such as cherry tomatoes, apples, etc.) that are similar to tomatoes in terms of image features, weight of ingredients, market time, etc. are identified.

[0137] If the first identification item is a location and the identification information is a second layer, determine other locations similar to the second layer (such as the first and third layers).

[0138] In implementation, for different types of ingredients, algorithms such as Euclidean distance, Earth Mover's Distance (EMD), and Manhattan distance are used to determine the similarity between different types of ingredients based on the target images of those ingredients. For example, when a user stores ingredients, the similarity between the target image corresponding to the recognition information and the images of the ingredients pre-stored in the database is calculated; when a user retrieves ingredients, the similarity between the target images during the storage and retrieval processes is calculated.

[0139] For the location of ingredients, the closer the other locations are to the identified location, the higher the similarity between the other locations and the identified information.

[0140] 2) Determine target candidate information based on the refrigerator's preset partitions;

[0141] For example, the refrigerator has multiple partitions with different functions, which are suitable for storing food with different characteristics. If the first identification item is the type and the second identification item is the location, and the location identification information is located in the first partition, all types of food that can be placed in the first partition can be identified as the target candidate information of the first identification item. If the first identification item is the location and the second identification item is the type, and the type identification information is tomato, the location corresponding to the partition that can store tomatoes can be identified as the target candidate information of the first identification item.

[0142] 3) Determine target candidate information based on preset user information;

[0143] For example, if the preset user information represents the user's eating habits, and the first identification item is category, the second identification item is location, and the location identification information is the second layer, then other types of ingredients that conform to the user's eating habits and are frequently placed in the second layer will be identified as the target candidate information for the first identification item.

[0144] 4) Determine target candidate information based on the layout of existing food items in the refrigerator;

[0145] For example, this method is only applicable to scenarios where users take out food. If the first identification item is the type and the second identification item is the location, and the location identification information is the second layer, then the food already in the second layer of the refrigerator is identified as the target candidate information of the first identification item.

[0146] 5) Based on the similarity between the identification information of the first identification item and the candidate information of the first identification item (denoted as the first dimension), the preset partition of the refrigerator (denoted as the second dimension), and the preset user information (denoted as the third dimension), determine the score corresponding to each dimension, and select the M target candidate information with the highest scores.

[0147] Specifically, for the first dimension, the greater the similarity, the higher the score; for the second dimension, the same partition corresponds to a higher score, while different partitions correspond to a lower score; for the third dimension, the more it matches the user's dietary habits, the higher the score.

[0148] Correspondingly, step S504 above can be implemented in, but is not limited to, the following ways:

[0149] The user modification information of the first identification item, the identification information of the second identification item, the target error type of the user modification information, and the target candidate information of the first identification item are determined as the target information.

[0150] The above solution addresses the scenario where user modifications to information are based on preset error messages in the first identification item. Although the user modifications are effective, they are not accurate. By predicting the target candidate information for the first identification item and using the target candidate information as the target information, it becomes easier for users to adjust their modified information later.

[0151] Optionally, for scenarios where no user modification information is received, this application embodiment provides a fourth method for managing food ingredients, such as... Figure 6 As shown, it includes the following steps:

[0152] Step S601: After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method.

[0153] Step S601 can be referred to in the above embodiments, and will not be repeated here.

[0154] Step S602: Determine the identification information of the plurality of identification items as the target information.

[0155] If no user modification information is received in the above scheme, that is, if the first identification item is not among the multiple identification items, it means that the user has not modified the identification information. The identification information of the second identification item is considered to be unchanged. Therefore, the identification information of the multiple identification items is used as the target information of the target ingredient.

[0156] In some alternative implementations, in the above... Figure 6 In addition to the above embodiments, the following steps are also included:

[0157] If the identification information of the multiple identification items contains identification information indicating that it is not identified, and the cumulative number exceeds a preset number, a reminder will be issued through a preset reminder method; wherein, the cumulative number is the number of times that the user modification information has not been received for a continuous period of time, and the identification information of the multiple identification items contains identification information indicating that it is not identified.

[0158] As mentioned above, the identification information for the second identification item is considered to be unmodifiable. For example, if there is no identification information indicating unrecognized items among the identification information of multiple identification items, it means that all identification items have been successfully identified, which is likely because all identification information is correct and the user believes that no modification is needed. Therefore, no reminder is required in this scenario.

[0159] If there are identification information indicating that a particular identification item was not identified among multiple identification items, it means that the identification of that item has failed and the identification information must be modified. The reason why the modification information has not been received from the user is because the user is not cooperating with the modification. It is necessary to count the cumulative number of times the user does not cooperate with the modification. If the cumulative number is too high, the user should be reminded.

[0160] The preset number of times mentioned above can be set according to the actual application scenario, and this embodiment does not make a specific limitation on it.

[0161] This embodiment does not specifically limit the above-mentioned preset reminder method, but provides an example:

[0162] 1) The refrigerator includes a voice playback device, which plays voice prompts to indicate reminders;

[0163] 2) The refrigerator includes a display screen that shows an interface representing reminders;

[0164] 3) The refrigerator includes a voice playback device and a display screen, which plays a voice indicating a reminder and / or displays an interface indicating a reminder on the display screen;

[0165] 4) The refrigerator includes a reminder device (such as an alarm clock, flashlight, etc.). Activate the reminder device.

[0166] The above-mentioned preset reminder methods are only illustrative examples. In actual applications, refrigerators may be equipped with other interactive devices, and corresponding reminder methods can be used. These will not be elaborated here.

[0167] In the above scheme, the identification information of the second identification item is considered to be unmodifiable, which may be because the user thinks it does not need to be modified or the user does not cooperate in making the modification. If there is identification information indicating that the identification information is not identified among multiple identification items, it means that the identification information needs to be modified. It can be determined that the user did not cooperate in making the modification because the modification information was not received. Based on this, it is necessary to count the cumulative number of times the user did not cooperate in making the modification. If the cumulative number is too high, the user needs to be reminded so as to better manage the ingredients in the future.

[0168] The above management method can be applied not only to food management in smart refrigerators, but also to the management of other items in other smart storage devices (such as smart vending machines, smart wine cabinets, smart wardrobes, and express delivery lockers). The specific implementation method can be referred to the above embodiments, which will not be repeated here.

[0169] Based on the same inventive concept, this application provides a food ingredient management device, see reference. Figure 7 As shown, the food ingredient management device 700 includes:

[0170] The notification module 701 is used to notify the identification information of multiple identification items of the target food in the refrigerator through a preset notification method after determining the identification information of the multiple identification items.

[0171] The information determination module 702 is used to determine the target information of the target ingredient based on the identification information of the second identification item and / or the user modification information of the first identification item; wherein, the first identification item is the identification item corresponding to the user modification information, and the second identification item is the other identification items among the plurality of identification items besides the first identification item.

[0172] In some optional implementations, if the user modification information is received, the information determination module 702 is specifically used for:

[0173] The user modification information of each first identification item and the identification information of each second identification item are determined as the target information.

[0174] In some optional implementations, before the information determination module 702 determines the user modification information of each first identification item and the identification information of each second identification item as the target information, it is further configured to:

[0175] Determine whether the user-modified information is among the preset correct information or preset incorrect information in the first identification item;

[0176] The information determination module 702 is specifically used for:

[0177] If the user modification information is among the preset correct information of the first identification item, then the user modification information of the first identification item and the identification information of the second identification item are determined as the target information; or

[0178] If the user modification information is in the preset error information of the first identification item, then the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information are determined as the target information; wherein, the target error type is determined based on the correspondence between the preset error information and the error type.

[0179] In some optional implementations, if the user modification information is in the preset error information of the first identification item, the information determination module 702 is further configured to: Before determining the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information as the target information, the information determination module 702 is further configured to:

[0180] Based on at least one of the following: the similarity between the identification information of the first identification item and the candidate information of the first identification item, the preset partition of the refrigerator, the preset user information, and the layout of existing food in the refrigerator, the target candidate information of the first identification item is determined.

[0181] The information determination module 702 is specifically used for:

[0182] The user modification information of the first identification item, the identification information of the second identification item, the target error type of the user modification information, and the target candidate information of the first identification item are determined as the target information.

[0183] In some optional implementations, if the user modification information is not received, the information determination module 702 is specifically used for:

[0184] The identification information of the plurality of identification items is determined as the target information.

[0185] In some optional implementations, the information determination module 702 is further configured to:

[0186] If the identification information of the multiple identification items contains identification information indicating that it is not identified, and the cumulative number exceeds a preset number, a reminder will be issued through a preset reminder method; wherein, the cumulative number is the number of times that the user modification information has not been received for a continuous period of time, and the identification information of the multiple identification items contains identification information indicating that it is not identified.

[0187] Since this device is the same as the device in the method of this application embodiment, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0188] Based on the same technical concept, this application also provides a smart refrigerator 800, such as... Figure 8 As shown, it includes at least one processor 801 and a memory 802 connected to at least one processor. In this embodiment, the specific connection medium between the processor 801 and the memory 802 is not limited. Figure 8 Taking the connection between the processor 801 and the memory 802 via bus 803 as an example, the bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0189] The processor 801 serves as the control center of the smart refrigerator. It connects to various parts of the refrigerator via various interfaces and lines, and performs data processing by running or executing instructions stored in the memory 802 and accessing data stored in the memory 802. Optionally, the processor 801 may include one or more processing units. The processor 801 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles issuing instructions. It is understood that the modem processor may not be integrated into the processor 801. In some embodiments, the processor 801 and the memory 802 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0190] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the food ingredient management method can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0191] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 802 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0192] In this embodiment, the memory 802 stores a computer program, which, when executed by the processor 801, causes the processor 801 to perform the following:

[0193] After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method;

[0194] Based on the identification information of the second identification item and / or the user modification information of the first identification item, the target information of the target ingredient is determined; wherein, the first identification item is the identification item corresponding to the user modification information, and the second identification item is the other identification items among the plurality of identification items besides the first identification item.

[0195] In some optional implementations, if the user modification information is received, the processor 801 specifically executes:

[0196] The user modification information of each first identification item and the identification information of each second identification item are determined as the target information.

[0197] In some optional implementations, before determining the user modification information of each first identification item and the identification information of each second identification item as the target information, the processor 801 further performs:

[0198] Determine whether the user-modified information is among the preset correct information or preset incorrect information in the first identification item;

[0199] The processor 801 specifically executes:

[0200] If the user modification information is among the preset correct information of the first identification item, then the user modification information of the first identification item and the identification information of the second identification item are determined as the target information; or

[0201] If the user modification information is in the preset error information of the first identification item, then the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information are determined as the target information; wherein, the target error type is determined based on the correspondence between the preset error information and the error type.

[0202] In some optional implementations, if the user modification information is in the preset error information of the first identification item, then before determining the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information as the target information, the processor 801 further executes:

[0203] Based on at least one of the following: the similarity between the identification information of the first identification item and the candidate information of the first identification item, the preset partition of the refrigerator, the preset user information, and the layout of existing food in the refrigerator, the target candidate information of the first identification item is determined.

[0204] The processor 801 specifically executes:

[0205] The user modification information of the first identification item, the identification information of the second identification item, the target error type of the user modification information, and the target candidate information of the first identification item are determined as the target information.

[0206] In some optional implementations, if the user modification information is not received, the processor 801 specifically executes:

[0207] The identification information of the plurality of identification items is determined as the target information.

[0208] In some alternative implementations, the processor 801 also performs:

[0209] If the identification information of the multiple identification items contains identification information indicating that it is not identified, and the cumulative number exceeds a preset number, a reminder will be issued through a preset reminder method; wherein, the cumulative number is the number of times that the user modification information has not been received for a continuous period of time, and the identification information of the multiple identification items contains identification information indicating that it is not identified.

[0210] Since the smart refrigerator is the same as the smart refrigerator in the method of this application embodiment, and the principle of the smart refrigerator in solving the problem is similar to that of the method, the implementation of the smart refrigerator can refer to the implementation of the method, and the repeated parts will not be described again.

[0211] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer program executable by a smart refrigerator. When the program is run on the smart refrigerator, it causes the smart refrigerator to perform the steps of the above-described food management method.

[0212] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0214] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0215] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0216] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0217] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for managing food ingredients, characterized in that, The method includes: After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method; Based on the identification information of the second identification item and / or the user modification information of the first identification item, the target information of the target ingredient is determined; wherein, the first identification item is the identification item corresponding to the user modification information, and the second identification item is the other identification items among the plurality of identification items besides the first identification item; If the user modification information is received, then based on the identification information of the second identification item and / or the user modification information of the first identification item, the target information of the target ingredient is determined, including: The user modification information of each first identification item and the identification information of each second identification item are determined as the target information; Before determining the user modification information of each first identification item and the identification information of each second identification item as the target information, the method further includes: Determine whether the user's modified information is among the preset correct information or preset incorrect information in the first identification item; wherein, the preset correct information includes multiple valid information indicating that the user's modification is correct; the preset incorrect information includes multiple valid information indicating that the user's modification is incorrect; Determining the target information as the user modification information of each first identification item and the identification information of each second identification item includes: If the user modification information is among the preset correct information of the first identification item, then the user modification information of the first identification item and the identification information of the second identification item are determined as the target information; or If the user modification information is in the preset error information of the first identification item, then the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information are determined as the target information; wherein, the target error type is determined based on the correspondence between the preset error information and the error type.

2. The method as described in claim 1, characterized in that, If the user modification information is in the preset error information of the first identification item, then before determining the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information as the target information, the method further includes: Based on at least one of the following: the similarity between the identification information of the first identification item and the candidate information of the first identification item, the preset partition of the refrigerator, the preset user information, and the layout of existing food in the refrigerator, the target candidate information of the first identification item is determined. The target information is determined by the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information, including: The user modification information of the first identification item, the identification information of the second identification item, the target error type of the user modification information, and the target candidate information of the first identification item are determined as the target information.

3. The method as described in claim 1, characterized in that, If no user modification information is received, then based on the identification information of the second identification item and / or the user modification information of the first identification item, the target information of the target ingredient is determined, including: The identification information of the plurality of identification items is determined as the target information.

4. The method as described in claim 3, characterized in that, The method further includes: If the identification information of the multiple identification items contains identification information indicating that it is not identified, and the cumulative number exceeds a preset number, a reminder will be issued through a preset reminder method; wherein, the cumulative number is the number of times that the user modification information has not been received for a continuous period of time, and the identification information of the multiple identification items contains identification information indicating that it is not identified.

5. A smart refrigerator, characterized in that, It includes at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the following: After determining the identification information of multiple identification items of the target food in the refrigerator, the identification information of the multiple identification items is notified through a preset notification method; Based on the identification information of the second identification item and / or the user modification information of the first identification item, the target information of the target ingredient is determined; wherein, the first identification item is the identification item corresponding to the user modification information, and the second identification item is the other identification items among the plurality of identification items besides the first identification item; If the user modification information is received, the processor will specifically execute: The user modification information of each first identification item and the identification information of each second identification item are determined as the target information; Before determining the user modification information of each first identification item and the identification information of each second identification item as the target information, the processor further executes: Determine whether the user's modified information is among the preset correct information or preset incorrect information in the first identification item; wherein, the preset correct information includes multiple valid information indicating that the user's modification is correct; the preset incorrect information includes multiple valid information indicating that the user's modification is incorrect; The processor specifically executes: If the user modification information is among the preset correct information of the first identification item, then the user modification information of the first identification item and the identification information of the second identification item are determined as the target information; or If the user modification information is in the preset error information of the first identification item, then the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information are determined as the target information; wherein, the target error type is determined based on the correspondence between the preset error information and the error type.

6. The intelligent refrigerator as described in claim 5, characterized in that, If the user modification information is in the preset error information of the first identification item, then before determining the user modification information of the first identification item, the identification information of the second identification item, and the target error type of the user modification information as the target information, the processor further executes: Based on at least one of the following: the similarity between the identification information of the first identification item and the candidate information of the first identification item, the preset partition of the refrigerator, the preset user information, and the layout of existing food in the refrigerator, the target candidate information of the first identification item is determined. The processor specifically executes: The user modification information of the first identification item, the identification information of the second identification item, the target error type of the user modification information, and the target candidate information of the first identification item are determined as the target information.

Images