Refrigerator food material placement recommendation method and controller

Abstract

The invention provides a refrigerator food material placement recommendation method and a controller, and relates to the technical field of kitchens. The method comprises the following steps: acquiring multi-layer live-action images, collected by an image collector, in the refrigerator; determining a food material name, a food material type and a food material state of each food material in the refrigerator by adopting a preset recognition model according to the multi-layer real-scene image in the refrigerator; determining the current position of each food material in the real-scene image according to the real-scene image and the layer where the real-scene image is located; according to the food material name, the food material type and the food material state of each food material and the layer label information of each layer in the refrigerator, the target layer of each food material is determined; and according to the current position of each food material and the target layer of each food material, food material placement recommendation information of the refrigerator is generated and pushed. Therefore, the food materials in the refrigerator are accurately recognized, the food material placement recommendation information is accurately generated, the food materials are prevented from being rotten, and fresh food materials are used in time.

Description

Technical Field

[0001] The present invention relates to the field of kitchen technology, and in particular to a method and a controller for recommending placing food in a refrigerator. Background Art

[0002] With the development of science and technology, kitchen equipment is becoming more and more intelligent. With the development of science and technology, people's diet is getting higher and higher, and the types of food are becoming more and more abundant, but the pace of life is getting faster and faster, so it is easy to develop various bad eating habits. In fact, everyone knows that most foods are better the fresher they are, not only with richer taste, but also with higher nutritional value. However, for many office workers, they are very busy at work every day, so they have no time to buy groceries every day, so refrigerators have become indispensable household appliances.

[0003] The existence of refrigerators has greatly extended the shelf life of food, so many people are accustomed to buying a large amount of food at one time and then storing it in the refrigerator. There are even many people who don’t know what things are suitable for putting in the refrigerator, so they will put everything in the refrigerator at once. However, they don’t know that some things can easily cause harm to the body if left in the refrigerator for too long.

[0004] There are often some leftovers in the refrigerator, and putting leftovers in the refrigerator will not only increase the power consumption of the refrigerator, but also cause an unpleasant smell inside the refrigerator. In fact, the refrigerator is not absolutely safe. There is no problem with keeping leftovers in the refrigerator for a short time, but once it exceeds 8 hours, it may breed a large amount of nitrite, and the longer the event, the more nitrite will be bred. If you continue to eat it, it will not only cause damage to the stomach and intestines, but also may cause cell diseases.

[0005] Therefore, there is an urgent need for a method for placing food in a refrigerator to remind users to promptly and reasonably handle the food. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the above-mentioned prior art and provide a refrigerator food placement recommendation method and controller to solve the problem of refrigerator food easily expiring in the prior art.

[0007] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0008] In a first aspect, an embodiment of the present application provides a method for recommending food placement in a refrigerator, which is applied to a controller in the refrigerator, wherein the refrigerator further includes an image collector, and the method includes:

[0009] Acquire the real scene images of multiple layers inside the refrigerator acquired by the image collector;

[0010] Using a preset recognition model, according to the multiple layers of real-life images inside the refrigerator, determine the name, type and state of each food inside the refrigerator;

[0011] Determine the current position of each food ingredient in the real scene image according to the real scene image and the layer where the real scene image is located;

[0012] Determine a target layer for each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient and the layer label information of each layer inside the refrigerator;

[0013] According to the current position of each ingredient and the target layer of each ingredient, recommended information on placing ingredients in the refrigerator is generated and pushed.

[0014] Optionally, the using of a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes:

[0015] Using the preset recognition model, obtaining the color, shape and texture features of the object in the real scene image;

[0016] The ingredient name, ingredient type and ingredient status of each ingredient in the refrigerator are determined according to the object color, object shape and object texture features in the real scene image.

[0017] Optionally, determining the target layer of each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient and the layer label information of each layer inside the refrigerator includes:

[0018] According to the ingredient name, ingredient type and ingredient state of each ingredient, a layer with the highest similarity to each ingredient is determined as the target layer of each ingredient.

[0019] Optionally, after determining the target layer of each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient and the layer label information of each layer inside the refrigerator, the method further includes:

[0020] According to the ingredient name, ingredient type and ingredient status of each ingredient and in accordance with a preset sorting rule, determine the target position of each ingredient in the target layer;

[0021] The generating and pushing the recommended information of placing the food in the refrigerator according to the current position of each food and the target layer of each food includes:

[0022] According to the current position of each ingredient and the target layer and target position of each ingredient, recommended information on placing ingredients in the refrigerator is generated and pushed.

[0023] Optionally, the refrigerator further includes: a humidity temperature sensor. Before using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the real-life images of multiple layers inside the refrigerator, the method further includes:

[0024] Acquire humidity information and temperature information of multiple layers inside the refrigerator collected by the humidity and temperature sensor;

[0025] The method of using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes:

[0026] The preset recognition model is used to determine the ingredient name, ingredient type and ingredient status of each ingredient inside the refrigerator based on the real-life images, humidity information and temperature information of the multiple layers inside the refrigerator.

[0027] Optionally, the refrigerator further includes: a gas sensor, and before using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the real-life images of multiple layers inside the refrigerator, the method further includes:

[0028] Acquire gas information of multiple layers inside the refrigerator collected by the gas sensor;

[0029] The method of using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes:

[0030] The preset recognition model is used to determine the ingredient name, ingredient type and ingredient status of each ingredient inside the refrigerator based on the real-life images of the multiple layers inside the refrigerator and the gas information.

[0031] Optionally, the refrigerator further includes: a weight sensor, and before using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the real-life images of multiple layers inside the refrigerator, the method further includes:

[0032] Obtaining weight information of multiple layers inside the refrigerator collected by the weight sensor;

[0033] The method of using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes:

[0034] The preset recognition model is used to determine the ingredient name, ingredient type and ingredient status of each ingredient inside the refrigerator based on the real-life images and weight information of the multiple layers inside the refrigerator.

[0035] Optionally, the refrigerator further includes: a Hall sensor, and before generating and pushing the recommended information on placing ingredients in the refrigerator according to the current position of each ingredient and the target layer of each ingredient, the method further includes:

[0036] Acquire the door opening and closing information of the refrigerator collected by the Hall sensor;

[0037] Determining refrigerator usage time according to the refrigerator door opening and closing information;

[0038] The generating and pushing the recommended information of placing the food in the refrigerator according to the current position of each food and the target layer of each food includes:

[0039] According to the usage time of the refrigerator, the current location of each ingredient and the target layer of each ingredient, recommended information on the placement of ingredients in the refrigerator is generated and pushed.

[0040] Optionally, before generating and pushing the recommended information on placing ingredients in the refrigerator according to the current position of each ingredient and the target layer of each ingredient, the method further includes:

[0041] Determining whether each food in the refrigerator is expired according to the food name, food type and food status of each food in the refrigerator;

[0042] If the expired food exists in the refrigerator, the food expiration information of the refrigerator is generated and pushed according to the current location of the expired food.

[0043] In a second aspect, an embodiment of the present application provides a controller, comprising: a processor and a storage medium, wherein the processor and the storage medium are communicatively connected via a bus, the storage medium stores program instructions executable by the processor, and the processor calls the program stored in the storage medium to execute the steps of the refrigerator food placement recommendation method as described in any one of the first aspects.

[0044] Compared with the prior art, this application has the following beneficial effects:

[0045] The present application provides a method and controller for recommending the placement of ingredients in a refrigerator. The method obtains multiple layers of real-life images of the inside of a refrigerator collected by an image collector; uses a preset recognition model to determine the name, type and state of each ingredient in the refrigerator based on the multiple layers of real-life images in the refrigerator; determines the current position of each ingredient in the real-life image based on the real-life image and the layer where the real-life image is located; determines the target layer of each ingredient based on the name, type and state of each ingredient, as well as the layer label information of each layer in the refrigerator; generates and pushes recommended information on the placement of ingredients in the refrigerator based on the current position of each ingredient and the target layer of each ingredient. Thus, the ingredients in the refrigerator are accurately identified, and the recommended information on the placement of ingredients is accurately generated, so that the user can adjust the placement of ingredients in the refrigerator based on the recommended information on the placement of ingredients, arrange them reasonably, prevent the ingredients from rotting, and use fresh ingredients in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A refrigerator provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of a method for recommending refrigerator food placement provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of a method for identifying the name, type and status of an ingredient provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a process for accurately placing food provided in an embodiment of the present application;

[0051] Figure 5 A flowchart of a method for determining food information provided in an embodiment of the present application;

[0052] Figure 6 A flowchart of another method for determining food information provided in an embodiment of the present application;

[0053] Figure 7 A flowchart of another method for determining food information provided in an embodiment of the present application;

[0054] Figure 8A flowchart of a method for pushing food placement recommendation information provided in an embodiment of the present application;

[0055] Fig. 9 A flowchart of another method for pushing food placement recommendation information provided in an embodiment of the present application;

[0056] Fig.10 A schematic diagram of a refrigerator food placement recommendation device provided in an embodiment of the present application;

[0057] Fig.11 A schematic diagram of a controller provided in an embodiment of the present application.

[0058] Icons: 1-controller, 2-image collector, 3-humidity and temperature sensor, 4-gas sensor, 5-weight sensor, 6-Hall sensor, 1001-acquisition module, 1002-first determination module, 1003-second determination module, 1004-third determination module, 1005-generation module, 1101-processor, 1102-storage medium. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0061] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0062] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0063] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0064] Before introducing a method for recommending the placement of food in a refrigerator provided in an embodiment of the present application, a refrigerator to which the method for recommending the placement of food in a refrigerator is applied is first introduced. Figure 1 A refrigerator is provided in an embodiment of the present application.

[0065] like Figure 1 As shown, the refrigerator includes: a controller 1 and an image collector 2. The controller 1 is connected to the image collector 2. The image collector 2 is installed inside the refrigerator and is used to photograph the food inside the refrigerator. For example, the image collector 2 can be installed according to the internal structure of the refrigerator, and it is required to be able to capture images at any position inside the refrigerator. For example, an image collector 2 is set on each layer of the refrigerator, or an image collector 2 is set at the top four corners of each layer of the refrigerator.

[0066] Further, continue to refer to Figure 1 In the subsequent embodiments, the refrigerator further includes: a humidity temperature sensor 3, a gas sensor 4, a weight sensor 5, and a Hall sensor 6. The humidity temperature sensor 3, the gas sensor 4, the weight sensor 5, and the Hall sensor 6 are all connected to the controller 1.

[0067] The following is an explanation of a recommended method for placing food in a refrigerator provided in an embodiment of the present application through a specific example. Figure 2 A flowchart of a method for recommending refrigerator food placement is provided in an embodiment of the present application. The method is executed by a controller having a computing and processing function.

[0068] like Figure 2 As shown, the method includes:

[0069] S101, obtaining real-life images of multiple layers inside a refrigerator collected by an image collector.

[0070] The image collector collects multiple layers of real-life images of the interior of the refrigerator at preset time intervals. For example, the preset time interval may be 12 hours or 24 hours.

[0071] The image collector transmits the acquired real-scene image to the controller.

[0072] S102: Using a preset recognition model, according to the multi-layer real-life image of the interior of the refrigerator, determine the name, type and state of each ingredient in the refrigerator.

[0073] The preset recognition model is a pre-trained model for identifying ingredients. The real scene image is input into the preset recognition model, and the name, type and state of each ingredient in the real scene image are output. For example, the name of the ingredient is apple, the type of ingredient is apple, and the state of the ingredient is cooked.

[0074] For example, the preset recognition model is a convolutional neural network model. To improve the accuracy of image recognition. By training a large number of food images, the neural network can learn and recognize various types of food, and can achieve good recognition results even when the food is randomly placed and partially blocked. The training process of the preset recognition model is similar to the existing neural network model training process, which will not be repeated here.

[0075] The preset recognition model is essentially a deep learning algorithm that learns from a database of millions of item images. The pattern recognition system can not only recognize existing foods, but also introduce new item categories based on the characteristics of these existing items (such as milk cartons, ketchup bottles, pickle jars, etc.). If no corresponding matching items are found in the local database, the extracted feature values ​​will be uploaded to the cloud server for further recognition and matching. Cloud recognition relies on more powerful computing resources and a richer database to improve the accuracy of food recognition.

[0076] S103: Determine the current position of each food ingredient in the real scene image according to the real scene image and the layer where the real scene image is located.

[0077] According to the collected real-life image, the horizontal and vertical coordinate values ​​of the food ingredients in the functional space are marked and determined. Specifically, the pixel points on the real-life image are collected to determine the two-dimensional coordinates of each food ingredient. Then, the current position of each food ingredient in the real-life image is determined in combination with the layer where the real-life image is located. That is, according to the current position of each food ingredient, the layer where each food ingredient is located can be determined, and then the food ingredient can be found according to the two-dimensional coordinates of the layer where the food ingredient is located, so that the user can find the food ingredient easily.

[0078] S104: Determine the target layer of each ingredient according to the ingredient name, ingredient type and ingredient status of each ingredient and the layer label information of each layer inside the refrigerator.

[0079] For example, the compartments inside the refrigerator are classified, and each layer has corresponding layer label information. For example, the classification includes fruit layer, vegetable layer, meat layer, and dairy layer, which helps to better manage and quickly access the ingredients. Perishable foods can also be placed in the coldest layer of the refrigerator: usually the temperature in the lower and back layers of the refrigerator is lower, which is suitable for storing perishable foods such as meat, dairy products, and seafood. This can extend the shelf life of the ingredients.

[0080] For example, other principles can also be considered to classify the compartments inside the refrigerator. For example, arrange them in order from top to bottom and from front to back: according to the first-in-first-out principle, place ingredients that are used less frequently or require a longer refrigeration time in places that are more difficult to access, such as the back or lower layer of the refrigerator. Frequently used ingredients are placed in easy-to-access locations, keeping commonly used ingredients close to the front for easy access. The refrigerator door is suitable for storing ingredients that are frequently opened, such as sauces, butter, beverages, etc. The layers with drawers can be used to store ingredients that need to maintain humidity, such as vegetables, fruits, and dairy products. Separate raw and cooked food. Raw meat often carries more germs. The cold air in the refrigerator sinks downward, so raw meat should be placed in the lower layer of the refrigerator. Cooked food and fruits need to be placed on the upper layer of the refrigerator to avoid cross-infection of germs on the lower layer.

[0081] The target layer of each ingredient is determined by comparing the ingredient name, ingredient type, and ingredient status of each ingredient, as well as the layer label information of each layer. For example, if the ingredient name is apple, the ingredient type is fruit, and the ingredient status is ripe, the target layer of the ingredient can be determined as the fruit layer. If the fruit layer has a ripe fruit layer and a late-ripening fruit layer, the target layer of the ingredient is determined as the ripe fruit layer.

[0082] Determine the target layer for each ingredient and place ingredients with the most similar classification and status on the same layer, which will help to better manage and quickly access the ingredients and delay the decay cycle of the ingredients.

[0083] S105: Generate and push recommended information on placing ingredients in the refrigerator based on the current location of each ingredient and the target layer of each ingredient.

[0084] The recommended information for placing ingredients includes both the current position of each ingredient and the target layer of each ingredient. When a user views the recommended information for placing ingredients, he or she can determine the current position and target layer of the ingredient based on the recommended information for placing ingredients, so as to adjust the placement of the ingredients.

[0085] For example, the food placement recommendation information can be pushed to the display screen of the refrigerator, and the food placement recommendation information can also be pushed to the user's client device (mobile phone), so that the user can view the food placement recommendation information in a timely manner.

[0086] In this way, the ingredients in the refrigerator can be accurately identified and recommended information on the placement of the ingredients can be accurately generated, making it convenient for users to adjust the placement of ingredients in the refrigerator based on the recommended information, arrange them reasonably, prevent the ingredients from rotting, and use fresh ingredients in a timely manner.

[0087] In summary, in this embodiment, the real-life images of multiple layers inside the refrigerator collected by the image collector are obtained; the preset recognition model is used to determine the name, type and state of each ingredient inside the refrigerator based on the real-life images of multiple layers inside the refrigerator; the current position of each ingredient in the real-life image is determined based on the real-life image and the layer where the real-life image is located; the target layer of each ingredient is determined based on the name, type and state of each ingredient, as well as the layer label information of each layer inside the refrigerator; and the recommended placement information of the ingredients in the refrigerator is generated and pushed based on the current position of each ingredient and the target layer of each ingredient. Thus, the ingredients in the refrigerator are accurately identified, and the recommended placement information of the ingredients is accurately generated, so that the user can adjust the placement of the ingredients in the refrigerator based on the recommended placement information of the ingredients, arrange them reasonably, prevent the ingredients from rotting, and use fresh ingredients in time.

[0088] In the above Figure 2 Based on the corresponding embodiments, the embodiments of the present application also provide a method for identifying the name, type and status of ingredients. Figure 3 A flowchart of a method for identifying the name, type and status of an ingredient provided in an embodiment of the present application. Figure 3 As shown, in S102, the preset recognition model is used to determine the name, type and state of each food in the refrigerator according to the real-life image of the multiple layers inside the refrigerator, including:

[0089] S201, using a preset recognition model to obtain object color, object shape and object texture features in a real scene image.

[0090] The preset recognition model processes the real-life image and can extract the color, shape, and texture features of the objects in the real-life image. These features are the basis for identifying the type and state of food. The preset recognition model can separate the target object from the complex background and perform accurate feature extraction on it.

[0091] For example, taking fruit as an example, the length, width, color, and texture features of the fruit in the real-life image are extracted to determine the circumference, area, and shape of the fruit.

[0092] S202: Determine the name, type and state of each ingredient in the refrigerator according to the object color, shape and texture features in the real scene image.

[0093] The extracted object color, object shape and object texture features are matched with the food information in the database to determine the name, type and status of each food in the refrigerator.

[0094] In summary, in this embodiment, a preset recognition model is used to obtain the color, shape and texture features of objects in the real scene image; based on the color, shape and texture features of objects in the real scene image, the name, type and state of each food in the refrigerator are determined. Thus, the name, type and state of the food are accurately determined.

[0095] In the above Figure 2 On the basis of the corresponding embodiment, in another embodiment of the present application, in S104, according to the ingredient name, ingredient type and ingredient state of each ingredient, and the layer label information of each layer inside the refrigerator, determining the target layer of each ingredient includes:

[0096] According to the ingredient name, ingredient type and ingredient state of each ingredient, the layer with the highest similarity to each ingredient is determined as the target layer of each ingredient.

[0097] For example, the ingredient name, ingredient type and ingredient state of the ingredient are used to generate an ingredient feature vector, and a label feature vector of the layer label information of each layer is generated. The similarity between each ingredient feature vector and multiple label feature vectors is calculated, and the layer corresponding to the label feature vector with the highest similarity is determined as the target layer.

[0098] In summary, in this embodiment, according to the name, type and state of each ingredient, the layer with the highest similarity to each ingredient is determined as the target layer of each ingredient. Thus, the target layer is accurately determined.

[0099] In the above Figure 2 On the basis of the corresponding embodiments, the embodiments of the present application also provide a method for accurately placing ingredients. Figure 4 A flow chart of a method for accurately placing food provided in an embodiment of the present application. Figure 4 As shown, after determining the target layer of each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient and the layer label information of each layer inside the refrigerator in S104, the method further includes:

[0100] S301. Determine a target position of each ingredient in a target layer according to the ingredient name, ingredient type and ingredient status of each ingredient and a preset sorting rule.

[0101] For example, the preset sorting rule may be to use the ingredients that are about to expire first in the order of shelf life from the nearest to the farthest, so as to reduce waste. The preset sorting rule may be to place the ingredients that are frequently used in an easily accessible place, and the ingredients that are not frequently used or used occasionally in an infrequently used area.

[0102] The target position is the position where the ingredients are placed in the target layer according to the preset sorting rules.

[0103] In S105, based on the current position of each ingredient and the target layer of each ingredient, the recommended information for placing ingredients in the refrigerator is generated and pushed, including:

[0104] S302: Generate and push recommended information on placing ingredients in the refrigerator based on the current location of each ingredient and the target layer and target location of each ingredient.

[0105] In summary, in this embodiment, according to the name, type and state of each ingredient, the target position of each ingredient in the target layer is determined according to the preset sorting rules; according to the current position of each ingredient and the target layer and target position of each ingredient, the recommended information for placing ingredients in the refrigerator is generated and pushed. Thus, the target position of the ingredients in the target layer is accurately determined, and the ingredients are accurately placed.

[0106] In the above Figure 2 Based on the corresponding embodiments, the embodiments of the present application also provide a method for determining food information. Figure 5 A flow chart of a method for determining food information provided in an embodiment of the present application. Figure 5 As shown, the refrigerator further includes: a humidity temperature sensor. Before using a preset recognition model in S102 to determine the name, type and state of each food in the refrigerator according to the real-life image of the multiple layers inside the refrigerator, the method further includes:

[0107] S401, obtaining humidity information and temperature information of multiple layers inside the refrigerator collected by a humidity and temperature sensor.

[0108] The humidity and temperature sensor collects the humidity and temperature information of multiple layers inside the refrigerator in real time, and transmits the humidity and temperature information to the controller to facilitate more accurate image recognition.

[0109] Different ingredients have different states at different temperatures and humidities. Obtaining humidity information and temperature information facilitates accurate identification of the ingredient state.

[0110] In S102, the preset recognition model is used to determine the name, type and state of each food in the refrigerator according to the multi-layer real-life image inside the refrigerator, including:

[0111] S402: Using a preset recognition model, according to the multiple layers of real-life images, humidity information, and temperature information inside the refrigerator, determine the name, type, and state of each ingredient in the refrigerator.

[0112] The introduction of humidity information and temperature information can more accurately determine the name, type and status of each ingredient in the refrigerator.

[0113] It should be noted that when pre-training the preset recognition model, humidity information and temperature information also need to be introduced. The training process of the preset recognition model introducing humidity information and temperature information is similar to the existing neural network model training process, which will not be repeated here.

[0114] In summary, in this embodiment, the refrigerator further includes: a humidity and temperature sensor to obtain humidity information and temperature information of multiple layers inside the refrigerator collected by the humidity and temperature sensor; and a preset recognition model to determine the name, type and state of each food in the refrigerator based on the real scene image, humidity information and temperature information of multiple layers inside the refrigerator. Thus, the name, type and state of each food in the refrigerator can be accurately determined.

[0115] In the above Figure 2 Based on the corresponding embodiment, the embodiment of the present application also provides another method for determining food information. Figure 6 A flowchart of another method for determining food information provided in an embodiment of the present application. Figure 6 As shown, the refrigerator further includes: a gas sensor. Before using a preset recognition model in S102 to determine the name, type and state of each food in the refrigerator according to the real-life images of multiple layers inside the refrigerator, the method further includes:

[0116] S501, obtaining gas information of multiple layers inside the refrigerator collected by the gas sensor.

[0117] The gas sensor collects multiple layers of gas information inside the refrigerator in real time and transmits the gas information to the controller to facilitate more accurate image recognition.

[0118] The smell of food is different in different food states. Obtaining gas information makes it easier to accurately identify the food state.

[0119] In S102, the preset recognition model is used to determine the name, type and state of each food in the refrigerator according to the multi-layer real-life image inside the refrigerator, including:

[0120] S502: Using a preset recognition model, according to the multi-layer real-life images and gas information inside the refrigerator, determine the name, type and status of each ingredient in the refrigerator.

[0121] The introduction of gas information can more accurately determine the name, type and status of each ingredient in the refrigerator.

[0122] It should be noted that when pre-training the preset recognition model, gas information also needs to be introduced. The training process of the preset recognition model introducing gas information is similar to the existing neural network model training process, which will not be repeated here.

[0123] In summary, in this embodiment, the refrigerator further includes: a gas sensor to obtain gas information of multiple layers inside the refrigerator collected by the gas sensor; and a preset recognition model to determine the name, type and state of each food in the refrigerator based on the real-life image and gas information of multiple layers inside the refrigerator. Thus, the name, type and state of each food in the refrigerator can be accurately determined.

[0124] In the above Figure 2 Based on the corresponding embodiment, the embodiment of the present application also provides another method for determining food information. Figure 7 A flowchart of another method for determining food information provided in an embodiment of the present application. Figure 7 As shown, the refrigerator further includes: a weight sensor. Before using a preset recognition model in S102 to determine the name, type and state of each food in the refrigerator according to the real-life image of the multiple layers inside the refrigerator, the method further includes:

[0125] S601. Obtain weight information of multiple layers inside the refrigerator collected by a weight sensor.

[0126] The weight sensor collects the weight information of multiple layers inside the refrigerator in real time and transmits the weight information to the controller to facilitate more accurate image recognition.

[0127] Different ingredients have different weight information. Obtaining weight information makes it easier to accurately identify the status of the ingredients.

[0128] In S102, the preset recognition model is used to determine the name, type and state of each food in the refrigerator according to the multi-layer real-life image inside the refrigerator, including:

[0129] S602: Using a preset recognition model, according to the real-life images and weight information of multiple layers inside the refrigerator, determine the name, type and status of each ingredient inside the refrigerator.

[0130] The introduction of weight information can more accurately determine the name, type and status of each ingredient in the refrigerator.

[0131] It should be noted that when pre-training the preset recognition model, weight information also needs to be introduced. The training process of the preset recognition model with weight information is similar to the existing neural network model training process, which will not be repeated here.

[0132] In summary, in this embodiment, the refrigerator further includes: a weight sensor, which obtains the weight information of multiple layers inside the refrigerator collected by the weight sensor; and a preset recognition model is used to determine the name, type and state of each food in the refrigerator based on the real-life image and weight information of multiple layers inside the refrigerator. Thus, the name, type and state of each food in the refrigerator can be accurately determined.

[0133] In the above Figure 2 On the basis of the corresponding embodiment, the embodiment of the present application also provides a method for pushing food placement recommendation information. Figure 8 A flowchart of a method for pushing food placement recommendation information provided in an embodiment of the present application. Figure 8 As shown, the refrigerator further includes: a Hall sensor. Before generating and pushing the recommended information of placing ingredients in the refrigerator according to the current position of each ingredient and the target layer of each ingredient in S105, the method further includes:

[0134] S701, obtaining the opening and closing information of the refrigerator door collected by the Hall sensor.

[0135] Door opening and closing information includes: door opening and closing time, door opening and closing frequency.

[0136] S702: Determine refrigerator usage time based on refrigerator door opening and closing information.

[0137] The user's refrigerator usage time can be determined based on the refrigerator door opening and closing information, for example, the refrigerator usage time is 18:00-23:00.

[0138] In S105, based on the current position of each ingredient and the target layer of each ingredient, the recommended information for placing ingredients in the refrigerator is generated and pushed, including:

[0139] S703: Generate and push recommended information on placing ingredients in the refrigerator based on the usage time of the refrigerator, the current location of each ingredient, and the target layer of each ingredient.

[0140] For example, the recommended information on placing ingredients in the refrigerator can be pushed within a preset time before the refrigerator is used, so that the user can view and adjust the placement of ingredients in the refrigerator in time. For example, the recommended information on placing ingredients in the refrigerator can be pushed within one hour before the refrigerator is used.

[0141] In summary, in this embodiment, the refrigerator further includes: a Hall sensor to obtain the door opening and closing information of the refrigerator collected by the Hall sensor; to determine the refrigerator usage time according to the door opening and closing information of the refrigerator; and to generate and push the recommended information of placing ingredients in the refrigerator according to the refrigerator usage time, the current position of each ingredient, and the target layer of each ingredient. Thus, it is convenient for the user to view and timely adjust the placement of ingredients in the refrigerator.

[0142] In the above Figure 2 On the basis of the corresponding embodiment, the embodiment of the present application also provides another method for pushing food placement recommendation information. Fig. 9 A flowchart of another method for pushing food placement recommendation information provided by an embodiment of the present application. Fig. 9 As shown, before generating and pushing the recommended information of placing ingredients in the refrigerator according to the current position of each ingredient and the target layer of each ingredient in S105, the method further includes:

[0143] S801. Determine whether each ingredient in the refrigerator is expired based on the ingredient name, ingredient type, and ingredient status of each ingredient in the refrigerator.

[0144] Each ingredient has its own expiration criteria. For example, the packaging of packaged food has a production date and expiration date, which can be obtained through image recognition. Fruits and vegetables are considered expired if they are rotten.

[0145] S802: If there are expired ingredients in the refrigerator, generate and push the expired ingredients information of the refrigerator according to the current location of the expired ingredients.

[0146] The expiration information of ingredients will be pushed and displayed to facilitate users to check and remove expired ingredients in time.

[0147] In summary, in this embodiment, whether each ingredient in the refrigerator is expired is determined based on the ingredient name, ingredient type, and ingredient status of each ingredient in the refrigerator; if there is expired ingredient in the refrigerator, the expired ingredient information of the refrigerator is generated and pushed based on the current location of the expired ingredient. Thus, it is convenient for users to check and remove expired ingredients in time.

[0148] The following is a description of a refrigerator food placement recommendation device, equipment, and storage medium provided by the present application for implementation. The specific implementation process and technical effects are described above and will not be repeated below.

[0149] Fig.10 A schematic diagram of a refrigerator food placement recommendation device provided in an embodiment of the present application, which is applied to a controller in a refrigerator, and the refrigerator also includes: an image collector, such as Fig.10 As shown, the device comprises:

[0150] The acquisition module 1001 acquires the real scene images of multiple layers inside the refrigerator acquired by the image collector.

[0151] The first determination module 1002 is used to determine the ingredient name, ingredient type and ingredient status of each ingredient in the refrigerator based on the multi-layer real-life image of the interior of the refrigerator using a preset recognition model.

[0152] The second determining module 1003 is used to determine the current position of each food in the real scene image according to the real scene image and the layer where the real scene image is located.

[0153] The third determination module 1004 is used to determine the target layer of each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient and the layer label information of each layer inside the refrigerator.

[0154] The generation module 1005 is used to generate and push recommended information on the placement of ingredients in the refrigerator according to the current location of each ingredient and the target layer of each ingredient.

[0155] Furthermore, the first determination module 1002 is specifically used to adopt a preset recognition model to obtain object color, object shape and object texture features in the real-scene image; and determine the ingredient name, ingredient type and ingredient status of each ingredient inside the refrigerator based on the object color, object shape and object texture features in the real-scene image.

[0156] Furthermore, the third determination module 1004 is specifically used to determine the layer with the highest similarity to each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient as the target layer of each ingredient.

[0157] Furthermore, the third determination module 1004 is specifically configured to determine a target position of each ingredient in the target layer according to the ingredient name, ingredient type and ingredient state of each ingredient and a preset sorting rule.

[0158] Furthermore, the generation module 1005 is specifically used to generate and push recommended information on the placement of ingredients in the refrigerator according to the current position of each ingredient and the target layer and target position of each ingredient.

[0159] Furthermore, the first determination module 1002, specifically used for the refrigerator, also includes: a humidity and temperature sensor, which obtains humidity information and temperature information of multiple layers inside the refrigerator collected by the humidity and temperature sensor; and uses a preset recognition model to determine the name, type and status of each ingredient inside the refrigerator based on the real-life image, humidity information and temperature information of multiple layers inside the refrigerator.

[0160] Furthermore, the first determination module 1002, specifically used for the refrigerator, also includes: a gas sensor to obtain gas information of multiple layers inside the refrigerator collected by the gas sensor; using a preset recognition model to determine the name, type and status of each ingredient inside the refrigerator based on the real-life images and gas information of multiple layers inside the refrigerator.

[0161] Furthermore, the first determination module 1002, specifically used for the refrigerator, also includes: a weight sensor, which obtains weight information of multiple layers inside the refrigerator collected by the weight sensor; and uses a preset recognition model to determine the ingredient name, ingredient type, and ingredient status of each ingredient inside the refrigerator based on the real-life images and weight information of multiple layers inside the refrigerator.

[0162] Furthermore, generation module 1005 is specifically used to obtain the opening and closing door information of the refrigerator collected by the Hall sensor; determine the refrigerator usage time based on the refrigerator opening and closing door information; generate and push the recommended information on the placement of ingredients in the refrigerator based on the refrigerator usage time, the current position of each ingredient and the target layer of each ingredient.

[0163] Furthermore, the generation module 1005 is specifically used to determine whether each ingredient in the refrigerator is expired based on the ingredient name, ingredient type and ingredient status of each ingredient in the refrigerator; if there is expired ingredient in the refrigerator, the ingredient expiration information of the refrigerator is generated and pushed based on the current location of the expired ingredient.

[0164] Fig.11 A schematic diagram of a controller provided in an embodiment of the present application, wherein the controller may be a device with computing and processing functions.

[0165] The controller includes: a processor 1101 and a storage medium 1102. The processor 1101 and the storage medium 1102 are connected via a bus.

[0166] The storage medium 1102 is used to store programs, and the processor 1101 calls the programs stored in the storage medium 1102 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0167] Optionally, the present invention also provides a storage medium, including a program, which is used to execute the above-mentioned method embodiment when executed by a processor. In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0170] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A method for recommending placing food in a refrigerator, characterized in that: The controller is applied to the refrigerator, the refrigerator further comprises: an image collector, and the method comprises: Acquire the real scene images of multiple layers inside the refrigerator acquired by the image collector; Using a preset recognition model, according to the multiple layers of real-life images inside the refrigerator, determine the name, type and state of each food inside the refrigerator; Determine the current position of each food ingredient in the real scene image according to the real scene image and the layer where the real scene image is located; Determine a target layer for each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient and the layer label information of each layer inside the refrigerator; According to the current position of each ingredient and the target layer of each ingredient, recommended information on placing ingredients in the refrigerator is generated and pushed.

2. The method according to claim 1, characterized in that The method of using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes: Using the preset recognition model, obtaining the color, shape and texture features of the object in the real scene image; The ingredient name, ingredient type and ingredient status of each ingredient in the refrigerator are determined according to the object color, object shape and object texture features in the real scene image.

3. The method according to claim 1, characterized in that The step of determining a target layer of each food ingredient according to the food ingredient name, food ingredient type and food ingredient state of each food ingredient and the layer label information of each layer inside the refrigerator includes: According to the ingredient name, ingredient type and ingredient state of each ingredient, a layer with the highest similarity to each ingredient is determined as the target layer of each ingredient.

4. The method according to claim 1, characterized in that: After determining the target layer of each ingredient according to the ingredient name, ingredient type and ingredient state of each ingredient and the layer label information of each layer inside the refrigerator, the method further includes: According to the ingredient name, ingredient type and ingredient status of each ingredient and in accordance with a preset sorting rule, determine the target position of each ingredient in the target layer; The generating and pushing the recommended information of placing the food in the refrigerator according to the current position of each food and the target layer of each food includes: According to the current position of each ingredient and the target layer and target position of each ingredient, recommended information on placing ingredients in the refrigerator is generated and pushed.

5. The method according to claim 1, characterized in that: The refrigerator further includes: a humidity temperature sensor. Before using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the real scene images of multiple layers inside the refrigerator, the method further includes: Acquire humidity information and temperature information of multiple layers inside the refrigerator collected by the humidity and temperature sensor; The method of using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes: The preset recognition model is used to determine the ingredient name, ingredient type and ingredient status of each ingredient inside the refrigerator based on the real-life images, humidity information and temperature information of the multiple layers inside the refrigerator.

6. The method according to claim 1, characterized in that The refrigerator further includes: a gas sensor. Before using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the real-life images of multiple layers inside the refrigerator, the method further includes: Acquire gas information of multiple layers inside the refrigerator collected by the gas sensor; The method of using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes: The preset recognition model is used to determine the ingredient name, ingredient type and ingredient status of each ingredient inside the refrigerator based on the real-life images of the multiple layers inside the refrigerator and the gas information.

7. The method according to claim 1, characterized in that The refrigerator further includes: a weight sensor. Before using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the real-life images of multiple layers inside the refrigerator, the method further includes: Obtaining weight information of multiple layers inside the refrigerator collected by the weight sensor; The method of using a preset recognition model to determine the name, type and state of each food in the refrigerator according to the multiple layers of real-life images in the refrigerator includes: The preset recognition model is used to determine the ingredient name, ingredient type and ingredient status of each ingredient inside the refrigerator based on the real-life images and weight information of the multiple layers inside the refrigerator.

8. The method according to claim 1, characterized in that The refrigerator further includes: a Hall sensor, and before generating and pushing the recommended information on placing ingredients in the refrigerator according to the current position of each ingredient and the target layer of each ingredient, the method further includes: Acquire the door opening and closing information of the refrigerator collected by the Hall sensor; Determining refrigerator usage time according to the refrigerator door opening and closing information; The generating and pushing the recommended information of placing the food in the refrigerator according to the current position of each food and the target layer of each food includes: According to the usage time of the refrigerator, the current location of each ingredient and the target layer of each ingredient, recommended information on the placement of ingredients in the refrigerator is generated and pushed.

9. The method according to claim 1, characterized in that: Before generating and pushing the recommended information on placing ingredients in the refrigerator according to the current position of each ingredient and the target layer of each ingredient, the method further includes: Determining whether each food in the refrigerator is expired according to the food name, food type and food status of each food in the refrigerator; If the expired food exists in the refrigerator, the food expiration information of the refrigerator is generated and pushed according to the current location of the expired food.

10. A controller, characterized in that: include: A processor and a storage medium, wherein the processor and the storage medium are connected to each other via a bus communication, the storage medium stores program instructions executable by the processor, and the processor calls the program stored in the storage medium to execute the steps of the refrigerator food placement recommendation method as described in any one of claims 1 to 9.

Images