Refrigerator food management method, equipment and refrigerator

Abstract

The present invention discloses a refrigerator food management method, device and refrigerator, the method comprising: obtaining an action process image of food being stored or taken out of a refrigerator compartment through a main camera arranged on the refrigerator; dividing the action process image into a first effective area, a second effective area and an invalid area, and performing image pattern recognition and sequence comparison on each area to determine the food information and food movement trajectory of the action process image; and determining the operation type of the action process image according to the food movement trajectory. Compared with the prior art, the refrigerator food management method of the present invention only needs to perform image pattern recognition and sequence comparison on the food, and can dynamically identify the storage and access of food in the refrigerator compartment, omitting the step of performing image pattern recognition on the hands, reducing the complexity of the recognition model, alleviating the recognition workload, and improving the recognition efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of household appliances, and in particular to a method and device for managing refrigerator food materials, and a refrigerator. Background Art

[0002] At present, most smart refrigerators have the function of identifying the ingredients inside.

[0003] In order to accurately identify the ingredients inside the refrigerator, one or more cameras are usually installed on each shelf of the refrigerator to take pictures of the ingredients. The pictures are processed to generate recognition results that are directly displayed on the display interface of the refrigerator or sent to a mobile terminal connected to the refrigerator, such as a mobile phone. However, this recognition method requires multiple cameras, which makes the structure of the refrigerator complex and increases the cost. At the same time, this method also has the problem of being unable to recognize stacked ingredients.

[0004] Another existing image recognition technology uses dynamic recognition technology to determine whether food is put in or taken out based on the hand's movements. This method uses one camera, which reduces costs and can solve the problem of stacked food being unable to be recognized. However, using this technology to recognize food inside the refrigerator requires training image recognition models for both the hand and the food at the same time, which makes image recognition complex and inefficient. In addition, this recognition method is not friendly to the recognition of food placed on the bottle holder of the door, and is prone to inaccurate detection and recognition due to the shooting angle. Summary of the invention

[0005] The object of the present invention is to provide a method, device and refrigerator for managing refrigerator food.

[0006] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a method for managing refrigerator food, the method comprising:

[0007] The main camera disposed on the refrigerator is used to obtain images of the process of storing or taking out food in the refrigerator compartment;

[0008] Dividing the action process image into a first valid area, a second valid area, and an invalid area, and performing image pattern recognition and sequence comparison on each area to determine the food information and food movement trajectory of the action process image;

[0009] Determining the operation type of the action process image according to the movement trajectory of the food;

[0010] According to the food information and the operation type of the action process image, the food statistics information located in the refrigerator room is managed.

[0011] As a further improvement of one embodiment of the present invention, the first valid area is an image area located outside the refrigerator, the second valid area is an image area located inside the refrigerator, and the invalid area is an area located on the door body after the refrigerator door is opened.

[0012] As a further improvement of an embodiment of the present invention, the “determining the operation type of the action process image according to the food movement trajectory” specifically includes:

[0013] If the food movement trajectory is that the food moves from the first effective area to the second effective area, then the operation type is a storage operation;

[0014] If the food movement trajectory is that the food moves from the second effective area to the first effective area, then the operation type is a taking-out operation;

[0015] If the starting point or the end point of the food movement trajectory is an invalid area, the operation type is an invalid operation.

[0016] As a further improvement of an embodiment of the present invention, the operation types of the action process image include a take-out operation, a deposit operation and an invalid operation.

[0017] As a further improvement of an embodiment of the present invention, the method further includes:

[0018] The auxiliary camera disposed on the refrigerator is used to obtain an image of the door shelf when the user closes the door, and the food information located on the shelf is determined.

[0019] As a further improvement of an embodiment of the present invention, the method further includes:

[0020] When the refrigerator door rotates to a second preset angle during closing, the auxiliary camera takes a picture to obtain an image of the door storage rack.

[0021] As a further improvement of an embodiment of the present invention, the method further includes:

[0022] When the opening angle of the refrigerator door exceeds a first predetermined angle, the main camera takes a picture to obtain an image of the action process of storing or taking out the food in the refrigerator compartment.

[0023] As a further improvement of an embodiment of the present invention, the main camera is installed at the top of the interior of the refrigerator compartment, and the direction of the main camera is perpendicular to the ground.

[0024] To achieve one of the above-mentioned purposes of the invention, one embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in any one of the above-mentioned refrigerator food management methods are implemented.

[0025] To achieve one of the above-mentioned objects of the invention, one embodiment of the present invention provides a refrigerator, which includes the above-mentioned electronic device.

[0026] Compared with the prior art, the refrigerator food management method of the present invention uses a main camera to shoot the food storage and access process in the refrigerator compartment, and divides the photographed photos into areas. Therefore, only image pattern recognition and sequence comparison of the food are required to dynamically identify the food storage and access in the refrigerator compartment, omitting the step of image pattern recognition on the hands, reducing the complexity of the recognition model, alleviating the recognition workload, and improving the recognition efficiency.

[0027] Furthermore, the main camera is used to dynamically identify the storage and access of food in the refrigerator compartment, and the auxiliary camera is used to statically identify the food on the door shelf, thereby performing a comprehensive identification of the food in the refrigerator to avoid incorrect food identification due to inaccurate identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the method for managing refrigerator food of the present invention.

[0029] Figure 2 It is a regional structure diagram of a picture taken by the main camera of the refrigerator of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0031] like Figure 1 As shown, the present invention provides a method for managing refrigerator food. The method uses a main camera to shoot the food storage and access process of the refrigerator compartment (mainly including the shelf and drawer area in the compartment), and divides the shot photos into areas. Therefore, only image pattern recognition and sequence comparison of the food are needed to dynamically identify the food storage and access of the refrigerator compartment, omitting the step of image pattern recognition of the hand, reducing the complexity of the recognition model, alleviating the recognition workload, and improving the recognition efficiency.

[0032] Furthermore, the main camera is used to dynamically identify the storage and access of food in the refrigerator compartment, and the auxiliary camera is used to statically identify the food on the door shelf, thereby performing a comprehensive identification of the food in the refrigerator to avoid incorrect food identification due to inaccurate identification.

[0033] The method comprises:

[0034] Step S100: obtaining an image of the process of food being put into or taken out of the refrigerator compartment through a main camera arranged on the refrigerator.

[0035] The main camera of the present invention is preferably a wide-angle camera, which can be installed at the top of the refrigerator compartment, or at other locations where images of the process of food being stored or taken out can be obtained. In a specific embodiment, the refrigerator is a single-door refrigerator, and the main camera is installed at the top of the refrigerator compartment near the middle of the frame of the door body.

[0036] In another specific embodiment, the refrigerator is a double-door refrigerator, and the main camera is installed on one side of the frame of the top of the refrigerator compartment close to the middle of the door body to avoid conflict with the refrigerator door closing device. Furthermore, the camera can be installed not on the frame, but in a position closer to the frame, so that after one side of the refrigerator door is closed, it is still possible to capture the action process of food being stored or taken out.

[0037] It should be noted that the main camera can be started after the refrigerator door is opened, and then when the opening angle of the refrigerator door exceeds the first predetermined angle, the main camera takes photos to obtain the action process image of the food being stored or taken out of the refrigerator compartment. That is, when the refrigerator door is opened beyond the first predetermined angle, the main camera starts to take photos (or pictures / images), and the main camera stops shooting until the opening angle of the refrigerator door is less than or equal to the first predetermined angle during the closing process. During the entire process of opening and closing the door, the main camera obtains multiple action process images of food being stored or taken out. The action process image refers to a group of process images of food being stored or taken out.

[0038] In another preferred embodiment, the main camera is oriented in a direction perpendicular to the ground.

[0039] Step S200: dividing the action process image into a first valid area, a second valid area and an invalid area, and performing image pattern recognition and sequence comparison on each area to determine the food information and food movement trajectory of the action process image.

[0040] Figure 2 It is a regional structure diagram of a picture taken by the main camera of the refrigerator of the present invention.

[0041] like Figure 2As shown, the action process image is divided into a first effective area 1, a second effective area 2 and an invalid area 3. Preferably, the first effective area 1 is an image area located outside the refrigerator, such as a photographed ground area. The second effective area 2 is an image area located inside the refrigerator, that is, when food is stored in or taken out of the compartment, it will pass through this area. The invalid area 3 is the area located at the door body after the refrigerator door is opened. It should be noted that the boundary line between the first effective area 1 and the second effective area 2 is the boundary line between the door body and the box body after the refrigerator door is closed.

[0042] After the image is divided into regions, it is necessary to perform image pattern recognition and sequence comparison on each region to determine the food information and food movement trajectory of the action process image.

[0043] In the existing solutions for dynamic identification of food materials, image pattern recognition of both hands and food materials is required to obtain the type of operation on the food materials. In the present invention, since the image is divided into regions, only image pattern recognition and sequence comparison of food materials are required to dynamically identify the storage and access of food materials in the refrigerator compartment, thus omitting the step of image pattern recognition of hands, reducing the complexity of the recognition model, alleviating the workload of recognition, and improving the recognition efficiency.

[0044] It should be noted that the food information includes the food name, and the food movement trajectory is moving from one area to another area in chronological order. In the present invention, food moving from one area to another area means that food from one area passes through the boundary line of two areas, that is, it is considered to move to another area.

[0045] In order to further improve the recognition efficiency and reduce the recognition workload, in a specific implementation, the action process image is cropped according to regions, and image pattern recognition is performed on the cropped image. After the food information is recognized in one region, image pattern recognition is no longer performed on other regions of the same image.

[0046] In a preferred embodiment, the continuous pictures taken by the main camera can be processed every other frame to further improve the recognition efficiency. Since the interval between pictures taken by the main camera is at the level of hundreds of milliseconds, and it is difficult for a person to move from one area to another in hundreds of milliseconds, after processing a picture, one or more frames can be processed every other frame (i.e., one or more pictures can be skipped).

[0047] Step S300: determining the operation type of the action process image according to the movement trajectory of the food.

[0048] The operation types of the action process image preferably include a take-out operation, a deposit operation and an invalid operation.

[0049] In a specific embodiment, if the food movement trajectory is that the food moves from the first effective area to the second effective area, the operation type is a deposit operation. It should be noted that as long as the food passes through the boundary line between the first effective area and the second effective area from the first effective area, the operation type is considered to be a deposit operation.

[0050] If the food movement trajectory is that the food moves from the second effective area to the first effective area, the operation type is a take-out operation. It should be noted that as long as the food passes through the boundary between the first effective area and the second effective area from the second effective area, the operation type is considered to be a take-out operation.

[0051] If the starting point or end point of the food movement trajectory is an invalid area, the operation type is an invalid operation. That is, as long as the food passes through the boundary line between the first / second valid area and the invalid area from the first / second valid area, or the food passes through the boundary line between the first / second valid area and the invalid area and enters the first / second valid area, it is considered an invalid operation. The invalid area is the position after the refrigerator door (door shelf) is opened. Since the present invention only dynamically identifies the food in the refrigerator compartment, and uses static identification for the food on the door shelf, therefore, whether taking food from the door shelf or putting food on the door shelf, it is not the object of dynamic identification of the present invention, so it is judged as an invalid operation, which does not affect the result of dynamic identification.

[0052] Step S400: managing statistical information of ingredients in the refrigerator compartment according to the ingredient information and operation type of the action process image.

[0053] The action images of the entire access process captured by the main camera can obtain information about the food entering and leaving the refrigerator compartment during the current door opening period, that is, storing certain food in the refrigerator compartment, taking certain food out of the refrigerator compartment, or storing certain food and taking out another or the same food. Based on the results of the previous statistics and the access situation during the current door opening period, the statistical information of the food in the refrigerator compartment can be obtained.

[0054] When the refrigerator door is opened to a certain angle, the food on the door shelf captured by the main camera is deformed, resulting in inaccurate recognition during the recognition process. In order to accurately manage the food in the entire refrigerator and solve the problem of inaccurate recognition of food on the door shelf, in a preferred embodiment, the method further includes:

[0055] Step S500: Acquire an image of the door shelf when the user closes the door through an auxiliary camera disposed on the refrigerator, and determine information about food on the shelf.

[0056] An auxiliary camera is provided on the refrigerator to solve the problem that the main camera shoots food on the door shelf, causing the food to deform and thus failing to accurately identify the food. In order to reduce the deformation of the food, it is preferred that the auxiliary camera is not facing the closed refrigerator door, but is offset outward at a certain angle (e.g., 15 degrees), and when the refrigerator door is closed to a second preset angle (e.g., 45 degrees), the auxiliary camera takes a picture to obtain a picture with a smaller degree of deformation of the food, thereby ensuring the accuracy of identifying the food on the door shelf. It is preferred that the auxiliary camera is also a wide-angle camera.

[0057] It should be noted that if it is a single-door refrigerator, take a photo each time the door is closed. If it is a double-door refrigerator, take a photo each time the door is closed, for a total of two photos.

[0058] The food information on the door body rack obtained based on the picture of the door body rack taken by the auxiliary camera is the food statistics information on the door body rack.

[0059] According to the food statistical information inside the refrigerator room and combined with the food statistical information on the door shelf, it is possible to obtain the food information inside the entire refrigerator and manage the food in the refrigerator.

[0060] The refrigerator food management method of the present invention uses a main camera to dynamically identify the storage and access of food in the refrigerator compartment, and combines with an auxiliary camera to statically identify the food on the door shelf, thereby performing all-round identification of the food in the refrigerator, avoiding food identification errors due to inaccurate identification, and improving identification accuracy.

[0061] In a specific embodiment, the main camera of the refrigerator is a 100-degree wide-angle anti-fog network camera, which is arranged at the top of the refrigerator compartment, and the main camera is oriented perpendicular to the ground. The auxiliary camera of the refrigerator is a 200-degree wide-angle anti-fog network camera, which is also arranged at the top of the refrigerator compartment. The auxiliary camera is not oriented directly toward the closed refrigerator door, but is offset outward by a certain angle of 15 degrees to ensure that when the refrigerator door is closed to 45 degrees, the photos taken by the auxiliary camera have a higher degree of restoration.

[0062] When the refrigerator door is opened, the main and auxiliary cameras are activated. When the opening angle of the refrigerator door exceeds a first predetermined angle, the main camera starts taking pictures, and when the opening angle of the refrigerator door is less than or equal to the first predetermined angle, the main camera stops taking pictures. During the process of opening and closing the refrigerator door, multiple action process images of the food in the refrigerator compartment being stored or taken out are obtained.

[0063] The action process image is divided into the first effective area, the second effective area and the invalid area, and image pattern recognition and sequence comparison are performed on each area. When a food material is recognized to pass through the boundary line between the first effective area and the second effective area from the first effective area, it is judged as a deposit operation on the food material; when a food material is recognized to pass through the boundary line between the first effective area and the second effective area from the second effective area, it is judged as a take-out operation on the food material; when a food material is recognized to pass through the boundary line between the first effective area and the invalid area from the first / second effective area, or the food material passes through the boundary line between the first / second effective area and the invalid area and enters the first / second effective area, it is judged as an invalid operation, which does not affect the result of dynamic recognition. Thus, the food material information and operation type are obtained according to the action process image.

[0064] The action images of the entire access process captured by the main camera can obtain information about the food entering and leaving the refrigerator compartment during the current door opening period, that is, storing certain food in the refrigerator compartment, taking certain food out of the refrigerator compartment, or storing certain food and taking out another or the same food. Based on the results of the previous statistics and the access situation during the current door opening period, the statistical information of the food in the refrigerator compartment can be obtained.

[0065] When the refrigerator door is closed, when the refrigerator door is at a 45-degree angle, the auxiliary camera takes pictures of the food on the door shelf. When the refrigerator door is closed, the main and auxiliary cameras are turned off. Based on the pictures taken by the auxiliary camera, the food information on the door shelf is determined.

[0066] According to the food statistical information inside the refrigerator room and combined with the food statistical information on the door shelf, it is possible to obtain the food information inside the entire refrigerator and manage the food in the refrigerator.

[0067] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements any one of the steps in the above-mentioned method for managing refrigerator food, that is, it implements the steps in any one of the technical solutions in the above-mentioned method for managing refrigerator food.

[0068] The present invention also provides a refrigerator, which comprises the electronic device described above.

[0069] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0070] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for managing refrigerator ingredients, It is characterized in that The method comprises: The main camera disposed on the refrigerator is used to obtain images of the process of storing or taking out food in the refrigerator compartment; Dividing the action process image into a first valid area, a second valid area, and an invalid area, and performing image pattern recognition and sequence comparison on each area to determine the food information and food movement trajectory of the action process image; Determining the operation type of the action process image according to the movement trajectory of the food; Managing statistical information of ingredients in the refrigerator compartment according to the ingredient information and operation type of the action process image; The first valid area is an image area outside the refrigerator, the second valid area is an image area inside the refrigerator, and the invalid area is an area located on the door body after the refrigerator door is opened; The “determining the operation type of the action process image according to the food movement trajectory” specifically includes: If the food movement trajectory is that the food moves from the first effective area to the second effective area, then the operation type is a storage operation; If the food movement trajectory is that the food moves from the second effective area to the first effective area, then the operation type is a taking-out operation; The food statistical information includes the food statistical information on the door shelf. The food statistical information on the door shelf is determined by obtaining an image of the door shelf when the user closes the door through an auxiliary camera arranged on the refrigerator.

2. According to the method for managing refrigerator food materials according to claim 1, It is characterized in that The “determining the operation type of the action process image according to the food movement trajectory” specifically includes: If the starting point or the end point of the food movement trajectory is an invalid area, the operation type is an invalid operation.

3. According to the method for managing refrigerator food materials according to claim 1, Features: The operation types of the action process image include a take-out operation, a deposit operation and an invalid operation.

4. According to the method for managing refrigerator food materials according to claim 1, It is characterized in that The method further comprises: When the refrigerator door rotates to a second preset angle during closing, the auxiliary camera takes a picture to obtain an image of the door storage rack.

5. According to the method for managing refrigerator food materials according to claim 1, It is characterized in that The method further comprises: When the opening angle of the refrigerator door exceeds a first predetermined angle, the main camera takes a picture to obtain an image of the action process of storing or taking out the food in the refrigerator compartment.

6. The method for managing refrigerator food according to claim 1, Features: The main camera is installed at the top of the refrigerator compartment, and the main camera is oriented perpendicular to the ground.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, It is characterized in that When the processor executes the program, the steps in the refrigerator food management method according to any one of claims 1 to 6 are implemented.

8. A refrigerator, It is characterized in that The refrigerator comprises the electronic device according to claim 7.

Images