Smart refrigerator and method for identifying and accessing food

Abstract

The present application discloses a smart refrigerator and a method for identifying and accessing food. In the present application, the refrigerator includes: a box body, a refrigeration part, and a door body is installed on the box body; a depth camera and a visible light camera are arranged on the box body, which collect depth images and visible light images in the front area of ​​the box body when the door body is opened; the depth camera and the visible light camera are connected to a controller, and the controller is configured to: obtain a first depth image collected by the depth camera and a first visible light image collected by the visible light camera; detect the first visible light image to obtain the type of each food in the first visible light image and the position area of ​​each food in the first visible light image; determine the 3D coordinate range of each food according to the 3D coordinate information of each pixel point in the first depth image and the position area of ​​each food; determine the food whose 3D coordinate range is located in the set space as the food currently being accessed, thereby improving the recognition rate of accessing food.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a smart refrigerator and a method for identifying and storing food. Background Art

[0002] With the development of the artificial intelligence industry, various home appliance companies are scrambling to apply artificial intelligence to the refrigerator field, so that refrigerators can have intelligent methods to manage food. The method of deep learning training on collected images gives refrigerators the ability to recognize food, actively manage the food stored in the refrigerator, and realize functions such as storage location recommendation and recipe recommendation.

[0003] There are many ingredients placed in the refrigerator. When identifying multiple ingredients in the image, it is impossible to distinguish the target ingredient currently being accessed, which affects the management of the refrigerator for the ingredients.

[0004] Therefore, how to improve the recognition rate of stored and accessed food is a problem that needs to be solved in the field of smart refrigerators. Summary of the invention

[0005] In the exemplary embodiments of the present application, a smart refrigerator and a method for identifying stored and accessed food are provided to improve the recognition rate of the accessed food.

[0006] In a first aspect, a smart refrigerator is provided, the refrigerator comprising: a box body and a refrigeration unit, wherein a door body is installed on the box body;

[0007] The box is provided with a depth camera and a visible light camera, which collect a depth image and a visible light image in the front area of ​​the box when the door is opened;

[0008] The depth camera and the visible light camera are connected to a controller, and the controller is configured as follows:

[0009] Acquire a first depth image acquired by the depth camera, and a first visible light image associated with the first depth image acquired by the visible light camera;

[0010] Detecting the first visible light image to obtain the type of each food in the first visible light image and the location area of ​​each food in the first visible light image;

[0011] Obtaining 3D coordinate information of each pixel point according to the first depth image;

[0012] Determine the 3D coordinate range of each food ingredient according to the 3D coordinate information of each pixel point and the position area of ​​each food ingredient in the first visible light image;

[0013] According to the 3D coordinate range of each ingredient, the ingredient whose 3D coordinate range is located in a set space is determined as the currently accessed ingredient; wherein the set space is located at the front side of the refrigerator and is the area through which the accessed ingredients pass during the food access process.

[0014] In some embodiments of the present application, the controller is further configured to:

[0015] If the 3D coordinate range of each food in the first visible light image is not within the set space, detecting the first visible light image to obtain the user's arm in the first visible light image;

[0016] Obtaining a 3D coordinate range of the user's arm according to the position area of ​​the user's arm in the first visible light image and the 3D coordinate information of each pixel point in the first depth image;

[0017] According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

[0018] In some embodiments of the present application, the controller is further configured to:

[0019] If the 3D coordinate range of each food in the first visible light image is not within the set space or the user's arm is not detected in the first visible light image, then acquiring N consecutive frames of depth images including the first depth image and N consecutive frames of visible light images associated with the N frames of depth images, where N is an integer greater than 1;

[0020] Detecting the N frames of visible light images respectively to obtain the type of each food in the N frames of visible light images and the location area of ​​each food in the N frames of visible light images;

[0021] Determine the 3D coordinate range of each food ingredient in the N depth images according to the 3D coordinate information of each pixel in the N depth images and the position area of ​​each food ingredient in the N visible light images;

[0022] According to the 3D coordinate range of each food, the food whose 3D coordinate range in the N frames of depth images has changed is determined as the food currently being accessed.

[0023] In some embodiments of the present application, the depth camera is any one of a binocular camera, a structured light camera, and a TOF camera.

[0024] In a second aspect, a smart refrigerator is provided, which is applied to a smart refrigerator, the refrigerator comprising: a box body and a refrigeration unit, wherein a door body is installed on the box body;

[0025] The box is provided with a depth camera and a visible light camera, which collect a depth image and a visible light image in the front area of ​​the box when the door is opened;

[0026] The depth camera and the visible light camera are connected to a controller, and the controller is configured as follows:

[0027] Acquire a first depth image acquired by the depth camera, and a first visible light image associated with the first depth image acquired by the visible light camera;

[0028] Detecting the first visible light image to obtain the type of each food in the first visible light image and the location of each food in the first visible light image, and obtaining the user's arm and the location of the user's arm in the first visible light image;

[0029] Obtaining 3D coordinate information of each pixel point according to the first depth image;

[0030] Determine the 3D coordinate range of each food ingredient according to the 3D coordinate information of each pixel and the position area of ​​each food ingredient in the first visible light image, and determine the 3D coordinate range of the user's arm according to the 3D coordinate information of each pixel and the position area of ​​the user's arm in the first visible light image;

[0031] According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

[0032] In a third aspect, a smart refrigerator is provided, the refrigerator comprising: a box body and a refrigeration unit, wherein a door body is installed on the box body;

[0033] The box is provided with a depth camera and a visible light camera, which collect a depth image and a visible light image in the front area of ​​the box when the door is opened;

[0034] The depth camera and the visible light camera are connected to a controller, and the controller is configured as follows:

[0035] Acquire N consecutive frames of depth images and N consecutive frames of visible light images associated with the N frames of depth images, where N is an integer greater than 1;

[0036] Detecting the N frames of visible light images respectively to obtain the type of each food in the N frames of visible light images and the location area of ​​each food in the N frames of visible light images;

[0037] Determine the 3D coordinate range of each food ingredient in the N depth images according to the 3D coordinate information of each pixel in the N depth images and the position area of ​​each food ingredient in the N visible light images;

[0038] According to the 3D coordinate range of each food, the food whose 3D coordinate range in the N frames of depth images has changed is determined as the food currently being accessed.

[0039] In a fourth aspect, a method for identifying and accessing food is provided, comprising:

[0040] Acquire a first depth image acquired by the depth camera, and a first visible light image associated with the first depth image acquired by the visible light camera;

[0041] Detecting the first visible light image to obtain the type of each food in the first visible light image and the location area of ​​each food in the first visible light image;

[0042] Obtaining 3D coordinate information of each pixel point according to the first depth image;

[0043] Determine the 3D coordinate range of each food ingredient according to the 3D coordinate information of each pixel point and the position area of ​​each food ingredient in the first visible light image;

[0044] According to the 3D coordinate range of each ingredient, the ingredient whose 3D coordinate range is located in a set space is determined as the currently accessed ingredient; wherein the set space is located at the front side of the refrigerator and is the area through which the accessed ingredients pass during the food access process.

[0045] In a fifth aspect, a method for identifying and accessing food is provided, comprising:

[0046] Acquire a depth image acquired by the depth camera, and a visible light image associated with the depth image acquired by the visible light camera;

[0047] Detecting the visible light image to obtain the type of each food in the visible light image and the location area of ​​each food in the visible light image, and obtaining the user's arm and the location area of ​​the user's arm in the visible light image;

[0048] Obtaining 3D coordinate information of each pixel point according to the depth image;

[0049] Determine the 3D coordinate range of each food ingredient according to the 3D coordinate information of each pixel and the position area of ​​each food ingredient in the visible light image, and determine the 3D coordinate information of the user's arm according to the 3D coordinate information of each pixel and the position area of ​​the user's arm in the visible light image;

[0050] According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

[0051] In a sixth aspect, a method for identifying and accessing food is provided, comprising:

[0052] Acquire N consecutive frames of depth images and N consecutive frames of visible light images associated with the N frames of depth images, where N is an integer greater than 1;

[0053] Detecting the N frames of visible light images respectively to obtain the type of each food in the N frames of visible light images and the location area of ​​each food in the N frames of visible light images;

[0054] Determine the 3D coordinate range of each food ingredient in the N depth images according to the 3D coordinate information of each pixel in the N depth images and the position area of ​​each food ingredient in the N visible light images;

[0055] According to the 3D coordinate range of each food, the food whose 3D coordinate range in the N frames of depth images has changed is determined as the food currently being accessed.

[0056] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute any one of the methods in the fourth to sixth aspects.

[0057] In the above-mentioned embodiment of the present application, the visible light image in the front area of ​​the cabinet captured by the visible light camera is detected to obtain the type of each food and the position area of ​​each food in the visible light image, and the 3D coordinate information of each pixel is obtained according to the depth image captured by the depth camera. The 3D coordinate range of each food is determined according to the 3D coordinate information of each pixel and the position area of ​​each food in the visible light image, and the area on the front side of the refrigerator where the food being accessed passes during the food storage and retrieval process is determined as the set space, so that the food whose 3D coordinate range is located in the set space is determined as the food currently being accessed, thereby improving the recognition rate of the accessed food.

[0058] On the basis of conforming to the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0060] Figure 1a A schematic diagram exemplarily shows a closed state of a smart refrigerator in an embodiment of the present application;

[0061] Figure 1b A schematic diagram exemplarily shows the open state of the smart refrigerator in the embodiment of the present application;

[0062] Figure 1c The schematic diagram of the three-dimensional coordinate system of the box 10 in the embodiment of the present application is exemplarily shown;

[0063] Figure 2 A schematic diagram of the front area of ​​the refrigerator in an embodiment of the present application is exemplarily shown;

[0064] Figure 3 The functional structure diagram of the controller of the smart refrigerator in the embodiment of the present application is exemplarily shown;

[0065] Figure 4 The schematic diagram of the process of the method for identifying and accessing food provided in the embodiment of the present application is exemplified;

[0066] Figure 5 The schematic diagram of judging the user's food storage and access actions in the embodiment of the present application is exemplified;

[0067] Figure 6 A schematic diagram showing an exemplary food testing result;

[0068] Figure 7 The schematic diagram of determining the currently accessed food provided by the embodiment of the present application is exemplarily shown;

[0069] Figure 8 The schematic diagram of determining the currently accessed food provided by the embodiment of the present application is exemplarily shown;

[0070] Fig. 9 The schematic diagram of the process of the method for identifying and accessing food provided in the embodiment of the present application is exemplified;

[0071] Fig.10 The flowchart of the method for identifying and accessing food provided in an embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0072] The following will provide a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0073] In the following, the terms "first", "second" are used for descriptive purposes only and should not be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0074] Intelligent management of food information in refrigerators has become one of the current trends in refrigerator development. Food management is based on identifying food access. Since the types of food stored in refrigerators are numerous and complex, and the food that users access each time is different, and the appearance of different food is similar, simple image recognition cannot correctly identify the food being accessed, which will lead to the inability of refrigerators to effectively manage food.

[0075] Based on this, an embodiment of the present application proposes a method for identifying and accessing food ingredients by combining depth images and visible light images. In the embodiment of the present application, the type information of the food ingredient is obtained according to the visible light image, and each obtained food ingredient is screened according to the depth image to obtain the currently accessed food ingredient. This can reduce background interference from other types of food ingredients in the smart refrigerator, thereby improving the recognition rate of the accessed food ingredients, and further realizing the intelligent management of food ingredients by the smart refrigerator.

[0076] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0077] Figure 1a and Figure 1b The structure of a smart refrigerator provided in an embodiment of the present application is exemplified.

[0078] like Figure 1a As shown, the smart refrigerator includes a box 10, a refrigeration unit (not shown in the figure) and other accessories (for example, a lighting lamp, a thermometer, etc. can be set in the box, which are not shown in the figure). The refrigeration system mainly consists of components such as a compressor, a condenser, an evaporator and a capillary throttle, forming a closed circulation system. Among them, the evaporator can be installed on the top of the inside of the smart refrigerator, and the other components are installed on the back of the smart refrigerator.

[0079] The box body 10 is installed with a door body 20, and the door body 20 may be further provided with a display screen 60, and the display screen 60 is coupled to the controller (such as through a circuit connection).

[0080] A visible light camera 30 may also be provided on the cabinet 10, and the visible light camera may collect visible light images in the front area of ​​the cabinet 10. The front area of ​​the cabinet 10 includes at least an area extending a certain distance from the first plane to the outside of the refrigerator, with the plane where the refrigerator door is located being the first plane. The visible light camera may collect visible light images of the area, that is, it may capture images of hand movements of the user during the process of storing and accessing food after opening the door 20, as well as images of the food stored and accessed.

[0081] Figure 2 The front area of ​​the cabinet 10 is shown as an example. As shown in the figure, the plane where the door of the smart refrigerator is closed is called plane H2, the plane d away from plane H2 is called plane H1, plane H1 is parallel to plane H2, and the space between plane H1 and plane H2 is the front area of ​​the cabinet 10. The visible light camera and the depth camera must at least be able to capture images of this area. Among them, the value of d can be set according to factors such as the human arm length parameter and the image recognition accuracy, for example, it can be set to between 25cm and 35cm.

[0082] A depth camera 40 may also be provided on the box 10, and the depth camera may collect a depth image in the front area of ​​the box 10. The depth camera 40 may be any one of a binocular camera, a structured light camera, and a TOF camera. Based on the 3D coordinate information of each pixel point in the depth image collected by the depth camera, a refrigerator coordinate system is established to restore the three-dimensional information of the refrigerator space.

[0083] Figure 1c The schematic diagram of the three-dimensional coordinate system of the box 10 is shown as an example. As shown in the figure, the origin of the box coordinate system is set at the center of the installation point of the depth camera 40. When the user faces the box, the X direction points to the user's right hand, the Y direction points to the user, and the Z direction points to the ground. To simplify the calculation, the coordinate system of the depth camera coincides with the coordinate system of the box. If the camera optical axis is not parallel to the Z axis and there is rotation and translation, the coordinate system conversion matrix can be used to convert the coordinate system.

[0084] After the cabinet coordinate system is established, the spatial position inside the refrigerator can be obtained through calculation or pre-calibration. Figure 1cIn the example of the user facing the refrigerator, the coordinates of the left side of the outer edge of the partition 1 in the refrigerator refrigeration area are (x1, y1, zs1), and the coordinates of the right side of the outer edge are (x2, y1, zs1); when the drawer in the refrigerator refrigeration area is closed, the coordinates of the upper left edge of the drawer are (x1, y1, zt1), and the coordinates of the lower left edge are (x1, y1, zt2); when the drawer in the refrigerator refrigeration area is fully opened, the coordinates of the upper left edge of the drawer are (x1, y2, zt3), and the coordinates of the lower left edge are (x1, y2, zt4). According to the spatial position coordinates and the position coordinates of the food, it can be judged whether the food is placed in the refrigerator.

[0085] In some embodiments, the visible light camera 30 and the depth camera 40 may be disposed at an upper portion of the housing 10 near the door 20 so as to capture images of the front area of ​​the housing 10 .

[0086] In some embodiments, the visible light camera 30 and the depth camera 40 may be arranged on a fixed component, and the fixed component may allow the camera to protrude a certain distance from the plane where the door 20 is located, so as to better capture the image in the front area of ​​the box 10. When the user opens the door, the camera may be triggered to turn on, and when the user closes the door, the camera may be triggered to turn off. The embodiment of the present application does not limit the shape, connection method and material of the fixed component.

[0087] In other embodiments, the visible light camera 30 and the depth camera 40 may be arranged on a movable part, and the movable part may pop out the camera when the door body 20 is opened, so that the camera protrudes a certain distance from the plane where the door body 20 is located, and when the door body 20 is closed, the movable part retracts the camera. The movable part may be linked with the door body 20 by mechanical connection or other means, so that the door body 20 triggers the control camera of the movable part when it is opened and closed. In some embodiments, the movable part may also include a camera protection cover, which is opened and started after the camera is popped out, and closed and turned off after the camera is retracted, thereby protecting the camera on the one hand and saving power consumption on the other hand.

[0088] like Figure 1b As shown, the box 10 of the smart refrigerator may include multiple compartments (such as compartments 50a to 50e in the figure) to facilitate users to classify and store different ingredients. In the embodiment of the present application, when the door 20 is opened during the food storage and access process, the controller controls the visible light camera and the depth camera to collect images in the front area of ​​the door 20.

[0089] It should be noted that Figure 1a and Figure 1b The structure of the smart refrigerator shown is only an example. The embodiment of the present application does not limit the size of the smart refrigerator, the number of doors (for example, it can be a single door or multiple doors), etc.

[0090] The smart refrigerator provided in the embodiment of the present application includes a controller (not shown in Figure 1a and Figure 1b ), the controller is coupled to the visible light camera and the depth camera (for example, connected by a circuit). When the door of the smart refrigerator is opened, the controller can detect the visible light image captured by the visible light camera to obtain at least one type of food and the location area of ​​the food in the image; and obtain the 3D coordinate range of each food according to the three-dimensional coordinate information of each pixel point in the depth image captured by the depth camera, and determine the currently accessed food according to the 3D coordinate range and the location area of ​​each food, thereby improving the recognition rate of the accessed food.

[0091] Based on the functions implemented by the above controller, Figure 3 The functional structure of the controller is shown as an example.

[0092] Figure 3 The functional structure diagram of the smart refrigerator controller in the embodiment of the present application is exemplarily shown. As shown in the figure, the controller 300 may include the following functional modules: an image acquisition module 301, an image detection module 302, a coordinate range determination module 303, and an access food determination module 304.

[0093] When the door of the smart refrigerator is opened and the user is accessing food, the image acquisition module 301 acquires a visible light image of the user's hand accessing food collected by visible light and a depth image of the user's hand accessing food collected by the depth camera, and sends the visible light image to the image detection module 302, and sends the depth image to the coordinate range determination module 303. The visible light image and the depth image are collected at the same time, that is, the visible light image and the depth image are temporally correlated.

[0094] The image detection module 302 detects the acquired visible light image to obtain the type of at least one food in the visible light image and the location area of ​​each food in the visible light image, and sends the obtained food type and the location area of ​​the food of the type to the coordinate range determination module 303.

[0095] The coordinate range determination module 303 obtains the 3D coordinate information of each pixel point according to the established box coordinate system and the depth image collected by the image acquisition module 301, and determines the 3D coordinate range of each ingredient according to the 3D coordinate information of each pixel point and the position area of ​​each ingredient in the visible light image.

[0096] When the refrigerator door is opened to access food, the access food determination module 304 can set the front side of the refrigerator and the area where the food being accessed passes through during the access process as the set space. According to the 3D coordinate range of each food determined by the coordinate range determination module 303, it is determined whether the 3D coordinate range is within the set space. If so, the food corresponding to the 3D coordinate range is determined as the currently accessed food, thereby improving the recognition rate of the accessed food.

[0097] Figure 4 The flowchart of the method for identifying and accessing food provided in an embodiment of the present application is exemplified.

[0098] As shown in the figure, the process may include the following steps:

[0099] S401: Acquire a first depth image acquired by a depth camera, and a first visible light image associated with the first depth image acquired by a visible light camera.

[0100] In this step, when the door of the smart refrigerator is opened and the user accesses food, the depth camera and visible light camera set on the box of the smart refrigerator are activated, the depth camera collects the first depth image of the user accessing food, and transmits the collected first depth image to the controller, and at the same time, the visible light camera collects the first visible light image of the user accessing food, and transmits the collected first visible light image to the controller. The first visible light image and the first depth image are temporally correlated.

[0101] In order to reduce the processing overhead of food image, in some embodiments, hand detection may be performed first, and when a hand is detected, image capture is performed. If a hand is detected, it indicates that the user is likely to be performing a food storage and access action.

[0102] by Figure 2 Taking the front area of ​​the refrigerator as an example, hand movements are detected in this area. Figure 5 As shown in FIG. 1 , when the user opens the refrigerator door and performs a food storage and retrieval action, a series of images captured by the visible light camera are recognized. If it is recognized that the user's hand first crosses plane H1 and then crosses plane H2, it is determined that the user has performed a storage action (such as Figure 5 If it is recognized that the user's hand first crosses plane H2 and then plane H1, it is determined that the user has performed a taking action. Each time the user operates, whether storing or taking food, the hand must first perform the storing action and then the taking action.

[0103] The controller recognizes the user's hand based on the first visible light image captured by the visible light camera, and then grabs the image. Further, when food is detected, the user's current operation behavior can be determined based on the user's current action. For example, if food is detected during the storage action, it is determined that the user is currently storing food, and if food is detected during the removal action, it is determined that the user is currently taking food.

[0104] In some embodiments, for the detection of hand access actions, when a hand is detected, an image is captured. The hand access action can be detected by a mature deep learning method to obtain an image of accessing food. The neural network used for image detection can be but is not limited to a (deep) neural network, a convolutional neural network, a deep belief network, a deep stack neural network, a deep fusion network, a deep recursive neural network, a deep recurrent neural network, a deep Bayesian neural network, a deep generative network, a deep reinforcement learning network structure or a derivative model thereof. Regardless of which deep neural network is used, it does not affect the essence of the embodiments of the present application.

[0105] S402: Detect the first visible light image to obtain the type of each food in the first visible light image and the location area of ​​each food in the first visible light image.

[0106] In this step, the first visible light image captured when the hand is detected can be used as a key frame image, and the key frame image is used to identify the type and location area of ​​the food. The recognition method can use traditional feature extraction (such as Histogram of Oriented Gradient (HOG), Local Binary Pattern (LBP)) and classification to realize the feature extraction of the first visible light image and the method of classifier learning and training, wherein the classifier can use support vector machine (SVM), adaboost, decision tree, etc. Deep learning training methods can also be used to identify food types and location areas, such as YOLO series algorithms, SSD algorithms, etc. It should be noted that the embodiments of the present application do not limit the detection method of food.

[0107] In the embodiment of the present application, when detecting the type of food, a detection frame may contain one food or multiple food of the same type. This mainly depends on factors such as the placement of the food and the degree of aggregation. Among them, the information of a food detection frame may include the position, size and type of food of the food detection frame. For example, the information of the i-th food detection frame can be expressed as (x i ,y i , w i ,h i , c i , pi ), where i represents the i-th food detection frame, i=1, 2, ..., (x i ,y i ,) is the center point coordinate of the food detection frame, (w i ,h i ) is the width and height of the food detection frame, c i is the type of food to which the detection frame belongs, p i is the probability (or confidence) of identifying it as the type of food.

[0108] Figure 6 A schematic diagram of a food detection result is shown as an example. As shown in the figure, there are multiple crabapples stacked together on the left side of the figure, and the food detection frame detected after the food detection is shown in the figure. The area of ​​detection frame 601 is relatively large, almost covering the crabapples stacked on the left side, and the area of ​​detection frames 602~604 is relatively small, which only includes a single crabapple with a relatively complete shape, and detection frames 602~604 are covered by detection frame 601 (as shown in the diagonal box in the figure). The type of food identified in each food detection frame in the figure is "crabapple".

[0109] When users store and retrieve food, they can store and retrieve multiple food at the same time (such as a box, a pile, a plate, a stick, a bundle or a bag, etc.). When determining the location area of ​​each food, a clustering algorithm can be used to merge the location areas of multiple food of the same type.

[0110] Still Figure 6 For example, after the food detection frame is detected, since the detection frames 602 to 604 are of the same type as the food in the detection frame 601 and are covered by the detection frame 601, the k-means algorithm can be used to merge the detection frames 602 to 604 and the detection frame 601 into a detection frame 605, that is, to obtain a food and a location area of ​​the food, such as Figure 6 Shown on the right.

[0111] S403: Obtain 3D coordinate information of each pixel point according to the first depth image.

[0112] In this step, the 3D coordinate information of each pixel in the first depth image is obtained according to the conversion relationship between the box coordinate system and the depth camera coordinate system.

[0113] S404: Determine the 3D coordinate range of each food ingredient according to the 3D coordinate information of each pixel point and the position area of ​​each food ingredient in the first visible light image.

[0114] In this step, the 3D coordinate information of each pixel point in the first depth image includes not only the 3D coordinate information of the currently accessed food, but also the 3D coordinate information of the background food, the 3D coordinate information of the user's arm, and the 3D coordinate information of each area in the refrigerator. Combined with the position area of ​​each food in the first visible light image obtained in S402, the 3D coordinate range of each food in the image is determined based on the conversion relationship between the 3D coordinates of the depth image and the pixel coordinates of the visible light image.

[0115] S405: According to the 3D coordinate range of each ingredient, the ingredient whose 3D coordinate range is located in a set space is determined as the currently accessed ingredient; wherein the set space is located at the front side of the refrigerator and is the area through which the accessed ingredients pass during the food access process.

[0116] When the user opens the refrigerator door to access food, the relative position of the area inside the refrigerator (such as partitions, drawers, refrigerator doors, etc.) and the depth camera is fixed. The area on the front of the refrigerator where the food is accessed during the access process can be set as a set space. During the user's access process, the food in the refrigerator area will not be in the set space, and only the food being accessed can pass through the set space. When the relative position of the area inside the refrigerator (such as partitions, drawers, refrigerator doors, etc.) and the depth camera changes, such as the position of the partition inside the refrigerator moves, the refrigerator door moves, etc., the position movement parameters can be detected in real time, and the position of the set space can be calculated in real time based on the spatial coordinate conversion relationship.

[0117] According to the 3D coordinate range of each ingredient obtained in S404, it is determined whether the 3D coordinate range of each ingredient is within the set space. If so, the ingredient is determined as the currently accessed ingredient. If not, the ingredient may be a background ingredient.

[0118] Figure 7 The schematic diagram for determining the currently accessed food is illustrated. As shown in the figure, the space between plane H3 and plane H4 is the set space, the 3D coordinate range of cabbage is (xf1~xf2, yf1~yf2, zf1~zf2), and the 3D coordinate range of eggs is (xe1~xe2, ye1~ye2, ze1~ze2). The 3D coordinate range of cabbage is within the set space (the set space does not match the built-in area of ​​the refrigerator), and the 3D coordinate range of eggs is not within the set space. It can be determined that cabbage is the currently accessed food, and eggs are background food.

[0119] In an embodiment of the present application, a depth image of the user during the process of accessing food is captured by a depth camera, and at the same time, a visible light image associated with the depth image is captured by a visible light camera, and the visible light image is detected to obtain at least one food in the visible light image and the position area of ​​each food in the visible light image, and the 3D coordinate information of each pixel in the depth image is combined to obtain the 3D coordinate range of each food, and the area on the front side of the refrigerator where the accessed food passes during the food access process is set as a set space, and the food whose 3D coordinate range is located in the set space is determined as the food currently being accessed, thereby realizing the determination of the accessed food through the visible light image and the depth image, and improving the accuracy of identifying the accessed food.

[0120] In some embodiments, during the process of storing and retrieving food, the user's arm is mostly included in the captured image. Food whose 3D coordinate range is not within the set space may also be the food currently being stored and retrieved. In this case, the food currently being stored and retrieved can be determined in combination with the user's arm information.

[0121] Specifically, if it is determined that the 3D coordinate range of each food in the first visible light image is not within the set space, the controller performs user arm detection on the acquired first visible light image to obtain the position area of ​​the user arm in the first visible light image, and obtains the 3D coordinate range of the user arm based on the position area of ​​the user arm in the first visible light image and the 3D coordinate information of each pixel in the first depth image. The 3D coordinate range of each food is obtained in combination with S404, and it is determined whether the 3D coordinate range of each food overlaps with the 3D coordinate range of the user arm and whether the overlap ratio is greater than the set threshold. If so, the food is determined as the currently accessed food; otherwise, the food may be a background food.

[0122] Figure 8 The schematic diagram of using the 3D coordinate range of the user's arm and each ingredient to determine the currently accessed ingredient is exemplified. As shown in the figure, the determined 3D coordinate range of the user's arm is (xh1~xh2, yh1~yh2, zh1~zh2), the 3D coordinate range of the ingredient pear is (xp1~xp2, yp1~yp2, zp1~zp2), and the 3D coordinate range of the ingredient apple is (xc1~xc2, yc1~yc2, zc1~zc2). The 3D coordinate range of the ingredient pear overlaps with the 3D coordinate range of the user's arm, and the overlap ratio with the arm is greater than the set threshold value T%, then the ingredient pear is determined as the currently accessed ingredient, and the ingredient apple is the background ingredient.

[0123] In an embodiment of the present application, each ingredient and its position area in the visible light image, as well as the user's arm and its position area in the visible light image are obtained by detecting the visible light image, and the 3D coordinate range of each ingredient and the 3D coordinate range of the user's arm are obtained according to the 3D coordinate information of each pixel in the depth image and the position area in the visible light image. The currently accessed ingredient is determined according to the overlapping ratio threshold of the 3D coordinate range of the ingredient and the 3D coordinate range of the user's arm, thereby improving the accuracy of identifying the accessed ingredient.

[0124] In other embodiments, the food accessed by the user is in motion, and its 3D coordinate range changes in consecutive frames of images, while the background food in the refrigerator area is stationary, and the 3D coordinate range of the background food does not change in consecutive frames of images. If the 3D coordinate range of each food is not within the set space, the currently accessed food can also be determined based on the changes in the 3D coordinate range of the food in the consecutive images.

[0125] Specifically, if it is determined that the 3D coordinate range of each food in the first visible light image is not within the set space or the user's arm is not detected in the first visible light image, the controller acquires N consecutive frames of depth images including the first depth image acquired by the depth camera, and at the same time, acquires N frames of visible light images acquired by the visible light camera, where N is an integer greater than 1, wherein the consecutive N frames of depth images are acquired at the same time as the N frames of depth images and are temporally correlated with each other; the N frames of visible light images are detected respectively to obtain the type of each food in the N frames of visible light images and the position area of ​​each food in the N frames of visible light images, and the detection process is shown in S402; the 3D coordinate range of each food in the N frames of depth images is determined according to the 3D coordinate information of each pixel point in the N frames of depth images and the position area of ​​each food in the N frames of visible light images, and the 3D coordinate range determination process is shown in S403-S404; it is determined whether the 3D coordinate range of each food changes in the N consecutive frames of depth images, and if it changes, it is determined that the food is the currently accessed food, and if it does not change, it is determined that the food is the background food.

[0126] by Figure 7 Taking the pear and apple in the figure as examples, the 3D coordinate range of the ingredient pear in the continuous N-frame depth image changes to (xpv1~xpv2, ypv1~ypv2, zpv1~zpv2), and the 3D coordinate range of the ingredient apple in the continuous N-frame depth image changes to (xc1~xc2, yc1~yc2, zc1~zc2). The 3D coordinate of the ingredient pear changes, and the ingredient pear is determined to be the ingredient currently being accessed.

[0127] In an embodiment of the present application, each ingredient and the position area of ​​each ingredient in the visible light image are obtained by detecting N consecutive frames of visible light images, and the 3D coordinate range of each ingredient is obtained based on the 3D coordinate information of each pixel point in N consecutive frames of depth images and the position area of ​​each ingredient in the visible light image, so that the ingredient whose 3D coordinate range has changed is determined as the currently accessed ingredient, thereby improving the accuracy of identifying the accessed ingredient.

[0128] In some embodiments, during the process of accessing food, if the food being accessed is very small, such as lychee, the coordinate range of the food overlapping with the user's arm may be smaller than the current threshold, and the food may be mistakenly identified as background food. Therefore, the food currently being accessed can be determined by the changes in the overlapping range of the user's arm and the food and the 3D left range of the food.

[0129] Specifically, if it is determined that the 3D coordinate range of each food ingredient in the first visible light image is not within the set space, the controller performs user arm detection on the first visible light image to obtain the user arm in the first visible light image and the position area of ​​the user arm in the first visible light image; and obtains the 3D coordinate range of the user arm according to the position area of ​​the user arm and the 3D coordinate information of each pixel in the first depth image; determines whether the 3D coordinate range of each food ingredient overlaps with the 3D coordinate range of the user arm and whether the overlap ratio is greater than a set threshold; if there is overlap but the overlap ratio is less than the set threshold, obtains N consecutive frames of depth images containing the first depth image collected by the depth camera and N consecutive frames of visible light images associated with the N frames of depth images collected by the visible light camera; detects the N frames of visible light images respectively to obtain the type of each food ingredient and the position area of ​​each food ingredient in the N frames of visible light images; determines the 3D coordinate range of each food ingredient in the N frames of depth images in combination with the 3D coordinate information of each pixel in the N frames of depth images; determines the food ingredient whose 3D coordinate range changes in the N consecutive frames of depth images as the currently accessed food ingredient.

[0130] In an embodiment of the present application, in the case where the 3D coordinate range of the food and the 3D coordinate range of the user's arm overlap but the overlapping area is smaller than a set threshold, the food whose 3D coordinate range changes in N consecutive frames of images is determined as the food currently being accessed, thereby avoiding determining the food currently being accessed as background food when the overlap ratio between the user's arm and the food is smaller than a threshold, thereby improving the accuracy of identifying the food being accessed.

[0131] The method for determining the currently accessed food in the embodiment of the present application can be used alone or in combination.

[0132] Fig. 9 The schematic diagram of the process of the method for identifying and accessing food provided in the embodiment of the present application is exemplarily shown. As shown in the figure, the process includes the following steps:

[0133] S901: Acquire a depth image acquired by a depth camera and a visible light image associated with the depth image acquired by a visible light camera.

[0134] S902: Detect the visible light image to obtain the type of each food in the visible light image and the location area of ​​each food in the visible light image, the user's arm and the location area of ​​the user's arm in the visible light image.

[0135] In this step, the process of detecting the user's arm may be the same as the method of detecting the type of food, or may be different from the method of detecting the type of food.

[0136] S903: Obtain 3D coordinate information of each pixel point according to the depth image.

[0137] S904: Determine the 3D coordinate range of each ingredient based on the 3D coordinate information of each pixel point and the position area of ​​each ingredient in the visible light image, and determine the 3D coordinate range of the user's arm based on the 3D coordinate information of each pixel point and the position area of ​​the user's arm in the visible light image.

[0138] In this step, the method for determining the 3D coordinate range of the user's arm is the same as the method for determining the 3D coordinate range of each food ingredient.

[0139] S905: According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

[0140] In the above embodiment, the related descriptions of S901-S905 are the same as those in the previous embodiment and will not be repeated here.

[0141] In an embodiment of the present application, each ingredient and its position area in the visible light image, as well as the user's arm and its position area in the visible light image, are detected by the visible light image to obtain; the 3D coordinate range of each ingredient and the 3D coordinate range of the user's arm are obtained based on the 3D coordinate information of each pixel in the depth image and the position area in the visible light image; the ingredient whose overlapping ratio with the 3D coordinate range of the user's arm is greater than a set threshold is determined to be the currently accessed ingredient, thereby improving the accuracy of identifying the accessed ingredient.

[0142] Fig.10 The following is a schematic diagram of another method for identifying and accessing food provided in an embodiment of the present application. As shown in the figure, the process includes the following steps:

[0143] S1001: Acquire N consecutive frames of depth images and N consecutive frames of visible light images associated with the N frames of depth images, where N is an integer greater than 1.

[0144] S1002: Detect N frames of visible light images respectively to obtain the type of each food in the N frames of visible light images and the location area of ​​each food in the N frames of visible light images.

[0145] S1003: Determine the 3D coordinate range of each food ingredient in the N frames of depth images according to the 3D coordinate information of each pixel in the N frames of depth images and the position area of ​​each food ingredient in the N frames of visible light images.

[0146] S1004: According to the 3D coordinate range of each ingredient, the ingredient whose 3D coordinate range in the N frames of depth images is changed is determined as the currently accessed ingredient.

[0147] In the above embodiment, the description of S1001-S1003 is the same as that of S401-S404, which will not be repeated here.

[0148] In an embodiment of the present application, N consecutive frames of depth images captured by a depth camera are obtained, and N consecutive frames of visible light images associated with the depth images captured by a visible light camera are obtained, and the N consecutive frames of visible light images are detected respectively to obtain each ingredient and the position area of ​​each ingredient in the visible light image, and the 3D coordinate range of each ingredient is obtained based on the 3D coordinate information of each pixel point in the N consecutive frames of depth images and the position area of ​​each ingredient in the visible light image, thereby determining the ingredient whose 3D coordinate range has changed as the currently accessed ingredient, thereby improving the accuracy of identifying the accessed ingredient.

[0149] The method for identifying stored and accessed food provided in the embodiment of the present application is applied to a smart refrigerator, so that the smart refrigerator can monitor the food information in the refrigerator in real time. When the stored food is insufficient, the smart refrigerator sends information to the user to replenish the food; when the stored food is sufficient, the smart refrigerator sends information to the user that there is no need to replenish the food, thereby realizing intelligent food management by the smart refrigerator.

[0150] Specifically, a smart home management application can be installed on the user terminal, through which the user can set up the smart refrigerator, or receive information sent by the smart refrigerator, or send instructions (such as query instructions) to the smart terminal. Among them, the user terminal information associated with the smart refrigerator (such as the call number of the user terminal) or the user's related information (such as the user's identification) can be set in the server in advance, so that the server can communicate with the user terminal of the user. Among them, the user terminal includes but is not limited to: mobile phones, PDAs, wearable devices, etc. Furthermore, the smart terminal can be connected to the server, and the server can communicate with the user terminal through the mobile communication network, so that the user can communicate remotely with the smart refrigerator through the user terminal.

[0151] For example, when the number of times a certain type or certain types of ingredients are stored in a smart refrigerator is greater than a set threshold, the smart refrigerator may send a prompt message to the server so that the server sends the prompt message to the target user terminal. The prompt message is used to prompt that the storage capacity of the corresponding type of ingredients is too high and the ingredients do not need to be replenished. When the number of times a certain type or certain types of ingredients are taken out of a smart refrigerator is greater than a set threshold, the smart refrigerator may send a prompt message to the server so that the server sends the prompt message to the target user terminal. The prompt message is used to prompt that the storage capacity of the corresponding type of ingredients is too low and the ingredients need to be replenished. Users can check the status of refrigerator ingredients through the APP anytime and anywhere. For example, when a user is buying ingredients outside, he can view the current ingredients in the smart refrigerator through the mobile phone application, thereby assisting the user in purchasing ingredients.

[0152] In the embodiment of the present application, the user can set personalized reminders for food purchase. The user can set reminders for food purchase for certain necessary ingredients according to their needs. When these ingredients are taken out frequently, reminder information will be displayed on the display screen of the smart refrigerator, and the reminder information can also be sent to the user's mobile phone to remind the user to replenish the ingredients as soon as possible, thereby realizing the intelligent management of food by the smart refrigerator.

[0153] According to another aspect of the exemplary implementation, the embodiment of the present application further provides a computer storage medium, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the processing method as described above.

[0154] Since the intelligent terminal and computer storage medium in the embodiments of the present application can be applied to the above-mentioned processing method, the technical effects that can be obtained can also refer to the above-mentioned method embodiments, and the embodiments of the present application will not be repeated here.

[0155] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0156] Although the specific implementation methods of the present application are described above, those skilled in the art should understand that these are only examples, and the protection scope of the present application is defined by the appended claims. Those skilled in the art may make various changes or modifications to these implementation methods without departing from the principles and essence of the present application, but these changes and modifications all fall within the protection scope of the present application.

Claims

1. A smart refrigerator, characterized in that: The refrigerator comprises: a box body and a refrigeration unit, wherein a door body is installed on the box body; The box is provided with a depth camera and a visible light camera, which collect a depth image and a visible light image in the front area of ​​the box when the door is opened; The depth camera and the visible light camera are connected to a controller, and the controller is configured as follows: Acquire a first depth image acquired by the depth camera, and a first visible light image associated with the first depth image acquired by the visible light camera; Detecting the first visible light image to obtain the type of each food in the first visible light image and the location area of ​​each food in the first visible light image; Obtaining 3D coordinate information of each pixel point according to the first depth image, wherein the 3D coordinate information includes 3D coordinate information of background food, 3D coordinate information of the user's arm, and 3D coordinate information of each area in the refrigerator; According to the conversion relationship between the 3D coordinates of the depth image and the pixel coordinates of the visible light image, the position area of ​​each food in the first visible light image is converted into 3D coordinate information to obtain the 3D coordinate range of each food; According to the 3D coordinate range of each food, the food whose 3D coordinate range is located in a set space is determined as the food currently being accessed; wherein the set space is located at the front side of the refrigerator and is the area through which the food being accessed passes during the food access process; if the 3D coordinate range of each food in the first visible light image is not within the set space, the first visible light image is detected to obtain the user's arm in the first visible light image; Obtaining a 3D coordinate range of the user's arm according to the position area of ​​the user's arm in the first visible light image and the 3D coordinate information of each pixel point in the first depth image; According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

2. The smart refrigerator according to claim 1, characterized in that: The controller is further configured to: If the 3D coordinate range of each food in the first visible light image is not within the set space or the user's arm is not detected in the first visible light image, then acquiring N consecutive frames of depth images including the first depth image and N consecutive frames of visible light images associated with the N frames of depth images, where N is an integer greater than 1; Detecting the N frames of visible light images respectively to obtain the type of each food in the N frames of visible light images and the location area of ​​each food in the N frames of visible light images; Determine the 3D coordinate range of each food ingredient in the N depth images according to the 3D coordinate information of each pixel in the N depth images and the position area of ​​each food ingredient in the N visible light images; According to the 3D coordinate range of each food, the food whose 3D coordinate range in the N frames of depth images has changed is determined as the food currently being accessed.

3. The smart refrigerator according to any one of claims 1 to 2, characterized in that: The depth camera is any one of a binocular camera, a structured light camera, and a TOF camera.

4. A smart refrigerator, applied to a smart refrigerator, characterized in that: The refrigerator comprises: a box body and a refrigeration unit, wherein a door body is installed on the box body; The box is provided with a depth camera and a visible light camera, which collect a depth image and a visible light image in the front area of ​​the box when the door is opened; The depth camera and the visible light camera are connected to a controller, and the controller is configured as follows: Acquire a first depth image acquired by the depth camera, and a first visible light image associated with the first depth image acquired by the visible light camera; Detecting the first visible light image to obtain the type of each food in the first visible light image and the location of each food in the first visible light image, and obtaining the user's arm and the location of the user's arm in the first visible light image; Obtaining 3D coordinate information of each pixel point according to the first depth image, wherein the 3D coordinate information includes 3D coordinate information of background food, 3D coordinate information of the user's arm, and 3D coordinate information of each area in the refrigerator; According to the conversion relationship between the 3D coordinates of the depth image and the pixel coordinates of the visible light image, the position area of ​​each food in the first visible light image is converted into 3D coordinate information to obtain the 3D coordinate range of each food, and according to the 3D coordinate information of each pixel point and the position area of ​​the user's arm in the first visible light image, the 3D coordinate range of the user's arm is determined; According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

5. A method for identifying and accessing food, characterized in that: include: Acquire a first depth image acquired by a depth camera, and a first visible light image associated with the first depth image acquired by a visible light camera; Detecting the first visible light image to obtain the type of each food in the first visible light image and the location area of ​​each food in the first visible light image; Obtaining 3D coordinate information of each pixel point according to the first depth image, wherein the 3D coordinate information includes 3D coordinate information of background food, 3D coordinate information of the user's arm, and 3D coordinate information of each area in the refrigerator; According to the conversion relationship between the 3D coordinates of the depth image and the pixel coordinates of the visible light image, the position area of ​​each food in the first visible light image is converted into 3D coordinate information to obtain the 3D coordinate range of each food; According to the 3D coordinate range of each food, the food whose 3D coordinate range is located in a set space is determined as the food currently being accessed; wherein the set space is located at the front side of the refrigerator and is the area through which the food being accessed passes during the food access process; if the 3D coordinate range of each food in the first visible light image is not within the set space, the first visible light image is detected to obtain the user's arm in the first visible light image; Obtaining a 3D coordinate range of the user's arm according to the position area of ​​the user's arm in the first visible light image and the 3D coordinate information of each pixel point in the first depth image; According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

6. A method for identifying and accessing food, characterized in that: include: Acquire a depth image acquired by a depth camera and a visible light image associated with the depth image acquired by a visible light camera; Detecting the visible light image to obtain the type of each food in the visible light image and the location area of ​​each food in the visible light image, and obtaining the user's arm and the location area of ​​the user's arm in the visible light image; Obtaining 3D coordinate information of each pixel point according to the depth image, wherein the 3D coordinate information includes 3D coordinate information of background food, 3D coordinate information of the user's arm, and 3D coordinate information of each area in the refrigerator; According to the conversion relationship between the 3D coordinates of the depth image and the pixel coordinates of the visible light image, the position area of ​​each food in the visible light image is converted into 3D coordinate information to obtain the 3D coordinate range of each food, and according to the 3D coordinate information of each pixel point and the position area of ​​the user's arm in the visible light image, the 3D coordinate information of the user's arm is determined; According to the 3D coordinate range of each ingredient, an ingredient whose 3D coordinate range overlaps with the 3D coordinate range of the user's arm and whose overlap ratio is greater than a set threshold is determined as the currently accessed ingredient.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause the computer to execute the method according to any one of claims 5 to 6.

Images