A refrigerator and food volume estimation method
Through camera module and depth detection technology, the identification of ingredients and hand postures is solved, and the problem that the refrigerator cannot accurately estimate the food volume is achieved, fast and simple food volume estimation is achieved, and the intelligent and user experience of refrigerator food management is improved.
Patent Information
- Application Number
- CN202010461454.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-05-27
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-05-27
AI Technical Summary
The existing refrigerator cannot accurately estimate the volume of ingredients placed in, which makes it difficult to recommend recipes and space allocation. Users need to manually enter the volume and have low accuracy, making the operation cumbersome.
The camera module collects food images, recognizes the food area and hand posture, combines depth detection technology to restore the three-dimensional space area of the food, calculates its volume, and accurately estimates it based on the food type and shape characteristics.
It realizes rapid and concise estimation of the volume of ingredients, improves the intelligence of refrigerator ingredient management, ensures the accuracy of recipe recommendations and space allocation, and improves user experience.
Smart Images

Figure CN113486689B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a method for estimating the volume of a refrigerator and food. Background Art
[0002] At present, with the development of the artificial intelligence industry, many manufacturers have applied artificial intelligence algorithms to the refrigerator field, so that the refrigerator has the function of intelligently managing food. Furthermore, the recognition processing center in the refrigerator can enable the refrigerator to identify the category of food through a deep learning algorithm and a food category recognition model. The food category recognition model is obtained by collecting a large number of food images for training, and then actively manages the food in the refrigerator according to the food category. The active management is that the recognition processing center implements storage location recommendation, recipe recommendation and other functions according to the category of food in the refrigerator. However, estimating the volume of food placed in the refrigerator is a major difficulty in the current management of refrigerator food. Since the recognition processing center cannot obtain the volume of food placed in the refrigerator, it is impossible to recommend recipes based on the stock of food in the refrigerator, and it is also impossible to determine the space allocation of each layer in the refrigerator to determine whether there is enough space in the current refrigerator to store food. The current refrigerator obtains volume information mostly by users self-estimating the volume and manually inputting it into the refrigerator, which has a very low accuracy. When the user actively inputs the volume of food, the operation is cumbersome, which is poor for user experience, and the operation of selecting and inputting different categories of food is more complicated. Therefore, how to simply and quickly estimate the volume of food has become a problem to be solved urgently. Summary of the invention
[0003] As described in the background technology, active management is for the recognition processing center to recommend storage locations and recipes based on the types of food in the refrigerator. However, estimating the volume of food in the refrigerator is a major difficulty in managing food in the refrigerator. Since the recognition processing center cannot obtain the volume of food in the refrigerator, most existing technologies rely on users to estimate the volume and manually input it into the refrigerator. The accuracy is very low, and the operation is cumbersome when the user actively inputs the volume of food. The user experience is poor, and the selection and input operation of different types of food is more complicated. The present invention is to estimate the volume of food simply and quickly through hand gestures and food types.
[0004] A first aspect of the present invention provides a refrigerator, comprising:
[0005] a housing, the housing comprising a storage chamber, the storage chamber having an opening;
[0006] A door body, movably connected to the housing, and used to cover the opening;
[0007] A camera module, used for collecting images and sending the images to a controller;
[0008] The controller is configured as:
[0009] Receiving an image sent by the camera module;
[0010] Identify the food area in the image and determine the coordinates of different points of the food in the food area;
[0011] Identifying a hand region in the image, and determining a hand posture according to the identified hand region;
[0012] The three-dimensional space area of the food is restored according to the coordinates of different points of the food in the food area and the hand gestures, and the volume of the food is determined according to the three-dimensional space area.
[0013] In some embodiments of the present application, the controller is further configured to:
[0014] Identifying the type and location of food in the image;
[0015] According to the identified food type and location, the identified food is edge marked to obtain the food area.
[0016] In some embodiments of the present application, the controller is further configured to:
[0017] determining coordinates of different points on the upper surface of the food according to the coordinates of different points of the food in the food area, and predicting an invisible area in the hand area according to the hand gesture;
[0018] Combining the upper surface and the invisible area, restoring the three-dimensional space area of the food;
[0019] The depth differences of relative points in the three-dimensional space area are accumulated to determine the volume of the food.
[0020] In some embodiments of the present application, the controller is further configured to:
[0021] Predicting coordinates of different points of the invisible area according to the visible area in the hand area, and adjusting the envelope formed by combining the upper surface and the invisible area according to the shape characteristics of the food, accumulating depth differences of relative points of the adjusted envelope, and determining the volume of the food; or
[0022] The coordinates of different points in the invisible area are predicted according to the visible area in the hand area, and the volume of the food is determined by accumulating the depth difference between the coordinates of different points on the upper surface and the coordinates of different points in the invisible area.
[0023] In some embodiments of the present application, the controller is further configured to:
[0024] Extracting feature data of the visible area of the hand area, matching it with feature data corresponding to different pre-established hand grips, and determining the hand grip according to the matching result;
[0025] The coordinates of different points in the invisible area are obtained according to the coordinates of different points in the visible area by using the pre-established mapping functions corresponding to different hand grip postures.
[0026] In some embodiments of the present application, the controller is further configured to:
[0027] The coordinates and feature data of different points in the visible area of the hand area are extracted, and input into a grip prediction model to obtain the hand grip and the coordinates of different points in the invisible area. The grip prediction model uses the coordinates and feature data of different points in the visible area that identify the hand grip and the coordinates of different points in the invisible area as input, and outputs the hand grip and the coordinates of different points in the invisible area as the goal to perform network model training, and the network model is a neural network model.
[0028] In some embodiments of the present application, the visible area feature data of the hand area includes at least one of the following:
[0029] The spatial coordinate values of fingers, palm, wrist and arm.
[0030] In some embodiments of the present application, the controller is further configured to:
[0031] Determine at least one of the total weight of the food, the total calories of the food, and the total nutritional content of the food according to the type of the food and the determined volume of the food;
[0032] A nutritional analysis is performed based on at least one of the total weight of the ingredients, the total calories of the ingredients, and the total nutritional content of the ingredients, and information is pushed based on the analysis results.
[0033] According to one aspect of an exemplary embodiment, a method for estimating food volume is provided, the method comprising:
[0034] Receiving an image sent by the camera module;
[0035] Identify the food area in the image and determine the coordinates of different points of the food in the food area;
[0036] Identifying a hand region in the image, and determining a hand posture according to the identified hand region;
[0037] The three-dimensional space area of the food is restored according to the coordinates of different points of the food in the food area and the hand gestures, and the volume of the food is determined according to the three-dimensional space area.
[0038] In the above-mentioned embodiments of the present application, a camera module arranged on the refrigerator collects images in the front area of the refrigerator to determine the type and volume of the stored ingredients, and the ingredient database of the refrigerator is updated based on the detected type of ingredients and the remaining storage space, thereby improving the intelligence of refrigerator ingredient management, and realizing functions such as storage location recommendation and recipe recommendation according to the type and quantity of ingredients in the refrigerator.
[0039] On the basis of conforming to the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is an application scenario diagram of a refrigerator;
[0041] Figure 2a is a structural schematic diagram of a refrigerator;
[0042] Figure 2b The present invention is a schematic diagram of a compartment structure of a refrigerator;
[0043] Figure 3 A schematic diagram of the front area of a refrigerator body;
[0044] Figure 4 A configuration block diagram of a refrigerator;
[0045] Figure 5 A block diagram of the architecture configuration of an operating system in a memory in a refrigerator;
[0046] Figure 6 A controller in a refrigerator is configured to execute steps;
[0047] Figure 7 A schematic diagram of the food area and surface coordinates in the captured image;
[0048] Figure 8 Flowchart of a complete method for estimating the volume of an ingredient. DETAILED DESCRIPTION
[0049] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0050] In the description of the present disclosure, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.
[0051] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present disclosure, unless otherwise specified, "plurality" means two or more.
[0052] In the description of the present disclosure, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.
[0053] In the description of the present disclosure, the coordinates of different points may be the coordinates of each point collected according to the sampling density, or may be the coordinates of some points after the coordinates of each point are screened. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.
[0054] In the prior art, since the recognition processing center cannot obtain the volume of the food placed in the refrigerator, it is impossible to recommend food based on the stock of food in the refrigerator, and it is also impossible to determine the space allocation of each layer in the refrigerator to determine whether there is enough space in the current refrigerator to store the food; currently, the refrigerator obtains volume information mostly by the user self-estimating the volume and manually inputting it into the refrigerator, and its accuracy is very low. When the user actively inputs the volume of the food, the operation is cumbersome, the user experience is poor, and the operation of selecting and inputting different categories of food is more complicated. The method provided in this embodiment is not only applicable to refrigerators, but also to equipment such as freezers.
[0055] Intelligent management of food information in refrigerators has become one of the current trends in refrigerator development. Food management is based on mastering the correct types and storage of food in the refrigerator. Since the types of food stored in the refrigerator are numerous and complex, and the amount of food stored and taken out by users each time is not fixed, and the unit attributes of some food are fuzzy, simple image recognition cannot correctly identify the specific volume of the food, which will lead to the inability of refrigerators to effectively manage food. If only a weight sensor is used to measure the weight of the food, the simple weight sensor can only measure the weight and cannot accurately identify the type of food.
[0056] Based on this, an embodiment of the present application proposes a method for estimating the volume of food combined with depth detection. In the embodiment of the present application, a camera module is used to capture food images and perform image recognition to obtain the type and volume of the food. By combining the type and volume of the food and the food feature database, the volume and distribution of various types of food placed in the refrigerator can be accurately grasped to realize the food management function of the refrigerator.
[0057] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0058] See also Figure 1 The figure shows an application scenario diagram of a refrigerator provided in some embodiments of the present disclosure.
[0059] The refrigerator 100 may provide functions including but not limited to food preservation and food volume estimation. The specific refrigerator type, size, power, etc. are not limited.
[0060] The refrigerator 100 also communicates data with the server 200 through a variety of communication methods. Here, the refrigerator 100 can be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 200 can provide various content and interactions to the refrigerator 100. In an embodiment, the refrigerator 100 can send and receive information, for example: receiving moving target recognition data, accessing a remotely stored digital media library, and sending image data to be detected. The server 200 can be a group or multiple groups, and can be one or more types of servers. The server 200 can be deployed locally or in the cloud, and functions such as image recognition are implemented through the server 200.
[0061] Specifically, Figure 2a and Figure 2b The structure of a refrigerator provided by an embodiment of the present application is illustrated in an exemplary embodiment.
[0062] like Figure 2aAs shown, the refrigerator includes a box body 110, a refrigeration unit (not shown in the figure) and other accessories (for example, a lighting lamp, a thermometer, etc. may be provided in the box body, which are not shown in the figure). The refrigeration system mainly comprises components such as a compressor, a condenser, an evaporator and a capillary throttle, forming a closed circulation system. Among them, the evaporator can be installed on the upper part of the refrigerator, and the other components are installed on the back of the refrigerator.
[0063] The box body 110 is installed with a door body 120 , and the door body 120 may be further provided with a display screen 150 , and the display screen 150 is coupled to the controller (eg, connected via a circuit).
[0064] The box body 110 may include a storage room. The appearance of the refrigerator 100 may be determined by the storage room defining the storage space and the door body 120 disposed in the storage room. In some embodiments, the storage room is a box body with an opening, which is formed by a storage room inner liner, a storage room outer shell, and a foaming layer located therebetween. The door body 120 is used to shield the opening of the storage room. The storage room is vertically divided into a freezer located below and a refrigerating room located above. Among them, the freezer and the refrigerating room may have independent storage spaces respectively.
[0065] In some embodiments, the door body 120 may include a freezer door body and a refrigerating door body. Also, the freezer compartment is defined at the lower side of the storage compartment and the opening of the freezer compartment may be selectively covered by the freezer door body.
[0066] In some embodiments, the refrigerating chamber is defined at the upper side of the storage chamber, that is, the refrigerating chamber is arranged above the freezing chamber. And, the opening of the refrigerating chamber is selectively covered by the refrigerating chamber door. In practical applications, the refrigerating chamber door is pivotally mounted on the refrigerating chamber, so that the opening of the refrigerating chamber can be selectively opened or closed by the refrigerating chamber door.
[0067] In some embodiments, the storage room of the refrigerator in the embodiment of the present invention may include a storage drawer and a first shelf and a second shelf located on the storage drawer. The first shelf and the second shelf of the storage drawer may be used to hold food (e.g., fruits, vegetables, etc.) respectively.
[0068] A camera module 130 may also be provided on the cabinet 110, and the camera module may capture images in the front area of the cabinet 110. The front area of the cabinet 110 includes at least an area extending a certain distance from the first plane to the outside of the refrigerator, with the plane where the refrigerator door is located being the first plane. The camera module may capture images of the area, that is, it may capture images of the hand movements of the user during the process of storing and accessing food after opening the door 120, as well as images of the food stored and accessed.
[0069] like Figure 2bAs shown, the refrigerator body 110 may include a plurality of compartments (such as compartments 50a to 50e in the figure) to facilitate users to classify and store different ingredients.
[0070] In some embodiments, the camera module 130 may be disposed at an upper portion of the housing 110 near the door 120 so as to be able to capture images in the front area of the housing 110 .
[0071] Figure 3 The schematic diagram of the front area of the cabinet 110 is shown in the embodiment. As shown in the figure, the plane captured by the camera module when the refrigerator door is closed is called plane H1. The camera module must be able to capture images of at least the plane H1 area. The distance between plane H1 and the refrigerator can be set according to factors such as the camera shooting range, the human arm length parameter, and the image recognition accuracy, for example, it can be set to between 25cm and 35cm.
[0072] In some embodiments, the camera module 130 may be disposed on a fixed component, which allows the lens of the camera module to protrude a certain distance from the plane where the door 120 is located, so as to better capture the image in the front area of the box 110. When the user opens the door 120 or triggers the food detection instruction, the camera module can be turned on, and when the user closes the door or triggers the food detection completion instruction, the camera module can be triggered to turn off. The embodiment of the present application does not limit the shape, connection method and material of the fixed component.
[0073] In other embodiments, the camera module 130 may be arranged on a movable part, and the movable part may pop out the camera module 130 when the user opens the door body 120 or triggers the food detection instruction, so that the camera module 130 protrudes a certain distance from the plane where the door body 120 is located, and when the user closes the door body 120 or triggers the food detection completion instruction, the movable part retracts the camera module 130. The movable part may be linked with the door body 120 by mechanical connection or other means, so that the door body 120 triggers the control camera module of the movable part when it is opened and closed. In some embodiments, the movable part may also include a camera protection cover, which is opened and the camera module is started after the camera module is popped out, and the protection cover is closed and the camera module is turned off after the camera module is retracted, thereby protecting the camera module on the one hand and saving power consumption on the other hand.
[0074] It should be noted that Figure 2a and Figure 2bThe structure of the refrigerator shown is only one embodiment. The embodiment of the present application does not limit the size of the refrigerator, the number of doors (for example, it can be a single door or multiple doors), etc. For example, in some embodiments, a radio frequency identification (RFID) reader is provided in the refrigerator, which can be used to read the RFID tag on the food packaging to obtain information such as the type and quantity of the food. In other embodiments, the refrigerator also has a voice function, which can recognize the input voice to obtain information such as the type and quantity of the food input by the user through voice.
[0075] The refrigerator provided in the embodiment of the present application includes a controller (not shown in Figure 2a and Figure 2b As shown in the figure, the controller is coupled to the camera module (for example, connected by a circuit). When the door of the refrigerator is opened or an identification instruction is received, the controller can identify the image captured by the camera module to obtain the type and volume of the accessed food; obtain the distribution of food on each compartment in the refrigerator body, and realize the food management function of the refrigerator according to the identified type and volume of the food to be placed and the distribution of food on each compartment, and perform nutritional analysis, and push information according to the analysis results.
[0076] An embodiment of the present invention provides a refrigerator, comprising:
[0077] a housing, the housing comprising a storage chamber, the storage chamber having an opening;
[0078] A door body, movably connected to the housing, and used to cover the opening;
[0079] A camera module, used for collecting images and sending the images to a controller;
[0080] The controller is configured as:
[0081] Receiving an image sent by the camera module;
[0082] Identify the food area in the image and determine the coordinates of different points of the food in the food area;
[0083] Identifying a hand region in the image, and determining a hand posture according to the identified hand region;
[0084] The three-dimensional space area of the food is restored according to the coordinates of different points of the food in the food area and the hand gestures, and the volume of the food is determined according to the three-dimensional space area.
[0085] In some embodiments of the present invention, the refrigerator may further include, in addition to the controller 410, a memory 420, a communicator 430, a user input interface 440, a user output interface 450, a power supply 460, an image acquisition interface 470, and a display 480. Figure 4 The embodiment shows some configuration block diagrams of refrigerators.
[0086] The controller 410 includes a random access memory (RAM) 411, a read only memory (ROM) 412, a processor 413, a communication interface and a communication bus. The controller 410 is used to control the operation and operation of the above-mentioned devices, as well as the communication cooperation between the internal components and the external and internal data processing functions.
[0087] In some embodiments of the present invention, the camera module may be a color camera, a depth camera, or a combination of the two. The color camera may be a normal color camera or a wide-angle color camera. The depth camera may be a binocular camera, a structured light camera, or a time of flight (TOF) camera.
[0088] Exemplarily, when an interaction of a user pressing a button 444 or touching the touch pad 442 is detected, the controller 410 may control the processor 413 to generate a signal corresponding to the detected interaction and send the signal to the display 480 so that the display 480 can display the corresponding content or screen.
[0089] In some embodiments, the processor 413 may be a central processing unit (CPU), a graphics processing unit (GPU), or a combination of a CPU and a GPU. The processor may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0090] The memory 420 is used to store various operating programs, data and applications driven and controlled under the control of the controller 410. The memory 420 can store various control signal instructions input by the user. In some embodiments, the memory is connected to the processor via a bus or other means, and at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor. In some embodiments, the memory can be a volatile memory (Volatile Memory), a non-volatile memory (Non-volatile Memory) or a combination thereof. The volatile memory can be a random access memory (Random-Access Memory, RAM), such as a static random access memory (Static Random Access Memory, SRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM). The non-volatile memory may be a read-only memory (ROM), such as a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The non-volatile memory may also be a flash memory, a magnetic memory, such as a magnetic tape, a floppy disk, or a hard disk. The non-volatile memory may also be an optical disk.
[0091] The communicator 430 may be a component for communicating with an external device or an external server according to various communication protocol types. For example, the refrigerator may send content data to an external device connected via the communicator 430, or browse and download content data from an external device connected via the communicator 430. The communicator 430 may include a network communication protocol module or a near field communication protocol module such as an infrared signal interface 431, a radio frequency signal interface 432, a WIFI module 433, a Bluetooth module 434, and a wired Ethernet communication protocol module 435, so that the communicator 430 may realize the communication of control signals and data signals with an external device or an external server according to the control of the controller 410. For example: in the case of an infrared signal interface, the infrared control signal sent by the user needs to be converted according to the infrared control protocol and then output to the controller 410. For example, in the case of a radio frequency signal interface, the instruction of the radio frequency control signal mode input by the user needs to be converted and then output to the controller 410. For example, the control signal of the external device controlling the refrigerator is received through the WIFI module 433, the Bluetooth module 434, and the wired Ethernet communication protocol module 435, and the control signal is processed and output to the controller 410.
[0092] The user input interface 440 may include at least one of a microphone 441, a touch pad 442, a sensor 443, a button 444, etc., so that the user can input user instructions for controlling the refrigerator into the refrigerator through voice, touch, gesture, pressing, etc.
[0093] The user output interface 450 outputs the user instructions received by the user input interface 440 to the controller 410, so as to control the processor 413 to execute the corresponding program steps through the controller 410. Alternatively, after the processor 413 executes the corresponding program steps, it is also possible to control the display 480 to display the corresponding screen or output the corresponding content through the user output interface. Here, the user output interface 450 may include an LED interface 451, a vibration interface 452 for generating vibrations, a sound output interface 453 for outputting sounds, etc. For example, a display device in an external device may receive an output signal such as audio, video or data from the user output interface 450, and display the output signal in the form of an image on the display device, output it in the form of audio at the sound output interface 453, or output it in the form of vibration at the vibration interface 452.
[0094] The image acquisition interface 470 is used to connect the camera module to the refrigerator by signal. For example, the detection image acquired by the camera module can be transmitted to the processor 413 in the controller 410 through the image acquisition interface 470 .
[0095] The display 480 is used to receive the image signal input by the processor 413 and display the video content, image and menu operation interface. The displayed video content may come from the video content processed by the processor 413, or may come from the video content input by the communicator 430 or the user input interface 440. The display 480 may also simultaneously display the user operation interface UI for controlling the refrigerator.
[0096] Furthermore, the display 480 may also include a display component for presenting images and a driving component for driving image display. Alternatively, if the display 480 is a projection display, it may also include a projection device and a projection screen.
[0097] The power supply 460 is used to provide operating power support for various components in the refrigerator under the control of the controller 410. The power supply 460 can be in the form of a battery and related control circuits.
[0098] Figure 5 The embodiment in the figure shows a block diagram of the architecture configuration of the operating system in the memory 420. The operating system architecture is composed of the application layer, the middleware layer and the kernel layer from top to bottom.
[0099] The application layer includes both built-in applications and non-system-level applications. It is responsible for direct interaction with users. The application layer may include multiple applications, such as settings applications, e-mail applications, media center applications, etc. These applications can be implemented as Web applications, which are executed based on the WebKit engine and can be developed and executed based on HTML5, Cascading Style Sheets (CSS) and JavaScript.
[0100] Here, HTML, the full name of which is HyperText Markup Language, is a standard markup language used to create web pages. Web pages are described by marking tags. HTML tags are used to describe text, graphics, animations, sounds, tables, links, etc. The browser will read the HTML document, interpret the content of the tags in the document, and display it in the form of a web page.
[0101] CSS, the full name of which is Cascading Style Sheets, is a computer language used to express the style of HTML files. It can be used to define style structures, such as fonts, colors, positions, etc. CSS styles can be directly stored in HTML web pages or in separate style files to control the styles in web pages.
[0102] JavaScript is a language used for web page programming. It can be inserted into HTML pages and interpreted and executed by browsers. The interactive logic of web applications is implemented through JavaScript. JavaScript can encapsulate JavaScript extension interfaces through browsers to achieve communication with the kernel layer.
[0103] The middleware layer can provide some standardized interfaces to support the operation of various environments and systems. For example, the middleware layer can be implemented as the Multimedia and Hypermedia Information Coding Experts Group (MHEG) middleware related to data broadcasting, as the DLNA middleware related to external device communication, or as the middleware that provides the browser environment for the application programs in the display device to run.
[0104] The kernel layer provides core system services, such as file management, memory management, process management, network management, system security permission management, etc. The kernel layer can be implemented as a kernel based on various operating systems, for example, a kernel based on the Linux operating system.
[0105] The kernel layer also provides communication between system software and hardware, and provides device driver services for various hardware, such as: providing display drivers for monitors, camera drivers for cameras, key drivers for remote controls, WiFi drivers for WiFi modules, audio drivers for audio output interfaces, power management drivers for power management (PM) modules, etc.
[0106] In some embodiments, the user can also input user commands in a graphical user interface (GUI) displayed on the display 480, and the controller 410 can receive the commands input by the user. Among them, the "user interface" is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display of an electronic device, wherein the control may include icons, controls, menus, tabs, text boxes, dialog boxes, status bars, channel bars, widgets, and other visual interface elements.
[0107] like Figure 6 As shown, the controller provided in some embodiments of the present application may be configured to execute the following program steps:
[0108] Step S601, receiving an image sent by the camera module;
[0109] The refrigerator receives a food detection instruction issued by a user through a user input interface, and sends a shooting instruction to the camera module. The user can input the user instruction of food detection into the refrigerator through voice, touch, gesture, pressing, etc., or when the camera module detects an image of the food to be detected in front of the refrigerator, the camera module sends a shooting instruction to receive the image sent by the camera module;
[0110] In some embodiments, the image sent may be a depth image or a color image, and its image format may be: JPEG, TIFF, RAW, BMP, GIF, PNG, etc.
[0111] Step S602, identifying the food area in the image, and determining the coordinates of different points of the food in the food area;
[0112] Among them, the image types can be divided into two categories. When there is a depth camera in the camera module, the coordinates of different points of the food in the food area can be directly obtained through the depth camera. However, when there is no depth camera in the camera module, the coordinates of different points in the color camera are calibrated according to the relative position relationship of the cameras in the camera module through a traditional color camera. The coordinates of different points of the food in the food area can be determined according to the relative position relationship and the calibrated coordinates. No matter which of the above methods is used to calculate the coordinates of different points in the food area, it does not affect the nature of the subsequent measurement of the food volume. The embodiment of the present invention preferably uses the depth camera in the camera module to obtain the coordinates of different points of the food in the food area.
[0113] In some embodiments, the controller is configured to:
[0114] Identifying the type and location of food in the image;
[0115] According to the identified food type and location, the identified food is edge marked to obtain the food area.
[0116] Specifically, the food image taken by the camera module is used to identify the type and position of the food in the image. The identification of the type and position of the food in the image can be determined by feature extraction, such as extracting Haar features (Haar-Like Features), LBP features (Local Binary Pattern), HOG features (Histogram Of Oriented Gradients), etc., or using a neural network algorithm, wherein the neural network can be but is not limited to (deep) neural networks, convolutional neural networks, deep belief networks, deep stacked neural networks, deep fusion networks, deep recursive neural networks, deep recurrent neural networks, deep Bayesian neural networks, deep generative networks, deep reinforcement learning and other network structures or their derivative models. Regardless of which deep neural network is used, it does not affect the essence of the embodiments of the present application, and the embodiments of the present invention preferably use a method of deep learning.
[0117] Specifically, at least one image of different types of food is collected from the Internet as an initial image as an identification sample, and the above initial images are classified according to the types of food, so that they can be divided into multiple initial image groups. Each initial image group can include multiple initial images, and the types of food in the same initial image group are the same, and the types of food in different initial image groups are different. Each initial image in each initial image group is clustered, and the image features of each initial image in the initial image group are clustered. The feature vector of each initial image in the initial image group can be obtained as a class feature vector of the food type, and the food type is identified according to the class feature vector and the edge of the identified food is marked to obtain the food area.
[0118] In some embodiments, the feature vectors of each initial image in each initial image group may be clustered using, but not limited to, k-means clustering, hierarchical clustering, FCM clustering, Gaussian mixture clustering, etc. In practical applications, the implementation method of clustering can be determined according to actual needs and is not limited here.
[0119] Step S603, identifying the hand region in the image, and determining the hand posture according to the identified hand region;
[0120] By using feature extraction or a neural network algorithm, hand area recognition can be obtained by the same neural network training as food area recognition, or by different network training. Hand recognition can also be combined with the hand posture estimation algorithm described below. Either method does not affect the essence of the present invention. Determining that the recognition of hand area images and the recognition of food areas are based on the same concept will not be repeated here, and those skilled in the art should be aware of this.
[0121] After determining the hand area, it is necessary to identify the hand posture. When the user holds food for identification under the camera module, the hand posture presented is different. According to the different hand postures, the common postures when storing and accessing food are classified. For example, when holding an orange, the palm is in a clenched posture because the orange is small, and when holding a grapefruit, the palm is usually fully extended because the grapefruit is large. Therefore, different hand posture models can be established according to the shape and size of the human hand from clenched to extended, such as 1) half-clenched hand, 2) extended palm, etc. The posture categories can be two or more categories, and those skilled in the art can set them according to different volume detection accuracies.
[0122] In some embodiments, hand posture estimation can be determined by model matching, and hand posture features in the current image are matched with features in a hand posture model library. The preset feature matching algorithm can adopt but is not limited to an Oriented Fast and Rotated BRIEF (ORB) algorithm, a Speeded Up Robust Features (SURF) algorithm, a Scale-invariant feature transform (SIFT) algorithm, etc., and the hand posture is determined according to the hand posture corresponding to the hand posture features in the matched hand posture model library;
[0123] In some embodiments, hand posture estimation can utilize the neural network training and learning method to classify different hand postures, construct a recognition classifier, quickly implement the update and optimization of the recognition classifier, and obtain the hand posture condition. The output result is the hand posture, such as half-grip, etc. Hand recognition can use the same set of neural network training classifiers as food recognition, collect a large number of hand posture images for classifier training, or use different network training to obtain. Hand posture recognition can also be combined with the above-mentioned food recognition algorithm to obtain. Either method does not affect the essence of the present invention.
[0124] In some embodiments, a neural network training classifier uses a small amount of hand gesture images to build a recognition classifier in a short time, quickly updating and optimizing the recognition classifier, and making up for the limitation of untimely classifier updates based on deep learning methods.
[0125] Step S604, restoring the three-dimensional space area of the food according to the coordinates of different points of the food in the food area and the hand gestures, and determining the volume of the food according to the three-dimensional space area.
[0126] determining coordinates of different points on the upper surface of the food according to the coordinates of different points of the food in the food area, and predicting an invisible area in the hand area according to the hand gesture;
[0127] Combining the upper surface and the invisible area, restoring the three-dimensional space area of the food;
[0128] The depth differences of relative points in the three-dimensional space area are accumulated to determine the volume of the food.
[0129] The coordinates of different points on the surface of the food are determined according to the coordinates of different points of the food in the food area, such as Figure 7 As shown, according to the coordinate values containing depth information that can be captured by the depth camera, restoring the three-dimensional space area of the food requires knowing the invisible area in the hand area.
[0130] The invisible area in the hand area is predicted according to the hand posture, the hand posture is recognized using the hand area image, the invisible area is determined according to the hand posture, and the upper surface of the food and the calculated invisible area are combined to jointly restore the three-dimensional space area of the food.
[0131] In some embodiments, when the palm area is visible, the coordinates of different points of the invisible area are predicted according to the visible palm area in the hand area, and the envelope formed by combining the upper surface and the invisible area is adjusted according to the shape characteristics of the food, and the depth differences of the relative points of the adjusted envelope are accumulated to determine the volume of the food; or
[0132] In other embodiments, when the palm area is invisible, the palm area is determined according to other visible areas in the hand area, and the coordinates of different points in the invisible area are predicted according to the visible palm area, and the envelope formed by combining the upper surface and the invisible area is adjusted according to the shape characteristics of the food, and the depth differences of the relative points of the adjusted envelope are accumulated to determine the volume of the food;
[0133] In other embodiments, when the palm area is invisible, the coordinates of different points of the invisible area are predicted according to other visible areas in the hand area, and according to the shape characteristics of the food, the envelope formed by combining the upper surface and the invisible area is adjusted, and the depth differences of the relative points of the adjusted envelope are accumulated to determine the volume of the food;
[0134] The coordinates of different points in the invisible area are predicted based on the visible area in the hand area, and the volume of the food is determined by accumulating the depth difference between the coordinates of different points on the upper surface and the coordinates of different points in the invisible area.
[0135] Among them, the visible area feature data of the hand area can be extracted and matched with the pre-established feature data corresponding to different hand grips, and the hand grip is determined according to the matching result;
[0136] The coordinates of different points in the invisible area are obtained according to the coordinates of different points in the visible area by using the pre-established mapping functions corresponding to different hand grip postures.
[0137] Specifically, since the size of the hand fluctuates within a small range, the relative distances and coordinates of the palm and the fingers, wrist, etc. can be obtained in the hand area, and one or more types of mapping relationships can be established to determine the invisible area; for example, in the present embodiment, two types of mapping relationships are established respectively, one type is to establish a mapping relationship for the hand grip, when it is determined that the hand is in a grip, a mapping table is established according to the distance between the fingers and the wrist, and an envelope surface network is mapped to the posture corresponding to each distance, and the coordinates of different points in the invisible area in the envelope surface network are combined with the upper surface and the invisible area to form an envelope body, and the volume of the food is determined according to the envelope body. As shown in Table 1, it is an envelope surface network mapping table.
[0138] Table 1 Envelope network mapping table
[0139]
[0140] Another type is to establish a wrist and palm coordinate transformation table for the unfolded hand, and establish a conversion relationship based on the coordinate information at the wrist to estimate the coordinates of the invisible area (all parts of the palm). The coordinates of different points in the invisible area can be determined by function mapping. For example, the coordinate at the wrist is W, and the coordinates of the invisible areas B and D are determined by function mapping, B=F1(W), D=F2(W), where mapping functions F1 and F2 are the same or different mapping functions, and can be freely set by technical personnel in this field according to the position of the invisible area. The coordinates of other positions on the bottom of the food can be obtained in a similar way, where the coordinates of the invisible areas B and D can be the coordinates of the palm blocked by the upper surface, or the coordinates of the invisible area blocked by the upper surface as the lower surface.
[0141] After determining the upper surface and the invisible area, the volume of the food is calculated based on the coordinates of different points on the upper surface and the lower surface. Since the depth of each point on the upper surface of the food is known, the relative depth between the food and the hand can be calculated. The depth of each point on the envelope surface is accumulated to calculate the volume of the food. For example, if the coordinates of the upper surface A (x1, y1, z1) and the coordinates of the lower surface B (x2, y2, z2) of the food are known, the coordinate difference between the upper surface coordinate A and the lower surface coordinate B can be obtained, and the distance between the two points can be calculated based on the coordinates. Then, using the form of integral accumulation, the distances between all the upper surface coordinates and the lower surface coordinates are accumulated to calculate the volume of the food.
[0142] In some embodiments, the envelope surface accumulation is corrected based on the type of food and the inherent characteristics of the food, such as the approximately spherical and approximately symmetrical characteristics of a pomegranate, to simplify the calculation process. After the envelope body is determined, the envelope body formed by combining the upper surface and the invisible area is adjusted according to the shape characteristics of the food. For example, the shape of the formed envelope body is elliptical, but the food identification category determines that the food is a pomegranate, and the shape characteristic of the pomegranate is a sphere, so the shape of the envelope body is adjusted.
[0143] In some embodiments, the coordinates and feature data of different points in the visible area of the hand area are extracted and input into a grip prediction model to obtain the hand grip and the coordinates of different points in the invisible area. The grip prediction model uses the coordinates and feature data of different points in the visible area that identify the hand grip and the coordinates of different points in the invisible area as input, and outputs the hand grip and the coordinates of different points in the invisible area as a target for network model training. The network model is a neural network model. The method for training a classifier based on a hand posture image has been described in the above embodiments. In this embodiment, the coordinates and feature data of different points in the visible area that identify the hand grip and the coordinates of different points in the invisible area are used as input, and the hand grip and the coordinates of different points in the invisible area are output as a target for network model training. This is based on the same concept as the above embodiments and will not be repeated here.
[0144] In some embodiments, the storage conditions provided by each storage location in the refrigerator can be preset. The design of the internal storage structure of refrigerators of different models may be different, so the storage conditions provided by the storage locations in the refrigerators of the corresponding models can be set according to the internal storage structures of refrigerators of different models. Specifically, a refrigerator information database can be preset, and the database can record relevant information about the storage conditions that can be provided by each storage location in the refrigerator of the corresponding type, such as temperature, closure state, capacity, etc., for one or more types of refrigerators.
[0145] In the embodiment of the present application, the storage condition requirements of different types of ingredients can be preset, and the ingredients to be planned are planned according to the size of the difference in storage conditions between different ingredients and whether they can be stored together. Optionally, there are also priority storage conditions, which may include one or multiple priority storage conditions of different dimensions, for example, user preferences, importance of ingredients, etc. The above conditions can be obtained through big data analysis or set by the user;
[0146] Specifically, a food database can be pre-set to record various types of food and the storage requirements of each food. According to the volume of the food and the distribution of the remaining space in the storage room, intelligent recommendations are made on where the food should be placed in the refrigerator, as shown in Table 2, which is a distribution table of the remaining space in the refrigerator.
[0147] Table 2 Distribution of remaining space in refrigerator
[0148] Types of food to be put Remaining space for food Storage location of food vegetable <![CDATA[0.5m 3 ]]> First compartment fruit <![CDATA[0.5m 3 ]]> Second compartment egg <![CDATA[0.2m 3 ]]> The third compartment …… …… ……
[0149] The storage priority of the ingredients can be calculated based on the identified types of ingredients, and the storage location of each ingredient can be determined by querying the remaining storage space in the refrigerator storage room. The priority of each storage location in the refrigerator for each ingredient to be planned can be sorted in the above manner to determine the current storage priority of the ingredients.
[0150] After calculating the priority of each ingredient, determine the appropriate storage location in the refrigerator for the ingredients to be planned based on the priority of each ingredient, the priority ranking of each storage location in the refrigerator, and the number of ingredients. Among them, the ingredient with the highest priority can be prioritized first according to the storage space of the ingredient in the refrigerator to determine the optimal storage location for the ingredient. If the number of the ingredient is large and there are still some ingredients left to be stored after the optimal storage location is full, then select the storage location with the next highest priority to allocate to the ingredient, and so on, until all the ingredients are allocated storage locations. Then select the ingredient with the next highest comprehensive priority, and allocate storage locations in the above manner until all the ingredients to be allocated are allocated storage locations, or until there are no storage locations available in the refrigerator.
[0151] In some embodiments, after determining the storage location of the food to be planned in the refrigerator, the storage location information of the food to be planned in the refrigerator can also be output so that the user can store the food according to the information. The embodiments of the present application can output information in a variety of ways. For example, the information can be displayed on a display screen on the refrigerator, or the above information can be broadcasted by voice. The query request sent by the user through the mobile terminal can also be accepted, and the above information can be sent to the user's mobile terminal in response to the request for the user to view. Of course, a combination of the above multiple methods can also be used to provide a query method for the user.
[0152] In some embodiments, the controller is further configured to: determine at least one of the total weight of the food, the total calories of the food, and the total nutritional content of the food according to the type of food and the determined volume of the food;
[0153] A nutritional analysis is performed based on at least one of the total weight of the ingredients, the total calories of the ingredients, and the total nutritional content of the ingredients, and information is pushed based on the analysis results.
[0154] Specifically, after obtaining the volume information of the ingredients, the weight, calories, nutritional components and other information of the ingredients can be estimated according to the volume of the ingredients. First, the density of the ingredients, the calories per unit of ingredients, the nutritional components and other information can be obtained through pre-measurement or literature query. Therefore, according to the volume of the ingredients, it can be obtained that the weight of the ingredients = density of the ingredients * volume of the ingredients, the calories of the ingredients = calories per unit weight of the ingredients * weight of the ingredients, and the nutritional content of the ingredients = nutritional content per unit weight of the ingredients * weight of the ingredients. In this way, the information of the ingredients can be obtained more comprehensively. When the user takes the ingredients, the user can be provided with dietary nutrition analysis and calorie analysis. When the quantity of ingredients is small or lacking, the user can be provided with purchase recommendation services, as shown in Table 3, which is a nutritional content table of ingredients inside the refrigerator.
[0155] Table 3 Nutritional content of food in refrigerator
[0156] Types of ingredients Weight of ingredients Calories of food Nutritional content of ingredients vegetable 5kg 200J * fruit 5kg 400J * egg 2kg 2000J * … … … …
[0157] In some embodiments of the present application, minimum storage capacity (weight) thresholds can be pre-set for different types of ingredients. When the ingredient database is updated, the controller determines the minimum storage capacity thresholds corresponding to each type of ingredient in the updated ingredient database. If it is determined that the weight of one or some types of ingredients is lower than the minimum storage capacity threshold of the corresponding ingredient, a prompt message can be displayed on the display screen of the smart terminal to remind the user that the storage capacity of this type of ingredient is too low, and the user can replenish the ingredients in time according to the prompt message.
[0158] Among them, the threshold value of the minimum storage volume corresponding to each type of food can be set by the user, and the user can make the above settings through the setting function provided by the refrigerator.
[0159] In some embodiments of the present application, a smart home management application can be installed on the user terminal, through which the user can set operations on the refrigerator, or receive information sent by the refrigerator, or send instructions (such as query instructions) to the smart terminal. Among them, the user terminal information associated with the refrigerator (such as the call number of the user terminal) or the user's related information (such as the user's identification) can be set in the server in advance, so that the server can communicate with the user terminal of the user. Among them, the user terminal includes but is not limited to: mobile phones, PDAs, wearable devices, etc. Furthermore, the smart terminal can be connected to the server, and the server can communicate with the user terminal through the mobile communication network, so that the user can communicate remotely with the refrigerator through the user terminal.
[0160] For example, when the weight of a certain type or certain types of food in the refrigerator is lower than the minimum storage threshold corresponding to the corresponding type of food, the refrigerator can send a prompt message to the server, so that the server sends the prompt message to the target user terminal, and the prompt message is used to prompt that the storage capacity of the corresponding type of food is too low. The user can also query the food database of the refrigerator through the user terminal. The query request can be sent to the refrigerator through the server, so that the refrigerator responds to the query request and sends the query result to the user terminal through the server. Check the status of refrigerator food anytime and anywhere through the APP. For example, when a user is buying food outside, he can view the types and storage of food currently available in the refrigerator through the mobile phone application, thereby assisting the user in purchasing food.
[0161] In the embodiment of the present application, the user can set personalized reminders for food purchase. The user can set reminders for food purchase for certain necessary ingredients according to their needs. When the weight of these ingredients is less than the minimum storage threshold, reminder information is displayed on the refrigerator display screen, and the reminder information can also be sent to the user's mobile phone to remind the user to replenish the ingredients as soon as possible.
[0162] The embodiment of the present invention provides a method for estimating the volume of food. Figure 8 As shown, it is a complete step flow chart;
[0163] Step S801, receiving an image sent by the camera module;
[0164] Step S802, identifying the food area in the image, determining the coordinates of different points of the food in the food area, and determining the type of the food;
[0165] Step S803, identifying the hand region in the image, and determining the hand posture according to the identified hand region. If it is determined that the hand is in a gripping posture, executing step S804, otherwise executing step S805;
[0166] Step S804, predicting the coordinates of different points of the invisible area according to the visible area in the hand area, and adjusting the envelope formed by combining the upper surface and the invisible area according to the shape characteristics of the food, accumulating the depth differences of the relative points of the adjusted envelope, and determining the volume of the food;
[0167] Step S805, predicting the coordinates of different points in the invisible area according to the visible area in the hand area, and accumulating the depth differences between the coordinates of different points on the upper surface and the coordinates of different points in the invisible area to determine the volume of the food;
[0168] Step S806, determining at least one of the total weight of the food, the total calories of the food, and the total nutritional content of the food according to the type of the food and the determined volume of the food, performing a nutritional analysis according to at least one of the total weight of the food, the total calories of the food, and the total nutritional content of the food, and pushing information according to the analysis result;
[0169] Step S807, intelligently recommend where in the refrigerator the food should be placed based on the food volume and the remaining space distribution in the storage room.
[0170] Based on the same inventive concept, some embodiments of the present application also provide a smart home system, including the above refrigerator provided in some embodiments of the present application. The principle of solving the problem by the smart home system is similar to that of the above refrigerator, so the implementation of the smart home system can refer to the implementation of the above refrigerator, and the repeated parts will not be repeated here.
[0171] In specific implementation, in some embodiments of the present application, the smart home system may also include: a washing machine, a television, an air conditioner, etc., which are not limited here.
[0172] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A refrigerator, characterized in that: The refrigerator comprises: a housing, the housing comprising a storage chamber, the storage chamber having an opening; A door body, movably connected to the housing, and used to cover the opening; A camera module, used for collecting images and sending the images to a controller; The controller is configured as: Receiving an image sent by the camera module; Identify the food area in the image and determine the coordinates of different points of the food in the food area; Identify the hand area in the image, extract visible area feature data of the hand area, match it with pre-established feature data corresponding to different hand grips, and determine the hand grip according to the matching result; Using the pre-established mapping functions corresponding to different hand grip postures, the coordinates of different points in the invisible area are obtained according to the coordinates of different points in the visible area; determining coordinates of different points on the upper surface of the food according to the coordinates of different points of the food in the food area, and predicting an invisible area in the hand area according to the hand posture; Combining the upper surface and the invisible area, restoring the three-dimensional space area of the food; The depth differences of relative points in the three-dimensional space area are accumulated to determine the volume of the food.
2. The refrigerator according to claim 1, characterized in that: The controller is further configured to: Identifying the type and location of food in the image; According to the identified food type and location, the identified food is edge marked to obtain the food area.
3. The refrigerator according to claim 1, characterized in that: The controller is further configured to: Predicting coordinates of different points of the invisible area according to the visible area in the hand area, and adjusting the envelope formed by combining the upper surface and the invisible area according to the shape characteristics of the food, accumulating depth differences of relative points of the adjusted envelope, and determining the volume of the food; or The coordinates of different points in the invisible area are predicted according to the visible area in the hand area, and the volume of the food is determined by accumulating the depth difference between the coordinates of different points on the upper surface and the coordinates of different points in the invisible area.
4. The refrigerator according to claim 1, characterized in that: The controller is further configured to: The coordinates and feature data of different points in the visible area of the hand area are extracted, and input into a grip prediction model to obtain the hand grip and the coordinates of different points in the invisible area. The grip prediction model uses the coordinates and feature data of different points in the visible area that identify the hand grip and the coordinates of different points in the invisible area as input, and outputs the hand grip and the coordinates of different points in the invisible area as the goal to perform network model training, and the network model is a neural network model.
5. The refrigerator according to claim 1 or 4, characterized in that: The visible area feature data of the hand area includes at least one of the following: The spatial coordinate values of fingers, palm, wrist and arm.
6. The refrigerator according to claim 1, characterized in that: The controller is also configured to: Determine at least one of the total weight of the food, the total calories of the food, and the total nutritional content of the food according to the type of the food and the determined volume of the food; A nutritional analysis is performed based on at least one of the total weight of the ingredients, the total calories of the ingredients, and the total nutritional content of the ingredients, and information is pushed based on the analysis results.
7. A method for estimating the volume of food, characterized in that: The method comprises: Receive images sent by the camera module; Identify the food area in the image and determine the coordinates of different points of the food in the food area; Identify the hand area in the image, extract visible area feature data of the hand area, match it with pre-established feature data corresponding to different hand grips, and determine the hand grip according to the matching result; determining coordinates of different points on the upper surface of the food according to the coordinates of different points of the food in the food area, and predicting an invisible area in the hand area according to the hand posture; Combining the upper surface and the invisible area, restoring the three-dimensional space area of the food; The depth differences of relative points in the three-dimensional space area are accumulated to determine the volume of the food.
