Method and device for detecting food volume, and kitchen appliance

Abstract

This application relates to the field of smart device technology and discloses a method and apparatus for detecting the volume of food, as well as a kitchen appliance. The method includes: acquiring an image of the working area of ​​the kitchen appliance where food is placed; training the image using a convolutional neural network model to determine image information of the food's position within the kitchen appliance, as well as segmented image information of the food's designated surface; determining area information of the food's designated surface based on the position image information and the segmented image information; and determining the food volume corresponding to the area information based on a stored correspondence between the food's area and volume. This improves the accuracy of determining food volume information.

Description

Technical Field

[0001] The present application relates to the technical field of smart devices, for example, to methods and devices for detecting the volume of food ingredients and kitchen appliances. Background Art

[0002] Currently, kitchen appliances, such as ovens, microwave ovens, air fryers, etc., all have the function of baking food. Generally, the type of food can be identified, and the size and weight of the food can be determined by a weight detection device. Thus, the set temperature and set time of the kitchen appliance for baking food can be determined to achieve the purpose of baking food.

[0003] However, adding a weight detection device, such as a gravity sensor, to a kitchen appliance will increase the manufacturing cost of the kitchen appliance. In addition, the performance and service life of the weight detection device will affect the size and weight of the food, resulting in inaccurate size and weight of the food. Summary of the Invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] The embodiments of the present disclosure provide a method, device, and kitchen appliance for detecting food volume, so as to solve the technical problem that kitchen appliances cannot accurately determine food volume information.

[0006] In some embodiments, the method comprises:

[0007] Acquire an image of a working area of ​​the kitchen appliance where food ingredients are placed;

[0008] Training the image using a convolutional neural network model to determine image information of the location of the food in the kitchen appliance and segmented image information of a set surface of the food;

[0009] determining area information of a set surface of the food according to the position image information and the segmented image information;

[0010] The volume of the food corresponding to the area information is determined based on the stored correspondence between the food area and the food volume.

[0011] In some embodiments, the apparatus comprises:

[0012] an image acquisition module configured to acquire an image of a working area of ​​the kitchen appliance where food ingredients are placed;

[0013] a model training module configured to train the image using a convolutional neural network model to determine image information of the location of the food in the kitchen appliance and segmented image information of a set surface of the food;

[0014] an area determination module configured to determine area information of a set surface of the food according to the position image information and the segmented image information;

[0015] The volume determination module is configured to determine the volume of the food corresponding to the area information according to the stored correspondence between the food area and the food volume.

[0016] In some embodiments, the food volume detection device includes a processor and a memory storing program instructions, and the processor is configured to execute the above-mentioned food volume detection method when executing the program instructions.

[0017] In some embodiments, the kitchen appliance includes: the above-mentioned device for detecting the volume of food.

[0018] The method, device, and kitchen appliance for detecting food volume provided by the embodiments of the present disclosure can achieve the following technical effects:

[0019] By using a convolutional neural network model to perform deep learning on images of the working area of ​​kitchen appliances, the volume information of the ingredients placed in the kitchen appliances can be obtained. This method does not require any additional hardware equipment. While identifying the type of ingredients, the volume and therefore the quality of the ingredients can be determined, which reduces the manufacturing resources of the kitchen appliances. In addition, it does not rely on the performance of weight measurement hardware equipment, which improves the accuracy of determining the volume information of the ingredients.

[0020] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation. In addition,

[0022] Figure 1 is a flow chart of a food volume detection method provided by an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a MobileNet model training provided by an embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a grid mapping matrix provided by an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of a binarization mask matrix provided by an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of a food mask image provided by an embodiment of the present disclosure;

[0027] Figure 6 This is a schematic diagram of a fitting function mapping provided by an embodiment of the present disclosure;

[0028] Figure 7 1 is a flow chart of a method for detecting food volume provided by an embodiment of the present disclosure;

[0029] Figure 8 1 is a structural diagram of a food volume detection device provided by an embodiment of the present disclosure;

[0030] Figure 9 1 is a structural diagram of a food volume detection device provided by an embodiment of the present disclosure;

[0031] Figure 10 It is a structural schematic diagram of a food volume detection device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0033] In the disclosed embodiment, an image of the working area of ​​the kitchen appliance is subjected to deep learning through a convolutional neural network model to obtain the volume of the food placed in the kitchen appliance. In this way, the quality of the food can be determined and further control of the baking of the food can be performed.

[0034] Figure 1 FIG. 1 is a flow chart of a food volume detection method provided by an embodiment of the present disclosure. Figure 1 As shown, the process of food volume detection includes:

[0035] Step 101: Acquire an image of a working area of ​​a kitchen appliance where food ingredients are placed.

[0036] In the disclosed embodiments, the kitchen appliances include ovens, microwave ovens, air fryers, and other appliances with baking functions. The kitchen appliances may be equipped with an image acquisition device, thereby enabling the image acquisition device to capture an image of the working area of ​​the kitchen appliance where food is placed.

[0037] Step 102: The image is trained using a convolutional neural network model to determine the image information of the position of the food in the kitchen appliance and the segmented image information of the set surface of the food.

[0038] Convolutional neural networks (CNNs) have been widely used in the field of computer image processing and have achieved good results. At present, in order to pursue classification accuracy, the model depth is getting deeper and deeper, and the model complexity is getting higher and higher. In this embodiment, different neural network models can be used to train images respectively to obtain image information of the position of ingredients in kitchen appliances, as well as segmentation image information of the set surface of ingredients. Among them, the image can be trained by a neural compression network model to obtain a grid mapping matrix corresponding to the hierarchical position of the ingredients in the kitchen appliances; the image can also be trained by an instance segmentation network model to obtain a binary mask matrix of the set surface of the ingredients.

[0039] Kitchen appliances, such as ovens and air fryers, may have one, two, or more grills, and ingredients can be placed on different grills. Therefore, the neural compression network model can be used to train images to obtain the hierarchical position of the ingredients in the kitchen appliance, and thus determine which grill layer the ingredients are on.

[0040] Among them, MobileNet is a lightweight neural compression network model with the advantages of small size and high precision. It can reduce response delay while ensuring accuracy, and can achieve the effect of rapid convergence. Using this lightweight network training can also reduce the amount of calculation, thereby reducing the requirements for hardware equipment.

[0041] Optionally, the image can be trained using a MobileNet model to obtain a grid mapping matrix corresponding to the hierarchical position of the food in the kitchen appliance. Of course, in other embodiments, other types of neural compression network models can also be trained on the image to obtain image information of the location of the food in the kitchen appliance.

[0042] Figure 2 This is a schematic diagram of a MobileNet model training provided by an embodiment of the present disclosure. After obtaining an image of the oven working area, it is input into the MobileNet model for training to determine the hierarchical position of the ingredients in the oven, such as Figure 2 As shown, the first layer, Layer 1, can be determined, and the following can be obtained: Figure 3The grid mapping matrix shown in the figure is a two-dimensional matrix of the same size as the image taken by the camera. The value of each element represents the actual area corresponding to the corresponding position.

[0043] Mask-RCNN is an instance segmentation network model. The Mask-RCNN algorithm can simultaneously perform segmentation, localization, and classification tasks, generating a mask image corresponding to the image. In this embodiment, only the predefined surfaces of the ingredients are annotated during Mask-RCNN training. In some embodiments, the predefined surfaces can be visible surfaces in the ingredient image. The trained Mask-RCNN can generate a corresponding binary mask matrix and identify the type of ingredient.

[0044] Optionally, the image is trained using a Mask-RCNN model to obtain a binary mask matrix of the set surface of the food and identify the type of food.

[0045] Figure 4 The obtained image is input into the Mask-RCNN model, and instance segmentation is performed according to each pixel in the food image, and information is accurately labeled, so that the type of food can be judged and the precise position segmentation can be achieved, and the following is obtained: Figure 4 The binarization mask matrix shown.

[0046] Of course, in some other embodiments, other types of instance segmentation network models can also be trained on images to obtain segmented image information of the set surface of the food.

[0047] Step 103: Determine the area information of the set surface of the food according to the position image information and the segmented image information.

[0048] Having obtained the positional image information and segmented image information of the food, the area information of the set surface of the food can be determined. This may include: performing a calculation, such as addition, multiplication, or weighted addition, on the positional image information and segmented image information corresponding to the same pixel in the positional image information and segmented image information. The information that satisfies the set conditions after the processing is then retained, and the area information of the set surface of the food is obtained based on the retained information. For example, information that is not equal to 0 or greater than a set value is retained, and the retained information is summed, subtracted, or weighted added to obtain the area information of the set surface of the food. Specifically, the positional image information and segmented image information corresponding to the same pixel are subjected to a first calculation, and the information that satisfies the set conditions after the processing is retained. A second calculation is then performed based on the retained information to obtain the area information of the set surface of the food.

[0049] In some embodiments, the grid mapping matrix and the binary mask matrix can be bitwise multiplied to obtain regional matrix information of the set surface of the food; the element values ​​in the regional matrix information can be summed to obtain the area information of the set surface of the food.

[0050] For example: Through the MobileNet model, the hierarchical position of the food in the kitchen appliance is determined, and the corresponding layer is read. Figure 3 The grid mapping matrix T shown i , get the value of each element of the grid mapping matrix, and then use the grid mapping matrix T i The binary mask matrix M obtained by Mask-RCNN segmentation j Multiply the bits to get the result matrix, and then mask the background area to keep only the corresponding area on the surface of the food. Thus, we can obtain Figure 5 The food mask image shown in FIG. The result matrix is ​​the regional matrix information of the set surface of the food.

[0051] Then, the element values ​​in the area matrix information are summed to obtain the area information of the set surface of the food. That is, the known result is centered on each element value s i , thus, the area of ​​the set surface of the food is S = ∑ i s i .

[0052] Step 104: Determine the volume of the food corresponding to the area information based on the stored correspondence between the food area and the food volume.

[0053] For each food ingredient, the mapping relationship between several food reference areas S and the actual volume V of the food ingredient, i.e. (S, V), can be obtained or saved in advance. Then, by performing function fitting on these mapping relationships, a fitting function is obtained, that is, the corresponding relationship between the food area and the food volume corresponding to each food ingredient is pre-configured and saved. Then, the fitting function obtained by fitting is used to estimate the food volume V′ according to the food area S, thereby determining the food volume corresponding to the area information.

[0054] Figure 6 This is a schematic diagram of a fitting function mapping provided by an embodiment of the present disclosure. Figure 6 As shown, if the area in the area information is determined to be 15, the corresponding food volume is 60.

[0055] In some embodiments, for different types of ingredients, different correspondences between ingredient areas and ingredient volumes can be pre-configured and saved. Thus, when the ingredient type information is determined through the Mask-RCNN model, the corresponding relationships that match the ingredient type information can be determined among the saved corresponding relationships, and then, in the matched corresponding relationships, the ingredient volume corresponding to the area information is determined.

[0056] As can be seen, in the disclosed embodiments, deep learning is performed on images of the working area of ​​the kitchen appliance using a convolutional neural network model to determine the volume information of the ingredients placed in the appliance. This information can then be used to further control the baking of the ingredients. This method can identify the type and volume of ingredients without requiring any additional hardware, reducing the manufacturing resources required for the kitchen appliance. Furthermore, it does not rely on the performance of weight measurement hardware, improving the accuracy of determining ingredient volume information.

[0057] The purpose of putting food into the kitchen appliance is to bake or roast the food. Therefore, after determining the volume of the food corresponding to the area information, the method further includes: controlling the baking of the food according to the volume of the food.

[0058] Optionally, when the food type information is determined by the Mask-RCNN model, the baking process corresponding to the food type information can be determined, and the food volume and food type can also be used to determine the mass of the food. That is, the determined volume V and the density ρ of this type of food are estimated to determine the food mass M = V*ρ. In this way, according to the quality of the food, the specific parameters in the baking process can be determined, such as baking time, baking temperature, etc., so that the baking control of the food placed in the food can be achieved by running the baking process with the determined parameters. Of course, the corresponding baking parameters such as baking time and baking temperature can also be determined according to the food volume of the food, and baking control can be performed directly.

[0059] Of course, the type of ingredients may also be obtained in other ways, for example, by determining the type of ingredients based on instruction information input by the user.

[0060] Clearly, determining the volume of ingredients can be used to further control baking. This allows for the determination of both the volume and mass of ingredients during ingredient identification, without requiring any additional hardware. This reduces manufacturing resources for kitchen appliances and improves the accuracy of ingredient specification by not relying on the performance of weight-measuring hardware.

[0061] The following operation procedures are summarized into specific embodiments to illustrate the food volume detection process provided by the embodiments of the present invention.

[0062] In one embodiment of the present disclosure, the oven stores the corresponding relationship between the food area and the food volume that matches the food type, which may include: Figure 6 The fitting function mapping relationship is shown.

[0063] Figure 7 FIG. 1 is a flow chart of a food volume detection method provided by an embodiment of the present disclosure. Figure 7 As shown, the food volume detection process includes:

[0064] Step 701: Acquire an image of the working area of ​​the oven where food is placed.

[0065] A camera is built into the oven so that an image of the working area of ​​the oven can be acquired by the camera.

[0066] Step 702: The image is trained using the MobileNet model to obtain a grid mapping matrix corresponding to the hierarchical position of the food in the kitchen appliance.

[0067] Step 703: The image is trained using the Mask-RCNN model to obtain a binary mask matrix of the set surface of the food and information about the type of food.

[0068] There is no limitation on the order of step 702 and step 703 , and they can be performed simultaneously or sequentially.

[0069] Step 704: perform bitwise multiplication on the grid mapping matrix and the binary mask matrix to obtain regional matrix information of the set surface of the food.

[0070] Step 705: summing the element values ​​in the area matrix information to obtain the area information of the set surface of the food.

[0071] Step 706: Determine the correspondence between the food area and the food volume that matches the food type information, and determine the food volume corresponding to the area information based on the matched correspondence.

[0072] It can be seen that in this embodiment, the image of the working area in the oven can be used to perform deep learning through a convolutional neural network model to obtain the volume information of the ingredients. In this way, the volume of the ingredients can be determined while identifying the type of ingredients. Therefore, the volume of the ingredients can be determined without adding any hardware equipment, reducing the manufacturing resources of the oven. Moreover, it does not rely on the performance of the weight measurement hardware equipment, thereby improving the accuracy of determining the volume information of the ingredients.

[0073] According to the above-mentioned food volume detection process, a device for food volume detection can be constructed.

[0074] Figure 8Schematic diagram of a food volume detection device provided by an embodiment of the present disclosure. Figure 8 As shown, the food volume detection device includes: an image acquisition module 810, a model training module 820, an area determination module 830 and a volume determination module 840.

[0075] The image acquisition module 810 is configured to acquire an image of a working area of ​​the kitchen appliance where food ingredients are placed.

[0076] The model training module 820 is configured to train the image through a convolutional neural network model to determine the image information of the position of the food in the kitchen appliance and the segmented image information of the set surface of the food.

[0077] The area determination module 830 is configured to determine the area information of the set surface of the food according to the position image information and the segmented image information.

[0078] The volume determination module 840 is configured to determine the volume of the food corresponding to the area information according to the stored correspondence between the food area and the food volume.

[0079] In some embodiments, the model training module 820 is specifically configured to train the image through a neural compression network model to obtain a grid mapping matrix corresponding to the hierarchical position of the food in the kitchen appliance; and to train the image through an instance segmentation network model to obtain a binary mask matrix of the set surface of the food.

[0080] In some embodiments, the area determination module 830 is specifically configured to perform bitwise multiplication on the grid mapping matrix and the binary mask matrix to obtain the area matrix information of the set surface of the food; and sum the element values ​​in the area matrix information to obtain the area information of the set surface of the food.

[0081] In some embodiments, the method further includes: a baking control module configured to control the baking of the ingredients according to the volume of the ingredients.

[0082] The following example illustrates the food volume detection process performed by the food volume detection device provided in the embodiment of the present disclosure.

[0083] Figure 9 Schematic diagram of a food volume detection device provided by an embodiment of the present disclosure. Figure 9 As shown, the food volume detection device includes: an image acquisition module 810 , a model training module 820 , an area determination module 830 and a volume determination module 840 , and may also include: a baking control module 850 .

[0084] The image acquisition module 810 may acquire an image of a working area of ​​the kitchen appliance where food ingredients are placed.

[0085] In this way, the model training module 820 can train the image through the MobileNet model to obtain the grid mapping matrix corresponding to the hierarchical position of the ingredients in the kitchen appliance, and train the image through the Mask-RCNN model to obtain the binary mask matrix of the set surface of the ingredients and the type information of the ingredients.

[0086] Therefore, the area determination module 830 can perform bitwise multiplication on the grid mapping matrix and the binary mask matrix to obtain the area matrix information of the set surface of the food, and sum the element values ​​in the area matrix information to obtain the area information of the set surface of the food.

[0087] The correspondence between the area and volume of the food items that matches the food item type has been pre-stored. Thus, the volume determination module 840 can determine the correspondence between the area and volume of the food items that matches the food item type information, and determine the food volume corresponding to the area information based on the matched correspondence.

[0088] Since the food type information is determined by the Mask-RCNN model, the baking process corresponding to the food type information can be determined, and the instruction information of the food can also be determined based on the density corresponding to the food type information and the volume information of the food. Therefore, the specific parameters in the baking process can be determined based on the food quality information, such as baking time, baking temperature, etc. Therefore, the baking control module 850 can control the baking of the food according to the baking process with determined parameters.

[0089] As can be seen, in this embodiment, the food volume detection device captures an image of the working area within the kitchen appliance and then uses a convolutional neural network model to perform deep learning to determine the volume information of the food within the appliance. This allows the determination of both the food type and the food volume, thereby further controlling the baking of the food. Without requiring any additional hardware, the food volume can be determined during the food type identification process, reducing the manufacturing resources of the kitchen appliance. Furthermore, the accuracy of determining food volume information is improved by not relying on the performance of weight measurement hardware.

[0090] The present disclosure provides a device for detecting the volume of food, the structure of which is as follows: Figure 10 Shown, including:

[0091] Processor 100 and memory 101 may also include a communication interface 102 and a bus 103. Processor 100, communication interface 102, and memory 101 may communicate with each other via bus 103. Communication interface 102 may be used for information transmission. Processor 100 may invoke logic instructions in memory 101 to execute the food volume detection method described in the above embodiment.

[0092] In addition, the logic instructions in the memory 101 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0093] Memory 101, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. Processor 100 executes the program instructions / modules stored in memory 101 to perform functional applications and data processing, thereby implementing the food volume detection method in the above-mentioned method embodiment.

[0094] The memory 101 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Furthermore, the memory 101 may include high-speed random access memory and non-volatile memory.

[0095] An embodiment of the present disclosure provides a kitchen appliance comprising the above-mentioned food volume detection device.

[0096] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned food volume detection method.

[0097] An embodiment of the present disclosure provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, enable the computer to execute the above-mentioned food volume detection method.

[0098] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0099] The technical solution of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code, or a transient storage medium.

[0100] The above description and accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments represent only possible variations. Unless expressly required, individual components and functions are optional, and the order of operations may vary. Portions and features of some embodiments may be included in or substituted for portions and features of other embodiments. The scope of the embodiments of the present disclosure includes the entire scope of the claims and all available equivalents thereof. When used in this application, although the terms "first," "second," etc. may be used in this application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element can be called a second element, and similarly, a second element can be called a first element, without changing the meaning of the description, as long as all occurrences of "first element" are consistently renamed and all occurrences of "second element" are consistently renamed. The first element and the second element are both elements, but they may not be the same element. Furthermore, the terms used in this application are only used to describe the embodiments and are not intended to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more of the associated listings. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups of these. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method or apparatus comprising the element. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can be referred to the description of the method part.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0102] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units can be merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of the present disclosure may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0103] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for detecting the volume of food, characterized in that: include: Acquire an image of a working area of ​​the kitchen appliance where food ingredients are placed; Determining, based on the image, image information of the location of the food in the kitchen appliance and segmented image information of a set surface of the food by a convolutional neural network model; determining area information of a set surface of the food according to the position image information and the segmented image information; Determining the volume of the food corresponding to the area information according to the stored correspondence between the food area and the food volume; Determining the image information of the position of the food in the kitchen appliance and the segmented image information of the set surface of the food includes: obtaining, based on the image, a grid mapping matrix corresponding to the hierarchical position of the food in the kitchen appliance using a neural compression network model; and obtaining, based on the image, a binarized mask matrix of the set surface of the food using an instance segmentation network model; The method of determining the area information of the set surface of the food includes: performing bitwise multiplication processing on the grid mapping matrix and the binary mask matrix to obtain the regional matrix information of the set surface of the food; and summing the element values ​​in the regional matrix information to obtain the area information of the set surface of the food.

2. The method according to claim 1, characterized in that After determining the volume of the food corresponding to the area information, the method further includes: Baking of the food is controlled according to the volume of the food.

3. A device for detecting the volume of food, characterized in that: include: an image acquisition module configured to acquire an image of a working area of ​​the kitchen appliance where food ingredients are placed; a model training module configured to determine, based on the image, image information of the position of the food in the kitchen appliance and segmented image information of a set surface of the food by using a convolutional neural network model; an area determination module configured to determine area information of a set surface of the food according to the position image information and the segmented image information; a volume determination module configured to determine the volume of the food corresponding to the area information based on the stored correspondence between the food area and the food volume; The model training module is specifically configured to obtain, based on the image, a grid mapping matrix corresponding to the hierarchical position of the food in the kitchen appliance using a neural compression network model; and obtain, based on the image, a binarized mask matrix of a set surface of the food using an instance segmentation network model; The area determination module is specifically configured to perform bitwise multiplication processing on the grid mapping matrix and the binary mask matrix to obtain the area matrix information of the set surface of the food; and sum the element values ​​in the area matrix information to obtain the area information of the set surface of the food.

4. The device according to claim 3, characterized in that Also includes: The baking control module is configured to control the baking of the food according to the volume of the food.

5. A device for detecting the volume of food, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to perform the method according to any one of claims 1 to 2 when executing the program instructions.

6. A kitchen appliance, characterized in that: Comprising the device as claimed in claim 3 or 5.

Images