Food volume, calorie calculation method, electronic device, electronic device and storage medium

Abstract

A method for calculating food volume and calories, an electronic device, an electronic apparatus and a storage medium. The method for calculating food volume comprises: obtaining actual size information of a three-dimensional reference object; obtaining a reference image and a food image corresponding to the three-dimensional reference object and the food to be estimated; and obtaining volume information of the food to be estimated according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the reference image and the food area size information obtained based on the food image. The method for calculating food volume reduces the complexity of operation and enhances the flexibility of application.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a method for calculating food volume and calories, an electronic device, an electronic device, and a storage medium. Background Art

[0002] With economic development, the dietary level of urban and rural residents has been significantly improved, but it has also led to obesity, high blood pressure, high blood lipids and other conditions in a large number of people, which has a negative impact on health; in addition, growing adolescents, pregnant women, weight loss and fitness people and the general public all need to manage their daily diet to achieve good health. For these people, it is necessary to estimate the calories of food intake, and the key to estimating food calories is to estimate the food volume. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a method for calculating the volume of food, comprising: obtaining actual size information of a three-dimensional reference object; obtaining a reference image and a food image corresponding to the three-dimensional reference object and the food to be estimated; and obtaining volume information of the food to be estimated based on the actual size information of the three-dimensional reference object, reference area size information obtained based on the reference image, and food area size information obtained based on the food image.

[0004] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, a reference image and a food image obtained by the same shooting method of the three-dimensional reference object and the food to be estimated are obtained, wherein the volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the one reference image, and the food area size information obtained based on the one food image; or multiple reference images and multiple food images obtained in pairs by the three-dimensional reference object and the food to be estimated in corresponding multiple shooting methods are obtained, wherein the volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the multiple reference images, and the food area size information obtained based on the multiple food images; wherein the same shooting method or each of the multiple shooting methods includes at least a shooting distance and a shooting direction.

[0005] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the multiple shooting methods include a first shooting method and a second shooting method, and the shooting direction of the first shooting method and the shooting direction of the second shooting method intersect each other; the multiple reference images include a first reference image and a second reference image respectively obtained using the first shooting method and the second shooting method; the multiple food images include a first food image and a second food image respectively obtained using the first shooting method and the second shooting method.

[0006] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, each of the multiple shooting methods also includes placing the three-dimensional reference object serving as the photographed object or the supporting surface of the food to be estimated, and the shooting direction of the first shooting method and the shooting direction of the second shooting method are perpendicular to each other, and the shooting direction of the first shooting method is perpendicular to the supporting surface of the first shooting method, and the shooting direction of the second shooting method is parallel to the supporting surface of the second shooting method.

[0007] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the reference image and the food image acquired in pairs corresponding to each other by the same shooting method are a pair of images corresponding to each other, or the reference image and the food image acquired in pairs corresponding to each other by the same shooting method are components of the same image.

[0008] For example, in the food volume calculation method provided in at least one embodiment of the present disclosure, obtaining the actual size information of the three-dimensional reference object includes: obtaining a plurality of three-dimensional reference object images parallel to the three-dimensional reference object and a first reference object whose reference size can be obtained, and obtaining the actual size information of the three-dimensional reference object based on the plurality of three-dimensional reference object images and the reference size.

[0009] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the actual size information of the stereoscopic reference object is obtained based on the multiple stereoscopic reference object images and the reference size, including: for each of the stereoscopic reference images, obtaining a first number of pixels corresponding to the reference size in the stereoscopic reference image, obtaining a second number of pixels in at least one dimension of the stereoscopic reference object in the stereoscopic reference image, obtaining a calculated size of the stereoscopic reference object in the stereoscopic reference image in at least one dimension based on the first number of pixels and the second number of pixels, and obtaining the actual size information of the stereoscopic reference object based on the multiple calculated sizes obtained from the multiple stereoscopic reference images.

[0010] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, obtaining the actual size information of the three-dimensional reference object includes: obtaining a three-dimensional reference object image in which the three-dimensional reference object is placed on one side of a second reference object whose reference area can be calculated; obtaining the actual size information of the three-dimensional reference object based on the three-dimensional reference object image and the reference area, wherein the three-dimensional reference object image includes information in the second reference object used to calculate the reference area.

[0011] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the actual size information of the three-dimensional reference object is obtained based on the three-dimensional reference object image and the reference area, including: obtaining the pixel area of ​​the second reference object in the three-dimensional reference object image; obtaining a first mapping ratio based on the pixel area of ​​the second reference object and the reference area; obtaining the pixel area of ​​the area occupied by the three-dimensional reference object in the three-dimensional reference object image; obtaining the volume of the three-dimensional reference object based on the first mapping ratio and the pixel area of ​​the area occupied by the three-dimensional reference object to obtain the actual size information of the three-dimensional reference object.

[0012] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the pixel value of the pixel point corresponding to the second reference object in the three-dimensional reference object image is within a first value range, and the step of obtaining the pixel area of ​​the area occupied by the three-dimensional reference object in the three-dimensional reference object image includes: taking the pixel points whose pixel values ​​of the pixel points in the three-dimensional reference object image do not satisfy the first value range as the area occupied by the three-dimensional reference object, so as to obtain the pixel area of ​​the area occupied by the three-dimensional reference object; or, extracting the area occupied by the three-dimensional reference object according to a deep learning model, so as to obtain the pixel area of ​​the area occupied by the three-dimensional reference object.

[0013] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the obtaining of the pixel area of ​​the second reference object in the stereoscopic reference object image includes: obtaining the minimum circumscribed rectangle of the second reference object in the stereoscopic reference object image and the pixel area of ​​the minimum circumscribed rectangle, and taking the pixel area of ​​the minimum circumscribed rectangle as the pixel area of ​​the second reference object, wherein the pixel value of the pixel point corresponding to the second reference object in the stereoscopic reference object image is within a first value range, and the minimum circumscribed rectangle of the second reference object is obtained based on the position of the pixel point whose pixel value is within the first value range, and the obtaining of the pixel area of ​​the area occupied by the stereoscopic reference object in the stereoscopic reference object image also includes: taking the pixel points whose pixel values ​​in the minimum circumscribed rectangle do not satisfy the first value range as the area occupied by the stereoscopic reference object, so as to obtain the pixel area of ​​the area occupied by the stereoscopic reference object.

[0014] For example, in the food volume calculation method provided in at least one embodiment of the present disclosure, the second reference object is a single-color paper with a standard size.

[0015] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the three-dimensional reference object includes a hand in a fist state and is processed in a spherical manner, and the pixel area of ​​the area occupied by the three-dimensional reference object in the three-dimensional reference object image is obtained, including: in response to the wrist part of the hand in the three-dimensional reference object image not being against the edge of the second reference object, obtaining a hand fist image according to a hand detection model, extracting the fist area in the hand fist image, and obtaining the pixel area of ​​the fist area as the pixel area of ​​the area occupied by the three-dimensional reference object; and in response to the wrist part of the hand in the three-dimensional reference object image being against the edge of the second reference object, taking the pixel points in the three-dimensional reference object image whose pixel values ​​do not meet the first value range as the fist area, and obtaining the pixel area of ​​the fist area as the pixel area of ​​the area occupied by the three-dimensional reference object.

[0016] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the acquiring of multiple reference images and multiple food images that are acquired in pairs corresponding to the stereoscopic reference object and the food to be estimated in multiple shooting methods includes: continuously shooting multiple images of the food to be estimated and the stereoscopic reference object simultaneously located on the same supporting surface, extracting multiple key images from the multiple images to obtain the multiple reference images and the multiple food images, wherein the key images include a corresponding pair of reference images and food images.

[0017] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the continuously shooting of multiple images of the food to be estimated and the three-dimensional reference object simultaneously located on the same supporting surface, and extracting multiple key images from the multiple images, includes: for each image in the multiple images, extracting the food area to be estimated occupied by the food to be estimated, and obtaining the pixel area of ​​the food area to be estimated corresponding to each image; based on the pixel area of ​​the food area to be estimated corresponding to each image, extracting an image in which a relative change of the pixel area of ​​the food area to be estimated is greater than a preset threshold value from the multiple images as one of the multiple key images, wherein the multiple key images include at least a first key image and a second key image, the first key image includes the first food image and the first reference image, and the second key image includes the second food image and the second reference image.

[0018] For example, in the food volume calculation method provided in at least one embodiment of the present disclosure, the volume information of the food to be estimated is obtained based on the actual size information of the three-dimensional reference object, the reference area size information obtained based on the reference image, and the food area size information obtained based on the food image, including: obtaining the pixel area of ​​the area occupied by the three-dimensional reference object of each reference image in the multiple reference images to obtain the reference area size information of the three-dimensional reference object; obtaining a second mapping ratio based on the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; for the first food image, extracting the food area to be estimated occupied by the food to be estimated to obtain the food area size information; obtaining food size information based on the food area size information and the second mapping ratio; for the multiple food images, extracting the food area to be estimated occupied by the food to be estimated, and determining the container shape of the food to be estimated according to the food area to be estimated corresponding to each image in the multiple food images; obtaining the volume information of the food to be estimated according to the container shape of the food to be estimated and the food size information.

[0019] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the volume information of the food to be estimated is obtained based on the actual size information of the three-dimensional reference object, the reference area size information obtained based on the multiple reference images, and the food area size information obtained based on the multiple food images, and also includes: for the second food image, extracting the food area to be estimated occupied by the food to be estimated, determining the height value of the food to be estimated based on the food area to be estimated and the second mapping ratio; and obtaining the volume information of the food to be estimated based on the shape of the container of the food to be estimated, the height value of the food to be estimated and the food size information.

[0020] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, determining the height value of the food to be estimated based on the food area to be estimated and the second mapping ratio includes: obtaining the number of pixels corresponding to each column of pixel points in the food area to be estimated; determining the height component of the food to be estimated based on the number of pixels corresponding to each column of pixel points; determining the height value of the food to be estimated based on the height component and the second mapping ratio.

[0021] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the pixel area of ​​the area occupied by the three-dimensional reference object of each reference image in the multiple reference images to obtain the reference area size information of the three-dimensional reference object includes: extracting the area occupied by the three-dimensional reference object for each reference image according to a deep learning model, and obtaining the pixel area of ​​the area occupied by the three-dimensional reference object corresponding to each reference image; and calculating the average value of the pixel areas of all the obtained areas occupied by the three-dimensional reference objects to obtain the reference area size information of the three-dimensional reference object.

[0022] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, the three-dimensional reference object includes a hand in a clenched fist state, and the pixel area of ​​the area occupied by the three-dimensional reference object of each reference image in the multiple reference images is obtained to obtain the reference area size information of the three-dimensional reference object, including: for each reference image, obtaining a clenched hand image according to a hand detection model, extracting the clenched fist area in the clenched hand image, and obtaining the pixel area of ​​the clenched fist area; calculating the average value of the pixel areas of all the obtained clenched fist areas to obtain the reference area size information of the three-dimensional reference object.

[0023] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, for the multiple food images, extracting the food area to be estimated occupied by the food to be estimated, and determining the container shape of the food to be estimated according to the food area to be estimated corresponding to each image in the multiple food images, including: extracting images in the same direction as the corresponding direction of the qth standard container in the standard container set from the multiple food images to obtain at least one container image to be estimated corresponding to the qth standard container, and for each of the container images to be estimated, extracting the food area to be estimated occupied by the food to be estimated; calculating the Euclidean distance between the food area to be estimated of the container image to be estimated and the shape edge of the qth standard container in the corresponding direction, respectively, to obtain at least one Euclidean distance value; superimposing the Euclidean distance values ​​corresponding to all directions of the qth standard container with the preset weight value corresponding to each direction to obtain the Euclidean distance sum value corresponding to the qth standard container; detecting the minimum value in the Euclidean distance sum value set corresponding to the standard container set, and taking the standard container corresponding to the minimum value as the container shape of the food to be estimated, wherein q is a positive integer.

[0024] For example, in the food volume calculation method provided in at least one embodiment of the present disclosure, the food to be estimated is placed in a container, and the known container shape of the food to be estimated is obtained or the container shape of the food to be estimated is determined according to the one food image, and the volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the reference image, and the food area size information obtained based on the food image, including: obtaining the pixel area of ​​the area occupied by the three-dimensional reference object of the one reference image to obtain the reference area size information of the three-dimensional reference object; obtaining a second mapping ratio according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; for the one food image, extracting the food area to be estimated occupied by the food to be estimated to obtain the food area size information; obtaining the food size information according to the food area size information and the second mapping ratio; obtaining the volume information of the food to be estimated according to the container shape of the food to be estimated and the food size information.

[0025] For example, in the food volume calculation method provided in at least one embodiment of the present disclosure, determining the container shape of the food to be estimated includes: obtaining a standard container recognition model based on a deep learning model, and determining the container shape of the food to be estimated according to the multiple food images and the standard container recognition model; or obtaining a standard container recognition model, and determining the container shape of the food to be estimated according to the one food image and the standard container recognition model.

[0026] For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, for the image to be processed including the food to be estimated, the extracting the food area to be estimated occupied by the food to be estimated includes: dividing the image to be processed into multiple area sets, performing similarity calculation on each of the area sets to obtain a similarity value for each of the area sets; merging area sets whose similarity values ​​meet a preset threshold to obtain multiple merged areas; taking the closed area with the largest pixel area in the merged area as the food area to be estimated of the image to be processed; wherein the similarity calculation includes one or more combinations of color similarity calculation, texture similarity calculation, size similarity calculation, and overlap similarity calculation. For example, in the food volume calculation method provided by at least one embodiment of the present disclosure, for the image to be processed including the food to be estimated, the extracting the food area to be estimated occupied by the food to be estimated includes: performing edge recognition on the image to be processed based on a deep convolutional network model to obtain the food area to be estimated of the food to be estimated in the image to be processed.

[0027] At least one embodiment of the present disclosure provides a method for calculating food calories, comprising: obtaining the type of food to be estimated; obtaining the volume of the food to be estimated according to the food volume calculation method described in any embodiment of the present disclosure; querying a food density table to determine the density of the food to be estimated according to the type of food to be estimated, and obtaining the mass of the food to be estimated based on the density of the food to be estimated and the volume of the food to be estimated; querying a food calorie table to determine the calorie density of the food to be estimated according to the type of food to be estimated, and obtaining the calories of the food to be estimated based on the calorie density of the food to be estimated and the mass of the food to be estimated.

[0028] At least one embodiment of the present disclosure provides an electronic device, comprising: a food volume calculation unit, the food volume calculation unit is configured to execute the food volume calculation method described in any embodiment of the present disclosure to obtain the volume information of the food to be estimated. For example, the electronic device provided in at least one embodiment of the present disclosure also includes a food calorie calculation unit, which is configured to: obtain the type of the food to be estimated; determine the density of the food to be estimated according to the type of the food to be estimated, and obtain the mass of the food to be estimated based on the density of the food to be estimated and the volume information of the food to be estimated; determine the calorie density of the food to be estimated according to the type of the food to be estimated, and obtain the calorie of the food to be estimated based on the calorie density of the food to be estimated and the mass of the food to be estimated.

[0029] For example, the electronic device provided in at least one embodiment of the present disclosure also includes an image acquisition unit, configured to obtain reference images and food images corresponding to the three-dimensional reference object and the food to be estimated; and an output unit, configured to display or output volume information and / or calorie information of the food to be estimated.

[0030] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transiently stores computer-executable instructions; and a processor, which is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the food volume calculation method according to any embodiment of the present disclosure or the food calorie calculation method according to any embodiment of the present disclosure.

[0031] At least one embodiment of the present disclosure provides a storage medium, wherein the storage medium stores computer executable instructions, and when the computer executable instructions are executed by a processor, the food volume calculation method according to any embodiment of the present disclosure or the food calorie calculation method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0033] Figure 1 A schematic flow chart of a method for calculating food volume provided in at least one embodiment of the present disclosure;

[0034] Figure 2 for Figure 1 An example flow chart of step S110 in the method for calculating the food volume is shown;

[0035] Figure 3A for Figure 1 An example flow chart of step S130 in the method for calculating the food volume is shown;

[0036] Figure 3B A standard container image of a hemispherical bowl provided for at least one embodiment of the present disclosure;

[0037] Figure 4A A schematic diagram of a fist-making image provided by at least one embodiment of the present disclosure;

[0038] Figure 4B Extraction provided for at least one embodiment of the present disclosure Figure 4A A schematic diagram of the fist-clenching region process in the hand-clenching image shown;

[0039] Figure 5A A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0040] FIG. 5B to FIG. 5F A product interface diagram for executing the food volume calculation method and the food calorie calculation method provided by at least one embodiment of the present disclosure;

[0041] Figure 6 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0042] Figure 7 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure;

[0043] Figure 8 A schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0045] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components.

[0047] The volume of food can be estimated through food photos and the calorie content of food can be obtained. For example, the volume of food can be obtained based on photos of a specific standard card and food taken at a certain angle range. This method requires a specific standard card as a reference and its usage scenarios are limited. In addition, since the specific standard card has a planar structure, it also has requirements on the shooting direction and needs to obtain images within a specific shooting angle range, which increases the complexity of the operation.

[0048] At least one embodiment of the present disclosure provides a method for calculating the volume of food. The method comprises: obtaining actual size information of a three-dimensional reference object; obtaining a reference image and a food image corresponding to the three-dimensional reference object and the food to be estimated; and obtaining volume information of the food to be estimated according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the reference image, and the food area size information obtained based on the food image.

[0049] The food volume calculation method does not limit the reference image and the food image to be in the same image, and does not limit the image acquisition to be completed in a specific shooting method, thereby reducing the complexity of the operation and enhancing the flexibility of the application.

[0050] The above-mentioned food volume calculation method is described in a non-limiting manner below through multiple embodiments and at least one example of each embodiment. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also belong to the scope of protection of the present disclosure.

[0051] Figure 1 A schematic flowchart of a method for calculating food volume provided for at least one embodiment of the present disclosure.

[0052] For example, Figure 1 As shown, the food volume calculation method provided by at least one embodiment of the present disclosure includes the following steps S110 to S130.

[0053] In step S110 , actual size information of the three-dimensional reference object is obtained.

[0054] In step S120, a reference image and a food image corresponding to the three-dimensional reference object and the food to be estimated are obtained. Here, the obtained reference image allows the reference area size information of the three-dimensional reference object to be calculated, and the obtained food image allows the food area size information of the food to be estimated to be calculated.

[0055] In step S130, volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the reference image, and the food area size information obtained based on the food image.

[0056] For example, steps S110 to S130 may be executed sequentially or in other adjusted orders. The embodiments of the present disclosure do not limit the execution order of the steps and may be adjusted according to actual conditions. For example, steps S110 to S130 may be implemented by a server or a local terminal, and the embodiments of the present disclosure do not limit this. For example, in some examples, the food volume calculation method provided by at least one embodiment of the present disclosure may selectively execute some of the steps S110-S130, or may execute some additional steps in addition to steps S110-S130, and the embodiments of the present disclosure do not specifically limit this.

[0057] For example, the three-dimensional reference object can be a three-dimensional object that can calculate volume information. For example, the three-dimensional reference object can be a sphere, a cylinder, an ellipsoid, a cube, etc. For example, the three-dimensional reference object can be selected as various items that are easy to obtain in people's lives, such as fruits such as apples or oranges, tennis balls or eggs, etc., and for example, it can be a hand in a clenched fist state that is approximately spherical. It should be noted that the shape of the three-dimensional reference object here is not limited to a standard shape. Three-dimensional reference objects that can be approximately spheres, cylinders, etc. can be used to implement the food volume calculation method disclosed in the present invention, and the present invention does not limit this. On the other hand, in the method disclosed in the present invention, the food to be estimated can be provided independently, such as food with a stable shape, such as steamed buns and bread, or it can be food contained in a container of a certain shape, such as rice, oatmeal, meat, etc., and the embodiments of the present invention do not limit this.

[0058] For example, for step S110, obtaining the actual size information of the three-dimensional reference object can be implemented in various appropriate ways. For example, the size of the three-dimensional reference object is a fixed known size pre-stored in a database, and obtaining the actual size information of the three-dimensional reference object includes retrieving the known size stored in the database; for example, the size of the three-dimensional reference object is a standard size, such as a football, a table tennis ball, etc., and obtaining the actual size information of the three-dimensional reference object includes obtaining the standard size of the three-dimensional reference object; for example, the size of the three-dimensional reference object can be obtained by pre-calculation and stored in the database, and obtaining the actual size information of the three-dimensional reference object includes retrieving the pre-calculated size stored in the database; for example, the size of the three-dimensional reference object can also be recalculated each time step S110 is executed to obtain the current size information of the three-dimensional reference object. For example, when calculating the size of the three-dimensional reference object, one or more sizes of the three-dimensional reference object can be obtained through a reference object with a known size or a size that can be directly obtained, and then the actual size information of the three-dimensional reference object can be calculated.

[0059] For example, in one example, a plurality of stereo reference images are obtained in which a stereo reference object is placed side by side with a first reference object whose reference size can be obtained, for example, the first reference object is placed side by side next to the stereo reference object, and during the shooting process, the first reference object and the stereo reference object are kept from blocking each other. For example, the plurality of stereo reference images are images taken at different angles, for example, a front view and a side view of the stereo reference object and the first reference object placed side by side, and the actual size information of the stereo reference object is obtained according to the plurality of stereo reference images and the reference size of the first reference object. For example, for each stereo reference image, a first number of pixels corresponding to the reference size of the first reference object in the stereo reference image and a second number of pixels in at least one dimension of the stereo reference object in the stereo reference image are obtained; based on the first number of pixels and the second number of pixels, a calculated size of the stereo reference object in at least one dimension in the stereo reference image is obtained; and the actual size information of the stereo reference object is obtained based on the plurality of calculated sizes obtained from the plurality of stereo reference images. For example, the first number of pixels corresponding to the reference size of the first reference object in the stereo reference image can be obtained by a deep learning model. Such a first reference object is, for example, any object with a known size (for example, a dimension such as length or width, etc., such as a ruler (or a triangle), A4 paper (or B5 paper, etc.), etc.

[0060] In another example, obtaining actual size information of a three-dimensional reference object includes: obtaining a three-dimensional reference object image in which the three-dimensional reference object is placed on one side of a second reference object whose reference area can be calculated; obtaining actual size information of the three-dimensional reference object according to the three-dimensional reference object image and the reference area, for example, the three-dimensional reference object image includes information in the second reference object used to calculate the reference area. For example, if the second reference object is a plane reference object, for example, the second reference object can be any plane object with a standard size, for example, it can be A4 paper (or B5 paper or other paper with a standard size), 120-type optical disc (with a diameter of 120 mm), etc., then one side of the second reference object refers to any side of the second reference object, and the reference area refers to the reference area calculated from any side of the second reference object; for example, if the second reference object is a three-dimensional reference object, for example, the second reference object is a three-dimensional object with a selected plane with known dimensions (such as length and width) in two dimensions, then one side of the second reference object refers to the selected plane in the second reference object, and the reference area of ​​the selected plane in the second reference object can be obtained according to the known dimensions in the two dimensions.

[0061] For example, when the second reference object is a rectangular object such as A4 paper (or B5 paper, etc.), the actual size information of the three-dimensional reference object is obtained according to the three-dimensional reference object image and the reference area, including: obtaining the pixel area of ​​the second reference object in the three-dimensional reference object image; obtaining the first mapping ratio based on the pixel area of ​​the second reference object and the reference area; obtaining the pixel area of ​​the area occupied by the three-dimensional reference object in the three-dimensional reference object image; obtaining the volume of the three-dimensional reference object according to the first mapping ratio and the pixel area of ​​the area occupied by the three-dimensional reference object, so as to obtain the actual size information of the three-dimensional reference object. For example, in these examples, the three-dimensional reference object can be a sphere, an ellipsoid, a cuboid, etc., and thus according to the first mapping ratio and the pixel area of ​​the area occupied by the three-dimensional reference object, the actual area of ​​the cross section of the three-dimensional reference object along the central axis can be obtained, thereby respectively obtaining the radius, the major diameter and the minor diameter, the length and the width, etc., and further obtaining the actual size of the three-dimensional reference object.

[0062] For example, the second reference object is a rectangular object such as A4 paper (or B5 paper, etc.), and the stereoscopic reference object image obtained by shooting has an angle error. If the pixel area of ​​the second reference object is calculated directly by the value of the pixel point, an error will occur. The second reference object can use the minimum bounding rectangle to select the range of the second reference object, and the pixel area of ​​the second reference object is obtained by calculating the pixel area of ​​the bounding rectangle, thereby obtaining a more accurate area of ​​the second reference object. For example, obtaining the pixel area of ​​the second reference object in the stereoscopic reference object image includes: obtaining the minimum bounding rectangle of the second reference object in the stereoscopic reference object image and the pixel area of ​​the minimum bounding rectangle, and using the pixel area of ​​the minimum bounding rectangle as the pixel area of ​​the second reference object. For example, the pixel value of the pixel point corresponding to the second reference object in the stereoscopic reference object image is within a first value range, and the minimum bounding rectangle of the second reference object is obtained according to the position of the pixel point with the pixel value within the first value range. In addition, obtaining the pixel area of ​​the area occupied by the stereoscopic reference object in the stereoscopic reference object image also includes: using the pixel points whose pixel values ​​in the minimum bounding rectangle do not meet the first value range as the area occupied by the stereoscopic reference object, so as to obtain the pixel area of ​​the area occupied by the stereoscopic reference object.

[0063] Further, for example, for the second reference object in the stereoscopic reference image, for example, a substantially monochrome image (e.g., monochrome (e.g., white) A4 paper, etc.), for example, the pixel value of the pixel point corresponding to the second reference object in the stereoscopic reference image is within a first value range. Correspondingly, obtaining the pixel area of ​​the region occupied by the stereoscopic reference object in the stereoscopic reference image includes: taking the pixel points whose pixel values ​​in the stereoscopic reference image do not satisfy the first value range as the region occupied by the stereoscopic reference object, so as to obtain the pixel area of ​​the region occupied by the stereoscopic reference object; or, extracting the region occupied by the stereoscopic reference object according to a deep learning model, and obtaining the pixel area of ​​the region occupied by the stereoscopic reference object, for example, performing target recognition on the stereoscopic reference image according to a deep learning model (e.g., faster-RCNN algorithm, YOLO algorithm) to extract the region occupied by the stereoscopic reference object, and then obtaining the pixel area of ​​the region occupied by the stereoscopic reference object.

[0064] For step S120, for example, the reference image and food image obtained may be one or multiple images. For example, a reference image and a food image obtained in pairs of a stereo reference object and a food to be estimated in one shooting method are obtained. For example, multiple reference images and multiple food images obtained in pairs of a stereo reference object and a food to be estimated in multiple shooting methods may be obtained. For example, multiple reference images and multiple food images may be obtained in pairs in multiple shooting methods. For example, multiple reference images and multiple food images may be obtained in pairs in multiple shooting methods. For example, multiple reference images and multiple food images are shot in pairs, and each pair corresponds to one shooting method. Different multiple pairs of reference images and food images are obtained by adjusting different shooting methods.

[0065] For example, the reference image and the food image acquired in pairs in the same shooting mode are a pair of images corresponding to each other, that is, the reference image and the food image are independently shot in the same shooting mode, thereby obtaining a pair of images corresponding to each other. Alternatively, the reference image and the food image acquired in pairs in the same shooting mode are components of the same image, that is, multiple combined images are shot in multiple shooting modes, each of which includes a three-dimensional reference object and the food to be estimated (for example, the two are placed side by side in any feasible manner), that is, each combined image includes a part of the reference image and a part of the food image, thereby obtaining a pair of images corresponding to each other. In this manner, the three-dimensional reference object and the food to be estimated can be placed in any feasible manner as long as the images of the two can be obtained, and in this manner, the shooting methods of the reference image and the corresponding food image are exactly the same.

[0066] The food volume calculation method provided by at least one embodiment of the present disclosure can separately obtain multiple food images and multiple reference images taken in the same shooting method, so that it is not necessary to shoot the food to be estimated and the reference object together at the same time when acquiring the image. The food images and reference images can be acquired separately or even in time as needed, which makes the use scenario of the food volume calculation method more flexible, the operation method is simple and fast, and the user experience is improved.

[0067] For example, the shooting method has multiple parameters, for example, at least the supporting surface on which the stereo reference object or the food to be estimated is placed, the shooting distance and the shooting direction. For example, the shooting distance is the shooting distance of the camera relative to the object to be shot, and the shooting direction is the shooting direction of the camera relative to the supporting surface. The shooting direction can also be understood as the angle between the line between the camera and the object to be shot and the supporting surface. For example, the shooting direction is perpendicular to the supporting surface or parallel to the supporting surface. For example, the shooting method can also include the focal length during shooting, etc. For example, for a corresponding pair of reference images and food images obtained by a shooting method, their shooting distances and shooting directions should be the same, and their supporting surfaces can be the same or different. For example, for multiple food images and multiple reference images obtained by different shooting methods, their shooting directions are different, and their shooting distances and supporting surfaces can be the same or different. For example, a reference image obtained by a first shooting method for a stereo reference object and a food image obtained by a first shooting method for the food to be estimated can be obtained respectively. The two images may not use the same supporting surface, but the shooting distances relative to the object to be shot and the shooting directions relative to the supporting surface in the two images are the same, thereby allowing the calculation required in the method of the embodiment of the present disclosure to be performed. The embodiments of the present disclosure do not limit the tools used for shooting. For example, cameras, video recorders, etc. of various specifications can be used.

[0068] For example, in some examples, in order to calculate the reference area size information of the three-dimensional reference object and the food area size information of the food to be estimated, it is necessary to obtain multiple reference images and multiple food images in pairs in multiple (for example, two or more) shooting modes. For example, the multiple shooting modes may include a first shooting mode and a second shooting mode, the shooting directions of the two shooting modes are different from each other, and the shooting direction of the first shooting mode and the shooting direction of the second shooting mode intersect with each other, that is, the shooting direction of the first shooting mode and the shooting direction of the second shooting mode have a certain angle, for example, the angle can be any angle between 0-90 degrees, so that the size information of at least two dimensions of the photographed object can be obtained by analyzing the photographed images, for example, the length information, width information and height information of the photographed object can be obtained by the image detection and recognition method. Correspondingly, the multiple reference images include the first reference image and the second reference image obtained using the first shooting mode and the second shooting mode respectively, and the first reference image and the second reference image allow the reference area size information of the three-dimensional reference object to be calculated; the multiple food images include the first food image and the second food image obtained using the first shooting mode and the second shooting mode respectively, and the first food image and the second food image allow the food area size information of the food to be estimated to be calculated. Since the shooting direction is not restricted to meet specific requirements, the user's operation difficulty is reduced and the application flexibility is enhanced.

[0069] For example, the shooting direction of the first shooting mode and the shooting direction of the second shooting mode are perpendicular to each other, and the shooting direction of the first shooting mode is perpendicular to the support surface of the first shooting mode, and the shooting direction of the second shooting mode is parallel to the support surface of the second shooting mode. In the two modes, the support surfaces are the same support surface or parallel to each other, so that, for example, the first shooting mode can obtain a front view of the object (i.e., the three-dimensional reference object and the food to be estimated), and the second mode can obtain a side view of the object, so that the size (such as the bottom area) obtained from the front view and the size (such as the height or thickness) obtained from the side view can be used to calculate the volume.

[0070] For example, in at least one example, obtaining a plurality of reference images and a plurality of food images obtained in pairs of a stereoscopic reference object and a food to be estimated in a plurality of shooting modes includes: continuously shooting a plurality of images of the food to be estimated and the stereoscopic reference object simultaneously located on the same support surface, extracting a plurality of key images from the plurality of images to obtain a plurality of reference images and a plurality of food images, wherein the key images include a corresponding pair of reference images and food images. Continuously shooting a plurality of images can obtain more comprehensive information about the food to be estimated, thereby obtaining a more accurate calculation result of the volume of the food to be estimated.

[0071] For example, continuously shooting multiple images of the food to be estimated and the three-dimensional reference object simultaneously located on the same support surface, extracting multiple key images from the multiple images, including: for each image in the multiple images, extracting the food area to be estimated occupied by the food to be estimated, and obtaining the pixel area of ​​the food area to be estimated corresponding to each image; according to the pixel area of ​​the food area to be estimated corresponding to each image, extracting an image in which the relative change of the pixel area of ​​the food area to be estimated in the multiple images is greater than a preset threshold value as one of the multiple key images, for example, the multiple key images at least include a first key image and a second key image, the first key image includes a first food image and a first reference image, and the second key image includes a second food image and a second reference image. For example, by shooting a video, multiple images of the food to be estimated and the three-dimensional reference object simultaneously located on the same support surface can be continuously shot. The relative change here refers to the difference in pixel area of ​​the food region to be estimated between multiple images. For example, after obtaining the pixel area of ​​the food region to be estimated in multiple images, the images are sorted according to the size of the pixel area of ​​the food region to be estimated (for example, ascending or descending order), and then a reference image is selected. From the sequence, an image whose pixel area change relative to the reference image is greater than a preset threshold is selected as the first key image, and so on and so forth to obtain multiple key images. Selecting an image with a larger difference in pixel area of ​​the food region to be estimated from multiple images as the key image can obtain a better food volume estimation result while reducing the number of image analyses.

[0072] For step S130, the reference area size information of the three-dimensional reference object can be calculated based on the reference image, and the food area size information of the food to be estimated can be calculated based on the food image, so that the volume information of the food to be estimated can be obtained based on the actual size information of the three-dimensional reference object, the reference area size information and the food area size information.

[0073] For example, in at least one of the above examples, the volume information of the food to be estimated can be obtained based on the actual size information of the three-dimensional reference object, the first reference image, the second reference image, the first food image, and the second food image.

[0074] For example, the food to be estimated is contained in a container of a certain shape, and the volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information and the food area size information, including: obtaining the pixel area of ​​the area occupied by the three-dimensional reference object of each reference image in multiple reference images to obtain the reference area size information of the three-dimensional reference object; obtaining a second mapping ratio according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; for the above-mentioned first food image, extracting the food area to be estimated occupied by the food to be estimated to obtain the food area size information; obtaining the food size information according to the food area size information and the second mapping ratio; for multiple food images, extracting the food area to be estimated occupied by the food to be estimated, and determining the container shape of the food to be estimated according to the food area to be estimated corresponding to each image in the multiple food images; obtaining the volume information of the food to be estimated according to the container shape of the food to be estimated and the food size information. For example, for the above-mentioned second food image, the food area to be estimated occupied by the food to be estimated is extracted, and the height value of the food to be estimated is determined according to the food area to be estimated and the second mapping ratio; and the volume information of the food to be estimated is obtained according to the shape of the container of the food to be estimated, the height value of the food to be estimated and the food size information.

[0075] For example, in the above example, the pixel area of ​​the area occupied by the stereo reference object of each reference image in the multiple reference images is obtained to obtain the reference area size information of the stereo reference object, including: extracting the area occupied by the stereo reference object for each reference image (for example, according to a deep learning model), obtaining the pixel area of ​​the area occupied by the stereo reference object corresponding to each reference image; calculating the average value of the pixel areas of the areas occupied by all the obtained stereo reference objects to obtain the reference area size information of the stereo reference object. The deep learning model used here is, for example, trained in advance using training data, and can divide the image into regions.

[0076] For instance, in the above example, the height value of the food to be estimated is determined according to the food area to be estimated and the second mapping ratio, including: obtaining the number of pixels corresponding to each column of pixels in the food area to be estimated; determining the height component of the food to be estimated according to the number of pixels corresponding to each column of pixels; determining the height value of the food to be estimated according to the height component and the second mapping ratio.

[0077] For example, for multiple food images, the shape of the container of the food to be estimated is determined according to the food area to be estimated corresponding to each image in the multiple food images, including: obtaining a standard container recognition model (such as a deep learning model), and determining the shape of the container of the food to be estimated according to the multiple food images and the standard container recognition model. The deep learning model can be trained in advance using training data including various container shapes, thereby detecting the shape of the container of the input image as a disk, a hemisphere, a truncated cone, etc. After the shape of the container is obtained, the volume can be calculated according to the shape.

[0078] For example, in the above example, for an image to be processed including food to be estimated, extracting the food area to be estimated occupied by the food to be estimated includes: dividing the image to be processed into multiple area sets, performing similarity calculation on each area set to obtain a similarity value for each area set; merging area sets whose similarity values ​​meet a preset similarity threshold to obtain multiple merged areas; and taking the closed area with the largest pixel area in the merged area as the food area to be estimated of the image to be processed. Here, the similarity calculation includes one or more combinations of color similarity calculation, texture similarity calculation, size similarity calculation, and overlap similarity calculation.

[0079] For example, for an image to be processed that includes food to be estimated, extracting a food region to be estimated occupied by the food to be estimated includes: performing edge recognition on the image to be processed based on a deep convolutional network model to obtain the food region to be estimated of the food to be estimated in the image to be processed.

[0080] For example, in at least one example, edge detection is performed on the first reference image and the second reference image through a deep learning model, and the pixel area of ​​the area occupied by the three-dimensional reference object in each reference image is obtained to obtain the reference area size information of the three-dimensional reference object. For example, the average value of the pixel area of ​​the area occupied by the three-dimensional reference object in the two reference images is calculated to obtain the reference area size information of the three-dimensional reference object; the second mapping ratio is obtained according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; then, for the first food image, edge detection is performed through a deep learning model to extract the food area to be estimated occupied by the food to be estimated, and the shape and size of the food area to be estimated are obtained. For example, if the food area to be estimated is circular, the maximum width of the food area to be estimated is detected as the diameter of the food area to be estimated, and then based on the second mapping ratio, the real diameter of the food to be estimated is obtained as the food size information; a pre-trained deep learning model (such as faster-RCNN algorithm, YOLO algorithm) is used to perform target recognition on multiple food images to obtain the shape of the food container to be estimated, for example, the food container to be estimated is a pot or bowl that can be approximated as a hemispherical shape; according to the food size information and the shape of the food container to be estimated, the volume information of the food to be estimated is obtained, for example, according to the hemispherical volume calculation formula and the real diameter of the estimated food, the volume information of the estimated food is obtained.

[0081] For example, in some other examples, step S130 may also include obtaining a second mapping ratio based on the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; identifying the food to be estimated in multiple food images through a deep learning model (such as a FasterR-CNN model) to obtain edge recognition results and the name of the food to be estimated, obtaining the food area size information of the food to be estimated based on the edge recognition results, thereby obtaining the food size information based on the food area size information to be estimated and the second mapping ratio, obtaining the shape of the food to be estimated based on the name of the food to be estimated, and obtaining the volume information of the food to be estimated based on the food size information and the shape of the food to be estimated. For example, the food size information includes the area of ​​the food to be estimated, and the food to be estimated is detected as an apple through the deep learning model. Since the apple can be approximated as a sphere, the volume information of the food to be estimated can be obtained.

[0082] In addition, it should be noted that, for step S130, the multiple reference images obtained allow the reference area size information of the three-dimensional reference object to be calculated, and the multiple food images obtained allow the food area size information of the food to be estimated to be calculated. The present disclosure does not limit the executor of obtaining the reference area size information based on the multiple reference images and obtaining the food area size information based on the multiple food images. For example, steps S110 to S130 are executed by a mobile terminal. At this time, the reference area size information of the three-dimensional reference object and / or the food area size information of the food to be estimated can be calculated by the mobile terminal, or calculated by the server and then sent to the mobile terminal.

[0083] For example, in other examples, the food image and reference image obtained are images directly above the three-dimensional reference object and the food to be estimated, that is, images with a shooting direction perpendicular to the support surface where the food to be estimated is located, the food to be estimated is placed in a container, and the shape of the container of the food to be estimated is known or the shape of the container of the food to be estimated can be determined based on a food image. In this case, only the one food image and the one reference image are needed to complete the calculation of the volume information of the food to be estimated. The volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information and the food area size information, including: obtaining the pixel area of ​​the area occupied by the three-dimensional reference object of the reference image taken to obtain the reference area size information of the three-dimensional reference object; obtaining the second mapping ratio according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; extracting the food area occupied by the food to be estimated for the one food image to obtain the food area size information; obtaining the food size information according to the food area size information and the second mapping ratio; obtaining the volume information of the food to be estimated according to the container shape of the food to be estimated and the food size information.

[0084] The following uses an example in which a three-dimensional reference object is a hand in a clenched fist that is approximately spherical, and a second reference object is a paper of standard size to specifically illustrate the method for calculating the food volume provided by the present disclosure. For the hand in a clenched fist that is approximately spherical, for example, it is treated as a standard sphere, but the embodiments of the present invention are not limited to this. For example, it can also be treated as an ellipsoid or a triangular prism, thereby obtaining the required size information and its approximate volume, which is used to estimate the food volume.

[0085] It should be pointed out that the present disclosure does not limit the type of image processed. In some examples, it can be a color image or a black and white grayscale image, and even in some examples, it can be a binary image. For color images, the pixel values ​​of the pixels mentioned in the present disclosure include a set of RGB pixel values. For black and white grayscale images, the pixel values ​​of the pixels are expressed in grayscale values.

[0086] In step S110 , actual size information of the three-dimensional reference object is obtained.

[0087] For example, in some examples, the hand in a clenched fist is first placed on the side of a second reference object whose reference area can be calculated, for example, the stereoscopic reference object is located in front of the second reference object, and a stereoscopic reference object image is captured, and the stereoscopic reference object image includes information in the second reference object used to calculate the reference area. For example, the second reference object is a monochrome (e.g., white) paper of standard size, such as A4 paper or B5 paper. For example, there may be patterns or other color patterns on the second reference object, but the patterns or other color patterns cannot exceed 5% of the total area of ​​the second reference object to avoid large detection errors when detecting the stereoscopic reference object. For example, the hand in a clenched fist can be placed in front of a monochrome paper of standard size, and a stereoscopic reference object image is captured, and the stereoscopic reference object image includes all the A4 paper information, that is, the entire length and width information of the A4 paper (the A4 paper may also be partially blocked).

[0088] The actual size information of the three-dimensional reference object is obtained according to the three-dimensional reference object image and the reference area of ​​the second reference object. For example, in at least one embodiment of the present disclosure, Figure 2 As shown, this step may specifically include steps S1101 to S1104.

[0089] In step S1101, the pixel area of ​​the second reference object in the stereoscopic reference object image is obtained.

[0090] For example, the second reference object is a single-color (eg, white, green, yellow, etc.) paper of standard size. For example, the second reference object is a white A4 paper, whose size is 210 mm×297 mm, so its reference area S A4 623.7cm 2 In order to better use the A4 paper as a reference, its area is used instead of its length and width for calculation, thereby avoiding the adverse effects of distortion caused by the shooting angle when shooting the image. In the stereoscopic reference image, the pixel values ​​corresponding to the pixel points of the second reference object are in the first value range (here, the range corresponding to white, for example, the R pixel value range is 250-255, the G pixel value range is 250-255, and the B pixel value range is 250-255). The positions of the pixel points whose pixel values ​​satisfy the first value range in the stereoscopic reference image are obtained, and the pixel areas S are obtained according to the positions of these pixel points. 矩形 .

[0091] In step S1102, a first mapping ratio is obtained based on the pixel area of ​​the second reference object and the reference area.

[0092] For example, the first mapping ratio can be calculated according to formula (1):

[0093]

[0094] In step S1103 , the pixel area of ​​the region occupied by the stereoscopic reference object in the stereoscopic reference object image is obtained.

[0095] For instance, in some examples, the wrist part of the hand in the clenched fist state in the stereoscopic reference object image is against the edge of the second reference object, and the area occupied by the stereoscopic reference object can be obtained based on the color difference between the hand in the clenched fist state and the second reference object, that is, the pixel value interval of the pixel points corresponding to the hand part in the clenched fist state in the stereoscopic reference object image is different from the pixel value interval of the pixel points corresponding to the second reference object part, and the fist area occupied by the hand in the clenched fist state is obtained by detecting the pixel values ​​of the pixel points of the stereoscopic reference object image.

[0096] For example, the second reference object is a single-color (e.g., white) paper of standard size, and the pixel values ​​of the pixel points in the three-dimensional reference object image that do not satisfy the first value range are taken as the fist area, and the pixel area of ​​the fist area is obtained as the pixel area S of the area occupied by the three-dimensional reference object (i.e., the hand in the fist state). ref .

[0097] In step S1104, the volume of the three-dimensional reference object is acquired according to the first mapping ratio and the pixel area of ​​the region occupied by the three-dimensional reference object to obtain actual size information of the three-dimensional reference object.

[0098] Since the hand in the clenched fist state can be approximated as a sphere, its orthographic projection in the 3D reference object image can be approximated as a circle. Therefore, through the circle area calculation formula, the pixel area S of the area occupied by the 3D reference object can be calculated according to the first mapping scale scale1 and the pixel area S of the area occupied by the 3D reference object. ref , calculate the radius r of the three-dimensional reference object according to formula (2):

[0099]

[0100] After obtaining the radius r of the three-dimensional reference object, the actual size information of the hand in the fist state is calculated according to the sphere volume calculation formula (3), that is, the actual size information of the three-dimensional reference object.

[0101]

[0102] For example, in some other examples, a color stereo reference object image such as a hand in a clenched fist may be grayscaled to obtain a black and white grayscale image; the grayscale image may be binarized to obtain an intermediate binary image; and finally the intermediate binary image may be eroded and expanded to fill the small holes in the intermediate binary image to obtain a binary image of the stereo reference object. For example, when executing step S110, in step S1101, the positions of the pixels whose grayscale values ​​of the pixels in the binary image of the stereo reference object meet the first value range (for example, the second reference object is white A4 paper, and the first value range is 250-255) are obtained, and the pixel area S of the second reference object is obtained according to the positions of these pixels. 矩形 ; Next, execute steps S1102-S1104 to obtain the actual size information of the three-dimensional reference object.

[0103] For example, in some other examples, the wrist portion of the hand in the clenched fist state in the stereoscopic reference object image does not touch the edge of the second reference object, and the stereoscopic reference object image includes the hand and arm portion in the clenched fist state. Since the area occupied by the hand in the clenched fist state in the stereoscopic reference object image includes not only the approximately spherical hand in the clenched fist state, but also the long arm portion, it is necessary to obtain the hand clenched fist image through the hand detection model, and then extract the clenched fist area in the hand clenched fist image, and accordingly remove the arm portion as much as possible to reduce the calculation error. For example, when executing step S110, first execute step S1101 to obtain the pixel area of ​​the second reference object in the stereoscopic reference object image; then, execute step S1102 to calculate the first mapping ratio; then, in step S1103, obtain the hand clenched fist image through the hand detection model, extract the clenched fist area in the hand clenched fist image, and obtain the pixel area of ​​the clenched fist area as the pixel area S of the area occupied by the stereoscopic reference object (i.e., the hand in the clenched fist state) ref ; Finally, execute step S1104 to obtain the actual size information of the three-dimensional reference object.

[0104] In contrast, for example, in some other examples, the second reference object is a monochrome background plate (e.g., a 120-inch optical disc printed in black) whose reference area can be calculated, and its shape may not be a rectangle. For example, when executing step S110, the shape of the second reference object (e.g., a circle) can be identified through a deep learning model, the second reference object area can be extracted, and the pixel area of ​​the second reference object area can be obtained, thereby obtaining a first mapping ratio based on the pixel area of ​​the second reference object area and the reference area of ​​the second reference object; then, the positions of pixels in the second reference object area that are different from the pixel values ​​of the second reference object (e.g., pixels whose R pixels are not 0, G pixels are not 0, and B pixels are not 0) are detected, thereby obtaining the pixel area S of the area occupied by the three-dimensional reference object (e.g., a hand in a clenched fist) ref; Finally, execute step S1104 to obtain the actual size information of the three-dimensional reference object.

[0105] Next, in step S120, a reference image and a food image corresponding to the three-dimensional reference object and the food to be estimated are acquired.

[0106] For example, the food to be estimated and the hand in the fist state are placed on the same support surface at the same time, for example, the two are placed side by side, for example, the hand is placed side by side with the fist next to the food to be estimated, and during the shooting process, the food to be estimated and the hand in the fist state are kept from blocking each other, and multiple images of the food to be estimated and the hand in the fist state are continuously shot. At the same time, the food to be estimated and the hand in the fist state should be kept on the same horizontal plane during the shooting process and will not block each other. For example, by shooting a video, multiple images of the food to be estimated and the hand in the fist state are continuously shot on the same support surface. For example, a video can be shot or other continuous shooting methods, starting from a direction perpendicular to the support surface where the food to be estimated and the hand in the fist state are located, sliding and shooting a video until it stops parallel to the direction of the support surface where the food to be estimated and the hand in the fist state are located, thereby obtaining multiple images, each image including a corresponding pair of reference images and food images. Since the hand is approximately spherical after being clenched, when collecting videos, it can be ensured that the camera does not affect the detection effect due to different shooting angles.

[0107] Next, multiple key images are extracted from the multiple images to obtain multiple reference images and multiple food images. For example, for each of the multiple images, the food region to be estimated occupied by the food to be estimated in each image is extracted, and the pixel area of ​​the food region to be estimated corresponding to each image is obtained; according to the pixel area of ​​the food region to be estimated corresponding to each image, an image whose relative change in the pixel area of ​​the food region to be estimated is greater than a preset threshold value is extracted from the multiple images as one of the multiple key images. The preset threshold value can be pre-selected based on experience, for example, 10% to 50%, and the number of key images is, for example, greater than or equal to 3. After the pixel area of ​​the food region to be estimated in the multiple images is obtained, the food region to be estimated is sorted according to the size of the pixel area (for example, ascending or descending order), and then a reference image is selected, and the first subsequent image whose relative change in pixel area relative to the reference image is greater than the preset threshold value is selected in the sequence, and then the second subsequent image whose relative change in pixel area relative to the first subsequent image is greater than the preset threshold value is selected in the sequence, and so on, thereby obtaining multiple key images.

[0108] For example, in one embodiment, first, an image whose shooting direction is perpendicular to the food to be estimated and the supporting surface of the hand in a clenched fist is extracted as the first key image; then, an image whose shooting direction is parallel to the food to be estimated and the supporting surface of the hand in a clenched fist is extracted as the second key image; then, an image in which the pixel area change of the food area to be estimated in the video relative to the first key image is greater than a threshold of 20% is extracted as the third key image; then, an image in which the pixel area change of the food area to be estimated in the video relative to the third key image is greater than 20% is extracted as the fourth key image, and so on, thereby obtaining multiple key images.

[0109] For example, each key image includes food information to be estimated and information of the hand in a clenched fist state, so that each key image includes a corresponding pair of reference images and food images. The food image is obtained by removing the hand part in the clenched fist state from each key image, thereby obtaining multiple corresponding reference images and food images based on multiple key images.

[0110] Finally, in step S130, the volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the reference image, and the food area size information obtained based on the food image.

[0111] Figure 3A for Figure 1 FIG. 1 is an example flow chart of step S130 in the method for calculating the food volume. Figure 3A As shown, the example of step S130 in the food volume calculation method may specifically include steps S1301 to S1306.

[0112] First, in step S1301, the pixel area of ​​the region occupied by the stereoscopic reference object in each reference image of the plurality of reference images is obtained to obtain reference region size information of the stereoscopic reference object.

[0113] For example, in some examples, a deep learning model (such as the faster-RCNN algorithm, the YOLO algorithm, etc.) can be used to extract the area of ​​a three-dimensional reference object (such as a hand in a clenched fist) for each reference image, and obtain the pixel area of ​​the area occupied by the three-dimensional reference object corresponding to each reference image; then, the average value of the pixel areas of the areas occupied by all the acquired three-dimensional reference objects is calculated to obtain the reference area size information of the three-dimensional reference object.

[0114] For example, in other examples, for each reference image, a hand clenched fist image can be obtained according to the hand detection model, the clenched fist area in the hand clenched fist image can be extracted, and the pixel area of ​​the clenched fist area can be obtained; then, the pixel area of ​​all the obtained clenched fist areas is averaged to obtain the reference area size information of the three-dimensional reference object. Then, in step S1302, the second mapping ratio is obtained according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object.

[0115] In the above example, the three-dimensional reference object is a hand in a clenched fist state which can be approximated as a sphere, so the real area S of the hand in the clenched fist state can be calculated based on the actual size information of the three-dimensional reference object. 手 The reference area size information of the three-dimensional reference object is the pixel area S of the area occupied by the hand in the fist state. pix , so the second mapping ratio can be calculated according to formula (4):

[0116]

[0117] Next, in step S1303, for the first food image, the food area to be estimated occupied by the food to be estimated is extracted to obtain food area size information.

[0118] For example, the first food image is an image whose shooting direction is perpendicular to the supporting surface where the food to be estimated is located, that is, an image directly above the food to be estimated.

[0119] For example, in some examples, for the first food image, when extracting the food region to be estimated occupied by the food to be estimated, a similarity calculation method can be used to obtain multiple regions with similar features such as color and texture, and the closed region with the largest area is selected as the food region to be estimated. For example, the first food image is first divided into multiple region sets, and a similarity calculation is performed on each region set to obtain a similarity value for each region set; then, the region sets whose similarity values ​​meet a preset similarity threshold are merged to obtain multiple merged regions; finally, the closed region with the largest pixel area in the merged region is used as the food region to be estimated for the first food image.

[0120] After obtaining the food region to be estimated of the first food image, the pixel area of ​​the food region to be estimated is extracted and used as the food region size information. For example, if the multiple food images are images of the food to be estimated placed in a hemispherical bowl, the image information directly above the food to be estimated obtained by the first food image is a circular food region to be estimated, and the pixel area of ​​the food region to be estimated is extracted as the food region size information.

[0121] For example, the similarity calculation method may use a selective search algorithm to segment the image of the food to be estimated, thereby obtaining the food region to be estimated. The selective search algorithm may include the following operations:

[0122] First, the dissimilarity between each pixel and the pixels that satisfy the connected domain relationship is calculated.

[0123] For example, the connected domain relationship can be four-connected, then the dissimilarity between each pixel point and the pixel points above, below, left, and right is calculated; for example, the connected domain relationship can be eight-connected, then the dissimilarity between each pixel point and the pixel points above, below, left, right, upper left, lower left, upper right, and lower right is calculated.

[0124] For example, the dissimilarity calculation may include one or more combinations of color similarity calculation, texture similarity calculation, size similarity calculation, and overlap similarity calculation. For example, the dissimilarity includes the sum of the results of color dissimilarity, texture dissimilarity, size dissimilarity, and overlap dissimilarity superimposed with corresponding weights.

[0125] It should be noted that the line between pixels is called an edge, and each edge corresponds to a pixel v i and v j , each edge corresponds to a dissimilarity value, which represents the dissimilarity between the two pixels corresponding to this edge.

[0126] Secondly, arrange the edges in ascending order according to the dissimilarity values ​​to obtain the edge set.

[0127] Then, traverse the edge set to make a merge judgment for each edge element.

[0128] For example, if the pixel v corresponding to the edge element i and v j Do not belong to the same area, and the dissimilarity value corresponding to the edge element is not greater than the pixel v i and v j Internal dissimilarity, then the pixel v i and v j Merge them into the same area; otherwise, continue traversing and make a merge judgment on the next edge element until the last edge element is traversed.

[0129] It should be noted that more information about the selective search algorithm can be found in conventional designs, and the above description of the present disclosure is merely an illustrative introduction.

[0130] For example, in other examples, when extracting the food area to be estimated occupied by the food to be estimated, the rough range of the food area to be estimated is first extracted according to the deep learning model, and then the food area to be estimated is obtained by the similarity calculation method for the rough range. This method of extracting the food area to be estimated by combining deep learning and similarity calculation can delete irrelevant background except the food area to be estimated, reduce the calculation error, and, in the case where the food to be estimated is contained in a container, can also improve the calculation error when the container is not full of the food to be estimated.

[0131] Next, in step S1304, the food size information is obtained according to the food area size information and the second mapping ratio.

[0132] For example, the multiple food images are images of the food to be estimated being placed in a hemispherical bowl, and the food area size information of the first food image is the pixel area S of the food area to be estimated. f According to the second mapping ratio, the actual area of ​​the food area to be estimated is S f *scale2, the actual radius r of the food area to be estimated can be obtained through the circular area calculation formula, that is, the food size information.

[0133] Next, in step S1305, for the multiple food images, the food area to be estimated occupied by the food to be estimated is extracted, and the container shape of the food to be estimated is determined according to the food area to be estimated corresponding to each of the multiple food images.

[0134] For example, in some examples, a standard container recognition model is obtained based on a deep learning model, and the container shape of the food to be estimated is determined based on multiple food images and the standard container recognition model. For example, a model for target recognition of common container data sets such as pots, bowls, plates, and basins can be trained based on the target recognition algorithm in the deep learning model (such as the faster-RCNN algorithm and the YOLO algorithm), thereby obtaining a standard container recognition model; multiple food images are input into the standard container recognition model for container type recognition, and the container shape of the food to be estimated is obtained.

[0135] For example, in some other examples, the Euclidean distance between the food image and the edge of the standard container image of the standard container pre-stored in the database is calculated, and the container shape of the food to be estimated is obtained according to the Euclidean distance corresponding to different standard containers. For example, the standard container set in the database pre-stores multiple standard container images, and each standard container image may include images from multiple angles, such as Figure 3BThe standard container image of a hemispherical bowl as an example of a standard container includes: (1) an image perpendicular to the viewing angle of the supporting surface on which the bowl is located (front view); (2) an image at a viewing angle of 45 degrees to the supporting surface on which the bowl is located (stereoscopic image); and an image parallel to the viewing angle of the supporting surface on which the bowl is located (side view).

[0136] First, extract images with the same direction as that corresponding to the qth standard container in the standard container set from multiple food images to obtain at least one container image to be estimated corresponding to the qth standard container. For example, extract the first food image and the second food image from the multiple food images as the container images to be estimated corresponding to the hemispherical bowl.

[0137] Next, for each container image to be estimated, the food region to be estimated occupied by the food to be estimated is extracted. For example, the food region to be estimated occupied by the food to be estimated is extracted using the food region extraction method described in step S1303 for the first food image and the second food image, respectively. For example, a deep learning model algorithm, a similarity calculation method, or a deep model combined with a similarity calculation algorithm may be used, which will not be described in detail here.

[0138] Next, the Euclidean distance between the food region to be estimated in the container image to be estimated and the shape edge of the qth standard container in the corresponding direction is calculated to obtain at least one Euclidean distance value. For example, the Euclidean distance between the first food image and Figure 3B The first Euclidean distance of the shape edge of the hemispherical bowl numbered (1) in the standard container image shown is calculated by comparing the second food image with Figure 3B The second Euclidean distance of the edge of the hemispherical bowl shape numbered (3) in the standard container image shown.

[0139] Next, the Euclidean distance values ​​corresponding to all directions of the qth standard container are superimposed with the preset weight values ​​corresponding to each direction and summed to obtain the Euclidean distance sum value corresponding to the qth standard container. For example, the product of the first Euclidean distance and the first weight value is added to the product of the second Euclidean distance and the second weight value to obtain the Euclidean distance sum value corresponding to the qth hemispherical bowl.

[0140] Finally, detect the minimum value in the Euclidean distance and value set corresponding to the standard container set, and use the standard container corresponding to the minimum value as the container shape of the food to be estimated. For example, according to the above steps, calculate the Euclidean distance and value set corresponding to all standard containers in the standard container set, and detect that the Euclidean distance and value corresponding to the hemispherical bowl is the smallest, then it is considered that the container shape of the food to be estimated is a bowl.

[0141] In step S1306, the volume information of the food to be estimated is obtained according to the shape of the container of the food to be estimated and the size information of the food.

[0142] For example, if the shape of the container of the food to be estimated is a bowl or pot that can be approximated as a hemispherical shape, the volume information of the food to be estimated can be obtained according to the real radius r of the area of ​​the food to be estimated and the hemispherical calculation formula.

[0143] For example, in some examples, the shape of the container of the food to be estimated is a basin that can be approximated as a cylinder. Since the volume calculation of the cylinder requires a height value, step S130 may also include: for the second food image, extracting the food area to be estimated occupied by the food to be estimated, and determining the height value of the food to be estimated according to the food area to be estimated and the second mapping ratio. At this time, in step S1306, the volume information of the food to be estimated is obtained according to the shape of the container of the food to be estimated, the height value of the food to be estimated, and the food size information. For example, the food size information includes the real radius r of the food to be estimated. According to the volume calculation formula of the cylinder, the real radius r of the food to be estimated, and the height value of the food to be estimated, the volume of the food to be estimated can be obtained.

[0144] For example, in some examples, the height value of the food to be estimated can be determined by obtaining the number of pixels corresponding to each column of pixels in the food area to be estimated. For example, the maximum value of the number of columns of pixels is selected as the height component of the food to be estimated, and the product of the height component and the second mapping ratio is used as the height value of the food to be estimated.

[0145] For example, in some examples, since the shooting direction of the second food image is not necessarily completely parallel to the support surface of the food to be estimated, the height component obtained by the maximum value of the number of column pixels may have a large error with the actual height, so the height value of the food to be estimated can be obtained by the height component of the edges on both sides of the food area to be estimated. At this time, since the change in the angle during shooting has little effect on the length of the edges on both sides of the cylinder, the height value obtained is more accurate. For example, the number of pixels corresponding to each column of pixels is arranged in the order of acquisition to obtain a column pixel number sequence; the first P elements and the last M elements in the column pixel number sequence are extracted, and the average of the P elements and the M elements is calculated as the height component of the food to be estimated; the product of the height component and the second mapping ratio is used as the height value of the food to be estimated.

[0146] For example, in some other embodiments, an image directly above the hand in a fist state perpendicular to the direction of the supporting surface of the food to be estimated is captured, in which the hand and the food to be estimated are placed side by side. The image directly above includes a reference image and a food image. Step S130 in the food volume calculation method may include: for the reference image in the image directly above, obtaining the minimum bounding rectangle of the hand in the fist state in the reference image, detecting the length and width of the minimum bounding rectangle through a deep learning model, and then calculating the area of ​​the minimum bounding rectangle, using the square root of the area of ​​the minimum bounding rectangle as the diameter of the hand in the fist state, thereby obtaining the area of ​​the hand in the fist state according to a circular area calculation formula to obtain the reference area size information of the three-dimensional reference object; obtaining a second mapping ratio according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object. The specific process is as described in step S1302 and will not be repeated here; for the food image, the area of ​​the food to be estimated occupied by the extracted food to be estimated is a circle, and the area of ​​the food to be estimated is obtained. Calculate the pixel area of ​​the food area to obtain the food area size information, so as to obtain the food size information according to the second mapping ratio and the food area size information, that is, the real area of ​​the food to be estimated. The specific process is as described in steps S1303 and S1304, which will not be repeated here; determine the container shape of the food to be estimated through a deep learning model according to a food image, for example, obtain the container shape as a plate, a bowl, etc.; obtain the volume information of the food to be estimated according to the container shape of the food to be estimated and the food size information, for example, the shape of the container is detected to be a plate, and the real diameter of the food to be estimated is calculated according to the real area of ​​the food to be estimated, and one tenth of the real diameter of the food to be estimated is used as the height of the food to be estimated, and the volume of the food to be estimated is calculated using a cylinder calculation formula. For example, the shape of the container is detected to be a bowl. Since the shape of the bowl can be approximated as a hemisphere, the real radius of the food to be estimated can be calculated according to the real area of ​​the food to be estimated, and then the volume information of the food to be estimated can be obtained according to the hemisphere volume calculation formula.

[0147] The method for calculating the food volume using the hand as a three-dimensional reference object disclosed in the above-mentioned embodiment makes the application scenarios of the food volume calculation method provided by at least one embodiment of the present disclosure more extensive. The food volume calculation can be completed with only the hand, which is flexible to use and easy to operate. In addition, since the shape of the three-dimensional reference object is approximately spherical, the accuracy of the food volume calculation is not affected by the shooting direction, reducing the difficulty of operation.

[0148] The hand detection model involved in the embodiments of the present disclosure and the method for obtaining the fist area according to the hand fist model are described in detail below with reference to the accompanying drawings, but the embodiments of the present disclosure are not limited to the following specific methods.

[0149] If the stereo reference object image includes not only the hand in a spherical fist state but also the arm part, it is necessary to obtain the hand fist image through the hand detection model, extract the fist area in the hand fist image, and obtain the pixel area of ​​the fist area. The following takes the stereo reference object image including the hand and arm parts as an example to specifically illustrate the implementation process of the hand detection model. It should be noted that the present disclosure is not limited to hand model detection on the stereo reference object image, and the hand image can also be obtained through the hand detection model for any image to be processed including the hand image, and the present disclosure does not limit this.

[0150] At present, general images are based on RGB (red, green, and blue) color space. In RGB color space, the skin color of human images is greatly affected by brightness, so it is difficult to separate skin color points from non-skin color points. In other words, in face images processed in RGB color space, skin color points are discrete points, and there are many non-skin color points embedded in the middle, which brings difficulties to skin color area calibration (such as face calibration, eye calibration, etc.). YCrCb color space is often used in face detection, because the influence of brightness can be ignored when converting RGB color space to YCrCb color space, and because YCrCb color space is little affected by brightness, skin color will produce good clustering, so that the three-dimensional color space can be mapped to a two-dimensional CrCb plane, so that the skin color points form a certain shape, so as to achieve the purpose of recognizing human images according to skin color.

[0151] "Y" in the YCrCb color space represents brightness, that is, the grayscale value of the pixel; while "Cr" and "Cb" represent chromaticity, which is used to describe the color and saturation of the image and is used to specify the color of the pixel. Among them, "Cr" reflects the difference between the red part of the RGB input signal and the brightness value of the RGB signal, that is, the red chromaticity component of the pixel, and "Cb" reflects the difference between the blue part of the RGB input signal and the brightness value of the RGB signal, that is, the blue chromaticity component of the pixel. The RGB signal brightness value is obtained by superimposing specific parts of the RGB input signal together. In other words, the YCrCb color space is a color model that separates brightness alone. Using this color model can make the skin color points not be affected by the brightness of light and difficult to separate.

[0152] For example, when a stereoscopic reference image is processed by a hand detection model to obtain a hand clenched fist image, the stereoscopic reference image is first mapped to the YCbCr color space to obtain a mapped image; then, the mapped image is projected on the CbCr plane to obtain a skin color sample image, which includes skin color sample points corresponding to the pixels of the stereoscopic reference image; finally, the skin color sample image is traversed, and in the process of traversing the skin color sample image, if the skin color sample point is located at the elliptical boundary of the skin pixel point and within the ellipse, the pixel value of the stereoscopic reference image pixel point corresponding to the skin color sample point is marked as the second value (for example, 0); if the skin color sample point is not located at the elliptical boundary of the skin pixel point and within the ellipse, the pixel value of the stereoscopic reference image pixel point corresponding to the skin color sample point is marked as the first value (for example, 255) to obtain a mask image corresponding to the stereoscopic reference image, which is a hand clenched fist image. For example, Figure 4A Schematic diagram of a clenched fist image, where the black area is the mask image corresponding to the hand and arm in a clenched fist state.

[0153] For example, after obtaining a hand clenching fist image, extracting the fist area includes the following operations: first, taking the pixel points in the hand clenching fist image whose grayscale values ​​are the second value as the hand area, for example, taking the pixel points in the hand clenching fist image whose grayscale values ​​are 0 grayscale as the hand area, that is, Figure 4A Then, the first side edge of the hand area is detected in the column detection direction of the hand area, and the slope between each M pixel points on the first side edge of the hand area is calculated to obtain multiple slope values; finally, according to the numerical relationship between the multiple slope values, the position of the edge pixel points of the fist area is determined to obtain the fist area in the hand fist image.

[0154] For example, the column detection direction is toward the wrist direction, the first side edge is the edge where the thumb of the hand is located, and the obtained multiple slope values ​​are N in total, and the mth slope value corresponds to the pixel points Dm+1 and Dm. For example, after obtaining N slope values, the N slope values ​​are sorted according to the column detection direction. If the m+1th slope value is greater than the mth slope value for the first time, the position of the pixel point Dm+1 corresponding to the mth slope value and the m+1th slope value is the position of the edge pixel point of the fist area, so that the fist area in the hand fist image is obtained according to the position of the edge pixel point.

[0155] Figure 4B Extraction provided for at least one embodiment of the present disclosure Figure 4AThe figure shows a schematic diagram of the fist area process in the hand clenching image, wherein pixel D1, pixel D2 and pixel D3 represent three pixels to be processed in the hand clenching image, which are 10 pixels apart from each other. Straight line L1 is the line connecting pixel D1 and pixel D2, and its slope is k1. Straight line L2 is the line connecting pixel D2 and pixel D3, and its slope is k2. The shaded area in the figure represents the extracted fist area, and the hand clenching image formed by the black lines is the hand area.

[0156] like Figure 4B As shown, the column detection direction is from top to bottom, and the first side edge is the left edge of the hand area in the figure. For the left edge of the hand area, the slope between every 10 pixels on the left edge is calculated to obtain multiple slope values. The slope value between pixel D1 and pixel D2 is calculated to be k1, and the slope value between pixel D2 and pixel D3 is k2. If the multiple slope values ​​are arranged in the column detection direction, the slope value k2 is greater than the slope value k1 for the first time, then pixel D2 is the edge pixel of the fist area, and thus Figure 4B The clenched fist area is shown in middle shaded area.

[0157] For example, when the image obtained when photographing food contains multiple types of food, such as multiple containers each containing their own food, you can first use food recognition, image segmentation and other technologies to divide the areas of the multiple foods separately, and then calculate the volume and calories of each food separately.

[0158] At least one embodiment of the present disclosure also provides a method for calculating food calories, including: obtaining the type of food to be estimated; using the above embodiment to obtain the volume of the food to be estimated; determining the density of the food to be estimated according to the type of food to be estimated, and obtaining the mass of the food to be estimated based on the density of the food to be estimated and the volume of the food to be estimated; determining the caloric density of the food to be estimated according to the type of food to be estimated, and obtaining the calories of the food to be estimated based on the caloric density of the food to be estimated and the mass of the food to be estimated.

[0159] At least one embodiment of the present disclosure further provides an electronic device, Figure 5A A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0160] The electronic device includes a food volume calculation unit, which is configured to execute the food volume calculation method provided by at least one embodiment of the present disclosure to obtain volume information of the food to be estimated. The electronic device may further include a food calorie calculation unit, which obtains the type of food to be estimated, determines the density of the food to be estimated according to the type of food to be estimated, obtains the mass of the food to be estimated based on the density of the food to be estimated and the volume information of the food to be estimated, and then determines the calorie density of the food to be estimated according to the type of food to be estimated, thereby obtaining the calorie of the food to be estimated based on the calorie density of the food to be estimated and the mass of the food to be estimated. The electronic device can be implemented as a server or a terminal device in different examples.

[0161] For example, when the electronic device is implemented as a server, other terminal devices (such as mobile phones, etc.) communicate with the server, for example, through a communication network, etc., for example, provide the server with reference images and food images corresponding to the three-dimensional reference object and the food to be estimated, or receive from the server the volume information of the food to be estimated obtained by the server through the above-mentioned food volume calculation unit, or further receive from the server the calories of the food to be estimated obtained by the server through the above-mentioned food calorie calculation unit. Correspondingly, in order to communicate with other terminal devices, in addition to the above-mentioned food volume calculation unit and / or food calorie calculation unit, the server may further include an image receiving unit to obtain the reference image and food image corresponding to the three-dimensional reference object and the food to be estimated from the communicating terminal device, and further include a calculation result output unit to output the calculated volume information of the food to be estimated and / or the calories of the food to be estimated to the communicating terminal device. For example, other terminal devices use their own image acquisition units (such as cameras) to capture reference images and food images corresponding to the three-dimensional reference object and the food to be estimated, or, after obtaining the above-mentioned calculation results, other device terminals provide them to users in the form of display, voice, printing, etc., for example, through their own display screens, microphones, print output units, etc. For example, when the electronic device is implemented as a terminal, as shown in the figure, the electronic device 500 includes an image acquisition unit 501, a food volume calculation unit 502, a food calorie calculation unit 503, and an output unit 504. For example, the electronic device 500 can be a mobile terminal, and the mobile terminal can be a mobile phone, a tablet computer, a smart watch, and other devices.

[0162] The image acquisition unit 501 is configured to acquire reference images and food images corresponding to the three-dimensional reference object and the food to be estimated. For example, the image acquisition unit 501 is a camera of a mobile terminal, and the reference image and food image in the food volume calculation method can be acquired by the camera of the mobile terminal.

[0163] The food volume calculation unit 502 is configured to execute the food volume calculation method provided by at least one embodiment of the present disclosure to obtain volume information of the food to be estimated.

[0164] The food calorie calculation unit 503 is configured to: obtain the type of food to be estimated; determine the density of the food to be estimated according to the type of food to be estimated, and obtain the mass of the food to be estimated based on the density of the food to be estimated and the volume information of the food to be estimated; determine the calorie density of the food to be estimated according to the type of food to be estimated, and obtain the calorie of the food to be estimated based on the calorie density of the food to be estimated and the mass of the food to be estimated.

[0165] The output unit 504 is configured to display or output the volume information and / or calorie information of the food to be estimated. For example, the output unit can be a display screen, a microphone, a printing output unit, etc., which provides the volume information and / or calorie information of the food to be estimated to the user by display, voice or printing, or the output unit can output the volume information and / or calorie information of the food to be estimated to other terminals for display or printing.

[0166] The above-mentioned food volume calculation unit, food calorie calculation unit, image receiving unit, calculation result output unit, etc. can be implemented by hardware, software, firmware or any combination of the three.

[0167] FIG. 5B to FIG. 5F A product interface diagram is shown that adopts the food volume calculation method and the food calorie calculation method provided by at least one embodiment of the present disclosure. For example, the food volume calculation method and the food calorie calculation method provided by at least one embodiment of the present disclosure are applied to an electronic device 500. For example, the electronic device 500 is a server, and the product interface diagram is an interface diagram of a terminal device used in conjunction with the electronic device 500. For example, the terminal device is a mobile phone, a tablet computer, a smart watch, or other device.

[0168] For example, a reference image and a food image corresponding to a three-dimensional reference object and the food to be estimated are captured by a camera of a terminal device, and the terminal device sends the reference image and the food image to an image receiving unit of a server through a communication network. The server calculates the food volume and the food calories according to the received food image and reference image through a food volume calculation unit and a food calorie calculation unit, and finally sends the calculation result to the terminal device through a calculation result output unit and displays it.

[0169] Figure 5BThis is the main interface of the terminal device. As shown in the figure, by clicking the "AI Recognition" button in the middle of the menu bar, you can take reference images and food images to calculate the volume of the food to be estimated and the calories of the food to be estimated according to the aforementioned embodiment; by clicking the "My" button in the lower right corner of the menu bar, you can enter the size information of the user's hand in a clenched fist state and store the size information in a personal database.

[0170] For example, when entering the size information of the user's hand in a clenched fist, Figure 5C As shown, the user enters a photo of the hand in a clenched fist state taken in front of an A4 paper according to the prompt, and the wrist overlaps with the bottom edge of the A4 paper, thereby obtaining the actual size information of the hand in the clenched fist state.

[0171] For example, when obtaining the volume and calories of the food to be estimated, after clicking the "AI Identification" button, Figure 5D and Figure 5E As shown, according to the prompt, an image of the hand in a clenched fist state and the food to be estimated is taken to obtain a reference image and a food image, and then the terminal device uploads the reference image and the food image to the server to calculate the volume of the food to be estimated and the calories of the food to be estimated, and the volume of the food to be estimated and the calories of the food to be estimated are obtained through the calculation result output unit of the server.

[0172] After obtaining the volume of the food to be estimated and the calories of the food to be estimated, Fig. 5F As shown, the mobile terminal displays the calculation results of the server, including the name of the food to be estimated, the calories of the food to be estimated, and other information.

[0173] For example, the calculation process of the volume of the food to be estimated and the calorie of the food to be estimated can also be implemented in the mobile terminal (without the need for a server). Alternatively, a part of the calculation process is set in the mobile terminal, while another part of the calculation process is executed in the server.

[0174] At least one embodiment of the present disclosure further provides an electronic device, Figure 6 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0175] For example, Figure 6 As shown, the electronic device includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004. The processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004. The components such as the processor 1001, the communication interface 1002, and the memory 1003 can also communicate with each other through a network connection. The present disclosure does not limit the type and function of the network.

[0176] For example, in at least one example, the electronic device also includes an image capturing device, such as at least one camera, so as to acquire various required images.

[0177] For example, the memory 1003 stores computer executable instructions non-transiently. When the processor 1001 is used to execute the computer executable instructions, the computer executable instructions are executed by the processor 1001 to implement the food volume calculation method according to any of the above embodiments. The specific implementation of each step of the food volume calculation method and related explanations can be found in the above-mentioned embodiments of the food volume calculation method, which will not be repeated here.

[0178] For example, the processor 1001 executes the program stored in the memory 1003 to implement the food volume calculation method, which is the same as the implementation method mentioned in the embodiment part of the aforementioned food volume calculation method, and will not be repeated here.

[0179] For example, the communication bus 1004 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industrial Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0180] For example, the communication interface 1002 is used to implement communication between the electronic device and other devices.

[0181] For example, the processor 1001 and the memory 1003 may be arranged on a server side (or a cloud side).

[0182] For example, the processor 1001 can control other components in the electronic device to perform the desired functions. The processor 1001 can be a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) can be an X86 or ARM architecture, etc.

[0183] For example, the memory 1003 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer executable instructions may be stored on the computer-readable storage medium, and the processor 1001 may run the computer executable instructions to implement various functions of the electronic device. Various applications and various data may also be stored in the storage medium.

[0184] For example, the electronic device can be a server, which, through data interaction with the terminal device, sends the food volume obtained by the food volume calculation method provided by at least one embodiment of the present disclosure and the food calorie obtained by the food calorie calculation method provided by at least one embodiment of the present disclosure to the terminal device for display.

[0185] For example, the server receives information such as pictures sent by the terminal device, performs a food volume calculation process, and sends the calculation result to the terminal device for display.

[0186] For example, for a detailed description of the process of an electronic device executing food volume calculation, reference may be made to the relevant description in the embodiment of the food volume calculation method, and repeated parts will not be repeated.

[0187] Figure 7 A schematic diagram of a computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 7 As shown, one or more computer executable instructions 1101 may be non-transitorily stored on the storage medium 1100. For example, when the computer executable instructions 1101 are executed by a processor, one or more steps in the food volume calculation method described above may be performed.

[0188] For example, the storage medium 1100 may be applied to the above-mentioned electronic device and / or the food volume calculation device 1400. For example, the storage medium 1100 may include the memory 1003 in the electronic device.

[0189] For example, the description of the storage medium 1100 may refer to the description of the memory in the embodiment of the electronic device, and the repeated parts will not be repeated.

[0190] Figure 8A schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure is shown. The electronic device provided by the present disclosure can be applied in an Internet system.

[0191] use Figure 8 The computer system provided in can implement the functions of the food volume calculation device and / or electronic device involved in the present disclosure. Such computer systems may include personal computers, laptops, tablets, mobile phones, personal digital assistants, smart glasses, smart watches, smart rings, smart helmets and any smart portable devices or wearable devices. The specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. This computer device can be a general-purpose computer device or a computer device with a specific purpose. Both computer devices can be used to implement the food volume calculation device and / or electronic device in this embodiment. The computer system may include any components that implement the information required to implement the food volume calculation currently described. For example, the computer system can be implemented by a computer device through its hardware devices, software programs, firmware and combinations thereof. For convenience, Figure 8 Only one computer device is drawn in the figure, but the relevant computer functions of the information required to realize the food volume calculation described in this embodiment can be implemented in a distributed manner by a group of similar platforms to disperse the processing load of the computer system.

[0192] like Figure 8 As shown, the computer system may include a communication port 250, connected to which is a network for realizing data communication. For example, the computer system can send and receive information and data through the communication port 250, that is, the communication port 250 can realize wireless or wired communication between the computer system and other electronic devices to exchange data. The computer system may also include a processor group 220 (i.e., the processor described above) for executing program instructions. The processor group 220 may be composed of at least one processor (e.g., a CPU). The computer system may include an internal communication bus 210. The computer system may include different forms of program storage units and data storage units (i.e., the memory or storage medium described above), such as a hard disk 270, a read-only memory (ROM) 230, and a random access memory (RAM) 240, which can be used to store various data files used for computer processing and / or communication, and possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260, which is used to realize input / output data flow between the computer system and other components (e.g., a user interface 280, etc.).

[0193] Typically, the following devices can be connected to the input / output component 260: input devices including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and a communication interface.

[0194] Although Figure 8 A computer system having various devices is shown, but it should be understood that the computer system is not required to have all of the devices shown, and the computer system may alternatively have more or fewer devices.

[0195] There are a few points to note about this disclosure:

[0196] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to the general design.

[0197] (2) For the sake of clarity, the thickness and size of layers or structures are exaggerated in the drawings used to describe the embodiments of the present invention. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element, or there may be intervening elements.

[0198] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0199] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A method for calculating food volume, comprising: Obtaining actual size information of a three-dimensional reference object; Acquire a reference image and a food image corresponding to the three-dimensional reference object and the food to be estimated; Obtaining volume information of the food to be estimated according to actual size information of the three-dimensional reference object, reference area size information obtained based on the reference image, and food area size information obtained based on the food image; The step of acquiring the reference image and the food image corresponding to the three-dimensional reference object and the food to be estimated includes: Acquire a reference image and a food image obtained by the same shooting method of the three-dimensional reference object and the food to be estimated, wherein the volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the one reference image, and the food area size information obtained based on the one food image; or Acquire a plurality of reference images and a plurality of food images obtained in pairs of the three-dimensional reference object and the food to be estimated in a plurality of shooting modes, wherein the volume information of the food to be estimated is obtained according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the plurality of reference images, and the food area size information obtained based on the plurality of food images; Wherein, the same shooting mode or each of the multiple shooting modes at least includes a shooting distance and a shooting direction; The multiple shooting modes include a first shooting mode and a second shooting mode, and a shooting direction of the first shooting mode and a shooting direction of the second shooting mode intersect each other; The multiple reference images include a first reference image and a second reference image acquired using the first shooting mode and the second shooting mode respectively; The plurality of food images include a first food image and a second food image acquired using the first photographing method and the second photographing method, respectively.

2. The method according to claim 1, wherein: Each of the plurality of shooting modes further includes placing the three-dimensional reference object or the supporting surface of the food to be estimated as the object to be shot, the shooting direction of the first shooting mode and the shooting direction of the second shooting mode are perpendicular to each other, and The shooting direction of the first shooting mode is perpendicular to the support surface of the first shooting mode, and the shooting direction of the second shooting mode is parallel to the support surface of the second shooting mode.

3. The method according to claim 1 or 2, wherein: The reference image and the food image acquired in pairs in the same shooting mode are a pair of images corresponding to each other, or The reference image and the food image, which are obtained in pairs in the same shooting mode, are components of the same image.

4. The method according to claim 1 or 2, wherein: The obtaining of actual size information of the three-dimensional reference object includes: Acquire a plurality of stereoscopic reference object images in which the stereoscopic reference object and a first reference object of obtainable reference size are placed side by side, The actual size information of the 3D reference object is acquired according to the multiple 3D reference object images and the reference size.

5. The method according to claim 4, wherein: The obtaining actual size information of the 3D reference object according to the plurality of 3D reference object images and the reference size includes: For each of the stereo reference images, obtaining a first number of pixels corresponding to the reference size in the stereo reference object image, and obtaining a second number of pixels in at least one dimension of the stereo reference object in the stereo reference object image, obtaining a calculated size of the stereoscopic reference object in the stereoscopic reference object image in the at least one dimension according to the first number of pixels and the second number of pixels, The actual size information of the stereoscopic reference object is obtained based on the plurality of calculated sizes obtained from the plurality of stereoscopic reference object images.

6. The method according to claim 1 or 2, wherein: The obtaining of actual size information of the three-dimensional reference object includes: Acquire a 3D reference object image in which the 3D reference object is placed on one side of a second reference object whose reference area can be calculated; The actual size information of the three-dimensional reference object is acquired according to the three-dimensional reference object image and the reference area, wherein the three-dimensional reference object image includes information in the second reference object used to calculate the reference area.

7. The method according to claim 6, wherein: The acquiring the actual size information of the three-dimensional reference object according to the three-dimensional reference object image and the reference area includes: Acquire a pixel area of ​​the second reference object in the stereoscopic reference object image; Obtaining a first mapping ratio based on a pixel area of ​​the second reference object and the reference area; Acquire a pixel area of ​​a region occupied by the stereoscopic reference object in the stereoscopic reference object image; The volume of the 3D reference object is acquired according to the first mapping ratio and the pixel area of ​​the region occupied by the 3D reference object to obtain actual size information of the 3D reference object.

8. The method according to claim 7, wherein: The pixel values ​​of the pixels corresponding to the second reference object in the stereoscopic reference object image are within a first value range, The step of obtaining a pixel area of ​​a region occupied by a 3D reference object in the 3D reference object image comprises: Taking the pixel values ​​of the pixels in the 3D reference object image that do not satisfy the first value range as the area occupied by the 3D reference object, so as to obtain the pixel area of ​​the area occupied by the 3D reference object; or The area occupied by the three-dimensional reference object is extracted according to the deep learning model, and the pixel area of ​​the area occupied by the three-dimensional reference object is obtained.

9. The method according to claim 7, wherein: The acquiring the pixel area of ​​the second reference object in the three-dimensional reference object image comprises: The minimum bounding rectangle of the second reference object in the stereoscopic reference object image and the pixel area of ​​the minimum bounding rectangle are obtained, and the pixel area of ​​the minimum bounding rectangle is used as the pixel area of ​​the second reference object, wherein the pixel value of the pixel point corresponding to the second reference object in the stereoscopic reference object image is within a first value range, and the minimum bounding rectangle of the second reference object is obtained according to the position of the pixel point whose pixel value is within the first value range, and, The step of obtaining a pixel area of ​​a region occupied by a 3D reference object in the 3D reference object image comprises: The pixel values ​​of the pixels in the minimum circumscribed rectangle that do not satisfy the first value range are taken as the area occupied by the three-dimensional reference object to obtain the pixel area of ​​the area occupied by the three-dimensional reference object.

10. The method according to claim 6, wherein: The second reference object is a single-color paper having a standard size.

11. The method according to claim 7, wherein: The three-dimensional reference object includes a hand in a clenched fist and is processed in a spherical manner. The step of obtaining a pixel area of ​​a region occupied by a 3D reference object in the 3D reference object image comprises: In response to the wrist portion of the hand in the stereoscopic reference object image not being against the edge of the second reference object, acquiring a fist image of the hand according to the hand detection model, extracting a fist area in the fist image of the hand, and acquiring a pixel area of ​​the fist area as the pixel area of ​​the area occupied by the stereoscopic reference object; and In response to the wrist portion of the hand in the stereoscopic reference object image resting against the edge of the second reference object, the pixel values ​​of the pixels in the stereoscopic reference object image that do not satisfy the first value range are taken as the fist area, and the pixel area of ​​the fist area is obtained as the pixel area of ​​the area occupied by the stereoscopic reference object.

12. The method according to claim 1 or 2, wherein: The step of acquiring a plurality of reference images and a plurality of food images obtained in pairs corresponding to the stereo reference object and the food to be estimated in a plurality of shooting modes comprises: Continuously photographing a plurality of images of the food to be estimated and the three-dimensional reference object simultaneously located on the same support surface, extracting a plurality of key images from the plurality of images to obtain the plurality of reference images and the plurality of food images, The key image includes a corresponding pair of reference images and a food image.

13. The method according to claim 12, wherein: The step of continuously shooting a plurality of images of the food to be estimated and the three-dimensional reference object being located on the same support surface at the same time, and extracting a plurality of key images from the plurality of images comprises: For each of the multiple images, extract the food region to be estimated occupied by the food to be estimated, and obtain the pixel area of ​​the food region to be estimated corresponding to each image; According to the pixel area of ​​the food area to be estimated corresponding to each image, an image whose relative change of the pixel area of ​​the food area to be estimated is greater than a preset threshold is extracted from the multiple images as one of the multiple key images, The multiple key images include at least a first key image and a second key image, the first key image includes the first food image and the first reference image, and the second key image includes the second food image and the second reference image.

14. The method according to claim 1, wherein: The obtaining the volume information of the food to be estimated according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the multiple reference images, and the food area size information obtained based on the multiple food images includes: Acquire the pixel area of ​​the region occupied by the three-dimensional reference object of each reference image in the plurality of reference images to obtain reference region size information of the three-dimensional reference object; Obtaining a second mapping ratio according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; For the first food image, extract the food area to be estimated occupied by the food to be estimated, and obtain the size information of the food area; Obtaining food size information according to the food area size information and the second mapping ratio; For the multiple food images, extract the food area to be estimated occupied by the food to be estimated, and determine the container shape of the food to be estimated according to the food area to be estimated corresponding to each of the multiple food images; The volume information of the food to be estimated is obtained according to the shape of the container of the food to be estimated and the food size information.

15. The method according to claim 14, wherein: The step of obtaining the volume information of the food to be estimated according to the actual size information of the three-dimensional reference object, the reference area size information and the food area size information further includes: For the second food image, extract the food area to be estimated occupied by the food to be estimated, and determine the height value of the food to be estimated according to the food area to be estimated and the second mapping ratio; and The volume information of the food to be estimated is obtained according to the shape of the container of the food to be estimated, the height value of the food to be estimated and the food size information.

16. The method according to claim 15, wherein: The step of determining the height value of the food to be estimated according to the food area to be estimated and the second mapping ratio includes: Obtain the number of pixels corresponding to each column of pixel points in the food area to be estimated; Determining the height component of the food to be estimated according to the number of pixels corresponding to each column of pixel points; The height value of the food to be estimated is determined according to the height component and the second mapping ratio.

17. The method according to claim 14, wherein: The acquiring the pixel area of ​​the region occupied by the three-dimensional reference object of each reference image in the plurality of reference images to obtain the reference region size information of the three-dimensional reference object includes: Extracting the area occupied by the three-dimensional reference object for each reference image, and obtaining the pixel area of ​​the area occupied by the three-dimensional reference object corresponding to each reference image; An average value of pixel areas of all acquired regions occupied by the three-dimensional reference objects is calculated to obtain reference region size information of the three-dimensional reference objects.

18. The method according to claim 14, wherein: The three-dimensional reference object includes a hand in a clenched fist state, The acquiring the pixel area of ​​the region occupied by the stereoscopic reference object of each reference image in the plurality of reference images to obtain the reference region size information of the stereoscopic reference object includes: For each of the reference images, a hand clenched fist image is obtained according to the hand detection model, a fist clenched fist area in the hand clenched fist image is extracted, and a pixel area of ​​the fist clenched fist area is obtained; The average value of the pixel areas of all acquired fist regions is calculated to obtain the reference region size information of the three-dimensional reference object.

19. The method according to claim 14, wherein: The step of extracting a food region to be estimated occupied by the food to be estimated from the plurality of food images, and determining a container shape of the food to be estimated according to the food region to be estimated corresponding to each of the plurality of food images, comprises: Extracting images in the same direction as the qth standard container in the standard container set from the multiple food images to obtain at least one container image to be estimated corresponding to the qth standard container, and extracting a food area to be estimated occupied by the food to be estimated for each container image to be estimated; respectively calculating the Euclidean distance between the food region to be estimated of the container image to be estimated and the shape edge of the qth standard container in the corresponding direction to obtain at least one Euclidean distance value; The Euclidean distance values ​​corresponding to all directions of the qth standard container are superimposed on the preset weight value corresponding to each direction and summed to obtain the Euclidean distance sum value corresponding to the qth standard container; Detecting the minimum value in the Euclidean distance and value set corresponding to the standard container set, and taking the standard container corresponding to the minimum value as the container shape of the food to be estimated, Wherein, q is a positive integer.

20. The method according to claim 1, wherein: The food to be estimated is placed in a container, and a known shape of the container of the food to be estimated is obtained or the shape of the container of the food to be estimated is determined according to the food image. The obtaining the volume information of the food to be estimated according to the actual size information of the three-dimensional reference object, the reference area size information obtained based on the one reference image, and the food area size information obtained based on the one food image comprises: Acquire the pixel area of ​​the region occupied by the three-dimensional reference object of the reference image to obtain reference region size information of the three-dimensional reference object; Obtaining a second mapping ratio according to the reference area size information of the three-dimensional reference object and the actual size information of the three-dimensional reference object; For the food image, extract the food area to be estimated occupied by the food to be estimated, and obtain the size information of the food area; Obtaining food size information according to the food area size information and the second mapping ratio; The volume information of the food to be estimated is obtained according to the shape of the container and the size information of the food.

21. The method according to claim 14 or 20, wherein: The step of determining the shape of the container of the food to be estimated comprises: Acquire a standard container recognition model, and determine the container shape of the food to be estimated according to the multiple food images and the standard container recognition model; or A standard container recognition model is obtained, and a container shape of the food to be estimated is determined according to the food image and the standard container recognition model.

22. The method according to any one of claims 14 to 20, wherein: For the image to be processed including the food to be estimated, extracting the food area to be estimated occupied by the food to be estimated includes: Dividing the image to be processed into a plurality of region sets, and performing similarity calculation on each of the region sets to obtain a similarity value of each of the region sets; Merging a set of regions whose similarity values ​​meet a preset threshold to obtain a plurality of merged regions; taking the closed area with the largest pixel area in the merged area as the food area to be estimated in the image to be processed; The similarity calculation includes one or more combinations of color similarity calculation, texture similarity calculation, size similarity calculation, and overlap similarity calculation.

23. The method according to any one of claims 14 to 20, wherein: For the image to be processed including the food to be estimated, extracting the food area to be estimated occupied by the food to be estimated includes: Perform edge recognition on the image to be processed based on a deep convolutional network model to obtain a food region to be estimated of the food to be estimated in the image to be processed.

24. A method for calculating food calories, comprising: Get the type of food to be estimated; Obtaining the volume of the food to be estimated according to the food volume calculation method according to any one of claims 1 to 23; Determining the density of the food to be estimated according to the type of the food to be estimated, and obtaining the mass of the food to be estimated based on the density of the food to be estimated and the volume of the food to be estimated; The calorie density of the food to be estimated is determined according to the type of the food to be estimated, and the calorie of the food to be estimated is obtained based on the calorie density of the food to be estimated and the mass of the food to be estimated.

25. An electronic device comprising: A food volume calculation unit, wherein the food volume calculation unit is configured to execute the food volume calculation method according to any one of claims 1 to 23 to obtain the volume information of the food to be estimated.

26. The electronic device according to claim 25, further comprising: Food calorie calculation unit, configured as: Obtaining the type of food to be estimated; Determining the density of the food to be estimated according to the type of the food to be estimated, and obtaining the mass of the food to be estimated based on the density of the food to be estimated and the volume information of the food to be estimated; The calorie density of the food to be estimated is determined according to the type of the food to be estimated, and the calorie of the food to be estimated is obtained based on the calorie density of the food to be estimated and the mass of the food to be estimated.

27. The electronic device according to claim 26, further comprising: An image acquisition unit, configured to acquire a reference image and a food image corresponding to the three-dimensional reference object and the food to be estimated; An output unit is configured to display or output the volume information and / or calorie information of the food to be estimated.

28. An electronic device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the food volume calculation method according to any one of claims 1 to 23 or the food calorie calculation method according to claim 24 is implemented.

29. A storage medium, wherein: The storage medium stores computer executable instructions, which, when executed by a processor, implement the food volume calculation method according to any one of claims 1 to 23 or the food calorie calculation method according to claim 24.

Images