Food nutrition detection method, electronic equipment and storage medium

Abstract

The invention discloses a food nutrition detection method, electronic equipment and a storage medium, and relates to the technical field of electronics, the method is applied to the electronic equipment, and the electronic equipment obtains a binocular image of food; image recognition and depth estimation are performed on the binocular image to obtain the types of the food materials in the food and the volumes of the various food materials, and then the total caloric value / nutritional ingredients of the food are determined according to the volumes of the various food materials and the preset nutritional ingredient information, so that the total caloric value of the food can be provided for a user, and the user experience is improved. Besides, when depth estimation is carried out through the binocular image, the depth values of the plurality of pixel points are determined firstly, and the depth values of other pixel points are estimated based on the depth values of the plurality of pixel points, so that the depth values of other pixel points can be obtained more accurately, the volume of the food can be determined more accurately, and the user experience is improved. The nutritional ingredients and the total calorie in the food can be accurately determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronics, and particularly relates to a food nutrition detection method, an electronic device and a storage medium. BACKGROUND

[0002] In recent years, users pay more and more attention to their own health, and many users have the needs of weight control, nutrition control, diet tracking and the like. These users pay much attention to the nutritional ingredients and the content of calories in the food consumed every day. For example, a user who wants to maintain a healthy weight will pay attention to the content of calories in the food, and a user who wants to control sugar will pay attention to the content of carbohydrates in the food. If an application program on a mobile phone can identify the food eaten by the user every day and can give the nutritional ingredients in the diet, the needs of the user can be better met.

[0003] At present, the method mainly captures a single food image through a camera, identifies the type of the food through the food image, and then determines the nutritional ingredients of the food. However, the result given by this method is the nutritional ingredients under a unit content, such as how many calories are contained in 100 grams of food. When the user wants to know the total calories of the food, the weight of the food needs to be manually weighed and the weight needs to be manually input or the total calories needs to be calculated according to the weight by the user, which is relatively cumbersome and is not very convenient for the user. SUMMARY

[0004] The present application provides a food nutrition detection method, an electronic device and a storage medium, which can more conveniently and accurately determine the total calories and other nutritional ingredient information in the food. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a food nutrition detection method, applied to an electronic device. The electronic device can detect a first operation performed by a user, the first operation indicating the electronic device to perform food nutrition detection. In response to the first operation, the electronic device acquires a first image through a first camera and acquires a second image through a second camera. The first image and the second image include the same food, and the food includes at least one food material. The electronic device performs image recognition on the first image and / or the second image to determine the category of each food material in the food and the pixel points of each category of food material in the first image and / or the second image. The electronic device fuses features of the first image and the second image, divides a feature map obtained by the feature fusion into a plurality of grid units, determines the depth values of a plurality of first pixel points corresponding to the plurality of grid units, wherein one grid corresponds to one first pixel point, and then determines the depth values of a plurality of second pixel points other than the plurality of first pixel points in the plurality of grid units according to the depth values of the plurality of first pixel points, thereby obtaining the depth values of each pixel point in each category of food material. The electronic device further determines the volume of each category of food material according to the depth values of each pixel point in each category of food material. The electronic device determines the total calories and / or other nutritional component information of the food according to the category of each food material, the volume of each category of food material, and preset nutritional component information. The electronic device displays a first interface, and the first interface includes the total calories and / or other nutritional component information of the food.

[0006] Based on the above technical solution, in response to the first operation, the electronic device can obtain binocular images (i.e., the first image and the second image) of the food through one shooting. Through image recognition and depth estimation on the binocular images, the depth values of each pixel point in each category of food material can be obtained, so that the volume of each category of food material can be obtained according to the depth values of each pixel point, and then the total calorie value and / or other nutritional components of the food can be determined. Thus, without the user weighing by himself, the total calorie and / or other nutritional component information of the food can be output to the user, and the user experience can be improved.

[0007] In addition, the method provided by the embodiment of the present application first determines the depth values of a plurality of first pixel points corresponding to a plurality of grid units through grid division. Since the depth estimation is based on binocular images, the absolute depth values of the plurality of first pixel points can be accurately obtained. After the plurality of grid units are divided, the depth values of the second pixel points can be estimated based on the depth values of the plurality of first pixel points, that is, the depth values of the second pixel points are determined by comprehensively considering the depth values of the plurality of first pixel points. Compared with directly estimating the depth values of the second pixel points, the depth values of the second pixel points calculated based on the depth values of the first pixel points in the grid units are more accurate, and the volume of each category of food material obtained according to the depth values is more accurate, and thus the total content of the total calories and other nutritional component information in the food can be more accurately determined.

[0008] In some implementations of the first aspect, in combination with the first aspect, the determining the depth values of the plurality of second pixel points in the plurality of grid cells other than the plurality of first pixel points according to the depth values of the plurality of first pixel points comprises: for any one of the plurality of grid cells, determining the depth values of the plurality of second pixel points in the any one of the plurality of grid cells other than the first pixel points according to the depth values of the first pixel points in the any one of the plurality of grid cells and the depth values of the first pixel points in other grid cells around the any one of the plurality of grid cells. In this way, the depth values of the plurality of second pixel points in the any one of the plurality of grid cells are determined in combination with the depth values of the first pixel points in other grid cells around the any one of the plurality of grid cells, so that when the depth value of the first pixel point in the any one of the plurality of grid cells is erroneous due to an unexpected situation, the influence of the erroneous depth value of the first pixel point in the any one of the plurality of grid cells can be reduced.

[0009] In some implementations of the first aspect, in combination with the first aspect and the above implementations, the determining the depth values of the plurality of second pixel points in the any one of the plurality of grid cells other than the first pixel points according to the depth values of the first pixel points in the any one of the plurality of grid cells and the depth values of the first pixel points in other grid cells around the any one of the plurality of grid cells comprises: determining a weighting coefficient corresponding to each of the first pixel points in the any one of the plurality of grid cells and the other grid cells; and determining the depth values of the second pixel points in the any one of the plurality of grid cells other than the first pixel points according to the depth values of the first pixel points in the any one of the plurality of grid cells and the depth values of the first pixel points in the other grid cells, and the weighting coefficient corresponding to each of the first pixel points. In this way, the depth values of the second pixel points are more accurate by adjusting the weighting coefficient.

[0010] In some implementations of the first aspect, in combination with the first aspect and the above implementations, the first pixel point can be a center point of the grid cell, so that the first pixel point can be determined more conveniently and the calculation speed can be accelerated.

[0011] In some implementations of the first aspect, in combination with the first aspect and the above implementations, the determining the volume of each type of food material according to the depth values of each pixel point in each type of food material comprises: determining three-dimensional coordinates of each pixel point in each type of food material in a three-dimensional space according to the depth values of each pixel point in each type of food material; generating three-dimensional point cloud data corresponding to each type of food material according to the three-dimensional coordinates of each pixel point in each type of food material in the three-dimensional space; and determining the volume of each type of food material according to the three-dimensional point cloud data corresponding to each type of food material. In this way, the volume of each type of food material can be accurately estimated by triangular facet three-dimensional reconstruction.

[0012] In some implementations of the first aspect and the above-mentioned implementations, after displaying the first interface, the method further includes: detecting a second operation on the first interface, the second operation being an operation of the user viewing the heat of the single type of food material; and in response to the second operation, displaying a second interface, the second interface including the total heat of the single type of food material and other nutritional component information of the single type of food material. In this way, after the food nutrition detection is completed, the user can not only obtain the total heat of the food in a timely manner, but also view the heat of each food material, so that the food material can be adjusted in a targeted manner when the total heat of the food is high or low, and the user experience can be improved.

[0013] In some implementations of the first aspect and the above-mentioned implementations, the method further includes: detecting a third operation on the first interface, the third operation indicating that the electronic device saves the food nutrition detection result; and in response to the third operation, saving the food nutrition detection result. In this way, the user can choose to save the nutrition detection result for subsequent viewing, and the user can determine the nutritional components and heat that he or she has ingested according to the calorie recognition results of multiple times, so as to adjust his or her diet in a timely manner.

[0014] In some implementations of the first aspect and the above-mentioned implementations, the method further includes: sending the locally saved food nutrition detection result to the cloud; receiving a diet suggestion sent by the cloud; and outputting prompt information to the user, the prompt information including the diet suggestion. In this way, personalized diet suggestions and nutrition guidance can be provided for the user according to the user's diet records, so that the user can better understand his or her own eating habits and help the user better achieve their health goals.

[0015] In a second aspect, the embodiments of the present application provide an electronic device, including: one or more processors; one or more memories; the memory stores one or more programs, when the one or more programs are executed by the processor, the electronic device executes the method of the first aspect.

[0016] In a third aspect, the embodiments of the present application provide a device, which is included in an electronic device, and the device has a function of realizing the behavior of the electronic device in the above aspects and possible implementation manners of the above aspects. The function can be realized by hardware, or realized by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0017] In a fourth aspect, the embodiments of the present application provide a computer-readable storage medium, the computer-readable storage medium stores instructions, when the instructions are run on a computer, the computer executes the method of the first aspect.

[0018] In a fifth aspect, the embodiments of the present application provide a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the method of the first aspect.

[0019] The technical effects obtained by the second aspect, the third aspect, the fourth aspect and the fifth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and thus are not described herein. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figures 1A-1D A set of user interface schematic diagrams provided by the embodiments of the present application;

[0021] Figures 2A-2C Another set of user interface schematic diagrams provided by the embodiments of the present application;

[0022] Figure 3 A flowchart of a food nutrition detection method provided by the embodiments of the present application;

[0023] Figure 4 A flowchart of image recognition and segmentation provided by the embodiments of the present application;

[0024] Figure 5 A structure diagram of a segmentation model provided by the embodiments of the present application;

[0025] Figure 6 A result diagram of image recognition and segmentation provided by the embodiments of the present application;

[0026] Figure 7 A structure diagram of an interaction model provided by the embodiments of the present application;

[0027] Figure 8 A schematic diagram of binocular recognition principle provided by the embodiments of the present application;

[0028] Figure 9 A structure diagram of a binocular recognition model provided by the embodiments of the present application;

[0029] Figure 10 A principle diagram of triangulation provided by the embodiments of the present application;

[0030] Figure 11 Another flowchart of a food nutrition detection method provided by the embodiments of the present application;

[0031] Figure 12 A hardware structure diagram of an electronic device provided by the embodiments of the present application;

[0032] Figure 13 A software system structure diagram of an electronic device provided by the embodiments of the present application;

[0033] Figure 14 A structural schematic diagram of an apparatus provided for an embodiment of the present application;

[0034] Figure 15 A structural schematic diagram of a chip provided for an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings. Hereinafter, the terms "first" and "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0036] In recent years, users pay more and more attention to their own health, and many users have needs such as weight control, nutrition control, diet tracking, etc. These users pay great attention to the nutritional ingredients and the content of calories in the food they eat every day. For example, users who want to maintain a healthy weight will pay attention to the calorie content in food; users who want to control sugar will pay attention to the content of carbohydrates in food. If the application program on the mobile phone can identify the food eaten by the user every day and give the nutritional ingredients in the diet, the needs of the user can be better met.

[0037] The current method mainly captures a single food image through a camera, identifies the type of food through the food image, and then determines the nutritional ingredients of the food of this type. However, no matter whether the food is a single food material (such as an apple) or multiple food materials (such as green pepper and shredded potato), the result given by this method is the calorie / nutritional ingredient of the food under unit content, such as how many calories are contained in 100 grams of food. When the user wants to know the total calorie of the food, the weight of the food needs to be weighed manually and the weight needs to be input manually or the total calorie needs to be calculated by the user according to the weight. The operation is relatively cumbersome and not very convenient for the user.

[0038] In addition, there is another method of obtaining food images and depth images before and after the user eats, obtaining the volume of the food through the food images and the depth images, determining the weight of the food according to the volume of the food, and then determining the nutritional ingredients of the food. However, this method requires the user to take multiple photos of the food, which is not very convenient for the user. In addition, this method uses monocular relative depth images when calculating the volume of the food, and cannot directly obtain the absolute depth information of the food. It also needs to combine a regression model for depth estimation, but the depth estimated by the regression model is not accurate enough, and the volume estimated by the regression model is not accurate enough.

[0039] Based on this, the embodiment of the present application provides a food nutrition detection method. When a user needs to determine the calories in food, the user can perform a first operation to instruct the electronic device to perform food nutrition detection. In response to the first operation, the electronic device obtains a first image through a first camera and a second image through a second camera, that is, obtains a binocular image. The first image and the second image include the same food, and the food includes at least one food material. Then, the first image and / or the second image are image-recognized by an image recognition model to determine the category of each food material in the food and the pixel points of each food material in the first image and / or the second image. The first image and the second image are feature-fused by a binocular absolute depth estimation network, and the feature map obtained by feature fusion is divided into a plurality of grid units to determine the depth values of a plurality of first pixel points corresponding to the plurality of grid units, wherein one grid corresponds to one first pixel point. Then, the electronic device determines the depth values of a plurality of second pixel points in the plurality of grid units except the plurality of first pixel points according to the depth values of the plurality of first pixel points, thereby obtaining the depth values of each pixel point in each food material. Further, the volume of each food material is determined according to the depth values of each pixel point in each food material. The total calories and / or other nutritional component information of the food are determined according to the category of each food material, the volume of each food material, and the preset nutritional component information. A first interface is displayed, and the first interface includes the total calories and / or other nutritional component information of the food.

[0040] Based on the above technical solution, in response to the first operation, the electronic device can obtain a binocular image (i.e., a first image and a second image) of the food through one shooting. Through image recognition and depth estimation on the binocular image, the depth values of each pixel point in each food material can be obtained, so that the volume of each food material can be obtained according to the depth values of each pixel point, and then the total calorie value and / or other nutritional components of the food can be determined. Thus, without the user weighing by himself / herself, the total calorie and / or other nutritional component information of the food can be output to the user, and the user experience can be improved.

[0041] In addition, the method provided by the embodiment of the present application first determines the depth values of a plurality of first pixel points corresponding to a plurality of grid units through grid division. Since the depth values of the plurality of first pixel points are accurately obtained based on the binocular image, the depth values of the second pixel points can be estimated based on the depth values of the plurality of first pixel points after the plurality of grid units are divided. That is, the depth values of the second pixel points are determined by comprehensively considering the depth values of the plurality of first pixel points. Compared with directly estimating the depth values of the second pixel points, the depth values of the second pixel points calculated based on the depth values of the first pixel points in the grid units are more accurate, and the volume of each food material obtained according to the depth values is more accurate, and thus the total content of the total calories and other nutritional component information in the food can be more accurately determined.

[0042] The method provided in the embodiment of the present application can be applied to an electronic device 100. The electronic device 100 may be, but is not limited to, a mobile phone, a tablet computer, a handheld computer, a laptop computer, an in-vehicle device, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR), a virtual reality (VR) device, etc. The present application does not limit the type of electronic device.

[0043] Taking a mobile phone as an example, the food nutrition detection (also known as calorie recognition) function provided in the embodiment of the present application is first described.

[0044] In one implementation, the user can enter the calorie recognition function interface from the camera, such as Figure 1A As shown in (a) of FIG, a camera application icon 101 is displayed on the desktop (user interface 100) of the electronic device. The camera application icon 101 can be used to trigger the launch of a camera application program (i.e., camera application). The electronic device can detect a user operation (e.g., click, voice control, gesture, etc.) on the camera application icon 101. In response to the user operation, the electronic device can display the following information: Figure 1A The user interface 200 shown in (b) is a default shooting mode shooting interface of the camera application.

[0045] like Figure 1A As shown in (b) of FIG, user interface 200 may include a preview window, camera mode options, and multiple function controls. The multiple function controls may include an album shortcut control, a shutter control, a camera flip control, and a function control 201 in the upper left corner of user interface 200. Function control 201 is used to activate the smart vision function of the electronic device.

[0046] The electronic device can detect a user operation (such as a click) on the function control 201, and in response to the user operation, the electronic device can display the following information: Figure 1B The user interface 300 shown in (a) in FIG. The user interface 300 is a shooting interface of the default recognition mode of the smart vision function.

[0047] like Figure 1BAs shown in (a) of FIG. 3, the user interface 300 can include a preview window, a photo album shortcut control, and an identification mode option 301, which can display one or more identification mode options. The one or more identification mode options can include an object identification mode option, a shopping mode option, a translation mode option, a code scanning mode option, a document scanning mode option, an ID card mode option, and the like.

[0048] The user can slide the identification mode option 301 left or right or click one of the identification mode options to select a desired identification mode. For example, the user can slide the identification mode option 301 right, and in response to the user operation, the electronic device can display a user interface 400 as shown in (b) of FIG. 3. Figure 1B As shown in (b) of FIG. 3, the user interface 400 can display a calorie mode option 3011, a text identification mode option, and an object identification mode option for the identification mode option 301. At this time, the cursor is under the calorie mode option 3011, indicating that the user has selected the calorie mode, and the user interface 400 is a shooting interface in the calorie identification mode.

[0049] As shown in (a) of FIG. 4, the user interface 400 can also include a preview window 401. The preview window 401 displays a prompt message, for example, "Please aim at the food to obtain calorie information". At this interface, the user can perform calorie identification on the food. Figure 1B

[0050] Specifically, the user can aim the camera at the food, as shown in (a) of FIG. 4. The preview window 401 can display the food to be identified 402, which includes a plurality of different types of food materials. For example, the food to be identified 402 can be a light meal, including bread, carrots, nuts, and cheese. The electronic device can perform nutrition detection on the food to be identified 402 according to the method provided in the embodiments of the present application, identify the type of each food material, the calorie / nutrient component corresponding to each food material, and the total calorie / nutrient component of all food materials. Then, the electronic device can display a user interface 500 as shown in (b) of FIG. 4, which can display the food nutrition detection result of the food to be identified 402. Figure 1C Figure 1C

[0051] As shown in (a) of FIG. 4, the user interface 400 can also include a preview window 401. The preview window 401 displays a prompt message, for example, "Please aim at the food to obtain calorie information". At this interface, the user can perform calorie identification on the food. Figure 1C ​​​As shown in (b) of FIG. 5, the user interface 500 includes a first prompt area 501, a food area 502, a second prompt area 503, and a save control 504. The first prompt area 501 displays the total calorie amount of all food materials (i.e., the total heat value), for example, the total calorie amount of the food to be identified 402 is 767 kilocalories (kcal). The second prompt area 503 displays the food name, the total amount of nutritional ingredients of all food materials, and the proportion of the daily intake amount that has been occupied by each nutritional ingredient, for example, the food to be identified 402 is a light meal, in which the total amount of fat is 53.0 grams (g), accounting for 62% of the daily intake amount; the total amount of carbohydrates is 37.5 g, accounting for 12.5% of the daily intake amount; and the total amount of protein is 39.8 g, accounting for 66% of the daily intake amount. Clicking the second prompt area 503 can also view the total amount of more nutritional ingredients contained in the food to be identified 402.

[0052] The food area 502 displays the plurality of food materials identified in the food to be identified 402, for example, the food to be identified 402 includes 4 food materials, bread, carrot, cheese, and nuts, which are labeled as ①, ②, ③, ④, and ⑤, respectively. It should be noted that the same food material can be in discontinuous regions in the image, and in some implementations, the discontinuous regions can be labeled. For example, the bread in the image is in two discontinuous regions, which can be labeled as ① and ⑤. In some implementations, each food material is identified, and it is determined whether there is a repeated identification in the discontinuous region. If there is a repeated identification, it indicates that the corresponding discontinuous region is the same food material, and the label can be combined into one.

[0053] The user can click the serial number of a food material (one way of the second operation), and in response to the user operation, the electronic device can display a user interface 600, which includes a third prompt area 601. The third prompt area 601 can display the weight of the food material, the heat value under the weight, and the nutritional ingredients under the weight. For example, as shown in (a) of FIG. 6, after the user clicks the serial number ②, the third prompt area 601 can display that the food material corresponding to the serial number ② is carrot, and the weight of the carrot is 46 g. The heat value contained in 46 g of carrot is 14.7 kcal, the carbohydrate contained in 46 g of carrot is 4.7 g, the protein contained in 46 g of carrot is 0.46 g, and the fat contained in 46 g of carrot is 0.092 g. Figure 1D As shown in (a) of FIG. 6, after the user clicks the serial number ②, the third prompt area 601 can display that the food material corresponding to the serial number ② is carrot, and the weight of the carrot is 46 g. The heat value contained in 46 g of carrot is 14.7 kcal, the carbohydrate contained in 46 g of carrot is 4.7 g, the protein contained in 46 g of carrot is 0.46 g, and the fat contained in 46 g of carrot is 0.092 g. Figure 1DAs shown in (b) of FIG. 4 , after the user clicks on sequence number ④, the third prompt area 601 may display that the ingredient corresponding to sequence number ④ is nuts, and the weight of the nuts is 20g. The caloric value of 20g of nuts is 150kcal, the carbohydrates are 3.06g, the protein is 5g, and the fat is 9.74g. The user can also obtain more detailed nutritional information by clicking on the third prompt area 601.

[0054] After the electronic device completes the food nutrition test, the user can click the save control 504 (third operation). In response to the user operation, the electronic device can save the nutrition test results, and then the user can record the daily diet. Furthermore, the electronic device can upload the diet record to the server, and the server can send the user's diet record to the nutritionist. The nutritionist can give personalized and reasonable diet suggestions based on the user's diet record, and then send the diet suggestions to the electronic device through the server. The electronic device can prompt the diet suggestions to the user. The user can promptly adjust his or her diet more reasonably based on the diet suggestions and recommended health recipes fed back by the nutritionist.

[0055] In other implementations, users can also pull down the main page of the mobile phone and click the icon to the right of the global search bar to quickly enter the above-mentioned smart vision function.

[0056] In other implementations, users can access the calorie recognition function from the sports health application and manage their diet records. Figure 2A As shown in (a) of FIG, the electronic device can detect a user operation (such as click, voice control, gesture, etc.) acting on the sports health application icon 102, and in response to the user operation, the electronic device can display the following Figure 2A The user interface 700 shown in (b) of FIG. The user interface 700 is the main interface of the sports health application.

[0057] like Figure 2A The main interface of the sports health application shown in (b) includes multiple functional controls, such as smart fat loss, diet diary 701, sleep-aiding music, etc. The electronic device can detect the user operation on the diet diary 701, and in response to the user operation, the electronic device can display the following information: Figure 2B The user interface 800 shown in (a) in FIG. 8 is the main interface of the diet diary function, where the user can record daily diet and view the calories consumed and consumed by the user.

[0058] like Figure 2BAs shown in (a) of FIG, the user interface 800 includes a photo identification control 801. The electronic device can detect a user operation on the photo identification control 801. In response to the user operation, the electronic device can display a Figure 2B In the user interface 900 shown in (b), the user can point the camera at the food to identify the calories of the food. The identification results can be referred to Figures 1C-1D After obtaining the nutritional test results, the user can save the nutritional test results.

[0059] The user interface 800 also includes a history control 802 . The user clicks the history control 802 to view recorded dietary information, including what food the user ate on which day, the total calories of the food, and the like.

[0060] The user interface 800 also includes a today's diet analysis control 803. The user clicks the today's diet analysis control 803 to view information such as the calorie value consumed today and the remaining calorie value that can be consumed.

[0061] The user interface 800 also includes a nutritionist suggestion control 804. The user clicks the nutritionist suggestion control 804 to view the diet suggestions and recommended healthy recipes fed back by the nutritionist. Figure 2C In the user interface 1000 shown, the dietician's dietary advice may be "Your current diet is nutritionally balanced, please continue to maintain it." After viewing the dietary advice, the user may also obtain a new dietary advice.

[0062] The following combination Figures 3 to 11 The specific implementation of the above embodiment is described.

[0063] Figure 3 This is a flow chart of a food nutrition detection method provided in an embodiment of the present application. Figure 3 The food nutrition detection method shown can be executed by an electronic device, or by a chip configured in the electronic device; Figure 3 The food nutrition detection method shown includes S101 to S106, which are described in detail below.

[0064] S101 : In response to detecting a first operation, acquiring a first image through a first camera and acquiring a second image through a second camera.

[0065] In one implementation, when a user needs to perform a nutritional test on a food, Figures 1A-2B The function shown in , the user can enter the corresponding function interface and aim the camera at the food to be detected (a method of the first operation). In response to the user operation, the electronic device can shoot the food to be detected to obtain the first image and the second image.

[0066] The electronic device includes a plurality of cameras, including a first camera and a second camera. The first camera can be a main camera, and the second camera can be a long-focus camera. The electronic device can capture a first image through the first camera and a second image through the second camera in response to one operation. The first image and the second image have the same image size. It should be understood that the electronic device obtains the first image and the second image in the same scene, so the first image and the second image include the same food, and there is a same food region between the first image and the second image. Because the first camera and the second camera are located differently on the electronic device, the position of the food in the first image is different from the position of the food in the second image, and there is a visual difference between the first image and the second image.

[0067] The food to be detected can include at least one food material. For example, the food to be detected is green pepper and shredded potato, which includes two food materials, green pepper and shredded potato.

[0068] In another implementation manner, the electronic device can obtain the first image and the second image from an album. The first image and the second image are obtained in advance by the first camera and the second camera and saved in the album. When the user enters the user interface shown in (b) of the first image and the second image in the album. Figure 1B When the user enters the user interface shown in (b) of the first image and the second image in the album. Figure 2B The user can click the album shortcut control to enter the album, and then select the first image and the second image in the album (another way of the first operation). In response to the user operation, the electronic device can obtain the first image and the second image from the album for nutrition detection.

[0069] S102, image recognition is performed on the first image and / or the second image to determine the category of each food material in the food and the pixel point of each food material in the first image and / or the second image.

[0070] The electronic device can input the first image and / or the second image into a segmentation model to segment the food in the first image and / or the second image according to the food material, so as to identify each food material in the food and segment the pixel point of each food material in the first image and / or the second image. The pixel point corresponds to a point on the food material.

[0071] It should be noted that in the embodiments of the present application, image recognition and segmentation can be performed on the first image to determine the pixel points of each food material in the first image. Alternatively, image recognition and segmentation can be performed on the second image to determine the pixel points of each food material in the second image. Alternatively, image recognition and segmentation can be performed on the first image and the second image respectively to determine the pixel points of each food material in the first image and the second image respectively.

[0072] Food recognition is a relatively complex target recognition. Due to the large variety of food, the granularity of the same food material in different foods is different, the combination of different food materials, different accessories, and different arrangements can all cause the visual difference of the same category of images to be very obvious. For example, fresh fruit salad presents different visual appearances due to the mixing of different fruit ingredients, making large-scale food image recognition more difficult.

[0073] First, food images have obvious global appearance, shape and other structural features, and there are large intra-class differences in these features. For example, biscuits have different sizes and shapes, and fried noodles have a variety of textures. Second, there are subtle differences in the details of food images, and some of the details are key local features. For example, relying only on global features is not enough to distinguish between "corn soup" and "egg soup", and further differences in food material information need to be explored. Finally, the proportion of different food materials in different food categories is different and does not always appear in all images of the category. For example, the proportion of cilantro in cilantro beef and old vinegar jellyfish is different, and the proportion of chili in chili potato strips and chili meat strips is different. Therefore, food image recognition needs to consider the characteristics of food images, explore the relationship between local features, highlight key local features, and consider different granularities of image features for recognition to improve the overall accuracy of food recognition.

[0074] Considering the above characteristics of food images, in the embodiments of the present application, a segment anything model (SAM) and a multi-modal large language model (LLM) are used to segment and recognize the first image. The SAM model is a multi-modal segmentation model that is mainly used to segment any target in an image at different granularities.

[0075] The following will be described in detail with the first image as an example. Figure 4 The following will be described in detail with the first image as an example. Figure 4 is a flowchart of image segmentation and recognition provided by the embodiments of the present application, as shown in Figure 4 S1021-S1028.

[0076] S1021: Input the first image into the SAM model.

[0077] S1022: The SAM model segments the first image to obtain masks of various food ingredients, categories of the food ingredients, and category confidences.

[0078] Figure 5 The structure of the SAM model provided in the embodiment of the present application is shown as follows: Figure 5 As shown, after the image to be identified is input into the SAM model, it passes through the image encoder to obtain image features (image embedding); then the descriptive text (prompt) entered by the user is processed by the text encoder (prompt encoder) to obtain text features (prompt embedding). The text features and image features are simultaneously input into the mask decoder module. The mask decoder uses the text features and image features obtained by the encoder to generate multiple effective masks of different granularities and output the segmented target body. The mask decoder can also perform semantic recognition, thereby providing semantic labels for the segmented targets and outputting the category corresponding to each target and the category confidence. For example, three segmentation results are shown in the figure, with category confidence scores of 1, 2, and 3 respectively.

[0079] The descriptive text input by the user may be of the type of mask, point, scratch, box, text, etc.

[0080] Text refers to the textual descriptions entered by the user to prompt and guide the SAM model. Users can choose from pre-set content, such as descriptive sentences from poetic Tang poetry, Song lyrics, Yuan opera, modern poetry, and prose. They can also search for desired text online using keywords. Alternatively, they can enter a description of the image content to be recognized, such as the name of the food to be segmented, or "carrot" as a text prompt.

[0081] Clicking refers to the user clicking on the image to be recognized to guide the SAM model to output the body of the click area. The user can click once or multiple times.

[0082] Swiping refers to the user swiping a range on the image to be recognized to guide the SAM model to output the subject of the swiping area.

[0083] The bounding box refers to a user drawing a box on the image to be recognized to guide the SAM model to segment the object in the box. In some implementations, for some scenes with complex background and multiple foreground regions, a target detection framework GroundingDINO can be used to process the input image to be recognized to obtain the bounding box of the target, and the bounding box coordinates of the target can be input into the SAM model as a prompt. In this way, the target mask with the required granularity can be more accurately obtained, and the user's satisfaction and the convenience of operation can be improved, and the advantages of the SAM method can be fully utilized to enable the SAM method to have the ability to locate the target.

[0084] In addition, the SAM model can also support mask input, and the mask is added to the text feature after a convolution (conv) process, and then the text feature is input into the mask decoder.

[0085] In some implementations, the user can not input the descriptive text, and the SAM model can automatically generate a uniform grid to divide the first image into small blocks, and take the coordinates of the intersection points of all grid lines as the descriptive text input into the SAM model, and then segment all objects in the image to be recognized.

[0086] In summary, when the image to be recognized is the first image, the first image is input into the SAM model, the SAM model segments and recognizes the food according to the ingredients, and the final segmentation result is the type, pixel point in the image and category confidence of each type of ingredient. Among them, the pixel point of each type of ingredient in the image is the pixel point included in the mask of each type of ingredient. The category confidence refers to the confidence that the ingredient corresponds to a certain type of ingredient, for example, the confidence that the ingredient is peanuts is 0.20.

[0087] It should be noted that the classification of ingredients can be that each type of named ingredient is a category, for example, lettuce is a category and cabbage is a category. Alternatively, ingredients with similar calorie values can be classified into one category, for example, lettuce and cabbage can be classified into one category. The classification of specific ingredients can be set according to actual needs.

[0088] For example, the first image is shown in (a) of FIG. 1, and the first image includes multiple ingredients such as bread, nuts, carrots, etc. The first image is input into the SAM model, and the mask segmentation result of the SAM model is shown in (b) of FIG. 1, and the output result of the SAM model is shown in (c) of FIG. 1. In actual applications, each type of ingredient can be identified, and the SAM model can output the identification of each type of ingredient for subsequent calorie calculation. Figure 6 Figure 6 For example, the first image is shown in (a) of FIG. 1, and the first image includes multiple ingredients such as bread, nuts, carrots, etc. The first image is input into the SAM model, and the mask segmentation result of the SAM model is shown in (b) of FIG. 1, and the output result of the SAM model is shown in (c) of FIG. 1. In actual applications, each type of ingredient can be identified, and the SAM model can output the identification of each type of ingredient for subsequent calorie calculation. Figure 6 For example, the first image is shown in (a) of FIG. 1, and the first image includes multiple ingredients such as bread, nuts, carrots, etc. The first image is input into the SAM model, and the mask segmentation result of the SAM model is shown in (b) of FIG. 1, and the output result of the SAM model is shown in (c) of FIG. 1. In actual applications, each type of ingredient can be identified, and the SAM model can output the identification of each type of ingredient for subsequent calorie calculation.​

[0089] S1023, determine whether the category confidence is greater than or equal to a first threshold.

[0090] The category confidence can be used to determine whether the category label of the current food material is reliable and whether it is a new category. If the category confidence of the food material is greater than or equal to the first threshold, it is considered that the SAM model has seen this type of food material, i.e., the training set contains pictures of this type of food, and the recognition result is reliable, and the category of the food material can be directly output.

[0091] If the category confidence of the food material is less than the first threshold, i.e., the confidence of a certain food material is low, it may be a new food material that the SAM model has not seen, and then the multi-modal large language model needs to be used for recognition again, and then the category of the food material is output.

[0092] The present application does not limit the value of the first threshold, for example, the first threshold can be 0.95.

[0093] S1024, in the case where the category confidence is greater than or equal to the first threshold, output the category of the food material.

[0094] S1025, in the case where the category confidence is less than the first threshold, determine the food material candidate region in the first image.

[0095] If the category confidence of the food material is less than the first threshold, a pre-trained visual encoder CLIP is used to extract features from the first image to obtain a visual representation of the first image, and then position encoding is added to obtain a food material candidate region proposal in the first image. The food material feature in the first image can be used as an input of a preset model (mask Q-former).

[0096] S1026, input the food material candidate region in the first image, the food material mask region, and the descriptive text corresponding to the food material mask region into the preset model to obtain an aggregated feature.

[0097] After the SAM model is finely segmented by the electronic device, the mask of each food material can be obtained. For food materials with a category confidence lower than the first threshold, the mask region thereof is determined and recorded as mask-proposal. The food material mask region is used as another input of the preset model.

[0098] In addition, when the SAM model is segmented, the descriptive text is input. The descriptive text corresponding to the food material mask region can be determined as a text modal input of the preset model in this stage.

[0099] The food material candidate region in the first image, the food material mask region, and the descriptive text corresponding to the food material mask region are input into a preset model. The preset model includes two learnable tasks, a mask query and a context query. The mask query mainly focuses on the mask region and performs a masked cross attention mechanism. The context query focuses on the proposal bounding box region outside the mask and performs a context cross attention mechanism. After the mask query and the context query pass through a self-attention mechanism layer, information interaction is performed in combination with the descriptive text. After the information interaction between different modalities, aggregated feature tokens can be obtained.

[0100] In S1027, the aggregated features are input into a large language model to determine the category of the food material.

[0101] In this stage, the large language model mainly converts the input tokens into corresponding category label names in a conditional probability, so as to determine the category of the food material with a category confidence less than a first threshold.

[0102] At this point, after the above operations, the first image can be recognized and segmented to obtain category information of various food materials, category confidence of the food materials, and regions of various food materials in the image, and further the volumes of the food materials can be determined.

[0103] In S103, the first image and the second image are feature fused, and a feature map obtained by the feature fusion is divided into a plurality of grid units. Depth values of a plurality of first pixels corresponding to the plurality of grid units are determined. Depth values of a plurality of second pixels in the plurality of grid units except the plurality of first pixels are determined according to the depth values of the plurality of first pixels.

[0104] In the embodiment of the application, the first model is used to perform binocular depth estimation on the first image and the second image to determine the real depth of each food material. Then, the three-dimensional information of various food materials can be determined according to the real depth of various food materials, and the volume of various food materials can be determined according to the three-dimensional information of the regions of various food materials.

[0105] First, the principle of binocular depth estimation is briefly described. The first camera and the second camera are installed at the same height, and the two left cameras have the same focal length f, internal and external parameters, and pixel number. The first camera can correspond to the left camera, and the second camera can correspond to the right camera. Binocular depth estimation is performed based on the left and right cameras.

[0106] AsFigure 8 As shown, the Z axis represents the depth, and the X axis represents the lateral dimension, and since the first camera and the second camera are installed at the same height, the dimension in the height direction is not needed. Point o in the figure is a point on the food material, i.e., the target for which the depth needs to be estimated, and (x, z) is the lateral coordinate and the depth coordinate of point o in the three-dimensional space. The distance between the left camera and the right camera in the X axis is denoted as the baseline B length, and the imaging coordinate of point o (x, z) on the imaging plane through the left camera is x l , and the imaging coordinate on the imaging plane through the right camera is x r , that is, the coordinates of the same object on the two imaging planes are different, and the difference is denoted as the disparity D = x l -x r . Since the focal length is constant, the following formulas (1) and (2) can be obtained.

[0107]

[0108] Therefore, the relationship between the disparity D and the depth z can be obtained as follows:

[0109]

[0110] wherein the focal length f and the baseline length B are known quantities. Therefore, the disparity D of point o on the first image and the second image is determined, and the depth coordinate z of point o is obtained. The depth refers to the distance from the object being photographed to the optical center of the camera.

[0111] Therefore, for each pixel point in the first image, a matching point can be found in the second image (i.e., both represent the same point on the food material), so that the corresponding x l and x r are determined. Since the depth of the food material is actually limited, the range of the disparity is also limited, and the disparity has a maximum value D max , and presents a geometric characteristic of being large near and small far, that is, there are multiple disparity levels, and the disparity D of each pixel point is valued in {0, 1, 2,..., D max}. Based on this, the first model can first calculate the matching error (Cost Volume) for each possible disparity, and then determine the depth of each point on the food material according to the matching error.

[0112] Figure 9 The structural schematic diagram of the first model provided by the embodiment of the application is shown in FIG. 1. Figure 9As shown in the figure, after the first image is input into the first model, it is first encoded by a pre-trained encoder (clip-vit) to obtain the feature representation of the first image's intermediate layer. Then, the spatial pyramid pooling (SPP) module performs feature compression to reduce the dimensionality of the intermediate layer features. SSP uses average pooling layers of varying sizes. A larger average pooling layer corresponds to a larger receptive field in the original image, meaning that the features obtained are more global, while a smaller average pooling layer means that the features obtained are more local. Then, a three-dimensional dilated convolution is performed to expand the receptive field while maintaining the feature map resolution, thus obtaining the first feature map.

[0113] Similarly, after the second image is input into the first model, it undergoes a pre-trained encoder (clip-vit), a spatial pyramid pooling module, and a 3D dilated convolution to produce a second feature map. The parameters of the networks used by the first and second images are shared. Both the first and second feature maps contain rich global and local information.

[0114] The features of the first feature map and the second feature map are fused and extracted, and the global and local features are combined to obtain the CostVolume. At this time, the Cost Volume is a 4-dimensional tensor Tensor. The Cost Volume contains the information between the first image and the second image and the information between different parallax levels. It should be noted that in the embodiment of the present application, if the aforementioned SAM model segments the first image, the features of the first image can be fused into the second image at this time, so that the depth map obtained subsequently corresponds to the second image and is in the same image coordinate system as the second image. If the aforementioned SAM model segments the second image, the features of the second image can be fused into the first image at this time, so that the depth map obtained subsequently corresponds to the first image and is in the same image coordinate system as the first image.

[0115] It should be noted that the embodiment of the present application does not limit the specific parameters of each network module in the first model (such as step size, convolution kernel, etc.), and can be set according to actual needs.

[0116] Afterwards, the Cost Volume is input into the linear combination module, which includes a three-dimensional convolution (3D CNN), a grid division module (bin module) and a depth calculation module.

[0117] Among them, the three-dimensional convolution module further fuses and extracts the information in the 4-dimensional cost volume to obtain a three-dimensional cost volume.

[0118] The grid partitioning module comprises a first multilayer perceptron (MLP), patch embeddings, a transformer encoder, and an MLP head.

[0119] The first MLP performs feature extraction and processing on the Cost Volume, and converts the Cost Volume into a 2-dimensional feature map. This allows the subsequent model part to better understand and process the features.

[0120] The patch embeddings are used to divide the 2-dimensional feature map into multiple grid cells bins, and convert each grid cell into a vector b. The size of each grid cell is adaptively determined, i.e., the size of the grid partitioning corresponding to different images is different. The parameters of the patch embeddings network can be learned in a pre-training process.

[0121] The transformer encoder captures the relationship between different positions in multiple vectors, encodes the input multiple vectors, and converts them into a context-rich representation.

[0122] The MLP head performs nonlinear transformation and aggregation on the features, and outputs a feature map with depth information after grid partitioning. After grid partitioning, the depth value of the grid center point c(b) is a known absolute depth value, where the grid center point corresponds to the first pixel point, and the absolute depth value is the true depth. This is equivalent to dividing the first image or the second image into multiple grid cells, and a grid center point corresponds to a pixel point on the first image or the second image. The depth value of this pixel point can be determined at this stage.

[0123] For a certain grid cell, the depth values of other pixel points (i.e., second pixel points) in the grid cell are calculated based on the depth value of the center point of the grid cell and the depth values of the center points of multiple grid cells around the grid cell. The depth value of the second pixel point is estimated as a linear combination of multiple known absolute depth values, and only the weighting coefficients p k That's it.

[0124] As Figure 9As shown, the depth value of c(b) is input into the depth calculation module, for each second pixel point, the depth calculation module uses a second multi-layer perception MLP to learn the mapping relationship between the input features (grid center point c(b)) and the output (weighting coefficient), and performs a normalization (Softmax) operation on the output of the second MLP to make the weighting coefficient form a valid probability distribution, so as to determine the weighting coefficient p of the plurality of known absolute depth values k Then, for a certain second pixel point, the final depth value of the second pixel point is calculated according to the linear combination of the grid cell center point depth values As shown in the following formula (4):

[0125]

[0126] Wherein, d max is the maximum depth value, d min is the minimum depth value. b k is the pixel width of each grid. That is, the aforementioned depth z. c(b k ) is the kth grid center point, k∈[1,N], i∈[1,k-1].

[0127] For example, for a second pixel point in a certain grid cell, the depth value of the second pixel point in the grid cell can be determined according to the depth value of the first pixel point in the grid cell and the depth value of the first pixel point in the four grid cells above, below, left and right of the grid cell. For example, for a second pixel point in a certain grid cell, the depth value of the second pixel point in the grid cell can be determined according to the depth value of the first pixel point in the grid cell and the depth value of the first pixel point in the nine grid cells around the grid cell. Which grid cells are used can be designed during model training.

[0128] The depth value of each second pixel point in the grid cell is calculated one by one to obtain the depth value of all pixel points, so that the first model can output the depth map corresponding to the first image or the second image after the above process.

[0129] Wherein, the parameters of the first MLP and the second MLP are different. The size of the feature map output by the grid division module may be different from the size of the first image, and the second MLP can process the size of the feature map output by the grid division module to the same size as the first image through convolution.

[0130] In other implementations, after dividing the grid, other pixel points in the grid cell can also be selected as the first pixel point, but the grid center point is convenient to determine and can speed up the calculation.

[0131] In other implementations, after the grid cells are divided, the depth values of the plurality of pixel points in the same grid cell are relatively close, and the depth values of other pixel points in the same grid cell can also be estimated according to the depth value of the grid center point in the grid cell.

[0132] The first model is obtained by pre-training based on label data. The trained first model can divide fine and dense grids, and can autonomously learn the weighting coefficients.

[0133] In summary, the first model uses a binocular depth estimation algorithm to calculate depth information. The SPP module and the atrous convolution in the first model utilize the ability of global context information to aggregate context of different scales and positions together, expand the receptive field, and extend the pixel-level features to the region-level features with different receptive field scales. The cost volume module is formed by combining global and local feature clues, and can be used for reliable disparity estimation. In addition, the first model uses a linear combination module to divide the depth of the image region by bins. By dividing fine and dense grids, each small region of the food material is located in a grid, and the depth value of the center point of the grid cell is accurately determined by estimation based on binocular images. By providing a depth range based on the grid cell, the depth values of other pixel points in the grid cell are determined by weighted estimation based on the absolute depth value of the center point of the grid cell, which can more accurately obtain the depth values of other pixel points, and further accurately determine the volume of the food. In particular, the food material is three-dimensional, and the depth near the edge of the food material is different. By dividing the grid to calculate the depth value, the depth value of the edge of the food material can be more accurately obtained.

[0134] Since the pixel points of each type of food material in the first image or the second image have been determined according to the SAM model, after the depth map corresponding to the first image or the second image is determined, the real depth value of each pixel point in each type of food material can be determined.

[0135] At this point, the depth value of each pixel point in each type of food material region can be obtained, and then the volume of each type of food material can be determined.

[0136] In S104, the volume of each type of food material is determined according to the real depth value of each pixel point in each type of food material.

[0137] It should be understood that the food is in a real three-dimensional space, and the above-mentioned depth value is the coordinate of each point in each type of food material in one dimension. In the embodiments of the present application, according to the conversion relationship between the camera coordinate system, the image coordinate system, the pixel coordinate system and the world coordinate system, the coordinates of each point in each type of food material in the other two dimensions can be determined, so that the three-dimensional coordinates (x, y, z) of each point in each type of food material in the three-dimensional space are obtained.

[0138] After that, as Figure 10As shown, all coordinate points are converged to form a three-dimensional point cloud data in three-dimensional space, and then the volume is determined. The three-dimensional point cloud data is subjected to Delaunay triangulation, and the point cloud data is converted into a triangular mesh surface formed by connecting points. Further, a fast algorithm related to computational geometry is used to estimate the volume of each food material region. Specifically, for each triangle, the area thereof can be calculated by using the Heron formula or vector cross product method. Then, for each triangle, the plane equation thereof can be calculated, which can be achieved by using the coordinates of the three vertices. For each triangle, the volume contribution thereof can be calculated by multiplying the area thereof with the distance from the plane thereof to a reference plane (usually a three-dimensional coordinate axis). For all triangles corresponding to a certain food material, the volume contributions of the triangles are added together, and thus the volume of the food material in the three-dimensional space is obtained. In the present application, only the region where the food material is located is subjected to volume estimation, and thus the calculation efficiency can be improved.

[0139] In S105, the total heat of the food is determined according to the volume of each food material and the preset nutrient component information.

[0140] In the embodiments of the present application, a nutrient component database can be constructed in advance, and the nutrient component database includes the name, category, code (number), preparation method, unit volume heat value, density, unit weight heat value, and various nutrient component contents under unit volume / weight of each food material, and the like. The various nutrient components can include carbohydrates, proteins, fats, sugar, total dietary fiber, cholesterol, calcium, iron, sodium, potassium, magnesium, phosphorus, niacin, folic acid, and the like.

[0141] For example, Table 1 shows some nutrient component information in the nutrient component database.

[0142] Table 1

[0143]

[0144]

[0145] As shown in Table 1, the food material categories included in the above-mentioned nutrient component database can include green vegetables, potatoes, carrots, shrimps, and the like, and each food material corresponds to a code. The number of the food material is the food material identifier, and the segmentation result of the food material corresponds to the food material identifier one by one. It should be noted that the electronic device can also use other tables or other forms to store the correspondence between the food material categories and the food material identifiers.

[0146] The category of the food material and the volume of each category of food material in the three-dimensional space can be obtained through the foregoing steps. At this stage, the heat of each category of food material can be determined according to the category of the food material and the unit volume of heat of each category of food material in the three-dimensional space from the nutrient database. The total heat of the food can be obtained by adding the heat of each category of food material. In the embodiment of the present application, the heat (unit: Joule) can also be understood as calorie, and 1 calorie is approximately equal to 4.1859 Joules.

[0147] For example, when the food includes two food materials, i.e., carrots and potatoes, the SAM model outputs the food material identifier 3, and the food material with the food material identifier 3 can be searched in the nutrient database to obtain the category of the food material as carrots and the unit nutrient component and / or unit calorie information of the carrots, and then the total heat and total carbohydrates, total protein, total fat, etc. contained in the carrots in the food can be determined according to the volume of the carrots. The SAM model also outputs the food material identifier 5, and the food material with the food material identifier 5 can be searched in the nutrient database to obtain the category of the food material as potatoes and the unit nutrient component and / or unit calorie information of the potatoes, and then the total heat and total carbohydrates, total protein, total fat, etc. contained in the potatoes in the food can be determined according to the volume of the potatoes. Then, the total heat of the food can be obtained by adding the total heat contained in the potatoes and the total heat contained in the carrots. The total protein of the food can be obtained by adding the total protein contained in the potatoes and the total protein contained in the carrots, and so on.

[0148] The electronic device can also search the density of each category of food material from the nutrient database according to the category of the food material, and determine the weight of each category of food material according to the volume of each category of food material in the three-dimensional space and the corresponding density. The heat of each category of food material in the food can be determined according to the weight of each category of food material and the corresponding unit weight heat value.

[0149] In practice, the nutrient component and heat information of the food material is also related to the preparation method of the food material, and the preparation method of the food material includes boiling, steaming, baking, stewing, etc. For example, under normal circumstances, the heat of the same food material prepared by steaming is less than the heat of the same food material prepared by stewing. In the embodiment of the present application, the different nutrient components and heat of the food material caused by different preparation methods are also considered, and different nutrient component information is stored in the nutrient database for the same food material with different preparation methods. Based on this, the preparation method can be identified in the embodiment of the present application, and the heat of the food material under the preparation method is determined, so that the nutrient component and / or heat of the food in the food image can be more accurately obtained.

[0150] In some implementations, the electronic device can determine the cooking method of an ingredient based on descriptive information input by the user. For example, the user can select information such as the cooking method, cuisine, and dish name as descriptive information for the first image and / or the second image. In other implementations, the electronic device determines a combination of ingredients based on the segmentation results of the SAM model, and then determines the cooking method, dish name, etc. based on the combination of ingredients.

[0151] Therefore, based on the above description, electronic devices can determine the total calories and total nutrients of food.

[0152] S106, displaying a first interface, which includes total calories and other nutritional information of the food.

[0153] After determining the total calories and total nutritional content of the food, the electronic device can display a first interface to present the nutritional test results of the food to the user through the first interface. The first interface can be as follows: Figure 1C As shown in (b) in FIG. Furthermore, users can more conveniently and simply obtain the total calories and other nutritional information in the food, which can improve the user experience.

[0154] Furthermore, the first interface also displays the serial number of each type of food. The user can click on the serial number. In response to the user operation, the electronic device can display the second interface, which presents the total calories and other nutritional information of each type of food to the user. The second interface can be as follows: Figure 1D shown.

[0155] In some implementations, after outputting the food identification result, if the user believes the result is incorrect, the user can modify it to the correct result. Furthermore, the electronic device can calculate the total calories of the food based on the modified result. The electronic device can also update the SAM model based on the modified result to subsequently improve the accuracy of food identification.

[0156] Furthermore, users can choose to save the nutritional test results for subsequent users to view, and users can determine the nutrients and calories they have consumed based on multiple calorie identification results so as to adjust their diet in a timely manner.

[0157] In some implementations, users can upload diet records in electronic devices to a server, which can send the user's diet records to a nutritionist. The nutritionist can provide the user with personalized diet recommendations and nutritional guidance based on the user's diet records, and then send the diet recommendations and nutritional guidance to the electronic device through the server. The electronic device can prompt the user with the diet recommendations and nutritional guidance, thereby guiding the user's daily diet, allowing the user to better understand their own eating habits, and helping the user to better achieve their health goals.

[0158] In some implementations, the electronic device can regularly perform statistical analysis on the nutritional test results saved by the user to obtain the various nutrients and / or calories consumed by the user from the diet during the statistical period, analyze whether the user's eating habits are healthy, and analyze whether the intake of various nutrients and / or calorie information is reasonable. Based on the statistical analysis results, the electronic device can send targeted health management reminder messages to the user so that the user can adjust his or her eating habits in a timely manner.

[0159] In summary, the present invention provides a method for detecting food nutrition. Figure 11 As shown, the method provided in an embodiment of the present application first captures a first image and a second image using a first camera and a second camera. Then, based on a SAM model and descriptive text input by the user, the food in the image is finely segmented and identified, obtaining the pixel points, categories, and category confidence levels of the ingredients. For ingredients with low category confidence levels, their categories are further determined by combining a large language model, improving the recognition accuracy of ingredients of unknown categories. Subsequently, based on binocular depth estimation, the first model is used to perform feature fusion on the first and second images. The feature map obtained after feature fusion is divided into grid cells, and the depth value of each pixel is determined through weighted estimation, obtaining high-precision depth information. Subsequently, the depth value of each type of ingredient is determined, and the three-dimensional coordinates of each type of ingredient are determined. A three-dimensional point cloud is generated based on the three-dimensional coordinates and triangulated. Three-dimensional reconstruction is performed using triangular facets to accurately estimate the volume of each type of ingredient. Subsequently, the food nutritional test results are determined based on the category of each ingredient, the volume of each ingredient, and preset nutritional information. The test results include the total calories of the food, the total nutritional content of the food, the calories of each ingredient, and the nutritional content of each ingredient. In addition, the food, nutritional content and calorie data obtained from the test can be shared with nutritionists, so that users can get timely dietary advice and recommended healthy recipes from nutritionists, and recommend recipes to users, so that users can adjust their diet more reasonably.

[0160] The above describes the food nutrition detection method provided in the embodiment of the present application. The following describes the electronic equipment and device for executing the above method provided in the embodiment of the present application.

[0161] Figure 12 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0162] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyro sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0163] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0164] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated into one or more processors.

[0165] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0166] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0167] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0168] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0169] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160, among others. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health (leakage, impedance), among others. In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0170] The wireless communication functionality of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, among others.

[0171] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, such that the electronic device 100 can communicate with a network and other devices through wireless communication technology.

[0172] The electronic device 100 implements the display function through the GPU, the display screen 194, and the application processor, among others. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0173] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), among others. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0174] The electronic device 100 can implement a photographing function through an ISP, a camera 193, a video codec, a GPU, a display 194, and an application processor, etc.

[0175] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0176] The camera 193 is used to capture still images or videos. Objects generate optical images through lenses and project them onto photosensitive elements. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, or other format image signal. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0177] The digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0178] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0179] The NPU is a neural-network (NN) computing processor that learns from the structure of biological neural networks, such as the transmission mode between human brain neurons, to quickly process input information and continuously self-learn. Through the NPU, the electronic device 100 can implement intelligent cognitive applications such as image recognition, face recognition, voice recognition, and text understanding, etc.

[0180] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0181] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0182] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0183] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals.

[0184] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0185] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0186] Microphone 170C, also known as a "microphone" or "speaker," is used to convert sound signals into electrical signals. When making a call or sending a voice message, a user can speak by placing their mouth close to microphone 170C to input the sound signal into microphone 170C.

[0187] The earphone interface 170D is used to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0188] The pressure sensor 180A is used to sense a pressure signal, and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The barometric pressure sensor 180C is used to measure the barometric pressure. In some embodiments, the electronic device 100 calculates the altitude, assists positioning and navigation by using the barometric pressure value measured by the barometric pressure sensor 180C.

[0189] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can detect the opening and closing of a flip leather cover by using the magnetic sensor 180D. The acceleration sensor 180E can detect the acceleration of the electronic device 100 in each direction (generally three axes). When the electronic device 100 is stationary, the acceleration sensor 180E can detect the size and direction of gravity. The distance sensor 180F is used to measure the distance. The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The fingerprint sensor 180H is used to collect a fingerprint. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locking, fingerprint photographing, fingerprint answering a call, and the like.

[0190] The temperature sensor 180J is used to detect the temperature. In some embodiments, the electronic device 100 performs a temperature processing strategy by using the temperature detected by the temperature sensor 180J. The touch sensor 180K is also referred to as a “touch device”. The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also referred to as a “touch panel”. The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a human vocal part vibration bone block.

[0191] The keys 190 include a power-on key, a volume key, and the like. The keys 190 can be mechanical keys. Alternatively, the keys 190 can be touch keys. The electronic device 100 can receive a key input, and generate a key signal input related to the user settings and function control of the electronic device 100.

[0192] The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompt, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The motor 191 can also correspond to different vibration feedback effects for touch operations acting on different regions of the display screen 194. Different application scenarios (such as time reminders, received messages, alarms, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0193] The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0194] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. The same SIM card interface 195 can simultaneously insert multiple cards. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calling and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0195] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 can include more or fewer components than the illustrated components, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0196] The software system of the electronic device 100 can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0197] Figure 13 is a software structure block diagram of the electronic device 100 of the embodiments of the present application.

[0198] A layered architecture divides software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0199] The application layer can include a series of application packages.

[0200] As shown in Figure 13 , the application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0201] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0202] As shown in Figure 13 , the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0203] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and take screenshots, etc.

[0204] The content provider is used to store and obtain data, and make the data accessible to the application. The data can include video, image, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc.

[0205] The view system includes visual controls, such as controls that display text, controls that display pictures, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon can include a view that displays text and a view that displays a picture.

[0206] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including call connection, call hang-up, etc.).

[0207] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

[0208] The notification manager enables applications to display notification information in the status bar, which can be used to convey alert-type messages that can automatically disappear after a brief stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the form of a figure or a scroll bar text in the top status bar of the system, such as a notification of an application running in the background, and can also be a notification that appears in the form of a dialog window on the screen. For example, the status bar prompts text information, issues a prompt sound, the electronic device vibrates, the indicator light flashes, etc.

[0209] The Android Runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0210] The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0211] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application framework layer into a binary file. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0212] The system library can include multiple functional modules. For example: surface manager, media library, three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.

[0213] The surface manager is used to manage the display subsystem, and provides 2D and 3D layer fusion for multiple applications.

[0214] The media library supports multiple commonly used audio, video format playback and recording, and static image files, etc. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0215] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0216] The 2D graphics engine is a drawing engine for 2D drawing.

[0217] The kernel layer is the layer between hardware and software. The kernel layer at least contains display driver, camera driver, audio driver, sensor driver.

[0218] Figure 14A structural schematic diagram of a food nutrition detection apparatus provided in an embodiment of the present application. The food nutrition detection apparatus can be an electronic device in the embodiment of the present application, or a chip or chip system in the electronic device. As shown in Figure 14 the food nutrition detection apparatus 1400 can include a processing unit 1401. The processing unit 1401 is configured to support the food nutrition detection apparatus 1400 to perform the processing steps described above.

[0219] In an implementation manner, the food nutrition detection apparatus 1400 further includes a storage unit 1402. The storage unit 1402 and the processing unit 1401 are connected through a line. The storage unit 1402 can include one or more memories, and the memory can be a device for storing programs or data in one or more devices or circuits. The storage unit 1402 can exist independently and be connected to the processing unit 1401 through a communication bus. The storage unit 1402 can also be integrated with the processing unit 1401.

[0220] The storage unit 1402 can store computer execution instructions of the method in the electronic device, so that the processing unit 1401 performs the method in the above embodiment. The storage unit 1402 can be a register, a cache or a random access memory (RAM), etc., and can also be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions.

[0221] Figure 15 A structural schematic diagram of a chip provided in an embodiment of the present application. As shown in Figure 15 the chip 1500 includes one or more (including two) processors 1501, a communication line 1502 and a communication interface 1503, and optionally, the chip 1500 further includes a memory 1504.

[0222] The method described in the embodiments of the present application can be applied to the processor 1501 or implemented by the processor 1501. The processor 1501 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit or the instruction in the form of software in the processor 1501. The processor 1501 described above can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices or discrete hardware components, and the processor 1501 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0223] The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing executed by a code processor, or executed by a combination of hardware and software modules in the code processor. Among them, the software module can be located in a storage medium mature in the art such as random access memory, read-only memory, programmable read-only memory or electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the storage 1504, and the processor 1501 reads the information in the storage 1504 and combines the hardware to complete the steps of the above method.

[0224] The processor 1501, the storage 1504 and the communication interface 1503 can communicate through the communication line 1502.

[0225] In the above embodiments, the instructions stored in the storage for the processor to execute can be realized in the form of a computer program product. Among them, the computer program product can be written in the storage in advance, or downloaded and installed in the storage in the form of software.

[0226] The embodiments of the present application further provide a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. For example, the available media can include magnetic media (for example, floppy disk, hard disk or magnetic tape), optical media (for example, digital versatile disc (DVD)), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0227] The embodiments of the present application provide an electronic device, which includes a processor and a memory. The memory is configured to store a computer program. The processor is configured to execute the computer program to perform the method described above.

[0228] The embodiments of the present application provide a chip. The chip includes a processor configured to invoke a computer program in a memory to perform the technical solutions in the above embodiments. The implementation principles and technical effects are similar to those of the above related embodiments, which will not be described here.

[0229] In the embodiments of the present disclosure, it should be understood that the disclosed apparatus / equipment and method can be implemented in other manners. For example, the embodiments of the apparatus / equipment described above are merely schematic. For example, the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0230] The embodiments of the present disclosure further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in each of the above-mentioned method embodiments. The computer readable storage medium stores a computer program or instructions. The computer program or instructions are executed by the processor to implement the above-mentioned method. The method described in the above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer readable medium or transmitted over a computer readable medium. The computer readable medium can include computer storage medium and communication medium, and can also include any medium that can transfer computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0231] As a possible design, the computer readable medium can include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disk storage; the computer readable medium can include magnetic disk storage or other magnetic storage devices. Moreover, any connection line can also be properly referred to as a computer readable medium. For example, if software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of the medium. As used herein, a disk and a disc include a compact disc (CD), a laser disc, an optical disc, a DVD, a floppy disk, and a Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer readable media.

[0232] Embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0233] In the above description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0234] It will be understood that the term "comprises" when used in this specification and the appended claims specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0235] It should also be understood that, in the description of the application, unless otherwise specified, " / " means or, such as A / B can mean A or B; "and / or" in this article is only a description of the relationship between the associated objects, which means any combination and all possible combinations of one or more of the related listed items, and includes these combinations, such as A and / or B, which can represent three cases of A alone, A and B together, and B alone.

[0236] As used in the description of the application and the appended claims, the term "if" can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0237] In the description of the application, the reference "one embodiment" or "some embodiments" and the like means that the particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, appearances of the phrases "in one embodiment" or "in some embodiments" or "in other embodiments" or "in still other embodiments" or the like in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise specified. The term "comprising" or "containing" or "including" or "having" or "comprises" or "contains" or "includes" or "has" or variations thereof herein are meant to be open-ended terms that can cover the presence of one or more recited elements or steps but do not exclude the presence of one or more other elements or steps.

[0238] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A food nutrition detection method, characterized in that: The method is applied to an electronic device, and includes: In response to detecting a first operation, acquiring a first image through a first camera of the electronic device and acquiring a second image through a second camera of the electronic device, wherein the first operation instructs the electronic device to perform food nutrition detection, the first image and the second image include the same food, and the food includes at least one ingredient; performing image recognition on the first image and / or the second image to determine the category of each food and the pixel points of each food in the first image and / or the second image; Performing feature fusion on the first image and the second image, dividing a feature map obtained by the feature fusion into a plurality of grid units, determining depth values ​​of a plurality of first pixels corresponding to the plurality of grid units, and determining depth values ​​of a plurality of second pixels in the plurality of grid units excluding the plurality of first pixels based on the depth values ​​of the plurality of first pixels; Determining the volume of each type of food according to the depth value of each pixel point in the each type of food; Determining the total calories and / or other nutritional information of the food based on the category of each ingredient, the volume of each ingredient, and preset nutritional information; A first interface is displayed, wherein the first interface includes total calories and / or other nutritional information of the food.

2. The method according to claim 1, characterized in that Determining depth values ​​of a plurality of first pixel points corresponding to the plurality of grid units, and determining depth values ​​of a plurality of second pixel points in the plurality of grid units other than the plurality of first pixel points based on the depth values ​​of the plurality of first pixel points, includes: For any one of the multiple grid units, determine the depth values ​​of multiple second pixel points other than the first pixel point in the any one grid unit based on the depth value of the first pixel point in the any one grid unit and the depth values ​​of the first pixel points in other grid units around the any one grid unit.

3. The method according to claim 2, characterized in that The determining, based on the depth value of the first pixel in the arbitrary grid unit and the depth values ​​of the first pixel in other grid units surrounding the arbitrary grid unit, the depth values ​​of a plurality of second pixel points in the arbitrary grid unit excluding the first pixel point, includes: Determining a weighting coefficient corresponding to each first pixel point in the arbitrary grid unit and the other grid units; Determine the depth value of the second pixel point other than the first pixel point in any one grid unit based on the depth value of the first pixel point in any one grid unit and the depth value of the first pixel point in the other grid units, and the weighting coefficient corresponding to each first pixel point.

4. The method according to any one of claims 1 to 3, characterized in that The first pixel point is the center point of the grid unit.

5. The method according to claim 1, wherein The determining the volume of each type of food according to the depth value of each pixel point in the each type of food includes: Determining the three-dimensional coordinates of each pixel point in each type of food in the three-dimensional space according to the depth value of each pixel point in the each type of food; Generating three-dimensional point cloud data corresponding to each type of food according to the three-dimensional coordinates of each pixel point in the three-dimensional space; The volumes of the various types of food are determined based on the three-dimensional point cloud data corresponding to the various types of food.

6. The method according to claim 1, characterized in that After displaying the first interface, the method further includes: A second operation is detected on the first interface, where the second operation is an operation by a user to view the calories of a single type of food; In response to the second operation, a second interface is displayed, which includes the total calories of the single type of food and other nutritional information of the single type of food.

7. The method according to claim 1, characterized in that The method further comprises: A third operation is detected on the first interface, the third operation instructing the electronic device to save the food nutrition test result; In response to the third operation, the food nutrition test result is saved.

8. The method according to claim 7, characterized in that The method further comprises: Send locally stored food nutrition test results to the cloud; receiving dietary recommendations sent by the cloud; Outputting prompt information to the user, wherein the prompt information includes the dietary advice.

9. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 8.

11. A computer program product, characterized in that The invention comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 8.

Images