Calorie estimation method and computer program product

Abstract

The invention provides a calorie estimation method and a computer program product, and relates to the technical field of large models. The method comprises the following steps: inputting an obtained food image into a pre-trained food identification model to obtain a food name and type description of at least one kind of food; then aiming at each kind of food, based on the food name and the type description of the food, searching the target calorie density of the food from a standard nutrition database; performing three-dimensional reconstruction processing on the food image to determine the surface area of each kind of food; calling a pre-trained quality estimation model based on the food image and the food name and the surface area of each kind of food to obtain a quality estimation value of each kind of food; and finally, based on the target calorie density and the mass estimation value of each food, calculating a calorie total value. According to the method, after food identification and calorie density searching, determination of the geometric morphology of the surface of the food is introduced, the overall precision and reliability of calorie estimation are effectively improved, and good adaptability is achieved.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and more specifically, to a calorie estimation method and computer program product. Background Technology

[0002] With the increasing awareness of public health, many people are beginning to pay attention to the calorie intake of food in their daily diet. Currently, existing calorie estimation methods mainly rely on traditional image recognition and classification technologies, combined with standard nutrient databases for lookup and matching. While these methods have certain convenience, they generally suffer from problems such as low estimation accuracy and poor adaptability. Summary of the Invention

[0003] The purpose of this invention is to provide a calorie estimation method and computer program product to improve the problems existing in the prior art.

[0004] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a calorie estimation method, comprising: The acquired food images are input into a pre-trained food recognition model to obtain the food name and type description of at least one food. For each food, the target calorie density of the food is retrieved from a standard nutrition database based on the food name and type description. The food images are subjected to 3D reconstruction processing to determine the surface area of ​​each food item; Based on the food images, the food names and surface areas of each food item, a pre-trained quality estimation model is invoked to obtain the quality estimate value of each food item. The total calorie value is calculated based on the target calorie density and mass estimate of each food.

[0005] In an optional implementation, the standard nutrition database includes a calorie mapping table and an inverted index table; each entry in the calorie mapping table reflects the mapping relationship between standard food names and calorie densities; the type description includes food type and ingredient list; The step of finding the target caloric density of a food from a standard nutrition database based on its food name and type description includes: The food name is standardized to obtain the target name; Search the heat mapping table based on the target name; If a matching entry exists in the heat map table, the heat density of the food is obtained from the matching entry; If the matching entry does not exist in the calorie mapping table, the calorie density of the food is determined from the inverted index table and the calorie mapping table based on the target name. If the ingredient list corresponding to the food is empty, then the caloric density of the food shall be used as the target caloric density; If the ingredient list corresponding to the food is not empty, then the caloric density of each ingredient in the ingredient list is found from the caloric mapping table, and the target caloric density is calculated based on the caloric density of the food and the caloric densities of each corresponding ingredient.

[0006] In an optional implementation, the inverted index table includes multiple name segments and an index value for each name segment appearing in each entry of the heat map table; The step of determining the caloric density of the food from the inverted index table and the caloric map table based on the target name includes: The target name is segmented to obtain at least two target words; Based on each target word segmentation, the corresponding candidate index value is retrieved from the inverted index table; Based on each candidate index value, the name of each candidate standard food and its caloric density are determined from the caloric mapping table. Calculate the Levenshtein edit distance between the target name and each of the candidate standard food names; The caloric density of the food is determined by the caloric density of the candidate food name with the smallest Levenshtein edit distance.

[0007] In an optional implementation, the step of calculating the target calorific density based on the calorific density of the food and the calorific densities of the corresponding ingredients includes: calculating an average value based on the calorific density of the food and the calorific densities of the corresponding ingredients to obtain the target calorific density.

[0008] In an optional implementation, the calorie mapping table further includes a weighting coefficient corresponding to each standard food name; the step of calculating the target calorie density based on the calorie density of the food and the calorie densities of the corresponding ingredients includes: The target calorific density is obtained by calculating the average value of the calorific density of the food and the calorific densities of the corresponding ingredients.

[0009] In an optional implementation, the step of performing three-dimensional reconstruction processing on the food image to determine the surface area of ​​each food item includes: The instance segmentation model is used to determine the image region containing each type of food from the food image; Using a monocular depth estimation model, the image region containing each food item is converted into a region depth image; Based on the depth images of each region, point cloud data for each type of food is extracted. After reconstructing the point cloud data of each food into a three-dimensional mesh using a three-dimensional reconstruction algorithm, the corresponding surface area is calculated based on each three-dimensional mesh.

[0010] In an optional implementation, the food recognition model is trained in the following manner: Obtain a first dataset; the first dataset includes a large number of first sample images and annotation data for each first sample image, the annotation data including the food name and type description of at least one food; Freeze the low-level feature extraction layers at the front end of the bottom-level visual encoder and text encoder in the first large language model, and freeze the parameters of all LayerNorm layers in the first large language model. Based on the dataset, a phased unfreezing strategy is used to progressively supervise and fine-tune the first large language model to obtain the food recognition model.

[0011] In an optional implementation, the quality estimation model is trained in the following manner: Acquire a second sample image and its label data; the label data includes the food name, three-dimensional surface area, and actual mass of at least one food. The second sample image and its corresponding food name and three-dimensional surface area are filled into a preset prompt word template to obtain inference prompt words; The reasoning prompts are input into the second language model multiple times to obtain multiple structured prediction results. The structured prediction results include the thought process and the predicted quality value of each food item. The reasoning prompts are used to instruct the second language model to estimate the food quality based on the input content and output the structured prediction results in strict accordance with the specified format requirements. Based on the actual quality of each food item in the labeled data and each of the structured prediction results, the reward value corresponding to each structured prediction result is calculated. Based on each reward value, the GRPO algorithm is used to update the parameters of the second language model to obtain the quality estimation model.

[0012] In an optional implementation, the step of calculating the reward value corresponding to each structured prediction result based on the actual quality and each of the structured prediction results includes: For each structured prediction result, the structured prediction result is parsed and its format is validated. If the format validation passes, the format correctness score corresponding to the structured prediction result is determined to be 1; if the format validation fails, the format correctness score is determined to be 0. The actual quality of each food item in the label data and the predicted quality value of each food item in the structured prediction results are input into a preset Gaussian function to obtain the quality accuracy score corresponding to the structured prediction results. The reward value corresponding to the structured prediction result is obtained by weighted summation of the format correctness score and the quality accuracy score.

[0013] In a second aspect, the present invention provides a computer program product that, when executed by a processor, implements the calorie estimation method described in the first aspect.

[0014] Compared with existing technologies, this invention provides a calorie estimation method and computer program product. First, the acquired food image is input into a pre-trained food recognition model to obtain the name and type description of at least one food. Then, for each food, the target calorie density is searched from a standard nutrition database based on its name and type description. Next, the food image undergoes 3D reconstruction to determine the surface area of ​​each food. Then, based on the food image, the name of each food, and the surface area, a pre-trained mass estimation model is invoked to obtain a mass estimate for each food. Finally, the total calorie value is calculated based on the target calorie density and mass estimate for each food. This invention, after food recognition and calorie density lookup, also introduces the determination of the food surface geometry, effectively improving the overall accuracy and reliability of calorie estimation, and also possesses good adaptability. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is one of the flowcharts illustrating a calorie estimation method provided in an embodiment of the present invention.

[0017] Figure 2 An example diagram of labeled data for a first sample image provided in an embodiment of the present invention.

[0018] Figure 3 This is a second schematic flowchart of a calorie estimation method provided in an embodiment of the present invention.

[0019] Figure 4 An example image showing the recognition result of a food image provided in an embodiment of the present invention.

[0020] Figure 5 An example diagram of a prompt word template provided in an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0026] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0027] As mentioned in the background section, existing calorie estimation mainly relies on traditional image recognition and classification techniques, combined with lookup matching from standard nutrient databases. While these methods offer some convenience, they generally suffer from low estimation accuracy and poor adaptability. This is because: firstly, two-dimensional images are easily affected by factors such as shooting angle, lighting, occlusion, and processing status, posing a challenge to the accurate identification of food types and quantities; secondly, traditional methods often lack the ability to model the actual geometric structure and quality of food, relying solely on image recognition results for quantitative inferences, ignoring the potentially complex relationship between morphological features and nutritional components.

[0028] Based on the discovery of the aforementioned technical problems, the inventors, through creative labor, proposed the following technical solutions to solve or improve these problems. It should be noted that the deficiencies in the solutions of the prior art are all results derived by the inventors after practical experience and careful research. Therefore, the discovery process of the aforementioned problems and the solutions proposed in the embodiments of this application below should be considered contributions made by the inventors to this application during the inventive process, and should not be construed as technical content known to those skilled in the art.

[0029] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a calorie estimation method provided in an embodiment of the present invention. The execution subject of this method can be a computing device such as a smartphone, personal laptop, personal computer, or server, or it can be a smart kitchen appliance such as a smart refrigerator, smart cooking equipment, or cooking robot. The method includes the following steps S301 to S305.

[0030] S301. Input the acquired food image into a pre-trained food recognition model to obtain the food name and type description of at least one food.

[0031] In this embodiment, the food recognition model can be obtained by supervised fine-tuning of a first large language model, which is a large model that supports multimodal input, and the type description can include food type and ingredient list.

[0032] Optionally, food types can include, but are not limited to, "ingredients," "finished dishes," "beverages," "desserts," "seafood," "vegetables," and "grains." Among these, the ingredient list for "ingredients" can be empty, while the ingredient lists for the other three categories are not empty. For example, the ingredient list for a "hamburger" can include bread, bun, beef patty, lettuce, and cheese.

[0033] S302. For each food, find the target caloric density of the food from a standard nutrition database based on the food name and type description.

[0034] In this embodiment, the target heat density is the calorie value per unit mass, and the unit can be kcal / 100g.

[0035] S303. Perform 3D reconstruction processing on the food image to determine the surface area of ​​each food item.

[0036] The surface area obtained reflects the physical dimensions of the food surface.

[0037] S304. Based on the food image, the food name and surface area of ​​each food, call the pre-trained quality estimation model to obtain the quality estimate value of each food.

[0038] It is understandable that this could be obtained by fine-tuning the second language model, which also supports multimodal input. During the fine-tuning phase, the second language model can learn the implicit relationships between visual appearance, semantic category, and geometric size, and can integrate multi-source information during the inference phase to output reasonable predictions of the quality of each food item.

[0039] S305. Calculate the total calorie value based on the target calorie density and mass estimate of each food.

[0040] The calorie estimation method provided in this invention first inputs the acquired food image into a pre-trained food recognition model to obtain the food name and type description of at least one food. Then, for each food, the target calorie density of the food is searched from a standard nutrition database based on its food name and type description. Next, the food image is subjected to three-dimensional reconstruction processing to determine the surface area of ​​each food. Then, based on the food image, the food name of each food, and the surface area, a pre-trained mass estimation model is invoked to obtain the mass estimate value of each food. Finally, the total calorie value is calculated based on the target calorie density and mass estimate value of each food. This invention, after food recognition and calorie density lookup, also introduces the determination of the food surface geometry, effectively improving the overall accuracy and reliability of calorie estimation, and also possesses good adaptability.

[0041] Here, we will first introduce the supervised fine-tuning process of the food recognition model, which may include the following steps S101 to S103.

[0042] S101. Obtain the first dataset; the first dataset includes a large number of first sample images and annotation data for each first sample image. The annotation data includes the food name and type description of at least one food.

[0043] This involves collecting multi-view images of over 300 common foods (no fewer than 1,000 images per category), and adding standardized text labels to each image to indicate the food name, food type, and ingredients, ultimately forming the first dataset.

[0044] Optionally, to improve annotation efficiency during the annotation phase, a semi-automatic annotation method can be adopted: First, an object detection algorithm (such as the Grounding-Dino algorithm) is used to automatically generate preliminary bounding boxes in each image, which are then manually refined to ensure positioning accuracy. Next, manual annotation can be performed based on the position of each bounding box. Subsequently, simple enhancements such as ±15° rotation, 80%–100% random cropping, and ±20% brightness fine-tuning are applied to each bounding box region to improve the model's adaptability to changes in shape and lighting. Finally, the first dataset is randomly divided into training, validation, and test sets in a training:validation:test ratio of 8:1:1.

[0045] For example, please see Figure 2 , Figure 2 The annotation data for two first sample images (i.e., image 1 and image 2) is shown in JSON format, as follows: Figure 2 Image 1 includes two dishes: white pomfret and large yellow croaker, both of which have empty ingredient lists. Image 2, on the other hand, only includes one dish: stewed pig's trotters with soybeans, whose ingredient list includes pig's trotters and soybeans. It should be noted that this example is merely illustrative and is not intended to be limiting.

[0046] S102. Freeze the low-level feature extraction layers at the front end of the bottom-level visual encoder and text encoder in the first language model, and freeze the parameters of all LayerNorm layers in the first language model.

[0047] For example, assuming the first major language model uses QWen2.5-vl 7B, the following freezing strategy is adopted: all LayerNorm layers are always frozen, and the low-level feature extraction layers at the front end of the text encoder and visual encoder are locked. The core reason for this approach is that the low-level feature extraction layers are mainly responsible for extracting basic text symbols (such as characters and word forms) and visual elements (such as edges and textures). These basic features have strong universality and stability and are the foundation of the model's understanding ability. Freezing and locking these layers can avoid destroying the basic feature extraction capabilities learned in the pre-training stage during fine-tuning, prevent overfitting to specific noise in the training data, reduce the parameter optimization space, improve training efficiency, and ensure that the model can still maintain good generalization ability when fine-tuning for specific tasks. It should be noted that this example is only an example, and the present invention does not limit the specific model used in the first major language model.

[0048] S103. Based on the dataset, a phased unfreezing strategy is adopted to progressively supervise and fine-tune the first language model to obtain the food recognition model.

[0049] Optionally, the fine-tuning optimizer can be AdamW, with its parameters set to... =0.9、 =0.999, weightdecay=0.01; initial learning rate set to 5× The batch size is 16; the learning rate is scheduled using a linear warm-up strategy for the first 500 steps, and then gradually reduced to 1× using a cosine decay method. .

[0050] The entire supervised fine-tuning process can be divided into three stages (the epoch below refers to the number of times the model completely traverses the training dataset once; one epoch means that the model completes one forward computation and one backward update for all samples in the training set): Phase 1 (5 epochs): Only the classification head of the first large language model and the last Cross-Attention layer are unfrozen, while the rest remain frozen; Phase Two (10 epochs): Following a "batch unfreezing" strategy, more Cross-Attention sub-layers and the Nth-to-last Transformer layer are unfrozen sequentially every 2 epochs (unfrozen down to the middle layers), and a "differentiated learning rate" is adopted: the top layer uses 5× Layers 3–5 use 1× , and at a lower level, use 5× ; Phase 3 (5 epochs): Unfreeze all Transformer layers and continue using the differential learning rate scheme described above to ensure that the underlying features remain stable and are not excessively disturbed.

[0051] The reason for adopting the phased unfreezing strategy is that high-level features are closer to specific tasks (such as food type identification), and unfreezing high-level features first allows the model to quickly learn task-related knowledge; while mid-level features have both generality and task relevance, and gradually unfreezing them can avoid learning instability caused by parameter mutations. At the same time, by controlling the update amplitude of low-level features through differentiated learning rates, we can ensure that basic features are not excessively interfered with by task data.

[0052] To ensure the accuracy of model recognition, the accuracy is verified based on the validation set after each stage: the Top-1 accuracy is calculated using three criteria: “label”, “type”, and “ingredient”, until the overall Top-1 accuracy of the validation set is ≥92%. If the verification of this stage is successful, the next stage can proceed.

[0053] Thus, by leveraging a streamlined and efficient training data collection and staged supervised fine-tuning (SFT) strategy, coupled with lightweight deployment, this method can achieve fast and stable food identification in multimodal input scenarios, providing high-quality and reliable input for subsequent calorie estimation.

[0054] In an optional implementation, the standard nutrition database may include a calorie mapping table and an inverted index table. The calorie mapping table contains a large number of entries, each reflecting the mapping relationship between a standard food name and its calorie density. Therefore, in Figure 1 Based on this, please refer to Figure 3 The above step S302 can be implemented in ways including S3021 to S3026.

[0055] S3021. Standardize the food name of the food to obtain the target name.

[0056] During the standardization process, it is necessary to remove stop words (i.e., modifiers) that do not affect nutritional content, such as "fresh," "cut into pieces," and "braised," and to standardize the numerical expression format (e.g., "3 eggs" is changed to "eggs"), thereby reducing the impact of semantic noise on the matching results and improving the consistency of subsequent searches.

[0057] S3022. Search the heat mapping table based on the target name.

[0058] S3023. If a matching entry exists in the heat mapping table, the heat density of the food is obtained from the matching entry.

[0059] The system uses the target name as a key to attempt an exact match in the calorie map. If a completely identical standard food name exists, the system directly locates the corresponding matching entry and extracts the food's calorie density. This search path is suitable for non-compound foods with standardized naming conventions and already entered into the system, enabling efficient and low-latency data access.

[0060] S3024. If there is no matching entry in the heat map table, the heat density of the food is determined from the inverted index table and the heat map table based on the target name.

[0061] When the target name indicates that the food is a compound food (i.e., the name involves multiple ingredients), a matching entry may not be found in the calorie map table. In this case, an inverted index table is needed.

[0062] The inverted index table is primarily used to support rapid candidate retrieval in fuzzy matching scenarios. It includes multiple name segments and the index value of each name segment appearing in each entry of the calorie mapping table, thus enabling reverse retrieval from local vocabulary to complete food entries. Therefore, step S3024 can be implemented using steps a1 to a5: Step a1: Perform word segmentation on the target name to obtain at least two target words; Step a2: Based on each target word segmentation, find the corresponding candidate index value from the inverted index table; Step a3: Based on each candidate index value, determine the name and calorie density of each candidate standard food from the calorie mapping table; Step a4: Calculate the Levenshtein edit distance between the target name and each candidate standard food name; Step a5: Take the calorie density of the candidate standard food name with the smallest Levenshtein edit distance as the calorie density of that food.

[0063] The Levenshtein edit distance reflects the minimum number of single-character edit operations required to transform one string into another, including insertion, deletion, or replacement operations. Therefore, the target name can be used as a baseline string, and each candidate food name can be compared pairwise to obtain the corresponding edit distance value. The smaller the value, the higher the similarity between the two at the character level.

[0064] S3025. If the ingredient list corresponding to the food is empty, then the caloric density of the food shall be used as the target caloric density.

[0065] S3026. If the ingredient list corresponding to the food is not empty, then look up the caloric density of each ingredient in the ingredient list from the caloric mapping table, and calculate the target caloric density based on the caloric density of the food and the caloric density of each corresponding ingredient.

[0066] In one scenario, the target calorific density can be obtained by calculating the average value based on the calorific density of the food and the calorific densities of the corresponding ingredients.

[0067] In another scenario, if the calorie mapping table also includes weight coefficients corresponding to each standard food name, then the target calorie density can be obtained by: finding the target name and the weight coefficients corresponding to each ingredient in the calorie mapping table, and then taking a weighted average of the calorie density of the food and the calorie density of each ingredient based on the found weight coefficients.

[0068] Based on steps S3021 to S3026 and their sub-steps, this invention employs methods such as name normalization, precise matching based on a heat mapping table, fuzzy matching based on an inverted index table, and consideration of the heat density of ingredients to ensure accurate heat density lookup in various complex scenarios.

[0069] For example, suppose the heat map table and the inverted index table include the contents shown in Table 1 below: Table 1 Heat Mapping Table and Inverted Index Table

[0070] Assuming the food image recognition result is as follows Figure 4 As shown, Figure 4 The target name for the first food is "tomato". Based on this, we can directly retrieve entry 1 (i.e., index value 1) in the calorie mapping table. Since the ingredient list for "tomato" is empty, the final target calorie density for tomato is 18kcal / 100g.

[0071] but, Figure 4 The target name for the second food is "Chicken Caesar Salad," which is a compound food. Since no corresponding entry can be found in the calorie mapping table based on this target name, it needs to be segmented into three target words: "chicken," "Caesar," and "salad." Based on these three target words, searches are performed in the inverted index table. The resulting candidate index values ​​are 2, 3, and 5, meaning the candidate standard food names include {"chicken," "Caesar Salad," and "fruit salad"}. The Levenshtein edit distance is smallest for "Caesar Salad," therefore, the calorie density of "Chicken Caesar Salad" can be considered to be 150 kcal / 100g.

[0072] At the same time, such as Figure 4 The ingredients list for "Chicken Caesar Salad" includes ["Lettuce", "Crucians", "Bacon"]. The calorie densities of these three ingredients can be found directly in the calorie mapping table, which are 15 kcal / 100g, 450 kcal / 100g, and 540 kcal / 100g, respectively. Then, the average calorie density of "Chicken Caesar Salad" and its various ingredients is calculated as: (150+15+450+540) / 4≈289, meaning that the final target calorie density of "Chicken Caesar Salad" is 289 kcal / 100g.

[0073] It should be noted that this example is for illustrative purposes only and is not intended to be limiting.

[0074] In optional implementations, due to the highly irregular shape of food and the frequent issues of occlusion, lighting changes, and background interference under actual shooting conditions, traditional two-dimensional image analysis methods struggle to accurately model its geometric features, leading to significant errors in surface area estimation and consequently affecting the reliability of the final heat estimation result. Therefore, this invention introduces a technical approach combining depth perception and three-dimensional reconstruction to improve the ability to reproduce the physical shape of food. Thus, the implementation process of step S303 can include S3031~S3034.

[0075] S3031. Use an instance segmentation model to determine the image region where each food item is located from the food image.

[0076] First, an instance segmentation model is used to segment the food image to extract a mask image corresponding to each food item. The white areas in this mask image represent the image regions where the food items are located. This boundary limitation ensures that depth information extraction focuses only on each food item, avoiding interference from irrelevant regions and reducing processing load. For example, the instance segmentation model can employ Mask R-CNN (Mask Region-based Convolutional Neural Network).

[0077] S3032. Using a monocular depth estimation model, the image region where each food is located is converted into a region depth image.

[0078] Monocular depth estimation models are deep learning models based on convolutional neural networks or Transformer architectures, capable of inferring the depth value distribution of individual pixels from a single-view color image or a cropped image patch. For example, monocular depth estimation models can employ the MiDaS model or the LeReS model.

[0079] S3033. Based on the depth images of each region, extract the point cloud data of each food item.

[0080] In this embodiment, based on the camera intrinsic parameter matrix and pixel coordinate mapping relationship, each pixel in the two-dimensional region depth image can be converted into three-dimensional point coordinates in space, thereby generating point cloud data corresponding to each type of food. This process involves a geometric transformation from the image coordinate system to the camera coordinate system, specifically including back projection operation, that is, combining the pixel position (u,v) with its corresponding depth value d to calculate the three-dimensional coordinates (x, y, z) in the world coordinate system. The specific process of converting to point cloud data is existing technology and will not be described in detail here.

[0081] S3034. After reconstructing the point cloud data of each food into a three-dimensional mesh using a three-dimensional reconstruction algorithm, calculate the corresponding surface area based on each three-dimensional mesh.

[0082] In this embodiment, a 3D reconstruction algorithm can be used to perform topological connections and surface fitting on the point cloud data of each food item, reconstructing it into a 3D mesh with a continuous surface structure. The reconstructed 3D mesh consists of vertices, edges, and faces (usually triangles), which can realistically restore the outer contour shape of the food and retain sufficient geometric details for surface area calculation.

[0083] The 3D reconstruction algorithm can employ the Marching Cubes algorithm, as it excels at rapidly generating continuous 3D mesh surfaces from discrete point clouds. Its voxel-based greedy strategy is highly adaptable to sparse point clouds and can supplement local details through interpolation, making it particularly suitable for reconstructing irregularly shaped and complex-edged objects such as food. Alternatively, the 3D reconstruction algorithm can also employ the PSR (Poisson Surface Reconstruction) algorithm, which constructs a smooth and continuous surface model by solving the Poisson equation. This effectively filters noise in the point cloud, is robust to local data loss caused by occlusion, generates a more complete closed mesh, and avoids holes or breaks in surface area calculations.

[0084] The reason this invention calculates surface area rather than food volume is that surface area is easier to extract stably from sparse depth data, specifically in the following aspects: (1) The surface area mainly depends on the geometric contour and topological structure of the object's surface. Even if the depth data is sparse (such as only obtaining the depth information of the key edges or local areas of the food), the surface connection relationship can still be completed by interpolation algorithm to form a continuous contour boundary. (2) Volume estimation requires complete three-dimensional spatial information of the object (including internal structure and overall scale), and has extremely high requirements for the density and completeness of depth data. Sparse data can easily lead to significant deviations in volume calculation. (3) Food ingredients often have natural deformation, partial occlusion (such as stacked vegetables) or depth noise caused by reflection. Surface area calculation only focuses on the geometric features of the visible surface, and is less affected by such interference, so the results are more stable.

[0085] In an optional implementation, for the quality estimation model in step S304 above, reinforcement learning can be used to fine-tune the instructions of the second language model. First, sample collection is required: Prepare different foods (such as tomatoes, Caesar salad, etc.) and take photos from multiple perspectives to obtain multiple second sample images. Then, each food needs to have its actual mass (unit: g) measured using a high-precision electronic scale (accuracy ≤ 0.1g). Additionally, point cloud data of each food item needs to be extracted from the second sample images using the methods described above or other methods (such as AR tools specifically designed for extracting 3D point clouds, like iPhone ARKit) to obtain the three-dimensional surface area (unit: g). This forms the label data for each second sample image, which may include the food name, three-dimensional surface area, and actual mass of at least one food.

[0086] For example, the label data for the two second sample images can be as follows ( / / and the text therein are for annotation purposes only): [ { "image_path": "0001.jpg", "foodlist": [ / / Tag data for the first image] { "label": "apple", "surface": 0.12, "mass": 120}, { "label": "banana", "surface": 0.14, "mass": 130} ] }, { "image_path": "0002.jpg", "foodlist": [ / / Tag data for the second image] { "label": "Stewed Pig's Trotters", "surface": 0.36, "mass": 350}, { "label": "Roasted Chicken Wings", "surface": 0.12, "mass": 120} ] } ] The label data shows that the first image of the second sample contains both apples and bananas, with the apple having a three-dimensional surface area of ​​0.12. The actual weight is 120g, while the banana's three-dimensional surface area is 0.14. The actual weight is 130g. The second sample image includes stewed pig's trotters and roasted chicken wings; the three-dimensional surface area of ​​the stewed pig's trotters is 0.36. The actual weight is 350g, while the three-dimensional surface area of ​​the roasted chicken wings is 0.12. The actual weight is 120g. It should be noted that this example is for illustrative purposes only and is not intended to be limiting.

[0087] The second major language model could be the food recognition model obtained through supervised fine-tuning, as described above. This ensures that the food names output by the model are consistent with the labeled data, enhancing semantic consistency and reducing the risk of semantic drift. Alternatively, the second major language model could be any other major model.

[0088] To improve the predictive ability of the model in the food quality estimation task, this invention introduces the GRPO (Group Relative Policy Optimization) algorithm as a reinforcement learning strategy to fine-tune the instructions of the second-largest language model. Unlike traditional policy gradient methods that rely on value functions, the GRPO algorithm optimizes the policy based on the relative reward ranking within the group, avoiding explicit Critic model modeling. The following section uses a second sample image and its label data as an example to illustrate the process of fine-tuning the instructions of the second-largest language model.

[0089] The process of fine-tuning the instructions of the second language model may include the following steps S201~S205.

[0090] S201. Obtain a second sample image and its label data, the label data including the food name, three-dimensional surface area and actual mass of at least one food.

[0091] S202. Fill the second sample image and the food name and three-dimensional surface area of ​​each food into the preset prompt word template to obtain the reasoning prompt words.

[0092] S203. Input the reasoning prompts into the second language model multiple times to obtain multiple structured prediction results. The structured prediction results include the thought process and the quality prediction value of each food item.

[0093] Among them, the inference prompts are used to instruct the second language model to estimate the food quality based on the input content and to output structured prediction results in strict accordance with the specified format requirements.

[0094] Since the GRPO algorithm optimizes its strategy based on the relative reward ranking within a group, it is necessary to input the inference prompts into the second largest language model multiple times to obtain multiple structured prediction results. For example, it can be input 8 or 16 times.

[0095] S204. Based on the actual quality of each food item in the label data and each structured prediction result, calculate the reward value corresponding to each structured prediction result.

[0096] In this embodiment, to effectively provide feedback and guidance to the model output, the present invention designs a dual reward mechanism that integrates the accuracy dimension of the output format and the accuracy dimension of the predicted numerical value. That is, the process of determining the corresponding reward value for each structured prediction result may include S204~S2043: S2041. The structured prediction result is parsed and its format is validated. If the format validation passes, the format correctness score corresponding to the structured prediction result is determined to be 1. If the format validation fails, the format correctness score is determined to be 0. Format validation primarily verifies whether the predicted quality values ​​for various foods in the structured prediction results are in a parsable JSON format and whether the thought process within them is correctly formatted. Therefore, the format correctness score (FormatReward) can be expressed as:

[0097] S2042. Input the actual quality of each food in the label data and the predicted quality value of each food in the structured prediction results into a preset Gaussian function to obtain the quality accuracy score corresponding to the structured prediction results. Considering the uncertainties inherent in image quality and 3D surface area, minimizing the absolute error is not the optimal goal. Therefore, a Gaussian function is used as a penalty term, allowing predicted values ​​to achieve high scores within the error range. That is, the formula for calculating the MassReward score is:

[0098] In the formula, This indicates the first [predictor's name] in a structured prediction result. Accurate rating of a type of food, This represents the quality accuracy score corresponding to a structured prediction result. , Each of the following is a structured prediction result. The estimated quality of a food and its corresponding actual quality. This is the standard deviation parameter, which can take a value of 1.

[0099] S2043. The format correctness score and the quality accuracy score are weighted and summed to obtain the reward value corresponding to the structured prediction result.

[0100] Among them, reward value The calculation formula is: Weighting coefficient It can take the value 0.2 or 0.3.

[0101] S205. Based on each reward value, the GRPO algorithm is used to update the parameters of the second language model to obtain the quality estimation model.

[0102] In this embodiment, the process of updating the parameters of the second language model using the GRPO algorithm based on each reward value is existing technology and will not be described in detail here.

[0103] This invention employs the GRPO algorithm and combines it with preset prompt word templates for instruction fine-tuning. This not only improves the model's adaptability to structured output tasks, but also ensures training stability while making the model more focused on fault tolerance and interpretability in actual task requirements. It is particularly suitable for multimodal calorie estimation and food quality modeling tasks involving image and language.

[0104] For example, the preset prompt word template can be as follows: Figure 5 As shown, Figure 5 middle, This indicates the food images to be inserted, while {name_list} represents the food name and surface area of ​​each food item to be inserted. The prompt word requires the output to include " <think> ……< / think> "The labels explain the thought process. It should be noted that this example is for illustrative purposes only and is not intended to be limiting."

[0105] In an optional implementation, corresponding to step S305 above, the calories of each food in the food image input can be calculated using the following formula, and then the total calories of the food image can be obtained by adding them together.

[0106]

[0107] In the formula, Indicates the calories in food. This indicates the estimated quality of the food. This indicates the target caloric density of the food.

[0108] It should be noted that the execution order of each step in the above method embodiments is not limited to what is shown in the accompanying drawings or step numbers, and the execution order of each step shall be subject to the actual application.

[0109] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 300 includes a processor 310, a memory 320, and a bus 330, with the processor 310 connected to the memory 320 via the bus 330.

[0110] The memory 320 can be used to store software programs or firmware, for example, the software program or firmware corresponding to the calorie estimation device 200 described above. The processor 310 executes various functional applications and data processing by running the software program stored in the memory 320 to implement the calorie estimation method provided in the embodiments of the present invention.

[0111] The memory 320 may be, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), FLASH (Flash Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0112] The processor 310 can be an integrated circuit chip with signal processing capabilities, capable of executing software programs, such as the software program corresponding to the calorie estimation device 200 described above. The processor 310 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), SoC (System on Chip), etc.; it can also be: DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0113] Understandable. Figure 6 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown. Figure 6 The components shown can be implemented using hardware, software, or a combination thereof.

[0114] This invention also provides a computer program product that, when executed by a processor, implements the calorie estimation method disclosed in the above embodiments.

[0115] In summary, this invention provides a calorie estimation method and computer program product. First, the acquired food image is input into a pre-trained food recognition model to obtain the name and type description of at least one food. Then, for each food, the target calorie density is searched from a standard nutrition database based on its name and type description. Next, the food image undergoes 3D reconstruction to determine the surface area of ​​each food. Then, based on the food image, the name of each food, and the surface area, a pre-trained mass estimation model is invoked to obtain a mass estimate for each food. Finally, the total calorie value is calculated based on the target calorie density and mass estimate for each food. This invention, after food recognition and calorie density lookup, also introduces the determination of the food surface geometry, effectively improving the overall accuracy and reliability of calorie estimation and possessing good adaptability.

[0116] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for estimating calories, characterized in that, include: The acquired food images are input into a pre-trained food recognition model to obtain the food name and type description of at least one food. For each food, the target calorie density of the food is retrieved from a standard nutrition database based on the food name and type description. The food images are subjected to 3D reconstruction processing to determine the surface area of ​​each food item; Based on the food images, the food names and surface areas of each food item, a pre-trained quality estimation model is invoked to obtain the quality estimate value of each food item. The total calorie value is calculated based on the target calorie density and mass estimate of each food.

2. The calorie estimation method according to claim 1, characterized in that, The standard nutrition database includes a calorie mapping table and an inverted index table; each entry in the calorie mapping table reflects the mapping relationship between standard food names and calorie densities; the type description includes food type and ingredient list; The step of finding the target caloric density of a food from a standard nutrition database based on its food name and type description includes: The food name is standardized to obtain the target name; Search the heat mapping table based on the target name; If a matching entry exists in the heat map table, the heat density of the food is obtained from the matching entry; If the matching entry does not exist in the calorie mapping table, the calorie density of the food is determined from the inverted index table and the calorie mapping table based on the target name. If the ingredient list corresponding to the food is empty, then the caloric density of the food shall be used as the target caloric density; If the ingredient list corresponding to the food is not empty, then the caloric density of each ingredient in the ingredient list is found from the caloric mapping table, and the target caloric density is calculated based on the caloric density of the food and the caloric densities of each corresponding ingredient.

3. The calorie estimation method according to claim 2, characterized in that, The inverted index table includes multiple name segments and the index value of each name segment appearing in each entry of the heat map table; The step of determining the caloric density of the food from the inverted index table and the caloric map table based on the target name includes: The target name is segmented to obtain at least two target words; Based on each target word segmentation, the corresponding candidate index value is retrieved from the inverted index table; Based on each candidate index value, the name of each candidate standard food and its caloric density are determined from the caloric mapping table. Calculate the Levenshtein edit distance between the target name and each of the candidate standard food names; The caloric density of the food is determined by the caloric density of the candidate food name with the smallest Levenshtein edit distance.

4. The calorie estimation method according to claim 2, characterized in that, The step of calculating the target calorific density based on the calorific density of the food and the calorific densities of the corresponding ingredients includes: The target calorific density is obtained by calculating the average value of the calorific density of the food and the calorific densities of the corresponding ingredients.

5. The calorie estimation method according to claim 2, characterized in that, The caloric mapping table also includes a weighting coefficient corresponding to each standard food name; the step of calculating the target caloric density based on the caloric density of the food and the caloric densities of the corresponding ingredients includes: The target calorie density is obtained by finding the target name and the weight coefficient corresponding to each ingredient in the calorie mapping table, and by performing a weighted average of the calorie density of the food and the calorie density of each corresponding ingredient based on the found weight coefficients.

6. The calorie estimation method according to claim 1, characterized in that, The step of performing three-dimensional reconstruction processing on the food image to determine the surface area of ​​each food item includes: The instance segmentation model is used to determine the image region containing each type of food from the food image; Using a monocular depth estimation model, the image region containing each food item is converted into a region depth image; Based on the depth images of each region, point cloud data for each type of food is extracted. After reconstructing the point cloud data of each food into a three-dimensional mesh using a three-dimensional reconstruction algorithm, the corresponding surface area is calculated based on each three-dimensional mesh.

7. The calorie estimation method according to claim 1, characterized in that, The food recognition model was trained in the following way: Obtain a first dataset; the first dataset includes a large number of first sample images and annotation data for each first sample image, the annotation data including the food name and type description of at least one food; Freeze the low-level feature extraction layers at the front end of the bottom-level visual encoder and text encoder in the first large language model, and freeze the parameters of all LayerNorm layers in the first large language model. Based on the dataset, a phased unfreezing strategy is used to progressively supervise and fine-tune the first large language model to obtain the food recognition model.

8. The calorie estimation method according to claim 1, characterized in that, The quality estimation model is trained in the following manner: Acquire a second sample image and its label data; the label data includes the food name, three-dimensional surface area, and actual mass of at least one food. The second sample image and its corresponding food name and three-dimensional surface area are filled into a preset prompt word template to obtain inference prompt words; The reasoning prompts are input into the second language model multiple times to obtain multiple structured prediction results, which include the thought process and the quality prediction value of each food item. The inference prompts are used to instruct the second language model to estimate food quality based on the input content and to output the structured prediction results in strict accordance with the specified format requirements. Based on the actual quality of each food item in the labeled data and each of the structured prediction results, the reward value corresponding to each structured prediction result is calculated. Based on each reward value, the GRPO algorithm is used to update the parameters of the second language model to obtain the quality estimation model.

9. The calorie estimation method according to claim 8, characterized in that, The step of calculating the reward value corresponding to each structured prediction result based on the actual quality and each of the structured prediction results includes: For each structured prediction result, the structured prediction result is parsed and its format is validated. If the format validation passes, the format correctness score corresponding to the structured prediction result is determined to be 1; if the format validation fails, the format correctness score is determined to be 0. The actual quality of each food item in the label data and the predicted quality value of each food item in the structured prediction results are input into a preset Gaussian function to obtain the quality accuracy score corresponding to the structured prediction results. The reward value corresponding to the structured prediction result is obtained by weighted summation of the format correctness score and the quality accuracy score.

10. A computer program product, characterized in that, When the computer program product is executed by a processor, it implements the calorie estimation method according to any one of claims 1-9.

Images