Intelligent food nutritional ingredient identification and nutrient intake evaluation method and system

Abstract

The invention discloses an intelligent food nutritional ingredient identification and nutrient intake evaluation method and system, and belongs to the technical field of nutrition evaluation. The method comprises the following steps: S1, constructing databases including a recipe database and a nutrient database; s2, obtaining pre-meal weights of various meals in the meal box during meal serving; s3, obtaining a meal picture before meal, and identifying the type of a recipe based on the recipe database; volume estimation is carried out based on the food pictures, and the approximate volume of each food before the meal is obtained; s4, obtaining a meal picture after the meal, and performing volume estimation based on the meal picture to obtain an approximate volume of each meal after the meal; s5, calculating the actual eating weight of each meal according to the approximate volume ratio of each meal before and after the meal and the weight obtained in S2; and S6, calculating actual intake nutrients based on the nutrient database and the actual edible weight. According to the invention, catering and nutrient intake of a patient can be conveniently and quickly managed, and rehabilitation and health care of the patient are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of nutritional assessment, and specifically relates to an intelligent food nutritional component identification and nutrient intake evaluation method and system. Background Art

[0002] The "2020 Report on the Nutrition and Chronic Disease Status of Chinese Residents" shows that overweight and obesity are becoming increasingly prominent among Chinese residents, and the prevalence of chronic diseases continues to rise. The 2021 "Scientific Research Report on the Dietary Guidelines for Chinese Residents" notes that unhealthy lifestyles remain prevalent among Chinese residents, with an imbalance between energy intake and expenditure. Residents consume a high amount of livestock meat, and the proportion of dietary fat energy supply continues to rise, reaching 34.6% in both urban and rural areas combined, exceeding the recommended upper limit of 30% in rural areas for the first time. Consumption of fruit, beans and soy products, and dairy products remains low, and dietary intake of vitamin A and calcium remains insufficient.

[0003] Numerous academic studies have demonstrated that the clinical application of digital nutritional dietary models improves the nutritional status of hospitalized patients, particularly those with metabolic diseases, and significantly regulates sugar and lipid metabolism. Nutritional interventions tailored to individual diseases can effectively meet nutritional needs during the recovery period. Providing customized nutritional intervention plans and personalized dietary counseling based on the severity of malnutrition can more effectively increase energy and protein intake and improve nutritional status. However, recent editions of the Dietary Guidelines for Chinese Residents and recommended intakes have all referenced data from Europe, the United States, Japan, and South Korea, making it difficult to accurately obtain intake data for large samples of the Chinese population.

[0004] Current methods for conducting dietary surveys in clinical practice include food records, 24-hour dietary recall, food frequency inquiries, and weighing. These traditional survey methods all suffer from issues such as inaccuracy, poor representativeness, the need for specialized training for investigators, and excessive time consumption, making them impractical for large-scale population surveys. Currently, hospital dietary supervision remains manual and conducted through spot checks. This can lead to issues such as changes in canteen menus, inaccurate side dish quantities, and inconsistent portion sizes. Feedback on dining data is typically provided in the form of weighing, which doesn't accurately reflect actual nutrient and energy intake, making it difficult to implement in hospitals with high traffic. The identification result is considered qualified when the similarity between the ingredient type identification data and the standard data in the database reaches 90% or higher; otherwise, the meal is reordered. Summary of the Invention

[0005] The present invention aims to solve one of the technical problems in the above-mentioned related art at least to a certain extent.

[0006] To this end, the purpose of the present invention is to provide an intelligent food nutrient identification and nutrient intake evaluation method and system, which can conveniently and quickly manage the patient's diet and nutrient intake, and facilitate the patient's recovery and health care.

[0007] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: An embodiment of the present invention provides an intelligent method for identifying nutritional components of food and evaluating nutrient intake, the method comprising the following steps: S1. Building a database, wherein the database includes a recipe database and a nutrient database; S2. Obtain the weight of each type of food in the lunch box when the meal is served; S3. Obtaining pre-meal meal images, identifying the recipe types based on a recipe database, and estimating the volume of the meal images to obtain the approximate volume of each pre-meal meal; S4. Obtaining pictures of meals after the meal, estimating the volume based on the pictures of the meals, and obtaining the approximate volume of each meal after the meal; S5. Calculate the actual consumption weight of each meal based on the approximate volume ratio of each meal before and after the meal and the weight obtained in S2; S6. Calculate the actual nutrient intake based on the actual consumption weight and the nutrient database.

[0008] In addition, the intelligent food nutrient component identification and nutrient intake evaluation method according to the present invention may also have the following additional technical features: In some embodiments, the process of building the recipe database includes: Determine appropriate nutrient intake based on the patient's condition and customize meal plans accordingly; Standardize the menu, put the prepared meals into standardized lunch boxes, and take pictures; The meal pictures of all the recipes are stored in a database to form a recipe database.

[0009] In some embodiments, before step S2, the method further includes: Medical staff will provide patients with menu recommendations based on their individual physical signs, disease conditions, existing diet structure and preferences, and activity levels, and will also deliver them to the nutrition canteen at the same time.

[0010] In some embodiments, the method of obtaining the meal image in step S3 and step S4 is to use a depth camera to capture the original RGB-D image.

[0011] In some embodiments, the SE_ResNet50_vd model is used in step S3 to identify the recipe type.

[0012] In some embodiments, the image segmentation in steps S3 and S4 is implemented using SAM; The SAM includes an encoder and a decoder; the encoder includes multiple convolutional layers and pooling layers for extracting image features; the decoder includes multiple deconvolutional layers and upsampling layers for restoring the feature map to the original image size and generating a segmentation result; The loss function of SAM is a multi-task loss function based on cross entropy, including pixel-level classification loss and bounding box-level regression loss.

[0013] In some embodiments, the approximate volume in steps S3 and S4 is obtained by performing image segmentation processing and super-resolution depth estimation on the meal image to obtain high-resolution point cloud data; obtaining three-dimensional spatial data of the dish based on the point cloud data, and obtaining the approximate volume of each meal; Super-resolution depth estimation is implemented using DELTAR.

[0014] In some embodiments, the method further comprises: S7. Send the actual nutrient intake data to medical staff for review and / or confirmation.

[0015] In some embodiments, the specific method of obtaining the weight of each type of meal in step S2 is: using a specific partitioned lunch box with marking points, placing the partitioned lunch box on a smart weighing device with a meal opening, placing meals in each partition in order, and measuring the weight each time a meal is placed, thereby obtaining the weight of each type of meal.

[0016] An embodiment of the present invention further provides an intelligent food nutrient component identification and nutrient intake assessment system, the system being used to implement the steps of any of the above-described intelligent food nutrient component identification and nutrient intake assessment methods; the system comprising: Database module, including recipe database and nutrient database; The weight acquisition module is used to obtain the weight of various meals in the lunch box; Picture acquisition module, used to obtain meal pictures before and after meals; A category recognition module is used to identify the category of recipes in meal images based on a recipe database; The meal size acquisition module is used to perform image segmentation and super-resolution depth estimation on pre- and post-meal meal images to obtain high-resolution point cloud data. Based on the point cloud data, the module then obtains the three-dimensional spatial data of the dish and the approximate volume of the dish. The actual consumption is obtained by multiplying the meal weight with the ratio of the pre- and post-meal volume. The nutrient calculation module is used to calculate the actual nutrient intake based on the actual consumption amount and the nutrient database.

[0017] In addition, the intelligent food nutrient component identification and nutrient intake evaluation system according to the present invention may also have the following additional technical features: In some embodiments, the system further comprises: The medical staff intervention module is used by medical staff to provide diet recommendations to patients based on their individual physical signs, disease conditions, existing diet structure and preferences, and activity levels, and to synchronize these recommendations to the nutrition canteen; and to receive the patient's actual nutrient intake data for review and / or confirmation.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: In the embodiments of the present invention, the intelligent food nutrient component identification and nutrient intake evaluation method provided can conveniently and quickly manage the patient's dining and nutrient intake through intelligent meal identification and consumption calculation, thereby facilitating the patient's recovery and health care. In an embodiment of the present invention, the provided intelligent method for identifying food nutrients and assessing nutrient intake calculates the actual consumed weight of each type of meal by multiplying the weight by the volume ratio before and after the meal. This method is more accurate than calculating by density (meal density can vary significantly depending on factors such as cooking methods and cooking time, resulting in non-fixed density). In the embodiment of the present invention, the provided intelligent food nutrient component identification and nutrient intake evaluation method can ensure the matching of each weight with the lunch box partition during the weighing process due to the presence of lunch box markings and the sequential partitioning of meals; due to the presence of lunch box markings, the matching of each meal type and lunch box partition can be ensured during image processing; thereby ensuring the accurate actual consumption of each type of meal.

[0019] The intelligent food nutrient identification and nutrient intake assessment system of the present invention is capable of implementing the steps of the intelligent food nutrient identification and nutrient intake assessment method, and thus has at least all the features and advantages of the intelligent food nutrient identification and nutrient intake assessment method, which are not further described here. Additional aspects and advantages of the present invention will be described in part in the following description and will become apparent from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of the steps of the intelligent food nutrient identification and nutrient intake evaluation method disclosed in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] The embodiments of the present invention are described in detail below through specific embodiments and application scenarios with reference to the accompanying drawings.

[0023] To address the deficiencies of the existing technology, the present invention aims to complete the standardized production of hospital treatment diets, meal distribution, supervision and feedback of dietary intake through the process management feedback of the intelligent recognition system, and comprehensively understand the actual nutrient intake of hospitalized patients. Relying on intelligent image recognition technology, accurate diet and nutrient surveys are conducted, effectively avoiding the reliance of traditional diet surveys on memory and description ability, eliminating the tedious weighing and measurement, and transferring a large amount of on-site workload to the system, filling the gap in rapid and accurate diet surveys, providing a theoretical basis for further clinical nutritional treatment, simplifying the diagnosis and treatment process, improving clinical diagnosis and treatment efficiency, and providing a theoretical basis for the formulation of national nutritional recommended intakes and corresponding policies.

[0024] See also Figure 1 As shown, in some embodiments of the present invention, an intelligent method for identifying food nutrients and evaluating nutrient intake is provided, which includes the following steps.

[0025] Step 1: Build a database, including a recipe database and a nutrient database; In some embodiments of the present invention, based on the needs of food nutrient identification and nutrient intake evaluation, all types of therapeutic diet recipes in the hospital are determined, the recipe contents and food ingredient ratios for different seasons are formulated, and the recipe standardization data is confirmed.

[0026] Furthermore, the nutrition canteen is used to standardize the recipes. After production, the recipes are placed in the canteen's standardized lunch boxes, and standard pictures of the food are taken. After completion, the photos are sorted and numbered, the ingredients of the standardized food are sorted out, and a recipe database is constructed.

[0027] In some embodiments of the present invention, a database of nutrient content in Chinese foods is established by combining nationally published standard databases such as the "Dietary Reference Intakes for Chinese Residents" and the "Chinese Food Composition Table." This database also takes into account factors such as the edible portion and cooking losses during the cooking process. This database is then mapped and matched with an established recipe database to form a nutrient database for corresponding recipes. Together, the nutrient database and recipe database form the system's identification database.

[0028] Step 2: Medical staff provide patients with recommended diets In some embodiments of the present invention, medical staff issue a detailed and targeted recommended recipe report based on the patient's individual physical signs, disease conditions, dietary structure and preferences, activity level and other specific conditions to guide the patient's diet. At the same time, the recipe information is synchronized to the nutrition canteen, and the nutrition canteen prepares food for each recipe according to standardized processes.

[0029] Step 3, canteen meal distribution and serving; this step collects data from the meal serving process to provide data support for subsequent evaluations; in actual use, through the intelligent recognition system, the hospital's more than 20 therapeutic diets are standardized, cooked, and packaged, and the amount and nutritional components of the meals finally delivered to the patients will be matched and compared with the personal nutrient intake set in the system. Through the intelligent recognition module, each therapeutic diet will be systematically matched before it is served. When the similarity between the identification data of the ingredient type and the standard data in the database reaches more than 90%, the identification result is considered qualified, otherwise the meal will be served again. More specifically: 3.1 Weighing meals as they are served: Zoned lunch boxes are used to hold meals. These are placed on a smart scale, which records weight changes in real time. When a food item is added, the system locks in after the weight stabilizes for three seconds. This weight is recorded as the food item and sent to the system's data processing module by the scale. By using markings on the lunch box and serving meals in the order of their zones, the measured weight can be ensured to match the zones of the lunch box.

[0030] 3.2 Image Acquisition In some embodiments of the present invention, a patient eats and orders food in a nutrition cafeteria, and a depth camera is used to capture raw RGB-D images. The depth camera is placed above the weighing area to automatically capture the lunch box and food.

[0031] This method of setting specific marks on the customized lunch box enables the positioning marks to be combined with the subsequent image processing to accurately segment the various areas of the lunch box, thereby marking each area with a serial number.

[0032] 3.3, Image segmentation, including positioning marking, pre-segmentation and SAM segmentation processing; In some embodiments of the present invention, edge detection and contour analysis techniques in OpenCV are used to process the data image, and combined with positioning marks, the bounding box coordinates of each part of the lunch box are obtained.

[0033] The SAM model (Segment Anything Model) supports segmentation using bounding boxes as hints, a technique called "hinted segmentation." This step uses the output contours from the previous step as input, and the SAM model generates a segmentation mask based on them.

[0034] This allows the precise outline of the food in the lunch box to be obtained.

[0035] SAM is an image segmentation model based on deep learning. In some embodiments of the present invention, its principles mainly include network structure, loss function, data enhancement and pre-training model. Through the comprehensive application of these technical means, SAM can achieve good results in various image segmentation tasks. SAM image segmentation mainly includes the following aspects: Network structure: SAM adopts an encoder-decoder structure similar to U-Net, where the encoder part consists of multiple convolutional layers and pooling layers to extract image features; the decoder part consists of multiple deconvolutional layers and upsampling layers to restore the feature map to the original image size and generate segmentation results; Encoder: Consists of multiple convolutional layers and pooling layers, used to extract image features. Each convolutional layer typically includes operations such as a convolution kernel, an activation function, and batch normalization to extract features and reduce the dimensionality of the input image. The pooling layer is used to downsample the feature map to reduce computational complexity and memory consumption. Decoder: Consists of multiple deconvolution layers and upsampling layers, used to restore the feature map to the original image size and generate segmentation results; each deconvolution layer usually includes operations such as deconvolution kernel, activation function and batch normalization, which are used to upsample and fuse the feature map; the upsampling layer is used to upsample the feature map to restore the original image size; Loss function: SAM uses a multi-task loss function based on cross-entropy, which includes pixel-level classification loss and bounding box-level regression loss; the classification loss is used to measure which category each pixel belongs to (such as foreground or background), and the regression loss is used to adjust the bounding box position of each pixel to better match the target; Data augmentation: To improve the robustness and generalization ability of the model, SAM uses a variety of data augmentation techniques, such as random rotation, scaling, cropping, flipping, color space transformation, and noise addition; Pre-trained model: To accelerate model training and improve segmentation accuracy, SAM usually uses pre-trained image classification models (such as ResNet, VGG, etc.) as the initial weights of the encoder to better extract image features.

[0036] 3.4. Types of Segmented Image Recognition In some embodiments of the present invention, the SE_ResNet50_vd model of the PaddlePaddle industrial-grade open source model library is used to perform image classification and recognition on the collected food images and annotations to confirm the food type.

[0037] In the above implementation, a residual structure is introduced into the SE_ResNet50_vd model, and a ResNet network is constructed by stacking multiple residual structures. Experiments have shown that the use of residual blocks can effectively improve convergence speed and accuracy. ResNet has ResNet structures with 18, 34, 50, 101, 152, and 200 layers, from small to large. The effectiveness of the ResNet series of models has been verified in different application scenarios, such as classification, detection, and segmentation. This series of models has very obvious advantages in speed and accuracy. Among the ResNet series of models, compared with other models, the ResNet_vd model has a very significant improvement in accuracy while the prediction speed remains almost unchanged.

[0038] 3.5 Deep Image Processing DELTAR Super-resolution Processing In some embodiments of the present invention, super-resolution depth estimation is performed on the collected original images (including RGB images and depth images) through the DELTAR deep learning model to obtain high-resolution point cloud data.

[0039] 3.6 3D Reconstruction Volume Calculation In some embodiments of the present invention, based on point cloud data, a convex hull algorithm is used to calculate the mapping matrix T between the RGB image and the depth image (derived from the point cloud data). The depth data is then mapped to the segmented areas of the dish, thereby obtaining pixel-level height information for the dish. Subsequently, the acquired three-dimensional spatial data of the dish is voxelized to perform a 3D reconstruction of the dish, and the approximate volume V is obtained by calculating the number of voxels within the dish.

[0040] Similarly, the weight, type, and volume information of other foods can be identified to obtain information about all foods.

[0041] Step 4: Patient post-meal information is sent back In some embodiments of the present invention, after the patient finishes eating, the depth camera is called to take a photo of the lunch box after the meal, and the collected RGB-D image data is sent back to the server for processing.

[0042] Furthermore, repeat steps 3.3, 3.5, and 3.6 to obtain the remaining volume of each food after the meal. Calculate the volume ratio of the remaining volume to the volume before the meal, and then multiply this volume ratio by the previously measured weight of the meal to obtain the actual consumption of the meal.

[0043] From this, the actual amount of each food consumed can be calculated.

[0044] Step 5: Nutritional Calculation In some embodiments of the present invention, pictures of dishes before and after the meal are compared, and the volume reduction ratio is calculated using the analyzed data to estimate the actual weight of food ingested.

[0045] Then, match the recipe database constructed in the previous step 1, comprehensively calculate the actual specific food types and weights of the patient's meal, and automatically calculate the nutrient intake of each type of food for the patient's meal based on the nutrient data of each food and recipe in the nutrient database, including energy, protein, fat, carbohydrates, vitamins, minerals, dietary fiber, etc.

[0046] Step 6: Issue a report In some embodiments of the present invention, medical staff can view basic dining data through the system, confirm the conditions before and after meals, verify the accuracy of the system feedback data, and the system provides medical staff with a capability portal.

[0047] Furthermore, after the medical staff has completed the confirmation, the system will automatically generate a guidance report according to the preset rules and push it to the corresponding medical staff for review, providing data support for subsequent clinical nutritional treatment. At the same time, it can be previewed and the report can be printed for delivery to the patient.

[0048] For the parts not described in detail in the present invention, including the convex hull algorithm, 3D reconstruction and volume estimation, reference may be made to the existing technology in the field. Other parts not described in detail are well-known technologies to those skilled in the art and are not limited in this embodiment and will not be described in detail here.

[0049] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. An intelligent method for identifying food nutrients and evaluating nutrient intake, characterized in that: The steps of the method include: S1. Building a database, wherein the database includes a recipe database and a nutrient database; S2. Obtain the weight of each type of food in the lunch box when the meal is served; S3. Obtaining pre-meal meal images, identifying the recipe types based on a recipe database, and estimating the volume of the meal images to obtain the approximate volume of each pre-meal meal; S4. Obtaining pictures of meals after the meal, estimating the volume based on the pictures of the meals, and obtaining the approximate volume of each meal after the meal; S5. Calculate the actual consumption weight of each meal based on the approximate volume ratio of each meal before and after the meal and the weight obtained in S2; S6. Calculate the actual nutrient intake based on the actual consumption weight and the nutrient database.

2. The intelligent food nutrient component identification and nutrient intake evaluation method according to claim 1, characterized in that: The recipe database construction process includes: Determine appropriate nutrient intake based on the patient's condition and customize meal plans accordingly; Standardize the menu, put the prepared meals into standardized lunch boxes, and take pictures; The meal pictures of all the recipes are stored in a database to form a recipe database.

3. The intelligent food nutrient component identification and nutrient intake evaluation method according to claim 1, characterized in that: Before step S2, the method further includes: Medical staff will provide patients with menu recommendations based on their individual physical signs, disease conditions, existing diet structure and preferences, and activity levels, and will also deliver them to the nutrition canteen at the same time.

4. The intelligent food nutrient component identification and nutrient intake evaluation method according to claim 1, characterized in that: The method of obtaining the meal pictures in step S3 and step S4 is to use a depth camera to take pictures and collect original RGB-D images.

5. The intelligent food nutrient component identification and nutrient intake evaluation method according to claim 1, characterized in that: In step S3, the SE_ResNet50_vd model is used to identify the recipe type.

6. The intelligent food nutrient component identification and nutrient intake evaluation method according to claim 1, characterized in that: Image segmentation in steps S3 and S4 is implemented using SAM; The SAM includes an encoder and a decoder; the encoder includes multiple convolutional layers and pooling layers for extracting image features; the decoder includes multiple deconvolutional layers and upsampling layers for restoring the feature map to the original image size and generating a segmentation result; The loss function of SAM is a multi-task loss function based on cross entropy, including pixel-level classification loss and bounding box-level regression loss.

7. The intelligent food nutrient component identification and nutrient intake evaluation method according to claim 1, characterized in that: The approximate volume in steps S3 and S4 is obtained by performing image segmentation and super-resolution depth estimation on the meal image to obtain high-resolution point cloud data; obtaining three-dimensional spatial data of the dish based on the point cloud data, and obtaining the approximate volume of each meal; Super-resolution depth estimation is implemented using DELTAR.

8. The intelligent food nutrient component identification and nutrient intake evaluation method according to claim 1, characterized in that: The method further comprises: S7. Send the actual nutrient intake data to medical staff for review and / or confirmation.

9. An intelligent food nutrient identification and nutrient intake evaluation system, characterized in that: The system is used to implement the steps of the intelligent food nutrient component identification and nutrient intake evaluation method according to any one of claims 1 to 8; the system comprises: Database module, including recipe database and nutrient database; The weight acquisition module is used to obtain the weight of various meals in the lunch box; Picture acquisition module, used to obtain meal pictures before and after meals; A category recognition module is used to identify the category of recipes in meal images based on a recipe database; The meal size acquisition module is used to perform image segmentation and super-resolution depth estimation on pre- and post-meal meal images to obtain high-resolution point cloud data. Based on the point cloud data, the module then obtains the three-dimensional spatial data of the dish and the approximate volume of the dish. The actual consumption is obtained by multiplying the meal weight with the ratio of the pre- and post-meal volume. The nutrient calculation module is used to calculate the actual nutrient intake based on the actual consumption amount and the nutrient database.

10. The intelligent food nutrient component identification and nutrient intake evaluation system according to claim 9, characterized in that: The system further comprises: The medical staff intervention module is used by medical staff to provide diet recommendations to patients based on their individual physical signs, disease conditions, existing diet structure and preferences, and activity levels, and to synchronize these recommendations to the nutrition canteen; and to receive the patient's actual nutrient intake data for review and / or confirmation.

Images