Intelligent food recognition and nutrition analysis method based on deep learning

Abstract

The application discloses a method for intelligent food recognition and nutrition analysis based on deep learning, and belongs to the technical field of image analysis.The method comprises the following steps: 1, collecting input data and pre-processing the data; 2, determining the position of a target in a scene and a mask area, and generating a dense depth map; 3, constructing a food three-dimensional point cloud, denoising the three-dimensional point cloud, and removing abnormal points; 4, obtaining three-dimensional features, extracting two-dimensional features from the image, and performing weight fusion; 5, splicing the obtained features, and strengthening the feature information after fusion; 6, regressing and predicting food weight and heat; and 7, optimizing model prediction and verifying the prediction effect.The method adopts a 2D+3D modal feature fusion mode, performs data enhancement on an original image, extracts 2D features, simultaneously generates a 3D point cloud, and performs multi-modal feature fusion to predict food heat and weight, so that the method can help consumers accurately estimate food heat, weight and other indexes, and formulate scientific diet methods and suggestions.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image analysis, and particularly relates to an intelligent food recognition and nutrition analysis method based on deep learning. BACKGROUND

[0002] In the field of food image analysis, traditional methods mainly focus on classification, detection, segmentation and recognition of two-dimensional images. With the development of deep learning technology, especially the wide application of convolutional neural networks, the accuracy and efficiency of image analysis have been significantly improved. However, food images not only contain rich visual information such as color, shape and texture, but also have physical properties such as volume, weight and calories, which often require more complex modeling methods. In order to effectively extract and predict physical properties such as volume, weight and calories from food images, traditional two-dimensional image processing methods may not provide sufficient spatial information. Therefore, in recent years, multi-modal learning methods based on depth estimation, three-dimensional point cloud generation and feature fusion have become a research hotspot.

[0003] Depth estimation technology is to infer the distance between each pixel and the camera from the 2D image, thereby generating a depth map. The depth map provides a dimension related to the spatial geometry of the objects in the image. These depth information can be generated by depth estimation algorithms. 3D point cloud is a data structure composed of multiple points with spatial coordinates, which can effectively describe the three-dimensional shape of the object. Through the depth map, each pixel in the two-dimensional image can be mapped to the three-dimensional space to obtain a set of point cloud data, and further extract the geometric features of the object. For two-dimensional images, using convolutional neural networks is the most common feature extraction method. CNN can learn local features such as edges, textures, colors, and global features in images. However, relying solely on the features of two-dimensional images cannot fully reflect the three-dimensional structure of food, so it is necessary to combine other information such as depth maps and point clouds to improve the prediction ability of the model. Traditional image feature extraction methods are insufficient to handle the three-dimensional information of food images. With the gradual popularization of 3D data, researchers have proposed various point cloud processing networks such as PointNet and PointNet++. These networks can process the spatial position of each point in the point cloud data and effectively extract the three-dimensional features of the food object, such as volume, shape, and surface roughness. In practical applications, multi-modal learning methods that combine 2D and 3D features are increasingly attracting attention. By fusing features from two-dimensional images and three-dimensional point clouds, the appearance and morphology of food, especially complex-shaped food, can be better captured. For example, by adding attention mechanisms or multi-scale feature learning methods to the model, the network can automatically select the most relevant 2D and 3D features for fusion in different tasks.

[0004] For food images, feature fusion techniques can combine the visual features of two-dimensional images with three-dimensional spatial information. Through feature fusion, the model's understanding of the appearance and geometry of objects can be enhanced, thereby improving the prediction ability of physical properties such as food volume, weight, and calories. After feature extraction and fusion, the next task is to predict physical properties such as volume, weight, and calories. Typically, a regression model such as a multi-layer perceptron (MLP) will input the fused features into a neural network, trained using a regression loss function such as mean squared error (MSE), and finally output the predicted value of the physical property.

[0005] This method has wide application potential in many fields, helping consumers estimate the calories, nutrients, volume, and weight of food, and thus developing a scientific diet plan. It can automatically estimate the amount and weight of food during production, improving production efficiency. Through intelligent devices for automatic detection and analysis of food, doctors and nutritionists can provide personalized dietary recommendations for patients. With the continuous advancement of deep learning and computer vision technology, multi-modal learning methods combining depth estimation, 3D point cloud generation, feature extraction, and regression prediction will become the mainstream of research. These methods will be increasingly applied to food image analysis, intelligent nutrition management, and health monitoring, driving the development of intelligent catering, food safety, and health management. SUMMARY

[0006] To solve the above problems in the prior art, the purpose of the present application is to provide an intelligent food recognition and nutrition analysis method based on deep learning.

[0007] The present application provides the following technical solution: an intelligent food recognition and nutrition analysis method based on deep learning, comprising the following steps:

[0008] S1, collect scene images as original input data, and pre-process the collected image data;

[0009] S2, determine the position and mask area of the target in the scene, and generate a dense depth map of the scene;

[0010] S3, construct a food three-dimensional point cloud, denoise the three-dimensional point cloud, and remove abnormal points;

[0011] S4, obtain three-dimensional features representing the spatial structure of the food, and use different CNN models to extract two-dimensional features from the image and perform weight fusion to obtain fused two-dimensional features;

[0012] S5, concatenate the obtained three-dimensional features and two-dimensional fused features, and use a self-attention mechanism to strengthen the feature information after fusion;

[0013] S6, based on the fused feature information, using a multi-layer perceptron model regression to predict food weight and calories;

[0014] S7, the model prediction is optimized and the prediction effect is verified.

[0015] Further, the specific process of S1 is as follows:

[0016] The scene image is collected by an RGB camera as input data, ensuring that the image quality is clear, the light is uniform, and the background is simple, and the obtained image is stored in an image database. The image processing tool is used for Gaussian filter denoising processing of the photographed image; the image size is uniformly adjusted to the model input size, and normalized processing is performed to make the pixel value in a unified range.

[0017] Further, the specific process of S2 is as follows:

[0018] S2.1, input the preprocessed image into the Mask R-CNN model, identify the food category in the image and determine the specific position and accurate code area of the target;

[0019] S2.2, adopt the MiDaS depth estimation model, predict the relative depth value of each pixel of the image through the depth convolutional neural network, and generate a dense depth map of the scene.

[0020] Further, the specific process of S3 is as follows:

[0021] Based on the mask determined in S2 and the generated depth map, the three-dimensional space coordinates of each pixel of the target are calculated using the camera internal parameters in the intelligent device, the food three-dimensional point cloud is constructed, and the constructed three-dimensional point cloud is denoised. Wavelet transform is used to remove abnormal points and improve the quality of point cloud data.

[0022] Further, the specific process of S4 is as follows:

[0023] S4.1, adopt the PointNet model to extract features from the denoised three-dimensional point cloud, and obtain three-dimensional features that can accurately represent the spatial structure of the food;

[0024] S4.2, the original image data is enhanced to improve the diversity of the data and the generalization ability of the model;

[0025] S4.3, different CNN models are used to extract two-dimensional features from the enhanced image, and the extracted different model features are fused to obtain fused two-dimensional features.

[0026] Further, the specific process of S5 is as follows:

[0027] The fusion features obtained in S4 are taken as input, a food database is combined, training data samples are constructed, real food weight and heat are taken as labels, the training data samples are input into a multilayer perception MLP, and food weight and heat are predicted by regression.

[0028] By adopting the above-mentioned technology, compared with the prior art, the beneficial effects of the present application are as follows:

[0029] The present application adopts a feature fusion method of 2D+3D two modalities, performs data enhancement on the original image, extracts 2D features by using different deep learning models, generates 3D point cloud by using depth estimation and mask image, and finally performs multi-modal feature fusion of 2D and 3D to predict food heat and weight. Through the method, the consumer can accurately estimate the indicators such as food heat and weight, so as to formulate a scientific diet method and suggestion. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The algorithm flowchart of the present application is shown in the figure;

[0031] Figure 2 The flowchart of the present application for online identification of food by mobile phone and analysis is shown in the figure;

[0032] Figure 3 The flowchart of the online and offline system of the present application is shown in the figure. DETAILED DESCRIPTION

[0033] The present application provides an intelligent food recognition and nutrition analysis method based on deep learning, and the algorithm flowchart is shown in the figure. Figure 1 Figure 2 The flowchart of the present application for online identification of food by mobile phone and analysis is shown in the figure, Figure 3 The flowchart of the online and offline system of the present application is shown in the figure. The specific implementation includes the following detailed steps:

[0034] Step 1. Obtain the original image data, collect the scene image by an RGB camera as input data, ensure that the image quality is clear, the light is uniform, and the background is simple, and store the obtained image in an image database, a total of 513 images of 11 different angles are collected.

[0035] Step 2. Use image processing tools to perform Gaussian filter noise reduction processing on the photographed image; uniformly adjust the image size to the model input size, and perform normalization processing to make the pixel value in a unified range.

[0036] Step 3. Target detection and instance segmentation, input the preprocessed image into the Mask R-CNN model, identify the food category in the image and determine the specific position and accurate mask area of the target, and the formula (1) is:

[0037] ​ (1)

[0038] wherein is the classification loss for identifying the target class; is the bounding box regression loss for determining the specific location of the target; is the mask prediction loss for determining the precise contour of the target.

[0039] Step 4. Adopt the MiDaS depth estimation model to predict the relative depth value of each pixel in the image through a deep convolutional neural network, and generate a dense depth map of the scene. Formula (2) is:

[0040] (2)

[0041] wherein , represents the depth value predicted by the network at the i,jth pixel, , represents the true depth value at the i,jth pixel.

[0042] Step 5. Based on the mask of step three and the depth map and corresponding depth value Z obtained in step four, then determine the ray direction of each pixel in the camera coordinate system with the help of the intrinsic parameters of the RGB camera as formula (3), finally multiply the direction vector with the corresponding depth value, then the two-dimensional pixel points can be stretched to the actual three-dimensional space, and then the spatial point set of the food surface is obtained; gathering these points constitutes the original three-dimensional point cloud of the food. Formula (3) is:

[0043] ;

[0044] ;

[0045] (3)

[0046] wherein x, y represent the pixel coordinates in the original image, C x、 C y modifying the horizontal and vertical coordinates of the image center in pixel coordinates, f x、 f y respectively represent the equivalent focal length of the camera in the x direction and the y direction, X and Y are the image plane coordinates after removing the principal point offset and unifying the horizontal and vertical focal length differences, and Z is the depth value.

[0047] Step 6. Denoise the 3D point cloud generated in Step 4 by using wavelet transform to remove outliers and improve the quality of the point cloud data. The processed point cloud is represented by formula (4):

[0048] (4)

[0049] in The original 3D point cloud is given, where W is the wavelet transform function and T is the threshold function. This is the inverse wavelet transform function.

[0050] To evaluate the quality of the generated 3D point cloud, the error between the actual volume label and the actual volume label is selected as the evaluation index (see Formula (5). It is compared with the traditional 3D point cloud generation method, stereo vision, and the deep learning method, U-Net, as shown in Table 1. The results show that the 3D point cloud obtained by Mask-RCNN and depth estimation Midas has the best effect.

[0051] (5)

[0052] in To predict volume, This represents the actual volume;

[0053] Table 1. Quality assessment of point clouds generated by different methods

[0054] .

[0055] Step 7. Use the PointNet model to extract features from the denoised 3D point cloud to obtain 3D features that accurately represent the spatial structure of the food. The experimental parameters are set as shown in Table 2. The Point aggregation feature is given by formula (6): (6)

[0056] in This indicates the midpoint of the point cloud. Local feature extraction function, Represents the characteristic transformation function;

[0057] Table 2 PointNet Model Training Parameter Settings

[0058] .

[0059] Step 8. Perform data augmentation on the original image, such as rotation, translation, and scaling, to increase data diversity and improve the model's generalization ability.

[0060] Step 9. Due to the large image dataset, three different CNN models, EfficientNet-B0, ResNet18, and MobileNetV2, were used to extract two-dimensional features from the enhanced images. The experimental parameters are shown in Table 3. The features extracted from the different models in Step 8 were weighted and fused to obtain fused two-dimensional features. The feature fusion formula is (7):

[0061] (7)

[0062] in , , The weights for feature fusion in each model represent the following:

[0063] ;

[0064] Table 3 CNN Model Training Parameter Settings

[0065] .

[0066] To compare the impact of different models on the image dataset, a control experiment was conducted. The image dataset was divided into training and validation sets in an 8:2 ratio, and the last fully connected layer of the model was changed to a linear layer for regression. The evaluation metrics were RMSE, MAE, and MAPE, and their calculation formulas were (8)-(10). The experimental results are shown in Tables 2 and 3. Table 4 shows the weight prediction comparison of different models, and Table 5 shows the heat prediction comparison of different models. In actual prediction, the EfficientNet-B0, MobileNetV2, and ResNet18 networks were used to extract multi-level visual features. Then, global average pooling was used at the end of each model to compress the spatial features into a fixed-length vector, and then a fully connected layer was connected to output a continuous heat value. The results show that the model with average weighted fusion achieved more accurate predictions of weight and volume.

[0067] (8)

[0068] (9)

[0069] (10)

[0070] in , , where n represents the true value, the predicted value, and the total number of samples;

[0071] Table 4 Comparison of weight predictions for different models on the validation set

[0072] ;

[0073] Table 5 Comparison of heat predictions by different models on the validation set

[0074] .

[0075] Step 10. The 3D features extracted in Step 7 and the 2D fusion features extracted in Step 9 are concatenated, and a self-attention mechanism is used to enhance the fused feature information. The attention mechanism is formula (11):

[0076] (11)

[0077] Where Q, K, and V represent the query vector, key vector, and value vector, respectively. For vector dimensions.

[0078] Step 11. Using the fusion features from Step 10 as input, and combining them with the existing food database, construct training data samples, using real food weight and calories as labels. Input the constructed training data samples into a multilayer perceptron (MLP) to regress and predict food weight and calories. The prediction formula (12) is:

[0079] (12)

[0080] Step 12. Compare the weight and calories predicted by the model with the actual labels. When the prediction error exceeds the set threshold (e.g., ±10%), recollect these samples and add them to the training data to improve the model accuracy.

[0081] Step 13. In order to analyze the role of the grouping method in the food prediction task, ablation experiments were conducted on different modal methods for comparison, as shown in Table 6. The results show that the same method is better than the calorie prediction in terms of food weight, and the multimodal method is better than the single modal method in terms of overall performance.

[0082] Table 6 Comparison of Multimodal and Single-Modal Methods

[0083] .

[0084] Step 14. Deploy the trained model weights to the smartphone. Specifically: Export the model file to the universal ONNX format; then compress it into a .ptlite file using the official conversion tool ONNX-TFLite, enabling quantization during conversion to significantly reduce file size and accelerate inference; then copy this lightweight model file into the assets directory of the Android project and import the corresponding inference library in the Gradle configuration; finally, instantiate the lightweight inferencer in the code to load the model file, thus accurately deploying it to the smartphone. The above steps are the offline stage. When the user takes a new image, the convolutional network model in the offline stage is large. To reduce the computational burden, a single CNN model is used to extract two-dimensional features. The captured image will be automatically input into the model already deployed on the phone after preprocessing steps 2 to 11 for prediction, automatically estimating the real-time display weight and heat, such as... Figure 2 The stage where users take photos and re-upload them for prediction is called the online stage, such as... Figure 3 .

Claims

1. A method for intelligent food identification and nutritional analysis based on deep learning, characterized in that, Includes the following steps: S1. Acquire scene images as raw input data and preprocess the acquired image data; S2. Determine the location and mask region of the target in the scene, and generate a dense depth map of the scene; the specific process of S2 is as follows: S2.1 Input the preprocessed image into the Mask R-CNN model to identify the food categories in the image and determine the specific location and precise code region of the target; S2.

2. Using the MiDaS depth estimation model, a deep convolutional neural network is used to predict the relative depth value of each pixel in the image, generating a dense depth map of the scene. S3. Construct a 3D point cloud of the food, denoise the 3D point cloud, and remove outliers; S4. Obtain the three-dimensional features representing the spatial structure of food, and use different CNN models to extract two-dimensional features from the image and perform weight fusion to obtain fused two-dimensional features; S5. The obtained three-dimensional features and two-dimensional fused features are spliced ​​together, and a self-attention mechanism is used to enhance the fused feature information. S6. Based on the fused feature information, regress to predict food weight and calories; S7. Optimize the model predictions and verify the prediction results.

2. The method for intelligent food identification and nutritional analysis based on deep learning according to claim 1, characterized in that, The specific process of S1 is as follows: The scene image is captured by an RGB camera as input data, and the acquired image is stored in an image database. The captured image is then processed by Gaussian filtering for noise reduction using image processing tools. The image size is adjusted to match the model input size and then normalized to ensure that the pixel values ​​are within a uniform range.

3. The method for intelligent food identification and nutritional analysis based on deep learning according to claim 1, characterized in that, The specific process of S3 is as follows: Based on the mask determined in S2 and the generated depth map, the three-dimensional spatial coordinates of each pixel of the target are calculated using the camera intrinsic parameters in the smart device, a three-dimensional point cloud of food is constructed, and the constructed three-dimensional point cloud is denoised by wavelet transform to remove outliers and improve the quality of point cloud data.

4. The method for intelligent food identification and nutritional analysis based on deep learning according to claim 3, characterized in that, The specific process of S4 is as follows: S4.

1. The PointNet model is used to extract features from the denoised 3D point cloud to obtain 3D features that can accurately represent the spatial structure of food. S4.2 Enhance the original image data to improve data diversity and the model's generalization ability; S4.

3. Use different CNN models to extract two-dimensional features from the enhanced image, and then perform weighted fusion of the extracted features from different models to obtain fused two-dimensional features.

5. The method for intelligent food identification and nutritional analysis based on deep learning according to claim 4, characterized in that, The specific process of S5 is as follows: Using the fusion features obtained in S4 as input, combined with the existing food database, training data samples are constructed. The actual food weight and calories are used as labels. The constructed training data samples are then input into a multilayer perceptron (MLP) to regress and predict food weight and calories.

Images