Weight prediction system based on multi-modal food data

Abstract

The present application belongs to the technical field of health management, and specifically relates to a weight prediction system based on multi-modal food data. The weight prediction system comprises a unified dietary feature learning module based on CLIP, a time series prediction model, and a dietary-weight loss calculation module. The present application explores the influence of dietary intake on weight prediction, and predicts the future weight change trend by using the weight change in a short period and the dietary record. The present application firstly extracts the unified features of historical food data through the dietary feature learning module, and then splices the features and the historical weight change sequence to serve as the input of the time series prediction model; the dietary feature learning module and the time series prediction model are trained through dietary-weight loss to establish the connection between dietary intake and weight change. The present application system is high in accuracy and strong in adaptability for weight prediction, and can be applied in the fields of health and medical treatment to conduct personalized health management.

Description

Technical Field

[0001] The present invention belongs to the field of health management technology, and in particular relates to a weight prediction system based on multimodal food data. Background Art

[0002] Food plays a vital role in human life, affecting quality of life and various physiological indicators, of which weight is one of the most basic and critical indicators. Literature [1] shows that the relative balance between energy expenditure and dietary intake determines weight gain or loss. It is worth noting that energy expenditure from non-exercise activities (such as sitting) usually accounts for a large part of total energy expenditure, far exceeding exercise expenditure [2]. Based on the above understanding, it is important and meaningful to study the impact of food intake on weight prediction.

[0003] Many studies have focused on various traditional tasks in the food domain, such as food classification [3, 4, 5], ingredient recognition [6, 7], recipe retrieval [8, 9], food volume prediction

[10] , and calorie and nutritional information estimation

[11] . However, these existing methods usually analyze each food image independently and do not study the impact of food intake on physiological indicators (such as weight) over a period of time. Another research direction is to predict the nutritional composition and calories of food from food images, which can help monitor intake patterns. Some studies first estimate the volume, such as voxels, point clouds, or 3D meshes, and then map it with the calorie data contained per unit volume. With the release of the Nutrition5K dataset

[12] , which provides fine-grained nutritional attributes, food quality, and food videos, some studies have predicted nutrition directly from images using neural networks. However, these studies only predict nutritional information and do not further link food intake with weight.

[0004] However, there are two major challenges in using food intake for weight prediction. The first is that food information can be expressed in a variety of forms, such as textual ingredient labels and food pictures; the second is how to model the relationship between food representation and weight prediction. To address the above challenges, this paper proposes a system for weight prediction based on multimodal food data. Specifically, this method introduces a diet feature learning module that uses the CLIP

[13] text or image encoder to extract a unified feature representation from various forms of historical food intake. In addition, we propose a diet-weight loss that enables the model to capture the correlation and dependency between food and weight changes. Importantly, the framework design of this method is compatible with any existing time series prediction model. Summary of the Invention

[0005] The object of the present invention is to provide a weight prediction system based on multimodal food data with high accuracy and strong adaptability.

[0006] The weight prediction system based on multi-modal food data provided by the present application is based on computer vision and time series prediction technology, and includes the following modules:

[0007] (1) A unified dietary feature learning module based on a CLIP model (hereinafter referred to as a dietary feature learning module). The module is used to obtain the dietary features of a meal from food pictures or food ingredient annotations; the dietary feature learning module includes a pre-trained CLIP encoder and a multi-layer perceptron; first, the multiple pictures or multiple ingredient texts of a meal are input into the CLIP model to obtain the features corresponding to each picture or each text, and then all the features are averaged and input into the multi-layer perceptron to further fuse and extract the final dietary features.

[0008] During the training process, the parameters of the CLIP model are fixed; the CLIP model is used as an encoder, and the purpose is to unify the feature extraction process of vision and text; this enables the present application to obtain the text form of the ingredient prediction result from the food picture using any ingredient prediction model, and input it as food information.

[0009] Considering that uploading pictures is more convenient for users than inputting ingredient texts, the module also supports predicting the ingredient text from the picture using any ingredient prediction model, and taking the predicted ingredient text as input. That is, the ingredient text can come from the user or the prediction result of the picture. The ingredient prediction model can use any existing model.

[0010] (2) A model-agnostic time series prediction model (hereinafter referred to as a time series prediction model). The time series prediction model is used to predict future time series from historical time series to obtain prediction results.

[0011] Specifically, after obtaining the dietary features of multiple days, the body weight values of the corresponding days are spliced and input into the time series prediction model to obtain the prediction results of the future multiple days.

[0012] The model can specifically use existing time series prediction models, such as NLnear

[14] with a linear structure, or iTransformer

[16] with a Transformer

[15] structure, etc.

[0013] (3) A diet-weight loss calculation module.

[0014] The diet-weight loss calculation module is used to train the dietary feature learning module and the time series prediction model; during the training process, the CLIP model is fixed, and the loss during the model training process is calculated, so that the model learns the relationship between the diet and the body weight change, thereby performing accurate weight prediction.

[0015] The application predicts the body weight value in the future T days based on the food intake and body weight value in the past L days. First, the diet feature learning module is used to extract the features of each meal, and the features are spliced with the body weight sequence as the input of the time series prediction model; during the training process, the CLIP model is fixed, and the diet-body weight loss is used to train the diet feature learning module and the time series prediction model, so that the model learns the relationship between meals and body weight changes, thereby accurately predicting the body weight.

[0016] L-T is expressed as a combination of different historical and future day arrays, L-T can be 3-3, 3-5, 3-7, 5-5, 5-7, 7-3 and 7-7, etc.

[0017] Further, the application provides a body weight prediction system based on multi-modal food data, wherein:

[0018] (I) The process of extracting food features by the diet feature learning module is:

[0019] Suppose the food information of the past L days is:

[0020]

[0021] Among them, Breakfast t , Lunch t and Supper t are the food information of the breakfast, lunch and dinner of the tth day; if the food information is food pictures, the food information of the breakfast, lunch and dinner can be respectively represented as:

[0022]

[0023]

[0024]

[0025] Among them, N b , N l and N s are the number of pictures of the three meals; if the food information is the raw material text, the food information of the breakfast, lunch and dinner can be respectively represented as:

[0026]

[0027]

[0028]

[0029] Among them, N b , N l and N s are the number of raw material texts of the three meals.

[0030] For food information input, the format of three meals needs to be consistent (all pictures or all text). Taking breakfast food pictures as an example, the specific process of extracting the food features of a meal is as follows:

[0031] (1) The Breakfast t input CLIP model

[13] to get the feature E n of each picture, which is represented as:

[0032] E n = CLIPEncoder(img n ), (1)

[0033] (2) Average the features of N b pictures:

[0034]

[0035] (3) Input the averaged feature into a multi-layer perceptron (MLP) to get the breakfast feature b t of the t-th day:

[0036]

[0037] Similarly, the lunch feature l t and the dinner feature s t of the t-th day can be obtained.

[0038] Similarly, the features of the three meals of the t-th day with text as input information can be obtained.

[0039] (II) The time series prediction module predicts the body weight and body weight change in the future period, and the specific process is as follows:

[0040] (1) Assume that the historical L-day body weight sequence is where w t represents the body weight value of the t-th day. Concatenate the food features and body weight of the historical L days to get the input of the time series model, which is represented as:

[0041]

[0042] (2) Input X into the time series prediction model M to get the output which is represented as:

[0043]

[0044] where and are the predicted body weight values from breakfast, lunch, dinner, and body weight, and M is the time series prediction model.

[0045] (Three) the diet-weight loss calculation module is used for modeling the relationship between body weight and food intake, and specifically comprises:

[0046] (1) Calculate the diet loss L diet :

[0047]

[0048] Where, Δ t is the weight change value on the t-th day: Δ t = w t -w t-1 ;

[0049] (2) Calculate the weight loss L weight :

[0050]

[0051] (3) Combine the diet loss and the weight loss to form the diet-weight loss:

[0052] L = λL weight +(1-λ)L diet , (8)

[0053] Wherein, λ is a hyperparameter, λ∈[0, 1].

[0054] The weight prediction system based on multi-modal food data provided by the application has the following specific test process:

[0055] (1) Take the weight and food data of L days as input to obtain the weight predicted by the framework;

[0056] (2) Take the weight prediction value obtained in step (1) as input to predict future weight;

[0057] (3) Repeat step (2);

[0058] (4) Calculate the mean square error of all prediction results and real weight values to evaluate the framework effect.

[0059] The present application firstly utilizes food intake to predict weight change trend, and proposes an effective weight prediction system. In the system, the diet feature learning module can uniformly encode food information of multiple modalities, effectively improving the weight prediction accuracy; through the design of a new loss function, the relationship between food intake and weight change is modeled; different time series prediction models are used as prediction models, which have strong compatibility. The use of food intake information improves the accuracy of weight prediction. The present application can be applied in the fields of health and medical care, and can be used for personalized health management; it can also help fitness enthusiasts to provide weight management and prediction services, so as to achieve the goal of weight gain or loss. In the diet and nutrition industry, the present application can help to predict the weight of customers and provide personalized diet and nutrition plans. Compared with not adding food intake information, the prediction accuracy and trend similarity of the present application are greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is the structure diagram of the weight prediction system based on multi-modal food data of the present application.

[0061] Figure 2 is the structural diagram of the diet feature learning module proposed by the present application.

[0062] Figure 3 is the working flow chart of the weight prediction system based on multi-modal food data of the present application. The three meal information and weight of the past L days are input. The three meal information first enters the diet feature learning module shown in Figure 2 , to obtain the feature of each meal, then all the three meal features and weight values are spliced into the time series prediction model to obtain the weight prediction result of the future T days. The diet-weight loss between the prediction result and the real weight is calculated to obtain the loss value, which is used for training the whole model.

[0063] Figure 4 is the prediction visualization of three users when L=3, T=3, the time series prediction model is Nlinear, and the food information is picture. DETAILED DESCRIPTION

[0064] The present application will be further described below by specific embodiments in conjunction with the drawings.

[0065] The present application uses dietary intake information to assist in weight prediction. A dietary feature learning module is used, which can input images or text; the dietary feature learning module extracts unified food features for different input modalities and is used for training. One meal information is first input to the CLIP module of the dietary feature learning module to obtain the features of each picture / each raw material text, then the features of multiple pictures / multiple raw material texts are averaged and input into a multilayer perceptron to obtain the final dietary features. The present application needs to collect a large amount of user food diet pictures and text annotations, and corresponding weight change values to construct a data set, and use the data set for training and testing. Assuming that a prediction system is trained to predict the weight of the next 3 days using 3-day weight and intake, the specific steps are as follows:

[0066] Step 1. Use the dietary feature learning module as a feature extractor to extract food features, and the specific process is as follows:

[0067] (1) Sample food intake and weight change of a plurality of users for 6 days as training data. The three meal pictures on the t-th day can be represented as:

[0068]

[0069]

[0070]

[0071] wherein N b , N l and N s are the number of three meal pictures. Each meal picture is input into the CLIP model of the dietary feature learning module to obtain the features of all pictures of the three meals. The feature of each picture is 512-dimensional, and the features of the three meals are N b ×512, N l ×512, N s ×512.

[0072] (2) Average the multiple features obtained for each meal to 1x512 dimensions;

[0073] (3) Input the averaged features into a multilayer perceptron to obtain the food features of each day of the past 3 days wherein b, 1, s represent breakfast, lunch and dinner, respectively;

[0074] (4) Concatenate the food features obtained in process (3) and the daily weight of the corresponding time period to obtain the input of the model:

[0075]

[0076] Step 2. Input X obtained in step 1 into the time series prediction model, where the time series prediction model is initialized with input days of 3 and output days of 5. Obtain the output:

[0077]

[0078] where, corresponding to the outputs of input b t , l t , s t , w t , respectively;

[0079] Step 3. Calculate the loss on the output and future weight, the specific process is as follows:

[0080] (1) Calculate the weight change Δ t of the t-th day:

[0081] Δ t = w t -w t-1 ,

[0082] (2) Calculate the diet loss:

[0083]

[0084] (3) Calculate the weight loss:

[0085]

[0086] (4) Merge the two parts of the loss, where λ takes the value of 0.25:

[0087] L = 0.25L weight + 0.75L diet ,

[0088] (5) Calculate the gradient to update the model parameters using the loss. The updateable model parameters include: the multi-layer perceptron parameters in the diet feature learning module, and the overall parameters of the time series prediction model.

[0089] Step 4. After the model is trained, test the model; the specific process is as follows:

[0090] (1) Test using user data in the test set. For a piece of user data, the food pictures in the previous 3 days are used to obtain the features according to step 1, and the previous 3 days of weight are spliced to obtain the input of the time series prediction model.

[0091] (2) Input the input data into the time series prediction model to obtain the predicted weight values from the 4th to the 6th day.

[0092] (3) Continue the food data from day 2 to day 4 (the food data from day 2, 3 are real intake, the food data from day 4 is assumed to be the same as day 1) to get the features from step 1 and concatenate with the weight data from day 2 to day 4 (the weight data from day 2, 3 are real data, the weight data from day 4 is the predicted data from step (2)), and input the time series prediction model to get the weight prediction from day 5 to day 7.

[0093] (4) Repeat step (2) until all the days recorded in the user data are predicted. The input food data from day i is assumed to be the same as day i mod 3, and the input weight data from day i is the average of all the weight prediction from day i.

[0094] (5) Input the weight data from day 1 to day 3 into the time series prediction model to get the weight prediction from day 4 to day 6, and then input the weight data from day 2 to day 4 into the time series prediction model to get the weight prediction from day 5 to day 7.

[0095] (6) Repeat step (5) until all the days recorded in the user data are predicted.

[0096] (7) Compare the error between the weight prediction from step (4) and the real weight, and the error between the weight prediction from step (6) and the real weight. The error is calculated by calculating the mean squared error between the prediction and the real value for each day and summing them up.

[0097] (8) The prediction from step (4) has smaller error compared to the prediction from step (6) and the real value. The numerical value and the trend are closer to the real value. Table 1 and Table 2 are the quantitative comparison between step (4) and step (6), and the mean absolute error and the mean squared error are chosen as the evaluation indicators. Nlinear

[14] and iTransformer

[16] are chosen as the time series prediction model, and the results are shown in Table 1 and Table 2, respectively.

[0098] Table 1

[0099]

[0100] Table 2

[0101]

[0102] Figure 4 is the prediction visualization of three users when L = 3, T = 3, the time series prediction model is Nlinear, and the food information is picture. Blue represents the real weight change, green represents the prediction result of step (4) without food information. Orange represents the prediction result with food information. Obviously, the orange legend is closer to the real weight change in both numerical value and trend.

[0103] References

[0104] [1] Jéquier, Eric, and Luc Tappy. "Regulation of body weight in humans." Physiological reviews 79.2 (1999): 451-480.

[0105] [2] Hamilton, Marc T., Deborah G. Hamilton, and Theodore W. Zderic. "Role of low energy expenditure and sitting in obesity, metabolic syndrome, type 2 diabetes, and cardiovascular disease." Diabetes 56.11 (2007): 2655-2667.

[0106] [3] Chen, Xin, et al. "Chinese food net: A large-scale image dataset for Chinese food recognition." arXiv preprint arXiv:1705.02743 (2017).

[0107] [4] Jiang, Shuqiang, et al. "Multi-scale multi-view deep feature aggregation for food recognition." IEEE Transactions on Image Processing 29 (2019): 265-276.

[0108] [5] Martinel, Niki, Gian Luca Foresti, and Christian Micheloni. "Wide-slice residual networks for food recognition." 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.

[0109] [6] Chen, Jingjing, and Chong-Wah Ngo. "Deep-based ingredient recognition for cooking recipe retrieval." Proceedings of the 24th ACM international conference on Multimedia. 2016.

[0110] [7] Chen, Jingjing, et al. "Zero-shot ingredient recognition by multi-relational graph convolutional network." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.

[0111] [8] Salvador, Amaia, et al. "Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021.

[0112] [9] Shukor, Mustafa, et al. "Transformer decoders with multimodal regularization for cross-modal food retrieval." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022.

[0113]

[10] Lo,Frank P-W.,et al."Point2volume:A vision-based dietaryassessment approach using view synthesis."IEEE Transactions on IndustrialInformatics 16.1(2019):577-586.

[0114]

[11] Meyers,Austin,et al."Im2Calories:towards an automated mobilevision food diary."Proceedings of the IEEE international conference oncomputer vision.2015.

[0115]

[12] Thames,Quin,et al."Nutrition5k:Towards automatic nutritionalunderstanding of generic food."Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition.2021.

[0116]

[13] Radford,Alec,et al."Learning transferable visual models fromnatural language supervision."International conference on machinelearning.PMLR,2021

[0117]

[14] Zeng,Ailing,et al."Are transformers effective for timeseriesforecasting?."Proceedings of the AAAI conference on artificialintelligence.Vol.37.No.9.2023.

[0118]

[15] Vaswani, A. "Attention is all you need." Advances in Neural Information Processing Systems (2017).

[0119]

[16] Liu, Yong, et al. "itransformer: Inverted transformers are effective for time series forecasting." arXiv preprint arXiv:2310.06625 (2023).

Claims

1. A weight prediction system based on multimodal food data, characterized in that: It is based on computer vision and time series prediction technology and includes the following modules: (1) A unified dietary feature learning module based on the CLIP model, hereinafter referred to as the dietary feature learning module, is used to obtain dietary features of a meal from food images or food ingredient annotations. The dietary feature learning module includes a pre-trained CLIP encoder and a multi-layer perceptron. First, multiple images of a meal or multiple ingredient texts are fed into the CLIP encoder as input to obtain the features corresponding to each image or each text. All features are then averaged and fed into the multi-layer perceptron to further fuse and extract the final dietary features. (2) Time series prediction model, which is used to predict future time series from historical time series and obtain prediction results; Specifically, after obtaining the dietary characteristics of multiple days, they are spliced ​​with the weight values ​​of the corresponding days and sent to the time series prediction model to obtain the prediction results for multiple days in the future; (3) Diet-weight loss calculation module, which is used to train the diet feature learning module and the time series prediction model. During the training process, the CLIP model is fixed and the loss during the model training process is calculated, so that the model learns the relationship between diet and weight change, thereby making accurate weight predictions; The process of extracting food features by the diet feature learning module is as follows: Assume that the food information for the historical L days is: Among them, Breakfast t , Lunch t and Supper t is the food information for breakfast, lunch, and dinner on day t; if the food information is a food picture, the food information for breakfast, lunch, and dinner is represented as follows: Among them, N b , N l and N s The number of pictures for each meal; if the food information is raw material text, the food information for breakfast, lunch, and dinner is represented as follows: Among them, N b , N l and N s The text quantity of the ingredients for the three meals respectively; For food information input, the format of the three meals must be consistent, that is, all pictures or all text; For breakfast food images, the specific process of extracting food features for a meal is as follows: (1) Breakfast t Input CLIP model to get the feature E of each picture n , expressed as: E n =CLIPEncoder(img n ), (1) (2) N b The average feature of the image: (3) Input the averaged features into the multi-layer perceptron (MLP) to obtain the breakfast feature b on day t t : Similarly, we can get the lunch feature l on day t t and dinner features t ; Similarly, we can get the features of the three meals on the tth day using text as input information; The time series prediction model predicts weight and weight changes over a period of time in the future. The specific process is as follows: (1) Assume that the weight sequence of historical L days is Among them, w t represents the weight value on day t; concatenate the food characteristics and weight of the historical L days to obtain the input of the time series model, which is expressed as: (2) Input X into the time series prediction model M and get the output Expressed as: in, and is the weight value predicted from breakfast, lunch, dinner and weight, and M is the time series prediction model; The diet-weight loss calculation module is used to model the relationship between body weight and food intake, specifically including: (1) Calculate dietary loss L diet : Among them, Δ t is the weight change value on day t: Δ t =w t -w t-1 ; (2) Calculate weight loss L weight : (3) Combining diet loss and weight loss to form diet-weight loss: L= λL weight +(1-λ)L diet , (8) Among them, λ is a hyperparameter, λ∈[0,1].

2. The weight prediction system based on multimodal food data according to claim 1, characterized in that: The prediction process is as follows: (1) Take L days of weight and food data as input and obtain the weight predicted by the framework; (2) Using the weight prediction value obtained in step (1) as input to predict future weight; (3) Repeat step (2); (4) Calculate the mean square error between all predicted results and the actual weight values ​​to evaluate the effect.

Images