Weight prediction system based on multi-modal food data

Abstract

The invention belongs to the technical field of health management, and particularly relates to a body weight prediction system based on multi-modal food data. The body weight prediction system comprises a unified diet feature learning module based on CLIP, a time sequence prediction model and a diet-body weight loss calculation module. According to the invention, the influence of diet intake on weight prediction is explored, and the change trend of future weight is predicted by using short-term weight change and diet records. According to the method, unified features of historical food data are extracted through a diet feature learning module, and then the features and a historical weight change sequence are spliced to serve as input of a time sequence prediction model; the relation between diet intake and weight change is established through a diet-weight loss training diet feature learning module and a time sequence prediction model. The system is high in accuracy and adaptability for weight prediction; the method can be applied to the fields of health and medical treatment for personalized health management.

Description

Technical Field

[0001] The present invention belongs to the technical field of health management, and in particular relates to a weight prediction system based on multimodal food data. Background Art

[0002] Food plays a vital role in human life, affecting the quality of life and various physiological indicators, among which weight is one of the most basic and critical indicators. Literature [1] shows that the relative balance between energy expenditure and dietary intake determines the gain or loss of weight. It is worth noting that energy expenditure from non-exercise activities (such as sitting) usually accounts for a large part of total energy expenditure, far exceeding exercise expenditure [2]. Based on the above understanding, it is important and meaningful to study the impact of food intake on weight prediction.

[0003] Many studies have focused on various traditional tasks in the food domain, such as food classification [3, 4, 5], ingredient recognition [6, 7], recipe retrieval [8, 9], food volume prediction

[10] , and calorie and nutritional information estimation

[11] . However, these existing methods usually analyze each food image independently and do not study the impact of food intake on physiological indicators (such as body weight) over time. Another research direction is to predict the nutritional composition and calories of food from food images, which can help monitor intake patterns. Some studies first estimate the volume, such as voxels, point clouds, or 3D meshes, and then map it with the calorie data contained per unit volume. With the release of the Nutrition5K dataset

[12] , which provides fine-grained nutritional attributes, food quality, and food videos, some studies have predicted nutrition directly from images using neural networks. However, these studies only predict nutritional information and do not further link food intake to body weight.

[0004] However, there are two major challenges in using food intake for weight prediction. The first is that food information can be expressed in a variety of forms, such as textual ingredient annotations and food pictures; the second is how to model the relationship between food representation and weight prediction. To address the above challenges, this paper proposes a system for weight prediction based on multimodal food data. Specifically, this method introduces a diet feature learning module that uses the CLIP

[13] text or image encoder to extract a unified feature representation from various forms of historical food intake. In addition, we propose a diet-weight loss that enables the model to capture the correlation and dependency between food and weight change. Importantly, the framework design of this method is compatible with any existing time series prediction model. Summary of the invention

[0005] The object of the present invention is to provide a weight prediction system based on multimodal food data with high accuracy and strong adaptability.

[0006] The weight prediction system based on multimodal food data proposed in the present invention is based on computer vision and time series prediction technology, and includes the following modules:

[0007] (1) A unified dietary feature learning module based on the CLIP model

[13] (hereinafter referred to as the dietary feature learning module). This module is used to obtain the dietary features of a meal from food images or food ingredient annotations. The dietary feature learning module contains a pre-trained CLIP encoder and a multi-layer perceptron. First, multiple images of a meal or multiple ingredient texts are fed into the CLIP model as input to obtain the features corresponding to each image or each text. All features are then averaged and fed into the multi-layer perceptron to be further fused and extracted into the final dietary features.

[0008] During the training process, the parameters of the CLIP model are fixed; the CLIP model is used as an encoder to unify the visual and text feature extraction processes; this allows the present invention to also use any raw material prediction model to obtain textual raw material prediction results from food images and use them as food information input.

[0009] Considering that uploading pictures is more convenient for users than inputting raw material text, this module also supports using any raw material prediction model to predict raw material text from pictures and use the predicted raw material text as input. That is, the raw material text can come from the user or from the prediction result of the picture. The raw material prediction model can use any existing model.

[0010] (2) Model-agnostic time series prediction model (hereinafter referred to as time series prediction model). The time series prediction model is used to predict future time series from historical time series to obtain prediction results;

[0011] Specifically, after obtaining the dietary characteristics of multiple days, they are concatenated with the weight values ​​of the corresponding days and sent to the time series prediction model to obtain the prediction results for multiple days in the future;

[0012] Specifically, the model can adopt an existing time series prediction model, such as NLnear

[14] with a linear structure, or iTransformer

[16] with a Transformer

[15] structure, etc.

[0013] (3) Diet-weight loss calculation module.

[0014] The diet-weight loss calculation module is used to train the diet feature learning module and the time series prediction model; during the training process, the CLIP model is fixed and the loss during the model training process is calculated, so that the model learns the relationship between diet and weight changes, thereby making accurate weight predictions.

[0015] The present invention predicts the weight value in the next T days based on the food intake and weight value in the past L days. First, the dietary feature learning module is used to extract the features of each meal, and it is spliced ​​with the weight sequence as the input of the time series prediction model; during the training process, the CLIP model is fixed, and the dietary feature learning module and the time series prediction model are trained using diet-weight loss, so that the model learns the relationship between diet and weight change, thereby accurately predicting weight.

[0016] Expressing LT as different combinations of historical and future days, LT can be 3-3, 3-5, 3-7, 5-5, 5-7, 7-3 and 7-7, etc.

[0017] Further: In the weight prediction system based on multimodal food data provided by the present invention:

[0018] (I) The process of extracting food features by the dietary feature learning module is as follows:

[0019] Assume that the food information for L days in history is:

[0020]

[0021] Among them, Breakfast t ,Lunch t and Supper t is the food information of breakfast, lunch and dinner on day t; if the food information is a food picture, the food information of breakfast, lunch and dinner can be expressed as:

[0022]

[0023]

[0024]

[0025] Among them, N b , N l and N s are the number of pictures of three meals respectively; if the food information is raw material text, the food information of breakfast, lunch and dinner can be expressed as:

[0026]

[0027]

[0028]

[0029] Among them, N b , N l and N s The text quantities of the ingredients for the three meals respectively.

[0030] For food information input, the format of the three meals must be consistent (all images or all text). Taking breakfast food images as an example, the specific process of extracting food features for one meal is as follows:

[0031] (1) Breakfast t Input the CLIP model

[13] to obtain the feature E of each image n , expressed as:

[0032] E n = CLIPEncoder(img n ), (1)

[0033] (2) N b The average feature of the image:

[0034]

[0035] (3) Input the averaged features into the multi-layer perceptron (MLP) to obtain the breakfast feature b of the tth day t :

[0036]

[0037] Similarly, we can get the lunch feature l on day t t and dinner features t .

[0038] Similarly, the features of the three meals on the tth day can be obtained using text as input information.

[0039] (ii) The time series prediction module predicts the weight and weight change in the future, and the specific process is as follows:

[0040] (1) Assume that the weight sequence of historical L days is Among them, w t Represents the weight value on day t. The food features and weight of the historical L days are concatenated to obtain the input of the time series model, which is expressed as:

[0041]

[0042] (2) Input X into the time series prediction model M and get the output It is expressed as:

[0043]

[0044] in, and is the weight value predicted from breakfast, lunch, dinner and weight, and M is the time series prediction model.

[0045] (III) The diet-weight loss calculation module is used to model the relationship between body weight and food intake, specifically including:

[0046] (1) Calculate dietary loss L diet :

[0047]

[0048] Among them, Δ t is the weight change value on day t: Δ t =w t -w t-1 ;

[0049] (2) Calculate weight loss L weight :

[0050]

[0051] (3) Combining diet loss and weight loss to form diet-weight loss:

[0052] L=λL weight +(1-λ)L diet , (8)

[0053] Among them, λ is a hyperparameter, λ∈[0,1].

[0054] The weight prediction system based on multimodal food data provided by the present invention has a testing process as follows:

[0055] (1) Take L days of weight and food data as input to obtain the weight predicted by the framework;

[0056] (2) using the predicted weight value obtained in step (1) as input to predict future weight;

[0057] (3) Repeat step (2);

[0058] (4) Calculate the mean square error between all predicted results and true weight values ​​to evaluate the effectiveness of the framework.

[0059] The present invention uses food intake to predict the trend of weight change for the first time, and proposes an effective weight prediction system. In this system, the dietary feature learning module can uniformly encode food information of multiple modes, effectively improving the accuracy of weight prediction; through the design of a new loss function, the relationship between food intake and weight change is modeled; different time series prediction models are used as prediction models, which have strong compatibility. Use food intake information to improve the accuracy of weight prediction. The present invention can be applied in the fields of health and medicine for personalized health management; it can also help fitness enthusiasts provide weight management and prediction services to achieve weight gain or weight loss goals. In the diet and nutrition industry, the present invention can help predict customer weight and provide personalized diet and nutrition plans. Compared with not adding food intake information, the prediction accuracy and trend similarity of the present invention are greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a structural block diagram of the weight prediction system based on multimodal food data of the present invention.

[0061] Figure 2 It is a structural diagram of the dietary characteristics learning module proposed in the present invention.

[0062] Figure 3 This is a flowchart of the weight prediction system based on multimodal food data of the present invention. The three meals information and weight of the past L days are used as input. The three meals information first enters Figure 2 The diet feature learning module shown in the figure obtains the features of each meal. Then, all the three meal features and weight values ​​are spliced ​​together and enter the time series prediction model to obtain the weight prediction results for the next T days. The diet-weight loss is calculated between the prediction results and the actual weight to obtain the loss value, which is used to train the entire model.

[0063] Figure 4 When L=3, T=3, the time series prediction model is Nlinear, and the food information is a picture, the predictions of three users are visualized. DETAILED DESCRIPTION

[0064] The present invention is further described below by means of specific embodiments in conjunction with the accompanying drawings.

[0065] The present invention uses dietary intake information to assist in weight prediction. A dietary feature learning module is used, and its input can be either an image or a text; the dietary feature learning module extracts unified food features for different input modalities and uses them for training. The food information of a meal is first input into the CLIP module of the dietary feature learning module to obtain the features of each picture / each raw material text, and then the features of multiple pictures / multiple raw material texts are averaged and input into a multi-layer perceptron to obtain the final dietary features. The present invention needs to collect a large number of user food diet pictures and text annotations, as well as the corresponding weight change values ​​to construct a data set, and use the data set for training and testing. Suppose a prediction system is trained to predict the weight of the next 3 days using 3 days of weight and intake. The specific steps are:

[0066] Step 1. Use the diet feature learning module as a feature extractor to extract food features. The specific process is as follows:

[0067] (1) Sample the food intake and weight changes of multiple users for 6 days as training data. The three meals pictures on day t can be expressed as:

[0068]

[0069]

[0070]

[0071] Among them, N b , N l and N s are the number of pictures of the three meals. Each meal picture is input into the CLIP model of the diet feature learning module to obtain the features of all pictures of the three meals. The feature of each picture is 512 dimensions, and the features of the three meals are N respectively. b ×512,N l ×512,N s ×512.

[0072] (2) The multiple features obtained for each meal are averaged into 1×512 dimensions;

[0073] (3) Input the averaged features into the multi-layer perceptron to obtain the food features of each of the three historical days Where b, 1, and s represent breakfast, lunch, and dinner, respectively;

[0074] (4) Combine the food features obtained in process (3) with the daily body weight of the corresponding time period to obtain the input of the model:

[0075]

[0076] Step 2. Input X obtained in step 1 into the time series forecasting model, where the time series forecasting model is initialized to input days 3 and output days 5. The output is:

[0077]

[0078] in, Corresponding to input b t , l t ,s t , w t Output:

[0079] Step 3. Calculate the loss of output and future weight. The specific process is:

[0080] (1) Calculate the weight change Δ on day t t :

[0081] Δ t =w t -w t-1 ,

[0082] (2) Calculate dietary losses:

[0083]

[0084] (3) Calculation of weight loss:

[0085]

[0086] (4) Combine the two losses, where λ is set to 0.25:

[0087] L=0.25L weight +0.75L diet ,

[0088] (5) Using the loss to calculate the gradient, the model parameters are updated. The model parameters that can be updated include: the multi-layer perceptron parameters in the dietary feature learning module and the overall parameters of the time series prediction model.

[0089] Step 4. After the model is trained, test the model; the specific process is as follows:

[0090] (1) Use the user data in the test set for testing. For one user data, the food pictures of the previous three days are used to obtain features as described in step 1 and are concatenated with the weight of the previous three days to obtain the input of the time series prediction model.

[0091] (2) Input the input data into the time series prediction model to obtain the predicted weight values ​​from the 4th to the 6th day.

[0092] (3) The food data from the 2nd to the 4th day (where the food data from the 2nd and 3rd day are the actual intake, and the food data from the 4th day are assumed to be the food intake from the 1st day) are continued to be characterized according to step 1 and concatenated with the weight from the 2nd to the 4th day (where the weight from the 2nd and 3rd day are the actual data, and the weight from the 4th day is the predicted data obtained in process (2)), and input into the time series prediction model to obtain the predicted weight values ​​from the 5th to the 7th day.

[0093] (4) Repeat process (2) until all days recorded by the user data are predicted. The food data entered for the i-th day is assumed to be the same as that for the i%3 (i mod 3) day. The weight data entered for the i-th day is the average of all weight predictions for that day, except for days 1 to 3.

[0094] (5) The user's weight for the previous three days is directly input into the time series prediction model to obtain the weight prediction values ​​for the 4th to 6th days, and then the weight for the 2nd to 4th days is input into the time series prediction model to obtain the weight prediction values ​​for the 5th to 7th days.

[0095] (6) Repeat process (5) until all days recorded by the user data are predicted.

[0096] (7) Compare the errors between the predicted weight and the actual weight obtained by process (4) and process (6) respectively. The error is calculated by calculating the mean square error between the predicted value and the actual value for each day and summing them up.

[0097] (8) Compared with the prediction results obtained by process (6), the prediction results of process (4) have a smaller error with the true value. Both the value and the trend are closer to the true value. Tables 1 and 2 are quantitative comparisons of processes (4) and (6), and mean absolute error and mean square error are selected as evaluation indicators. Nlinear

[14] and iTransformer

[16] are selected as time series prediction models, and the results are shown in Tables 1 and 2 respectively.

[0098] Table 1

[0099]

[0100] Table 2

[0101]

[0103] Figure 4The predictions of three users are visualized when L=3, T=3, the time series prediction model is Nlinear, and the food information is pictures. Blue represents the actual weight change, and green represents process (4), that is, the prediction results without food information. Orange represents the prediction results with food information added. Obviously, the orange legend is closer to the actual weight change in both value and trend.

[0104] References

[0105] [1]Jéquier, Eric, and Luc Tappy. "Regulation of body weight in humans." Physiological reviews 79.2 (1999): 451-480.

[0106] [2] Hamilton, Marc T., Deborah G. Hamilton, and Theodore W. Zderic. "Role oflow energy expenditure and sitting in obesity, metabolic syndrome, type 2 diabetes, and cardiovascular disease." Diabetes 56.11 (2007): 2655-2667.

[0107] [3]Chen, Xin, et al. "Chinesefoodnet: A large-scale image dataset for Chinese food recognition." arXiv preprint arXiv:1705.02743(2017).

[0108] [4] Jiang, Shuqiang, et al. "Multi-scale multi-view deep feature aggregation for food recognition." IEEE Transactions on Image Processing 29(2019):265-276.

[0109] [5]Martinel,Niki,Gian Luca Foresti,and Christian Micheloni."Wide-slice residual networks for food recognition."2018 IEEE Winter conference onapplications of computer vision(WACV).IEEE,2018.

[0110] [6]Chen,Jingjing,and Chong-Wah Ngo."Deep-based ingredient recognitionfor cooking recipe retrieval."Proceedings of the 24th ACM internationalconference on Multimedia.2016.

[0111] [7]Chen,Jingjing,et al."Zero-shot ingredient recognition by multi-relational graph convolutional network."Proceedings of the AAAI Conference onArtificial Intelligence.Vol.34.No.07.2020.

[0112] [8]Salvador,Amaia,et al."Revamping cross-modal recipe retrieval withhierarchical transformers and self-supervised learning."Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition.2021.

[0113] [9]Shukor,Mustafa,et al."Transformer decoders with multimodalregularization for cross-modal food retrieval."Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2022.

[0114]

[10] Lo,Frank P-W.,et al."Point2volume:A vision-based dietaryassessment approach using view synthesis."IEEE Transactions on IndustrialInformatics 16.1(2019):577-586.

[0115]

[11] Meyers,Austin,et al."Im2Calories:towards an automated mobilevision food diary."Proceedings of the IEEE international conference oncomputer vision.2015.

[0116]

[12] Thames,Quin,et al."Nutrition5k:Towards automatic nutritionalunderstanding of generic food."Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition.2021.

[0117]

[13] Radford,Alec,et al."Learning transferable visual models fromnatural language supervision."International conference on machinelearning.PMLR,2021

[0118]

[14] Zeng,Ailing,et al."Are transformers effective for timeseriesforecasting?."Proceedings of the AAAI conference on artificialintelligence.Vol.37.No.9.2023.

[0119]

[15] Vaswani,A."Attention is all you need."Advances in NeuralInformation Processing Systems(2017).

[0120]

[16] Liu,Yong,et al."itransformer:Inverted transformers are effectivefor time series forecasting."arXiv preprint arXiv:2310.06625(2023)。

Claims

1. A weight prediction system based on multimodal food data, characterized in that: It is based on computer vision and time series prediction technology and includes the following modules: (1) A unified diet feature learning module based on the CLIP model, hereinafter referred to as the diet feature learning module, is used to obtain the diet features of a meal from food pictures or food ingredient annotations; the diet feature learning module includes a pre-trained CLIP encoder and a multi-layer perceptron; first, multiple pictures of a meal or multiple ingredient texts are sent as input to the CLIP encoder to obtain the features corresponding to each picture or each text, and then all the features are averaged and sent to the multi-layer perceptron to further fuse and extract the final diet features; (2) Time series prediction model, which is used to predict future time series from historical time series and obtain prediction results; Specifically, after obtaining the dietary characteristics of multiple days, they are concatenated with the weight values ​​of the corresponding days and sent to the time series prediction model to obtain the prediction results for multiple days in the future; (3) A diet-weight loss calculation module is used to train the diet feature learning module and the time series prediction model. During the training process, the CLIP model is fixed and the loss during the model training process is calculated, so that the model learns the relationship between diet and weight change, thereby making accurate weight predictions.

2. The weight prediction system based on multimodal food data according to claim 1, characterized in that: The process of extracting food features by the diet feature learning module is as follows: Assume that the food information for L days in history is: Among them, Breakfast t ,Lunch t and Supper t is the food information of breakfast, lunch and dinner on day t; if the food information is a food picture, the food information of breakfast, lunch and dinner is represented as follows: Among them, N b , N l and N s are the number of pictures of three meals respectively; if the food information is raw material text, the food information of breakfast, lunch and dinner is represented as follows: Among them, N b , N l and N s The textual quantities of the ingredients for the three meals are respectively; For food information input, the format of the three meals must be consistent, that is, all pictures or all text; For breakfast food images, the specific process of extracting food features for a meal is as follows: (1) Breakfast t Input CLIP model to get the feature E of each picture n , expressed as: E n =CLIPEncoder(img n ), (1) (2) N b The average feature of the image: (3) Input the averaged features into the multi-layer perceptron (MLP) to obtain the breakfast feature b of the tth day t : Similarly, we can get the lunch feature l on day t t and dinner features t ; Similarly, the features of the three meals on the tth day can be obtained using text as input information.

3. The weight prediction system based on multimodal food data according to claim 2, characterized in that: The time series prediction module predicts the weight and weight change in the future. The specific process is as follows: (1) Assume that the weight sequence of historical L days is Among them, w t represents the weight value on day t; concatenate the food characteristics and weight of the historical L days to obtain the input of the time series model, which is expressed as: (2) Input X into the time series prediction model M and get the output It is expressed as: in, and is the weight value predicted from breakfast, lunch, dinner and weight, and M is the time series prediction model.

4. The weight prediction system based on multimodal food data according to claim 3, characterized in that: The diet-weight loss calculation module is used to model the relationship between body weight and food intake, specifically including: (1) Calculate dietary loss L diet : Among them, Δ t is the weight change value on day t: Δ t =w t -w t-1 ; (2) Calculate weight loss L weight : (3) Combining diet loss and weight loss to form diet-weight loss: L= λL weight +(1-λ)L diet , (8) Among them, λ is a hyperparameter, λ∈[0,1].

5. The weight prediction system based on multimodal food data according to claim 4, characterized in that: The prediction process is as follows: (1) Take L days of weight and food data as input to obtain the weight predicted by the framework; (2) using the predicted weight value obtained in step (1) as input to predict future weight; (3) Repeat step (2); (4) Calculate the mean square error between all predicted results and the actual weight values ​​to evaluate the effect.

Images