A food nutrition assessment method based on multi-scale information fusion
Through the multimodal fusion algorithm and the bidirectional feature pyramid fusion algorithm, the problem of insufficient fusion of RGB images and deep images is solved, the accuracy of nutritional evaluation is improved, and an effective basis for formulating healthy recipes is provided.
Patent Information
- Application Number
- CN202411093062.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-09
AI Technical Summary
Existing vision-based nutrition evaluation technology failed to fully consider the relationship between RGB image features and depth image features, resulting in insufficient fusion and insufficient multi-scale fusion, which in turn leads to inaccurate prediction results of various nutrient content in food and the inability to formulate healthy recipe.
Using multimodal fusion algorithm and bidirectional feature pyramid fusion algorithm, RGB image features and depth image features of each scale are extracted, and more relevant nutritional assessments are generated through channel attention weight weighting processing, multi-layer convolutional styling and bidirectional pyramid enhancement. feature.
By considering the relationship between food RGB images and depth image features, the semantic and detailed characteristics of nutrition information are enhanced, the accuracy of nutrition evaluation is improved, and an effective basis for formulating healthy recipes is provided.
Smart Images

Figure CN118644850B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deep learning and food analysis, and in particular to a food nutrition assessment method based on multi-scale information fusion. Background Art
[0002] In the past decade, some studies have found that diet is closely related to health, and the relationship between diet and health has attracted more and more attention; many studies have shown that maintaining good physical health and preventing diseases can be achieved through a healthy diet. In other words, an unhealthy diet can have a negative impact on mental health and physical health; therefore, how to formulate a healthy recipe is an urgent issue to be addressed.
[0003] Formulating a healthy diet necessarily requires considering the calories and various nutrient contents of food. For this reason, nutrition assessment technology came into being. Traditional nutrition assessment technology often analyzes the nutrient content of food based on the chemical or physical properties of food, and most of them are used for scientific research and require professional experimental equipment, making them difficult to popularize. With the development of deep learning technology, some studies have applied deep learning technology to nutrition assessment. As a result, vision-based nutrition assessment technology has emerged. Only pictures are needed as input to obtain food-related nutrient information in the picture. Today, the popularity of mobile terminals makes it very simple to obtain pictures, which makes deep learning nutrition assessment technology very easy to popularize, solving the problem that traditional nutrition assessment technology is difficult to popularize.
[0004] At present, most methods of nutrition assessment technology based on vision only extract RGB image features of food images, and use RGB image features as input to predict the content of each nutrient in food. However, using RGB images as two-dimensional images will inevitably lack some three-dimensional information in terms of volume. In order to solve this problem, some studies have adopted RGB image features and depth image features for nutrition assessment, and used depth images as additional three-dimensional information. However, this type of method simply inputs the depth image as the fourth channel into the feature extractor or directly adds the RGB image features to the depth image features, without considering the relationship between the RGB image features and the depth image features. This leads to insufficient fusion of the RGB image features and the depth image features, and insufficient multi-scale fusion, which results in inaccurate prediction results of the content of each nutrient in food, which is not in line with reality, and thus makes it impossible to formulate a healthy recipe. Summary of the invention
[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem that the relationship between RGB image features and depth image features is not taken into account in the prior art, resulting in insufficient fusion of RGB image features and depth image features, and insufficient multi-scale fusion, which leads to inaccurate and unrealistic prediction results of the content of each nutrient in food, and thus makes it impossible to formulate a healthy recipe.
[0006] In order to solve the above technical problems, the present invention provides a food nutrition assessment method based on multi-scale information fusion, comprising:
[0007] Obtain food RGB images and depth images; use feature extractors to extract RGB image features and depth image features at various scales;
[0008] Based on the multimodal fusion algorithm, the RGB image features and depth image features of each scale are fused to obtain the target fusion features of each scale, including:
[0009] After global average pooling of the deep image features at each scale, multi-layer channels and nonlinear activation functions are used to obtain the channel attention weights corresponding to each scale. The RGB image features at each scale are weighted using the channel attention weights corresponding to each scale to obtain the first fusion features at each scale.
[0010] After concatenating the depth image features and RGB image features of each scale, multi-layer convolution is used to obtain the second fusion features of each scale;
[0011] Add the first fusion feature, the second fusion feature, and the RGB image feature of each scale to obtain the target fusion feature of each scale;
[0012] According to the multi-scale feature fusion algorithm based on the bidirectional feature pyramid, the target fusion features of each scale are enhanced to obtain the bidirectional enhanced fusion features of each scale, including:
[0013] The preset first convolution kernel is used to unify the number of channels of the target fusion features of each scale to obtain the channel fusion features of each scale;
[0014] The channel fusion features of the last scale are used as the unidirectional enhanced fusion features of the last scale, and then the bilinear interpolation is used to enlarge the size of the unidirectional enhanced fusion features of the last scale; the unidirectional enhanced fusion features of the last scale after the scale enlargement are added to the channel fusion features of the corresponding scale of the previous level to obtain the unidirectional enhanced fusion features of the corresponding scale of the previous level, and so on, until the unidirectional enhanced fusion features of the first scale are obtained;
[0015] The unidirectional enhanced fusion features of the first scale are used as the bidirectional enhanced fusion features of the first scale, and then the preset second convolution kernel is used to scale down the bidirectional enhanced fusion features of the first scale; the scaled-down bidirectional enhanced fusion features of the first scale are added and fused with the unidirectional enhanced fusion features of the corresponding scale of the next level to obtain the bidirectional enhanced fusion features of the second scale, and so on, until the bidirectional enhanced fusion features of the last scale are obtained;
[0016] The bidirectional enhanced fusion features of each scale are pooled and concatenated to obtain the prediction features; the prediction features are input into the multi-task learning model, and after multiple linear layers and nonlinear activation functions, the predicted values of various nutrients in food are output.
[0017] Preferably, after performing global average pooling on the depth image features of each scale, a multi-layer channel and a nonlinear activation function are used to obtain a channel attention weight corresponding to each scale; and the RGB image features of each scale are weighted by using the channel attention weight corresponding to each scale to obtain the first fusion feature of each scale, including:
[0018] After performing a global average pooling operation on the deep image features of the i-th scale, a multi-layer channel and a nonlinear activation function are used to obtain the channel attention weight corresponding to the i-th scale, which is expressed as:
[0019] ;
[0020] ;
[0021] in, Represents the pooling weight obtained after global average pooling of the deep image features of the i-th scale; Represents the depth image feature of the i-th scale; represents global average pooling; Represents the channel attention weight corresponding to the i-th scale; represents the first fully connected layer corresponding to the i-th scale; represents the second fully connected layer corresponding to the i-th scale; express Non-linear activation function; express Non-linear activation function;
[0022] The RGB image features of the i-th scale are weighted using the channel attention weight corresponding to the i-th scale to obtain the first fusion feature of the i-th scale, which is expressed as:
[0023] ;
[0024] in, represents the first fusion feature of the i-th scale; Represents the RGB image features of the i-th scale; Indicates that the RGB image features of the i-th scale are weighted by using channel-by-channel multiplication operation;
[0025] Get the first fusion features of each scale in turn , n represents the total number of all scales.
[0026] Preferably, after concatenating the depth image features and RGB image features of each scale, multi-layer convolution is used to obtain the second fusion features of each scale, which includes:
[0027] The depth image feature of the i-th scale is concatenated with the RGB image feature of the i-th scale to obtain the concatenated feature of the i-th scale, which is expressed as:
[0028] ;
[0029] in, Represents a splicing operation; Represents the RGB image features of the i-th scale; Represents the depth image feature of the i-th scale; represents the splicing feature of the i-th scale;
[0030] Based on the splicing features of the i-th scale, multi-layer convolution is used to obtain the second fusion features of the i-th scale, which is expressed as:
[0031] ;
[0032] in, Represents multi-layer convolution operations; represents the second fusion feature of the i-th scale;
[0033] The second fusion features of each scale are obtained in turn , n represents the total number of all scales.
[0034] Preferably, the first fusion feature, the second fusion feature, and the RGB image feature of the i-th scale are added to obtain the target fusion feature of the i-th scale, which is expressed as:
[0035] ;
[0036] in, represents the first fusion feature of the i-th scale; represents the second fusion feature of the i-th scale; Represents the RGB image features of the i-th scale; represents the target fusion feature of the i-th scale; Represents element-wise addition operation;
[0037] Obtain target fusion features of each scale in turn , n represents the total number of all scales.
[0038] Preferably, the preset first convolution kernel is used to unify the number of channels of the target fusion features of each scale to obtain the channel fusion features of each scale; the channel fusion features of the last scale are used as the unidirectional enhanced fusion features of the last scale, and then the unidirectional enhanced fusion features of the last scale are enlarged by bilinear interpolation; the unidirectional enhanced fusion features of the last scale after the scale enlargement are added to the channel fusion features of the corresponding scale of the previous level to obtain the unidirectional enhanced fusion features of the corresponding scale of the previous level, and so on, until the unidirectional enhanced fusion features of the first scale are obtained, including:
[0039] Use a 1×1 convolution kernel to fuse features of targets at all scales The number of channels is unified, and the target fusion features of each scale with the same number of channels are obtained as the channel fusion features of each scale ; Wherein, the 1×1 convolution kernel is a preset first convolution kernel;
[0040] Bilinear interpolation is used to enhance the fusion features of the (i+1)th scale in one direction. Perform a scale-up operation; add the channel fusion feature of the i-th scale to the unidirectional enhanced fusion feature of the (i+1)-th scale after scale-up to obtain the unidirectional enhanced fusion feature of the i-th scale, which is expressed as:
[0041] ;
[0042] in, represents the unidirectional enhanced fusion feature of the i-th scale; represents the channel fusion feature of the i-th scale; when hour, ; Represents element-wise addition operation; Represents a linear interpolation operation; represents the unidirectional enhanced fusion feature of the (i+1)th scale; Represents the unidirectional enhanced fusion feature of the (i+1)th scale after scale enlargement;
[0043] Obtain the unidirectional enhanced fusion features of each scale in turn , n represents the total number of all scales.
[0044] Preferably, the unidirectional enhanced fusion feature of the first scale is used as the bidirectional enhanced fusion feature of the first scale, and then a preset second convolution kernel is used to scale down the bidirectional enhanced fusion feature of the first scale; the scaled-down bidirectional enhanced fusion feature of the first scale is added and then fused with the unidirectional enhanced fusion feature of the corresponding scale of the next level to obtain the bidirectional enhanced fusion feature of the second scale, and so on, until the bidirectional enhanced fusion feature of the last scale is obtained, which includes:
[0045] A 3×3 convolution kernel with a step size of 2 is used to enhance the bidirectional fusion features of the (i-1)th scale. The scale is reduced; the unidirectional enhanced fusion feature of the i-th scale is added and fused with the bidirectional enhanced fusion feature of the (i-1)-th scale after the scale reduction to obtain the bidirectional enhanced fusion feature of the i-th scale, which is expressed as:
[0046] ;
[0047] in, represents the bidirectional enhanced fusion feature of the i-th scale; represents the unidirectional enhanced fusion feature of the i-th scale; when hour, ; Represents element-wise addition operation; Indicates that the fusion operation uses a 3×3 convolution kernel with a stride of 1; Indicates that the scale reduction operation of the bidirectional enhanced fusion feature of the (i-1)th scale uses a 3×3 convolution kernel with a step size of 2; represents the bidirectional enhanced fusion feature of the (i-1)th scale; represents the bidirectional enhanced fusion feature of the (i-1)th scale after the scale is reduced; the 3×3 convolution kernel with a step size of 2 is the preset second convolution kernel;
[0048] Obtain bidirectional enhanced fusion features of each scale in turn , n represents the total number of all scales.
[0049] Preferably, the feature extractor is a pre-trained feature extraction network; wherein two feature extractors with different parameters but the same structure are used to extract RGB image features and depth image features respectively.
[0050] Preferably, the bidirectional enhanced fusion features of each scale are pooled and concatenated to obtain prediction features, which are expressed as follows:
[0051] ;
[0052] ;
[0053] ;
[0054] in, Represents the final fusion feature of the i-th scale after the global average pooling operation; represents the bidirectional enhanced fusion feature of the i-th scale; represents the global average pooling operation; Represents the final fusion features of each scale after the global average pooling operation; Represents a splicing operation; Represents the predicted feature.
[0055] Preferably, the prediction features are input into the multi-task learning model, and after multiple linear layers and nonlinear activation functions, the predicted values of various nutrients in the food are output, and the expressions are:
[0056] ;
[0057] ;
[0058] in, Represents the predicted features after mapping; represents the prediction feature; represents the weight of the first linear layer; represents the bias of the first linear layer; , Represents the first linear layer and the second linear layer corresponding to predicting the j-th nutrient in the multi-task learning model; represents the predicted value of the jth nutrient; Indicates the types of nutrients as calories, mass, fat, carbohydrates, and protein; express Non-linear activation function.
[0059] Preferably, the loss function of the multi-task learning model is expressed as:
[0060] ;
[0061] ;
[0062] in, represents the total loss of all nutrients; Indicates the number of types of all nutrients; Indicates the number of all samples in a batch; represents the loss of the jth nutrient; represents the true value of the jth nutrient in the kth sample in a batch; represents the predicted value of the jth nutrient of the kth sample in a batch; Indicates the types of nutrients as calories, mass, fat, carbohydrates, and protein.
[0063] The above technical solution of the present invention has the following beneficial effects compared with the prior art:
[0064] The food nutrition assessment method based on multi-scale information fusion described in the present invention extracts multi-scale features of food RGB images and depth images; since food RGB images have richer information than depth images, for feature pairs of each scale, the RGB image features of the food are used as the main features, and the depth image features are used as the auxiliary features. The RGB image features and the depth image features of each scale are fused through the designed multimodal fusion algorithm to obtain target fusion features of each scale; since small-scale features have more semantic information, the semantic information of features of each scale is enhanced through a feature pyramid fusion method from top to bottom, and since large-scale features have richer detail information, the semantic information of features of each scale is enhanced through a reverse feature pyramid fusion method from bottom to top. The detailed information of the scale features is finally enhanced through the designed multi-scale feature fusion algorithm based on the bidirectional feature pyramid to obtain the bidirectional enhanced fusion features of each scale; based on the bidirectional enhanced fusion features of each scale, a multi-task learning method is adopted to predict the information of various nutrients in food at one time; the present invention provides additional three-dimensional spatial information as auxiliary information through the depth image features of the food, and fuses the RGB features of the food as the main features. Considering the relationship between the RGB image features and the depth image features of the food, a bidirectional feature pyramid is used to enhance the features of each scale, thereby obtaining features that are more relevant to the nutritional information, achieving better nutritional evaluation effects, and providing an effective basis for formulating reasonable and healthy recipes. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:
[0066] Figure 1 It is a process schematic diagram of a food nutrition assessment method based on multi-scale information fusion provided by the present invention;
[0067] Figure 2 yes Figure 1 Schematic diagram of the fusion operation of RGB image features and depth image features;
[0068] Figure 3 is a schematic diagram of lateral connection during multi-scale fusion; Figure 3 (a) shows a schematic diagram of the partial process of fusing feature pyramid features from top to bottom to enhance semantic information; Figure 3(b) in the figure is a schematic diagram showing the partial process of feature fusion from bottom to top of feature pyramid to enhance detail information. DETAILED DESCRIPTION
[0069] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0070] Reference Figure 1 As shown, Figure 1 A schematic diagram of a food nutrition assessment method based on multi-scale information fusion provided by the present invention; specifically comprising:
[0071] S1: Obtain food RGB images and depth images; use feature extractors to extract RGB image features and depth image features at various scales;
[0072] Among them, since the information contained in features of different scales is different, feature maps of different levels and scales in the feature extractor are output; data enhancement operations such as random horizontal flipping are performed on the food RGB image and depth image; based on the enhanced food RGB image and depth image, a feature extractor with the same structure but no shared parameters is used to extract the food RGB image features and depth image features; specifically including:
[0073] Pre-train feature extraction network for classification tasks on classification datasets;
[0074] Perform data augmentation operations such as random inversion on food RGB images and depth images;
[0075] Two pre-trained feature extraction networks are used to extract RGB image features and depth image features respectively, and RGB image features of various scales are obtained. With deep image features , the specific formula is as follows:
[0076] ;
[0077] ;
[0078] ;
[0079] ;
[0080] in, They are RGB image and depth image respectively; They are the pre-trained feature extraction network corresponding to RGB feature extraction and the pre-trained feature extraction network corresponding to deep image feature extraction; They are RGB image multi-scale feature set and depth image multi-scale feature set respectively; , Respectively represent the RGB image features and depth image features of the i-th scale, , From 1 to n, the scale keeps getting smaller, and the actual setting , that is, using features of four scales;
[0081] S2: Based on the multimodal fusion algorithm, the RGB image features and depth image features of each scale are fused to obtain the target fusion features of each scale, including:
[0082] S21: After global average pooling of the deep image features of each scale, multi-layer channels and nonlinear activation functions are used to obtain the channel attention weight corresponding to each scale; the channel attention weight corresponding to each scale is used to weight the RGB image features of each scale to obtain the first fusion feature of each scale, that is:
[0083] After performing a global average pooling operation on the deep image features of the i-th scale, a multi-layer channel and a nonlinear activation function are used to obtain the channel attention weight corresponding to the i-th scale, which is expressed as:
[0084] ;
[0085] ;
[0086] in, Represents the pooling weight obtained after global average pooling of the deep image features of the i-th scale; Represents the depth image feature of the i-th scale; represents global average pooling; Represents the channel attention weight corresponding to the i-th scale; represents the first fully connected layer corresponding to the i-th scale; represents the second fully connected layer corresponding to the i-th scale; express Non-linear activation function; express Non-linear activation function;
[0087] The RGB image features of the i-th scale are weighted using the channel attention weight corresponding to the i-th scale to obtain the first fusion feature of the i-th scale, which is expressed as:
[0088] ;
[0089] in, represents the first fusion feature of the i-th scale; Represents the RGB image features of the i-th scale; Indicates that the RGB image features of the i-th scale are weighted by using channel-by-channel multiplication operation;
[0090] Get the first fusion features of each scale in turn , n represents the total number of all scales;
[0091] S22: After concatenating the depth image features and RGB image features of each scale, multi-layer convolution is used to learn the fusion between the RGB image features and the depth image features, thereby obtaining the second fusion features of each scale, namely:
[0092] The depth image feature of the i-th scale is concatenated with the RGB image feature of the i-th scale to obtain the concatenated feature of the i-th scale, which is expressed as:
[0093] ;
[0094] in, Represents a splicing operation; Represents the RGB image features of the i-th scale; Represents the depth image feature of the i-th scale; represents the splicing feature of the i-th scale;
[0095] Based on the splicing features of the i-th scale, multi-layer convolution is used to obtain the second fusion features of the i-th scale, which is expressed as:
[0096] ;
[0097] in, Represents multi-layer convolution operations; represents the second fusion feature of the i-th scale;
[0098] The second fusion features of each scale are obtained in turn , n represents the total number of all scales;
[0099] S23: Add the first fusion feature, the second fusion feature, and the RGB image feature of each scale to obtain the target fusion feature of each scale, that is:
[0100] The first fusion feature, the second fusion feature, and the RGB image feature of the i-th scale are added to obtain the target fusion feature of the i-th scale, and its expression is:
[0101] ;
[0102] in, represents the first fusion feature of the i-th scale; represents the second fusion feature of the i-th scale; Represents the RGB image features of the i-th scale; represents the target fusion feature of the i-th scale; Represents element-wise addition operation;
[0103] Obtain target fusion features of each scale in turn , n represents the total number of all scales;
[0104] Among them, since food RGB images have richer information than depth images, for feature pairs of each scale, food RGB image features are used as the main features, and depth image features are used as auxiliary features. Figure 2 As shown, the food RGB image features and the depth image features are fused to obtain the target fusion features of the food;
[0105] S3: According to the designed multi-scale feature fusion algorithm based on bidirectional feature pyramid, the target fusion features of each scale are enhanced to obtain bidirectional enhanced fusion features of each scale, including:
[0106] S31: Using the preset first convolution kernel, unify the number of channels of the target fusion features of each scale to obtain the channel fusion features of each scale, that is:
[0107] Use a 1×1 convolution kernel to fuse features of targets at all scales The number of channels is unified, and the target fusion features of each scale with the same number of channels are obtained as the channel fusion features of each scale ; Wherein, the 1×1 convolution kernel is a preset first convolution kernel;
[0108] S32: The channel fusion feature of the last scale is used as the unidirectional enhanced fusion feature of the last scale, and then the bilinear interpolation is used to enlarge the size of the unidirectional enhanced fusion feature of the last scale; the unidirectional enhanced fusion feature of the last scale after the scale enlargement is added to the channel fusion feature of the corresponding scale of the previous level to obtain the unidirectional enhanced fusion feature of the corresponding scale of the previous level, and so on, until the unidirectional enhanced fusion feature of the first scale is obtained, that is:
[0109] Bilinear interpolation is used to enhance the fusion features of the (i+1)th scale in one direction. Perform a scale-up operation; add the channel fusion feature of the i-th scale to the unidirectional enhanced fusion feature of the (i+1)-th scale after scale-up to obtain the unidirectional enhanced fusion feature of the i-th scale, which is expressed as:
[0110] ;
[0111] in, represents the unidirectional enhanced fusion feature of the i-th scale; represents the channel fusion feature of the i-th scale; when hour, ; Represents element-wise addition operation; Represents a linear interpolation operation; represents the unidirectional enhanced fusion feature of the (i+1)th scale; Represents the unidirectional enhanced fusion feature of the (i+1)th scale after scale enlargement;
[0112] Obtain the unidirectional enhanced fusion features of each scale in turn , n represents the total number of all scales;
[0113] S33: The unidirectional enhanced fusion feature of the first scale is used as the bidirectional enhanced fusion feature of the first scale, and then the preset second convolution kernel is used to scale down the bidirectional enhanced fusion feature of the first scale; the scaled-down bidirectional enhanced fusion feature of the first scale is added and fused with the unidirectional enhanced fusion feature of the corresponding scale of the next level to obtain the bidirectional enhanced fusion feature of the second scale, and so on, until the bidirectional enhanced fusion feature of the last scale is obtained, that is:
[0114] A 3×3 convolution kernel with a step size of 2 is used to enhance the bidirectional fusion features of the (i-1)th scale. The scale is reduced; the unidirectional enhanced fusion feature of the i-th scale is added and fused with the bidirectional enhanced fusion feature of the (i-1)-th scale after the scale reduction to obtain the bidirectional enhanced fusion feature of the i-th scale, which is expressed as:
[0115] ;
[0116] in, represents the bidirectional enhanced fusion feature of the i-th scale; represents the unidirectional enhanced fusion feature of the i-th scale; when hour, ; Represents element-wise addition operation; Indicates that the fusion operation uses a 3×3 convolution kernel with a stride of 1; Indicates that the scale reduction operation of the bidirectional enhanced fusion feature of the (i-1)th scale uses a 3×3 convolution kernel with a step size of 2; represents the bidirectional enhanced fusion feature of the (i-1)th scale; represents the bidirectional enhanced fusion feature of the (i-1)th scale after the scale is reduced; the 3×3 convolution kernel with a step size of 2 is the preset second convolution kernel;
[0117] Obtain bidirectional enhanced fusion features of each scale in turn , n represents the total number of all scales;
[0118] Among them, since small-scale features have more semantic information, the semantic information of features at each scale is enhanced through the top-to-bottom feature pyramid fusion method. Since large-scale features have richer detail information, the detail information of features at each scale is enhanced through the bottom-to-top reverse feature pyramid fusion, and finally the enhanced multi-scale features are obtained through the bidirectional feature pyramid fusion; some operations such as Figure 3 As shown; among them, Figure 3 (a) shows a schematic diagram of the partial process of fusing feature pyramid features from top to bottom to enhance semantic information; Figure 3 (b) is a schematic diagram showing a partial process of enhancing detail information by fusing feature pyramid features from bottom to top;
[0119] S4: Pooling and concatenation of the bidirectional enhanced fusion features at each scale are performed to obtain the prediction features, which are expressed as follows:
[0120] ;
[0121] ;
[0122] ;
[0123] in, Represents the final fusion feature of the i-th scale after the global average pooling operation; represents the bidirectional enhanced fusion feature of the i-th scale; represents the global average pooling operation; Represents the final fusion features of each scale after the global average pooling operation; Represents a splicing operation; represents the prediction feature;
[0124] The prediction features are input into the multi-task learning model, and after multiple linear layers and nonlinear activation functions, the predicted values of various nutrients in the food are output. The expressions are:
[0125] ;
[0126] ;
[0127] in, Represents the predicted features after mapping; represents the prediction feature; represents the weight of the first linear layer; represents the bias of the first linear layer; , Represents the first linear layer and the second linear layer corresponding to predicting the j-th nutrient in the multi-task learning model; represents the predicted value of the jth nutrient; Indicates the types of nutrients as calories, mass, fat, carbohydrates, and protein; express Non-linear activation function;
[0128] Among them, the loss function expression of the multi-task learning model is:
[0129] ;
[0130] ;
[0131] in, represents the total loss of all nutrients; Indicates the number of types of all nutrients; Indicates the number of all samples in a batch; represents the loss of the jth nutrient; represents the true value of the jth nutrient in the kth sample in a batch; represents the predicted value of the jth nutrient of the kth sample in a batch; Indicates the types of nutrients as calories, mass, fat, carbohydrates, and protein.
[0132] Traditional nutrition assessment technology is mostly used in scientific research and requires professional experimental equipment and corresponding knowledge reserves, which makes it difficult to popularize. However, with the application of deep learning technology in the field of nutrition assessment, only pictures can be used as input to obtain food-related nutrient information. Nowadays, the popularity of mobile terminals makes it very convenient to obtain pictures. Vision-based nutrition assessment technology solves the problem of being difficult to popularize. For vision-based nutrition assessment technology, most methods currently only use food RGB images as input, and food RGB images as two-dimensional images inevitably lack some volumetric three-dimensional information; for methods that use food RGB images and depth images, the fusion of RGB image features and depth image features is not considered enough, and multi-scale feature fusion is not sufficient.
[0133] According to the food nutrition assessment method based on multi-scale information fusion provided by the present invention, the multi-scale features of the food RGB image and the depth image are first extracted, and the depth image features are used to provide additional three-dimensional spatial information as auxiliary information. The RGB image features of the food are fused as the main features, and then the fused multi-scale features are enhanced using a bidirectional feature pyramid to enhance the information of each scale, which is finally used for nutrition assessment. The present invention introduces additional spatial auxiliary information, considers the relationship between the food RGB image features and the depth image features, uses a bidirectional feature pyramid to enhance the features of each scale, and uses the bidirectional enhanced fusion features after pooling as the final fusion features of the food, thereby obtaining features that are more relevant to the nutrition information and achieving better nutrition assessment results.
[0134] In order to verify the effectiveness of the present invention, the PMAE index is used to evaluate the effectiveness of the present invention based on the Nutrition5k dataset. The specific PMAE index results are shown in Table 1.
[0135] Among them, the expression of PMAE index is:
[0136] ;
[0137] ;
[0138] in, Respectively represent the predicted value and true value of the kth sample in a batch; Represents the number of all samples in a batch; k represents the kth sample among all samples in a batch; represents the mean absolute error.
[0139] Table 1 PMAE index results
[0140] Heat (PMAE%) Quality (PMAE%) Fat (PMAE%) Carbohydrate (PMAE%) Protein (PMAE%) Average (PMAE%) 14.0 10.3 22.6 19.5 20.2 17.3
[0141] The present invention uses food RGB images and depth images as input, uses a feature extractor to extract features of different scales, and uses the spatial information in the food depth image to enhance and supplement the food RGB image features. Then, by considering that features of different scales have an impact on the prediction results, a bidirectional feature pyramid module is designed to fuse features of different scales, and finally multi-task learning is used to predict calories, mass, protein, fat, and carbohydrates. The present invention has a significant improvement in PMAE indicators, which proves the effectiveness of the present invention.
[0142] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.
Claims
1. A food nutrition assessment method based on multi-scale information fusion, It is characterized in that include: Get food RGB image and depth image; Use the feature extractor to extract RGB image features and depth image features at various scales; Based on the multimodal fusion algorithm, the RGB image features and depth image features of each scale are fused to obtain the target fusion features of each scale, including: After global average pooling of the deep image features at each scale, multi-layer channels and nonlinear activation functions are used to obtain the channel attention weights corresponding to each scale. The RGB image features at each scale are weighted using the channel attention weights corresponding to each scale to obtain the first fusion features at each scale. After concatenating the depth image features and RGB image features of each scale, multi-layer convolution is used to obtain the second fusion features of each scale; Add the first fusion feature, the second fusion feature and the RGB image feature of each scale to obtain the target fusion feature of each scale; According to the multi-scale feature fusion algorithm based on the bidirectional feature pyramid, the target fusion features of each scale are enhanced to obtain the bidirectional enhanced fusion features of each scale, including: The preset first convolution kernel is used to unify the number of channels of the target fusion features of each scale to obtain the channel fusion features of each scale; The channel fusion features of the last scale are used as the unidirectional enhanced fusion features of the last scale, and then the bilinear interpolation is used to enlarge the size of the unidirectional enhanced fusion features of the last scale; the unidirectional enhanced fusion features of the last scale after the scale enlargement are added to the channel fusion features of the corresponding scale of the previous level to obtain the unidirectional enhanced fusion features of the corresponding scale of the previous level, and so on, until the unidirectional enhanced fusion features of the first scale are obtained; The unidirectional enhanced fusion features of the first scale are used as the bidirectional enhanced fusion features of the first scale, and then the preset second convolution kernel is used to scale down the bidirectional enhanced fusion features of the first scale; the scaled-down bidirectional enhanced fusion features of the first scale are added and fused with the unidirectional enhanced fusion features of the corresponding scale of the next level to obtain the bidirectional enhanced fusion features of the second scale, and so on, until the bidirectional enhanced fusion features of the last scale are obtained; The bidirectional enhanced fusion features of each scale are pooled and concatenated to obtain the prediction features; the prediction features are input into the multi-task learning model, and after multiple linear layers and nonlinear activation functions, the predicted values of various nutrients in food are output.
2. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that After globally averaging the deep image features at each scale, a multi-layer channel and a nonlinear activation function are used to obtain the channel attention weight corresponding to each scale; The RGB image features of each scale are weighted by using the channel attention weights corresponding to each scale, and the first fusion features of each scale are obtained, including: After performing a global average pooling operation on the deep image features of the i-th scale, a multi-layer channel and a nonlinear activation function are used to obtain the channel attention weight corresponding to the i-th scale, which is expressed as: in, represents the pooling weight obtained after global average pooling of the deep image features of the i-th scale; f d,i represents the deep image feature of the i-th scale; GAP(·) represents global average pooling; w i represents the channel attention weight corresponding to the i-th scale; FC i,1 represents the first fully connected layer corresponding to the i-th scale; FC i,2 represents the second fully connected layer corresponding to the i-th scale; ReLU(·) represents the ReLU nonlinear activation function; Sigmoid(·) represents the Sigmoid nonlinear activation function; The RGB image features of the i-th scale are weighted using the channel attention weight corresponding to the i-th scale to obtain the first fusion feature of the i-th scale, which is expressed as: in, represents the first fusion feature of the i-th scale; f r,i represents the RGB image features of the i-th scale; * represents the weighted processing of the RGB image features of the i-th scale by channel-by-channel multiplication operation; Get the first fusion features of each scale in turn n represents the total number of all scales.
3. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that After the depth image features and RGB image features of each scale are spliced, multi-layer convolution is used to obtain the second fusion features of each scale, including: The depth image feature of the i-th scale is concatenated with the RGB image feature of the i-th scale to obtain the concatenated feature of the i-th scale, which is expressed as: in, Indicates the splicing operation; f r,i represents the RGB image features of the i-th scale; f d,i represents the depth image feature of the i-th scale; f t,i represents the splicing feature of the i-th scale; Based on the splicing features of the i-th scale, multi-layer convolution is used to obtain the second fusion features of the i-th scale, which is expressed as: Among them, θ m,i Represents multi-layer convolution operations; represents the second fusion feature of the i-th scale; The second fusion features of each scale are obtained in turn n represents the total number of all scales.
4. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that The first fusion feature, the second fusion feature, and the RGB image feature of the i-th scale are added to obtain the target fusion feature of the i-th scale, and its expression is: in, represents the first fusion feature of the i-th scale; represents the second fusion feature of the i-th scale; f r,i represents the RGB image features of the i-th scale; F i represents the target fusion feature of the i-th scale; Represents element-wise addition operation; Obtain the target fusion features F of each scale in turn = {F 1 ,…,F n }, n represents the total number of all scales.
5. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that The preset first convolution kernel is used to unify the number of channels of the target fusion features of each scale to obtain the channel fusion features of each scale; the channel fusion features of the last scale are used as the unidirectional enhanced fusion features of the last scale, and then the unidirectional enhanced fusion features of the last scale are enlarged by bilinear interpolation; the unidirectional enhanced fusion features of the last scale after the scale enlargement are added to the channel fusion features of the corresponding scale of the previous level to obtain the unidirectional enhanced fusion features of the corresponding scale of the previous level, and so on, until the unidirectional enhanced fusion features of the first scale are obtained, including: A 1×1 convolution kernel is used to fuse the features of targets at each scale {F 1 ,…,F n The number of channels of each scale is unified, and the target fusion features of each scale with the same number of channels are obtained as the channel fusion features of each scale {F 1,1 , ... F n,1 }; wherein the 1×1 convolution kernel is a preset first convolution kernel; Bilinear interpolation is used to enhance the unidirectional fusion feature F at the (i+1)th scale i+1,2 Perform a scale-up operation; add the channel fusion feature of the i-th scale to the unidirectional enhanced fusion feature of the (i+1)-th scale after scale-up to obtain the unidirectional enhanced fusion feature of the i-th scale, which is expressed as: Among them, F i , 2 represents the unidirectional enhanced fusion feature of the i-th scale; F i , 1 represents the channel fusion feature of the i-th scale; when i=n, F n,2 =F n,1 ; represents the element-wise addition operation; μ(·) represents the linear interpolation operation; F i+1,2 represents the unidirectional enhanced fusion feature of the (i+1)th scale; μ(F i+1,2 ) represents the unidirectional enhanced fusion feature of the (i+1)th scale after scale enlargement; The unidirectional enhanced fusion features F of each scale are obtained in turn 2 ={F 1,2 , ..., F n,2 }, n represents the total number of all scales.
6. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that The unidirectional enhanced fusion features of the first scale are used as the bidirectional enhanced fusion features of the first scale, and then a preset second convolution kernel is used to reduce the scale of the bidirectional enhanced fusion features of the first scale; The bidirectional enhanced fusion features of the first scale after the scale reduction are added and fused with the unidirectional enhanced fusion features of the corresponding scale of the next level to obtain the bidirectional enhanced fusion features of the second scale, and so on until the bidirectional enhanced fusion features of the last scale are obtained, including: A 3×3 convolution kernel with a step size of 2 is used to enhance the bidirectional fusion feature F of the (i-1)th scale. i-1,3 The scale is reduced; the unidirectional enhanced fusion feature of the i-th scale is added and fused with the bidirectional enhanced fusion feature of the (i-1)-th scale after the scale reduction to obtain the bidirectional enhanced fusion feature of the i-th scale, which is expressed as: Among them, F i,3 represents the bidirectional enhanced fusion feature of the i-th scale; F i,2 represents the unidirectional enhanced fusion feature of the i-th scale; when i=1, F 1,3 =F 1,2 ; Represents element-wise addition operation; ∈ i Indicates the 3×3 convolution kernel with a step size of 1 used in the fusion operation; ω i represents the 3×3 convolution kernel with a step size of 2 used for the scale reduction operation of the bidirectional enhanced fusion feature of the (i-1)th scale; F i-1,3 represents the bidirectional enhanced fusion feature of the (i-1)th scale; ω i (F i-1,3 ) represents the bidirectional enhanced fusion feature of the (i-1)th scale after the scale is reduced; the 3×3 convolution kernel with a step size of 2 is the preset second convolution kernel; The bidirectional enhanced fusion features F of each scale are obtained in turn 3 ={F 1,3 ,…,F n,3 }, n represents the total number of all scales.
7. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that The feature extractor is a pre-trained feature extraction network; wherein two feature extractors with different parameters but the same structure are used to extract RGB image features and depth image features respectively.
8. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that The bidirectional enhanced fusion features of each scale are pooled and concatenated to obtain the prediction features, which are expressed as follows: F f,i =GAP(F i,3 ) F f ={F f,1 ,…,F f,n } Among them, F f,i represents the final fusion feature of the i-th scale after the global average pooling operation; F i,3 represents the bidirectional enhanced fusion feature of the i-th scale; GAP(·) represents the global average pooling operation; F f Represents the final fusion features of each scale after the global average pooling operation; Indicates splicing operation; F final Represents the predicted feature.
9. The food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that The prediction features are input into the multi-task learning model, and after multiple linear layers and nonlinear activation functions, the predicted values of various nutrients in the food are output, and their expressions are: F' final =ReLU(W 1 ·F final +b 1 ) Among them, F' final Represents the predicted features after mapping; F final represents the prediction feature; W 1 represents the weight of the first linear layer; b 1 represents the bias of the first linear layer; Represents the first linear layer and the second linear layer corresponding to predicting the j-th nutrient in the multi-task learning model; represents the predicted value of the j-th nutrient; j∈{cal,mass,fat,carb,protein} indicates that the nutrient types are calories, mass, fat, carbon water and protein; ReLU(·) represents the ReLU nonlinear activation function.
10. A food nutrition assessment method based on multi-scale information fusion according to claim 1, It is characterized in that The loss function of the multi-task learning model is expressed as: Among them, L total represents the total loss of all nutrients; H represents the number of types of all nutrients; K represents the number of all samples in a batch; l j represents the loss of the jth nutrient; represents the true value of the jth nutrient in the kth sample in a batch; represents the predicted value of the jth nutrient of the kth sample in a batch; j∈{cal,mass,fat,carb,protein} indicates that the nutrient types are calories, mass, fat, carbon water and protein.
Citation Information
Patent Citations
RGBD saliency detection method based on multi-scale feature fusion
CN111242138A
Text detection method based on attention feature fusion and hole residual feature enhancement
CN113486890A
Food nutritional ingredient content prediction method and system based on cross-modal attention mechanism
CN114529790A
Multi-modal marine target detection method based on multi-scale capsule and Bi-FPN
CN115471676A
Food evaluation method and device, electronic equipment and storage medium
CN116883991A
