A system and method for managing ingredient intake based on images

Abstract

A system and method for image-based ingredient intake management, the method comprising the following steps: acquiring an image of one or more foods to be identified by an image acquisition module from an image recognition module; obtaining multi-layered image visual features by performing feature acquisition on the image through the image recognition module, the image visual features including a first feature layer and a second feature layer; performing food identification based on the image visual features; and displaying the name of the food to be identified and its nutritional information on a display unit on a terminal device via a graphical user interface.

Description

Technical Field

[0001] This invention relates to a system and method for users to manage ingredient intake, and more particularly to a system and method for managing ingredient intake based on food photographs. Background Technology

[0002] With globalization and varying personal preferences, human dietary habits have become increasingly diverse and rapidly evolving. Furthermore, with rising health awareness, people are paying more attention to the various components and calories in their diets, which is especially important for groups requiring precise dietary control, such as athletes and patients. However, many foods in daily life lack clear labeling or are processed to the point where their composition is unclear. Therefore, it's impossible to accurately know their ingredients and nutrient content, easily leading to unknowingly consuming excessive amounts of specific nutrients or insufficient amounts. Moreover, the general public lacks nutritional knowledge and understanding, making it impossible to independently determine the nutrient content of their daily diet. Summary of the Invention

[0003] The main objective of this invention is to solve the problem of not being able to easily know the amount of nutrients ingested in daily diet.

[0004] This invention discloses a method for image-based component intake management, comprising the following steps: acquiring an image of one or more foods to be identified, captured by an image acquisition module, from an image recognition module; obtaining multi-layered image visual features by performing feature acquisition on the image through the image recognition module, the image visual features including a first feature layer and a second feature layer, wherein the first feature layer is located after the second feature layer; performing food identification based on the image visual features, the food identification including: performing a name matching algorithm on the first feature layer of the image visual features to associate the one or more foods to be identified with one or more food names stored in a food database; and performing a nutritional prediction algorithm on the second feature layer of the image visual features to associate the one or more foods to be identified with one or more nutritional information in the food database; and displaying the food name and nutritional information of the foods to be identified through a graphical user interface on a terminal device.

[0005] In one embodiment, the multi-layer image visual feature has n layers, the first feature layer is the image visual feature of the nth layer and is a 1×512 feature vector, and the second feature layer is the image visual feature of the (n-1)th layer and is a 1×768 feature vector.

[0006] In one embodiment, the nutrition prediction algorithm obtains one or more unit component information and a weight information by performing a unit component prediction and a weight prediction on the second feature layer, respectively. The nutrition information is a product of the unit component information and the weight information. The unit component information includes one unit of calories, one unit of carbohydrates, one unit of protein and one unit of lipid of the food to be identified, and the weight information is a predicted weight of the food to be identified.

[0007] In one embodiment, the name matching algorithm includes the following steps: the first feature layer is input to an encoder, and a feature vector of the first feature layer is multiplied by a first weight matrix to obtain a first output feature vector, thereby filtering out multiple important information highly associated with the meal name; the first output feature vector is input to a decoder and multiplied by a second weight matrix to obtain a second output feature vector, thereby reconstructing a second output feature vector with the same dimension as the first feature layer based on the first output feature vector; the second output feature vector is superimposed with the first feature layer to output a third output feature vector to a meal name recognizer; and the meal name recognizer multiplies the third output feature vector with a third weight matrix from a text encoder to obtain a distribution vector of dimension 1×N, where the highest score in the distribution vector is a recognition result, and N is the number of meal names in the multiple meal name lists stored in the food database; wherein, the nutrition prediction algorithm includes a unit component prediction and a weight prediction, and the unit component prediction includes the following steps: the second feature layer is input to a nutrition encoder, and a feature vector of the second feature layer is multiplied by a first weight matrix to obtain a first output feature vector, thereby filtering out multiple important information highly associated with the meal name; the first output feature vector is input to a decoder and multiplied by a second weight matrix to obtain a second output feature vector, thereby reconstructing a second output feature vector with the same dimension as the first feature layer; the second output feature vector is superimposed with a first vector from the first feature layer to output a third output feature vector to a meal name recognizer; and the meal name recognizer multiplies the third output feature vector with a third weight matrix from a text encoder to obtain a distribution vector of dimension 1×N, where the highest score in the distribution vector is a recognition result, and N is the number of meal names in the multiple meal name lists stored in the food database; wherein, the nutrition prediction algorithm includes a unit component prediction and a weight prediction, and the unit component prediction includes the following steps: the second feature layer is input to a nutrition encoder, and a feature vector of the second feature layer is superimposed by a first weight matrix from a text encoder to obtain a second output feature vector, thereby filtering out multiple important information highly A fourth output feature vector is obtained by doubling the fourth weight matrix to filter out multiple important pieces of information highly correlated with food components; the fourth output feature vector is input into one or more nutrient component predictors, and the fourth output feature vector is doubly multiplied with one or more fifth weight matrices to output one or more first predicted values, so as to obtain the first predicted values ​​based on the fourth output feature vector, and the first predicted values ​​reflect the unit nutrient information of the one or more foods to be identified; wherein, the weight prediction includes the following steps: the second feature layer is input into a weight encoder, a feature vector of the second feature layer is doubly multiplied with a fifth weight matrix to obtain a fifth output feature vector to filter out multiple important pieces of information highly correlated with food weight; the fifth output feature vector is superimposed with the second feature layer to output a sixth output feature vector to a weight predictor; and the weight predictor is doubly multiplied with a seventh weight matrix to obtain a second predicted value, so as to obtain the second predicted value based on the sixth output feature vector, and the second predicted value reflects the predicted weight of the one or more foods to be identified.

[0008] The present invention also discloses a non-transitory machine-readable medium storing a set of instructions, the set of instructions including: one or more instructions, when executed by a processor of a terminal device associated with a user, causing the terminal device to perform: acquiring from an image recognition module an image captured by an image acquisition module including one or more foods to be identified; obtaining multi-layer image visual features by performing feature acquisition on the image through the image recognition module, the image visual features including a first feature layer and a second feature layer; performing food identification based on the image visual features, the food identification including: performing a name matching algorithm on the first feature layer of the image visual features to associate the one or more foods to be identified with one or more food names stored in a food database; and performing a nutritional prediction algorithm on the second feature layer of the image visual features to associate the one or more foods to be identified with one or more nutritional information in the food database; and displaying the food name and nutritional information of the foods to be identified via a graphical user interface on a display unit on the terminal device.

[0009] In one embodiment, the multi-layer image visual feature has n layers, the first feature layer is the image visual feature of the nth layer and is a 1×512 feature vector, and the second feature layer is the image visual feature of the (n-1)th layer and is a 1×768 feature vector.

[0010] In one embodiment, the nutrition prediction algorithm obtains one or more unit component information and a weight information by performing a unit component prediction and a weight prediction on the second feature layer, respectively. The nutrition information is a product of the unit component information and the weight information. The unit component information includes one unit of calories, one unit of carbohydrates, one unit of protein and one unit of lipid of the food to be identified, and the weight information is a predicted weight of the food to be identified.

[0011] In one embodiment, the name matching algorithm includes the following steps: the first feature layer is input to an encoder, and a feature vector of the first feature layer is multiplied by a first weight matrix to obtain a first output feature vector, thereby filtering out multiple important information highly associated with the meal name; the first output feature vector is input to a decoder and multiplied by a second weight matrix to obtain a second output feature vector, thereby reconstructing a second output feature vector with the same dimension as the first feature layer based on the first output feature vector; the second output feature vector is superimposed with the first feature layer to output a third output feature vector to a meal name recognizer; and the meal name recognizer multiplies the third output feature vector with a third weight matrix from a text encoder to obtain a distribution vector with dimension 1×N, where the highest score in the distribution vector is a recognition result, and N is the number of meal names in the multiple meal name lists stored in the food database; wherein, the nutrition prediction algorithm includes a unit component prediction and a weight prediction, and the unit component prediction includes the following steps: the second feature layer is input to a nutrition encoder, and a feature vector of the second feature layer is multiplied by a first weight matrix from a first weight matrix to obtain a first output feature vector, thereby filtering out multiple important information highly associated with the meal name; the first output feature vector is input to a decoder and multiplied by a second weight matrix to obtain a second output feature vector, thereby reconstructing a second output feature vector with the same dimension as the first feature layer; the second output feature vector is superimposed with a first vector from the first feature layer to output a third output feature vector to a meal name recognizer; and the meal name recognizer multiplies the third output feature vector with a third weight matrix from a text encoder to obtain a distribution vector with dimension 1×N, where the highest score in the distribution vector is a recognition result, and N is the number of meal names in the multiple meal name lists stored in the food database; wherein, the nutrition prediction algorithm includes a unit component prediction and a weight prediction, and the unit component prediction includes the following steps: the second feature layer is input to a nutrition encoder, and a feature vector of the second feature layer is superimposed by a first weight matrix from a first weight matrix; the second output feature vector is superim A fourth output feature vector is obtained by the inner product of four weight matrices to filter out multiple important pieces of information highly correlated with food components; the fourth output feature vector is input into one or more nutrient component predictors, and the fourth output feature vector is inner-producted with one or more fifth weight matrices to output one or more first predicted values, so as to obtain the first predicted values ​​based on the fourth output feature vector, and the first predicted values ​​reflect the unit nutrient information of one or more foods to be identified; wherein, the weight prediction includes the following steps: the second feature layer is input into a weight encoder, a feature vector of the second feature layer is inner-producted with a fifth weight matrix to obtain a fifth output feature vector to filter out multiple important pieces of information highly correlated with food weight; the fifth output feature vector is superimposed with the second feature layer to output a sixth output feature vector to a weight predictor; and the weight predictor inner-products the sixth output feature vector with a seventh weight matrix to obtain a second predicted value, so as to obtain the second predicted value based on the sixth output feature vector, and the second predicted value reflects the predicted weight of one or more foods to be identified.

[0012] This invention further discloses a system for image-based component ingestion management, comprising: an image recognition module, which acquires an image captured by an image acquisition module, including one or more food items to be identified; the image recognition module performs feature acquisition on the image to obtain multi-layer image visual features, the image visual features including a first feature layer and a second feature layer, wherein the first feature layer is located after the second feature layer; and a meal name recognition module, comprising an adapter, a meal name recognizer, and a text encoder; the adapter extracts a meal name association feature from the first feature layer; the meal name recognizer uses the meal name association feature and the text encoder generates a feature based on N known meal names. The transpose of the matrix is ​​used to obtain a 1×N distribution vector, and the one with the highest probability in the distribution vector is used as a recognition result; and a nutrition recognition module includes a nutrition encoder, multiple nutrient component predictors, a weight encoder and a weight predictor. The nutrition encoder extracts a nutrient association feature from the second feature layer. The nutrient component predictor obtains multiple nutrient information based on the nutrient association feature. The weight encoder extracts a weight association feature from the second feature layer. The weight predictor obtains a predicted weight of the food to be identified based on the weight association feature. The nutrition recognition module obtains complete nutrient information based on the nutrient information and the predicted weight.

[0013] In one embodiment, the image visual features are multi-layered, with n layers. The first feature layer is the image visual features of the nth layer and is a 1×512 feature vector, and the second feature layer is the image visual features of the (n-1)th layer and is a 1×768 feature vector. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a system according to an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of a method according to an embodiment of the present invention.

[0016] Figure 3 This is a schematic diagram of an embodiment of the present invention.

[0017] Figure 4 This is a schematic diagram of image visual features according to an embodiment of the present invention.

[0018] Figure 5 This is a schematic diagram of an algorithm according to an embodiment of the present invention.

[0019] Figure 6 This is a schematic diagram of a name matching algorithm according to an embodiment of the present invention.

[0020] Figure 7 This is a schematic diagram of nutritional information according to an embodiment of the present invention.

[0021] Figure 8 This is a schematic diagram of a nutrient prediction algorithm according to an embodiment of the present invention. Detailed Implementation

[0022] The terminology used herein is for the purpose of illustrating particular embodiments only and is not intended to limit the invention. Unless the context otherwise indicates, the singular forms “a” and “the” used herein may also include the plural forms.

[0023] This document uses flowcharts to illustrate the steps performed according to the embodiments. It should be understood that the steps do not necessarily have to be performed precisely in the order described herein; instead, these steps can be performed in reverse order or simultaneously. Furthermore, other steps can be added to the process, or one or more steps can be removed from these processes. A detailed description and technical content of the present invention are now provided in conjunction with the accompanying drawings.

[0024] Figure 1 The present invention is illustrated in the following system diagram. The system may include a terminal device 10 associated with a user, an application 10a, a graphical user interface 10b, and a server 20. The terminal device 10 may be a mobile communication device (such as a mobile phone, tablet, laptop, or wearable electronic device) or a computer. The terminal device 10 may include a processor 11, a memory 12, a storage medium 13, an input interface 14, an output interface 15, a display unit 16, and an image acquisition module 17 (such as a camera lens). The application 10a is installed on the terminal device 10 and is configured to display the graphical user interface 10b on the display unit 16 of the terminal device 10. The server 20 may include a processor 21, a memory 22, and a storage medium 23. The terminal device 10 is connected to the server 20 via a network N. The server 20 may further include or be connected to a database 30, and the server 20 is configured to execute a food database module 20a, an image recognition module 20b, and a food identification module 20c according to one or more instructions. In this example, the terminal device 10 may be a computer device, including a non-transitory machine-readable medium. The image recognition module 20b may be a visual feature extractor. The non-transitory machine-readable medium stores a set of instructions, which include one or more instructions. When the one or more instructions are executed by the processor 11 of the user's terminal device 10, the terminal device 10 performs the methods exemplified below.

[0025] Figure 2This is a schematic diagram of a method in one example of the present invention. In operation 100, the image acquisition module 17 of the terminal device 10 is used to photograph one or more food items 41, 42 to be identified, thereby obtaining an image 43 including one or more of the food items 41, 42 to be identified. The image 43 can be a static photograph or a dynamic video, such as... Figure 3 As shown.

[0026] In operation 200, the image recognition module 20b is used to perform feature acquisition on the image 43 to obtain multi-layer image visual features 50, such as... Figure 4 As shown, the image visual feature 50 has n layers, where n is an integer greater than 1. The image visual feature 50 includes a first layer to an nth layer. The order of the first layer to the nth layer is obtained according to the output order of the feature acquisition of the image 43 by the image recognition module 20b. In this example, the dimension of the first layer to the (n-1)th layer is a feature vector with a dimension of 1×768, and the dimension of the nth layer is a feature vector with a dimension of 1×512. In some examples, the image visual feature 50 includes a first feature layer 51 and a second feature layer 52, which are defined as the image visual feature 50 of the nth layer and the (n-1)th layer, respectively. The first feature layer 51 and the second feature layer 52 can be regarded as the last layer and the second-to-last layer of the image visual feature 50.

[0027] In this example, the image recognition module 20b performs the feature acquisition based on a contrastive language-image pre-training (CLIP) related artificial intelligence (AI) model, such as the Chinese CLIP Vision Encoder, to generate the image visual features 50 from the first layer to the nth layer.

[0028] like Figure 5 As shown, operation 300a involves using the food recognition module 20c to perform a name matching algorithm on the first feature layer 51, which finds the associated food names in a food database based on the image features of one or more foods 41 and 42 to be identified in the image 43. Operation 300b involves using the food recognition module 20c to perform a nutrition predicting algorithm on the second feature layer 52, which predicts the associated nutritional information based on the image features of the foods 41 and 42 to be identified in the image 43. In some embodiments, the food database may be located in the food database module 20a and accessed; or it may be located in the database 30 and accessed via the server 20, and the food database stores pre-stored or subsequently added food names and nutritional information.

[0029] See Figure 5 The food recognition module 20c identifies multiple image features 41a, 41b, 41c, and 41d of the food to be identified 41 in the first feature layer 51, and multiple image features 42a, 42b, 42c, and 42d of the food to be identified 42 in the first feature layer 51. After the name matching algorithm, multiple associated features 41x, 41y, and 41z of the food to be identified 41 and multiple associated features 42x, 42y, and 42z of the food to be identified 42 can be obtained respectively. The associated features 41x, 41y, and 41z can obtain a food name 41n associated with a food item 41-1 (e.g., hamburger), and the associated features 42x, 42y, and 42z can obtain a food name 42n associated with a food item 42-1 (e.g., roast chicken). In one example, the image features 41a, 41b, 41c, 41d and the image features 42a, 42b, 42c, 42d are processed and calculated to obtain the associated features 41x, 41y, 41z and the associated features 42x, 42y, 42z. The associated features 41x, 41y, 41z and the associated features 42x, 42y, 42z are used as the basis for predicting the food names 41n, 42n.

[0030] Figure 6 This example illustrates an operational structure 21a for executing the name matching algorithm. The operational structure 21a includes an adapter 211, a meal name recognizer 212, and a text encoder 213. The adapter 211 includes an encoder 211a and a decoder 211b. The first feature layer 51 (with a 1×512 feature vector) is input to the adapter 211. The encoder 211a performs an inner product of the vector dimension of the first feature layer 51 with a first weight matrix (512×128 in this example) to reduce it to a 1×128 first output feature vector. The decoder 211b then performs an inner product of the first output feature vector with a second weight matrix (128×512 in this example) to enlarge it to a 1×512 second output feature vector. The dimension of the second output feature vector is the original size, i.e., the same as that of the first feature layer 51.

[0031] The adapter 211 is based on the information bottleneck principle. Utilizing the filtering effect generated during training, it extracts a meal name association feature from the first feature layer 51. This meal name association feature is information related to the meal name; for example, a large number of dense, granular objects might be associated with rice-based meals, while multiple linear objects might be associated with noodle-based meals. In other words, after processing by the encoder 211a and decoder 211b, the features associated with the meal name are extracted from the first feature layer 51 and represented in the second output feature vector.

[0032] The encoder 211a reduces the first feature layer 51 from a 1×512 dimension to a 1×128 first output feature vector system to filter out multiple important pieces of information highly correlated with the dish name (such as image features of the food surface or edges), and discards information that is not highly correlated with or uncorrelated with the dish name (such as background such as the table). The decoder 211b expands the first output feature vector from a 1×128 dimension back to a 1×512 second output feature vector system to focus on and emphasize these important pieces of information. In this way, through feature reconstruction by the encoder 211a and the decoder 211b, the second output feature vector can better reflect the features related to dish name recognition.

[0033] Next, the meal name association feature (i.e., the second output feature vector) output by the adapter 211 is superimposed with the input image features (the first feature layer 51) by a first vector summation SP1 to perform a residual connection and strengthen the feature signal. The third output feature vector after the first vector summation SP1 is further transmitted to the meal name recognizer 212 for recognition. The meal name recognizer 212 is a third weight matrix (in this example, the dimension is 512×N), where N is the number of meal names in the multiple meal name lists 60 stored in the food database, and is an integer greater than 1. The text encoder 213 represents the meal name list 60 as the third weight matrix and provides it to the meal name recognizer 212.

[0034] The third output feature vector, obtained by superimposing the first vector with SP1, is passed to the meal name recognizer 212 and inverted with the third weight matrix to obtain a 1×N distribution vector. Finally, the meal name with the highest score (highest probability) is selected from the 1×N distribution vector as a recognition result 61. In this example, the text encoder 213 uses the Chinese CLIP text encoder.

[0035] Back Figure 5The food recognition module 20c also identifies multiple image features 41A, 41B, 41C, 41D of the food to be identified 41 in the second feature layer 52 and multiple image features 42A, 42B, 42C, 42D of the food to be identified 42 in the second feature layer 52. Through the nutrition prediction algorithm, a unit component prediction and a weight prediction are performed on the image features 41A, 41B, 41C, 41D and the image features 42A, 42B, 42C, 42D, respectively. Multiple associated features 41X, 41Y, 41Z of the food to be identified 41 and multiple associated features 42X, 42Y, 42Z of the food to be identified 42 are obtained respectively. That is, the nutrition prediction algorithm includes the unit component prediction and the weight prediction. In one example, the image features 41A, 41B, 41C, 41D and the image features 42A, 42B, 42C, 42D are processed and calculated to obtain the associated features 41X, 41Y, 41Z and the associated features 42X, 42Y, 42Z. The associated features 41X, 41Y, 41Z serve as the basis for predicting the weight information 410 and multiple unit component information 411, 412, 413, 414 of the food to be identified 41. The associated features 42X, 42Y, 42Z serve as the basis for predicting the weight information 420 and multiple unit component information 421, 422, 423, 424 of the food to be identified 42.

[0036] The weight information 410 and 420 are the predicted total weight (in g) of the food items 41 and 42 to be identified, respectively. The unit composition information 411 and 421 are the unit calories (in Kcal / g) of the food items 41 and 42 to be identified, respectively. The unit composition information 412 and 422 are the unit carbohydrates of the food items 41 and 42 to be identified, respectively. The unit composition information 413 and 423 are the unit protein of the food items 41 and 42 to be identified, respectively. The unit composition information 414 and 424 are the unit lipids of the food items 41 and 42 to be identified, respectively. The food recognition module 20c can calculate multiple nutritional information 4101, 4102, 4103, 4104, 4201, 4202, 4203, and 4204 corresponding to the foods to be identified 41 and 42 based on the product of the weight information 410 and 420 and the unit component information 411, 412, 413, 414, 421, 422, 423, and 4204. Figure 7 As shown, in detail, the nutritional information 4101, 4102, 4103, and 4104 are the total calories, total carbohydrates, total protein, and total lipids of the food to be identified 41, respectively, and the nutritional information 4201, 4202, 4203, and 4204 are the total calories, total carbohydrates, total protein, and total lipids of the food to be identified 42, respectively.

[0037] Figure 8This example illustrates an operational structure 21b used to execute the nutrition prediction algorithm. The operational structure 21b includes a nutrition encoder 214, multiple nutrient predictors 215a, 215b, 215c, and 215d, a weight encoder 216, and a weight predictor 217. The combination of the nutrition encoder 214 and the nutrient predictors 215a, 215b, 215c, and 215d is used to estimate calorie and nutrient (e.g., carbohydrates, proteins, lipids) information based on the second feature layer 52. When the second feature layer 52 (with a 1×768 feature vector) is input to the nutrition encoder 214, the nutrition encoder 214 performs an inner product of the input 1×768 feature vector with a fourth weight matrix (768×256 in this example) to obtain a 1×256 fourth output feature vector. The nutrition encoder 214 is used to filter out multiple important pieces of information that are highly correlated with food components and discard information that is poorly correlated or uncorrelated with food components. Dimensional encoding can convert the surface information of the second feature layer 52 into feature information associated with food ingredients. For example, a yellow, serrated surface and edges may be associated with high-calorie fried food; a light-colored liquid in a container may be associated with low-calorie beverages. Next, each of the nutrient component predictors 215a, 215b, 215c, and 215d performs an inner product of the fourth feature vector (dimension 1×256) output by the nutrient encoder 214 with a fifth weight matrix (dimension 256×1 in this example) to calculate one or more first predicted values. In this example, there are four nutrient component predictors 215a, 215b, 215c, and 215d, so there are also four fifth weight matrices and four first predicted values, which are the calories per gram (unit component information 411, 421), carbohydrates per gram (unit component information 412, 422), fat per gram (unit component information 413, 423), and protein per gram (unit component information 414, 424) reflected by the second feature layer 52. The nutrient component predictors 215a, 215b, 215c, and 215d can be determined according to the quantity of the nutrient component. The nutrient component predictors 215a, 215b, 215c, and 215d are used to convert the feature information related to food components extracted by the nutrient encoder 214 into actual predicted values.

[0038] This example uses calories, carbohydrates, proteins, and lipids as examples of nutrients, but the invention is not limited to these. Depending on the needs of actual application, it may also include other components, such as minerals, vitamins, dietary fiber, etc. On the other hand, the nutrients referred to here are not necessarily the food components that are beneficial to the human body, but should cover other food components.

[0039] The weight encoder 216 performs an inner product of the input 1×768 feature vector (the second feature layer 52) with a sixth weight matrix (768×768 in this example) to obtain a fifth output feature vector with a dimension of 1×768. The weight encoder 216 is used to filter out multiple important pieces of information that are highly correlated with the weight of the food and discard information that is not highly correlated with or has no correlation with the weight of the food. In other words, the weight encoder 216 acquires feature factors related to weight, such as: light and shadow, container size, reference objects, etc. Furthermore, using residual connection technology, the feature factor is superimposed with the original input image features (the second feature layer 52) by a second vector SP2, and a sixth output feature vector (dimension 1×768) is output to the weight predictor 217. The sixth output feature vector is then multiplied by a seventh weight matrix (768×1 in this example) to calculate a second predicted value. This second predicted value represents or includes the predicted weight of the food items 41 and 42 to be identified, thereby obtaining the weight information 410 and 420. The weight predictor 217 is used to convert the feature information related to the food weight extracted by the weight encoder 216 into actual predicted values.

[0040] Finally, the predicted weights (weight information 410, 420) are multiplied by M with the individual single gram nutrient values ​​(unit composition information 411, 412, 413, 414, 421, 422, 423, 424) to obtain complete nutrient composition information, including calorie information 71, carbohydrate information 72, protein information 73, and lipid information 74.

[0041] In one example, the nutrient encoder 214 and the nutrient component predictors 215a, 215b, 215c, 215d based on a unit component prediction model, and the weight encoder 216 and the weight predictor 217 based on a weight prediction model, learn from each other interactively during training; however, in some examples, the unit component prediction model and the weight prediction model may also be learned independently.

[0042] The unit composition prediction model and the weight prediction model can be pre-learned from data in the food database to pre-map image features to various information, such as calories, nutrients, weight, etc. In some cases, the encoders and predictors may not necessarily need to interact with the food database during operation and can execute independently. Alternatively, in other cases, the encoders and predictors may interact with the food database during operation.

[0043] In one example, the first weight matrix, the second weight matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the sixth weight matrix are transformation matrices, which serve as the feature transformation basis for the encoder 211a, the decoder 211b, the meal name recognizer 212, the nutrition encoder 214, the nutrition component predictors 215a, 215b, 215c, 215d, and the weight encoder 216, respectively. The dimensions of these matrices mentioned above are merely examples, and the dimensions may differ from those in the examples above. On the other hand, those skilled in the art will understand that the contents of the first weight matrix, the second weight matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the sixth weight matrix are transformation matrices obtained through actual training and learning, which will not be elaborated further here.

[0044] Please refer to the reply. Figure 2 Finally, operation 400 involves the display unit 16 on the terminal device 10 displaying the food names 41n and 42n of the foods to be identified, as well as the nutritional information 4101, 4102, 4103, 4104, 4201, 4202, 4203, and 4204, via the graphical user interface 10b. Figure 7 As shown. In summary, this invention analyzes the food to be identified through vision and displays it on a terminal device, which can help users understand their dietary status, such as calorie intake. Furthermore, this invention analyzes the names and nutritional content of different feature layers (first feature layer and second feature layer) of the visual features of the image, and can perform feature matching of text and images separately, thereby improving matching and prediction efficiency.

[0045] 100, 200, 300a, 300b, 400: Operation

[0046] 10: Terminal device

[0047] 10a: Application

[0048] 10b: Graphical User Interface

[0049] 11: Processor

[0050] 12: Memory

[0051] 13: Storage medium

[0052] 14: Input Interface

[0053] 15: Output Interface

[0054] 16: Display Unit

[0055] 17: Image Acquisition Module

[0056] 20: Server

[0057] 20a: Food Database Module

[0058] 20b: Image Recognition Module

[0059] 20c: Food Recognition Module

[0060] 21: Processor

[0061] 21a, 21b: Operational Structure

[0062] 211: Adapter

[0063] 211a: Encoder

[0064] 211b: Decoder

[0065] 212: Restaurant Name Recognizer

[0066] 213: Text Encoder

[0067] 214: Nutrition Encoder

[0068] 215a, 215b, 215c, 215d: Nutritional Component Predictors

[0069] 216: Weight Encoder

[0070] 217: Weight Predictor

[0071] 22: Memory

[0072] 23: Storage medium

[0073] 30: Database

[0074] 41, 42: Foods to be identified

[0075] 41a, 41b, 41c, 41d, 42a, 42b, 42c, 42d: Image features

[0076] 41A, 41B, 41C, 41D, 42A, 42B, 42C, 42D: Image Features

[0077] 41x, 41y, 41z, 42x, 42y, 42z: Association features

[0078] 41X, 41Y, 41Z, 42X, 42Y, 42Z: Association Features

[0079] 41-1, 42-1: Food Items

[0080] 41n, 42n: Food names

[0081] 410, 420: Weight Information

[0082] 411, 412, 413, 414, 421, 422, 423, 424: Unit composition information

[0083] 4101, 4102, 4103, 4104, 4201, 4202, 4203, 4204: Nutritional Information

[0084] 43: Image

[0085] 50: Image visual features

[0086] 51: First Feature Layer

[0087] 52: Second Feature Layer

[0088] 60: Menu

[0089] 61: Recognition Results

[0090] 71: Calorie Information

[0091] 72: Carbohydrate Information

[0092] 73: Protein Information

[0093] 74: Lipid Information

[0094] N: Network

[0095] SP1: First Vector Superposition

[0096] SP2: Superposition of the second vector

Claims

1. A method for image-managed component ingestion, characterized in that, Includes the following steps: An image of one or more food items to be identified is obtained from an image recognition module and captured by an image acquisition module. The image recognition module performs feature acquisition on the image to obtain multi-layer image visual features, which include a first feature layer and a second feature layer, wherein the first feature layer is located after the second feature layer. Food identification is performed based on the visual features of the image. The food identification includes: A name matching algorithm is applied to the first feature layer of the image's visual features to associate the one or more foods to be identified with one or more food names stored in a food database; and A nutritional prediction algorithm is applied to the second feature layer of the image's visual features to associate the one or more foods to be identified with one or more nutritional information in the food database; and A display unit on a terminal device displays the name of the food to be identified and its nutritional information via a graphical user interface.

2. The method for image-managed component acquisition according to claim 1, characterized in that, The image visual features are multi-layered, with n layers. The first feature layer is the image visual features of the nth layer and is a 1×512 feature vector. The second feature layer is the image visual features of the (n-1)th layer and is a 1×768 feature vector.

3. The method for image-managed component acquisition according to claim 1, characterized in that, The nutrition prediction algorithm obtains one or more unit component information and one weight information by performing a unit component prediction and a weight prediction on the second feature layer, respectively. The nutrition information is a product of the unit component information and the weight information. The unit component information includes one unit of calories, one unit of carbohydrates, one unit of protein and one unit of lipid of the food to be identified. The weight information is a predicted weight of the food to be identified.

4. The method for image-managed component acquisition according to claim 1, characterized in that, The name matching algorithm includes the following steps: The first feature layer is input to an encoder. A feature vector of the first feature layer is multiplied by a first weight matrix to obtain a first output feature vector, so as to filter out multiple important pieces of information that are highly related to the name of the meal. The first output feature vector is input into a decoder and is multiplied by a second weight matrix to obtain a second output feature vector, so as to reconstruct the second output feature vector with the same dimension as the first feature layer based on the first output feature vector; The second output feature vector is superimposed with the first feature layer to produce a third output feature vector, which is then fed into a restaurant name recognizer. The meal name recognizer multiplies the third output feature vector with a third weight matrix from a text encoder to obtain a distribution vector of dimension 1×N. The highest score in the distribution vector is a recognition result, where N is the number of meal names in the multiple meal name lists stored in the food database. The nutrient prediction algorithm includes a unit component prediction and a weight prediction. The unit component prediction includes the following steps: The second feature layer is input to a nutrition encoder. A feature vector from this second feature layer is multiplied by a fourth weight matrix to obtain a fourth output feature vector, which filters out multiple important pieces of information highly correlated with food components; and The fourth output feature vector is input to one or more nutrient component predictors. The fourth output feature vector is inner productted with one or more fifth weight matrices to output one or more first predicted values. The first predicted values ​​are obtained by transforming the fourth output feature vector. The first predicted values ​​reflect the unit nutrient information of the one or more foods to be identified. The weight prediction includes the following steps: The second feature layer is input to a weight encoder. A feature vector of the second feature layer is multiplied by a fifth weight matrix to obtain a fifth output feature vector, so as to filter out multiple important pieces of information that are highly correlated with the weight of the food. The fifth output feature vector is superimposed with the second feature layer to produce a sixth output feature vector, which is then fed into a weight predictor; and The weight predictor multiplies the sixth output feature vector with a seventh weight matrix to obtain a second predicted value, which is then transformed based on the sixth output feature vector to obtain the second predicted value, which reflects the predicted weight of the one or more food items to be identified.

5. A non-transitory machine-readable medium for storing a set of instructions, characterized in that, This set of instructions includes: One or more instructions, when executed by a processor of a terminal device associated with a user, cause the terminal device to perform: An image of one or more food items to be identified is obtained from an image recognition module and captured by an image acquisition module. The image recognition module performs feature acquisition on the image to obtain multi-layer image visual features, which include a first feature layer and a second feature layer. Food identification is performed based on the visual features of the image. The food identification includes: A name matching algorithm is applied to the first feature layer of the image's visual features to associate the one or more foods to be identified with one or more food names stored in a food database; and A nutritional prediction algorithm is applied to the second feature layer of the image's visual features to associate the one or more foods to be identified with one or more nutritional information in the food database; and A display unit on a terminal device displays the name of the food to be identified and its nutritional information via a graphical user interface.

6. The non-transitory machine-readable medium according to claim 5, characterized in that, The image visual features are multi-layered, with n layers. The first feature layer is the image visual features of the nth layer and is a 1×512 feature vector. The second feature layer is the image visual features of the (n-1)th layer and is a 1×768 feature vector.

7. The non-transitory machine-readable medium according to claim 5, characterized in that, The nutrition prediction algorithm obtains one or more unit component information and one weight information by performing a unit component prediction and a weight prediction on the second feature layer, respectively. The nutrition information is a product of the unit component information and the weight information. The unit component information includes one unit of calories, one unit of carbohydrates, one unit of protein and one unit of lipid of the food to be identified. The weight information is a predicted weight of the food to be identified.

8. The non-transitory machine-readable medium according to claim 5, characterized in that, The name matching algorithm includes the following steps: The first feature layer is input to an encoder. A feature vector of the first feature layer is multiplied by a first weight matrix to obtain a first output feature vector, so as to filter out multiple important pieces of information that are highly related to the name of the meal. The first output feature vector is input into a decoder and is multiplied by a second weight matrix to obtain a second output feature vector, so as to reconstruct the second output feature vector with the same dimension as the first feature layer based on the first output feature vector; The second output feature vector is superimposed with the first feature layer to produce a third output feature vector, which is then fed into a restaurant name recognizer. The meal name recognizer multiplies the third output feature vector with a third weight matrix from a text encoder to obtain a distribution vector of dimension 1×N. The highest score in the distribution vector is a recognition result, where N is the number of meal names in the multiple meal name lists stored in the food database. The nutrient prediction algorithm includes a unit component prediction and a weight prediction. The unit component prediction includes the following steps: The second feature layer is input to a nutrition encoder. A feature vector from this second feature layer is multiplied by a fourth weight matrix to obtain a fourth output feature vector, which filters out multiple important pieces of information highly correlated with food components; and The fourth output feature vector is input to one or more nutrient component predictors. The fourth output feature vector is inner productted with one or more fifth weight matrices to output one or more first predicted values. The first predicted values ​​are obtained by transforming the fourth output feature vector. The first predicted values ​​reflect the unit nutrient information of the one or more foods to be identified. The weight prediction includes the following steps: The second feature layer is input to a weight encoder. A feature vector of the second feature layer is multiplied by a fifth weight matrix to obtain a fifth output feature vector, so as to filter out multiple important pieces of information that are highly correlated with the weight of the food. The fifth output feature vector is superimposed with the second feature layer to produce a sixth output feature vector, which is then fed into a weight predictor; and The weight predictor multiplies the sixth output feature vector with a seventh weight matrix to obtain a second predicted value, which is then transformed based on the sixth output feature vector to obtain the second predicted value, which reflects the predicted weight of the one or more food items to be identified.

9. A system for image-managed component ingestion, characterized in that, include: An image recognition module acquires an image captured by an image acquisition module, which includes one or more food items to be identified. The image recognition module performs feature acquisition on the image to obtain multi-layer image visual features, which include a first feature layer and a second feature layer, wherein the first feature layer is located after the second feature layer. A meal name recognition module includes an adapter, a meal name recognizer, and a text encoder. The adapter extracts meal name association features from a first feature layer. The meal name recognizer obtains a 1×N distribution vector based on the meal name association features and a transpose matrix generated by the text encoder based on N known meal names. The highest probability value in this distribution vector is used as a recognition result. A nutrition identification module includes a nutrition encoder, multiple nutrient component predictors, a weight encoder, and a weight predictor. The nutrition encoder extracts a nutrient association feature from the second feature layer, the nutrient component predictor obtains multiple nutrient information based on the nutrient association feature, the weight encoder extracts a weight association feature from the second feature layer, and the weight predictor obtains a predicted weight of the food to be identified based on the weight association feature. The nutrition identification module obtains complete nutrient information based on the nutrient information and the predicted weight.

10. The system for image-managed component acquisition according to claim 9, characterized in that, The image visual features are multi-layered, with n layers. The first feature layer is the image visual features of the nth layer and is a 1×512 feature vector. The second feature layer is the image visual features of the (n-1)th layer and is a 1×768 feature vector.

Images