Automated diet control
Multimodal image text matching is performed through deep learning models, which solves the error problem in diet picture recognition and information extraction, realizes more accurate diet data processing, and improves the reliability and safety of diet control.
Patent Information
- Application Number
- CN202310723135.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2023-06-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art has errors and information loss in diet picture recognition and information extraction, resulting in inaccurate diet data recorded by users and affecting the effect of diet control.
The deep learning model is used to match multimodal image text, and the user's dietary images and descriptions are matched through training models. The model is adjusted using user feedback to ensure that the matching consistency reaches a given success rate and provide more accurate dietary data processing.
It improves the accuracy and reliability of diet data processing, reduces errors, provides safer and more efficient diet control services, and meets the needs of precise diet monitoring.
Smart Images

Figure CN118942083A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of diet calculation, and more particularly, to a method and system for analyzing diet pictures and information and calculating a health-optimized diet plan based on the results. Background Art
[0002] For patients with diabetes or obesity, it is necessary to control blood sugar, nutrition and other indicators, not to eat too much or too little. Daily diet needs to be precisely controlled, even according to professional medical recipes every day.
[0003] There are many computer programs on the market that help patients control their daily diet, especially apps for mobile phones. Some apps require patients to log every meal, some apps have pre-built recipes to follow, and some apps require users to take pictures of their meals and clock them in daily. When users use them, there are often problems such as wrong descriptions, missing descriptions, or wrong or blurry pictures of the photographed meals.
[0004] The known solutions to those problems in the prior art require the user to first enter his diet description, and then the information extraction module of the app extracts the food name A1 and the food quantity A2 from it. Then, the user is guided by the app to take a picture of the food using a mobile phone, and the image object detection module extracts the food name B1 and the food quantity B2 corresponding to the food in the picture, and pairs A1, B1, and A2 with B2. The app then compares the input data, and if inconsistent, indicates that the picture detection result and the user input are inconsistent, and provides feedback to the user to confirm the reshoot or re-recording of the input data. This approach requires long data processing and may suffer from severe information loss. In particular, there may be errors and information loss when extracting the food name and food quantity from the image object detection, and there may be errors and information loss when extracting the food name and food quantity from the information extraction module. This information loss between modules will lead to many errors in comparison or verification, which will have a negative impact on use. Summary of the invention
[0005] Therefore, the task of the present patent application is to provide a method and a system for executing the method in order to improve the necessary data processing and provide a safer, more efficient and reliable service for human dietary control.
[0006] This task has been solved by a method for training a deep learning model for controlling a human diet plan provided by software on a computer, the method comprising the following method steps: providing a multimodal model sample, comprising at least one digital food image and a dietary description of the food in the image; performing a multimodal image-text matching with the deep learning model, run by software executed on the computer, wherein a food image and its corresponding dietary description text from the multimodal model sample are input to the model, and the model outputs a prediction of whether the image and its textual description match; validating the model prediction by a user, wherein as feedback to the model training process, the user confirms the predicted consequence or rejects it; and redoing the multimodal image-text matching and model validation until the consistency predicted by the model reaches a given success rate. The described invention uses a machine learning model (in particular, a deep learning model) to recognize previously provided dietary text descriptions and dietary images, and compare them with images and descriptions of the corresponding current meal that the user intends to eat, to determine whether the dietary requirements extracted by the computer from the previously provided dietary text descriptions and dietary images are met. To do so, a specific way of training the deep learning model is hereby disclosed, so as to make the model more adaptable to this specific scenario. The computer used may be any suitable kind of computer, but it is strongly recommended to use a modern mobile phone, where the corresponding application software (App) can be easily applied and the necessary digital camera for taking digital images is usually already provided. The given target success rate is flexible, but should be at least 90%, since otherwise an App using the trained model would be quite useless, since in delicate use cases such as diet monitoring, too many errors would be unacceptable to the user.
[0007] Advantageous and therefore preferred further developments of the invention emerge from the associated subclaims as well as from the description and the associated drawings.
[0008] One of those preferred further developments of the disclosed model training method comprises: for positive feedback from the user, the corresponding food images and their corresponding description texts used for model prediction are kept within the training process of the model, while for negative feedback from the user, the images and description texts are removed from the training process via a computer.
[0009] Another of those preferred further developments of the disclosed model training method comprises providing the multimodal model samples by retrieving them from a database.
[0010] Another of those preferred further developments of the disclosed model training method includes: the multimodal model samples are created by the following method steps: creating a food picture library; collecting digital food images, which may have one food, multiple foods or multiple food categories, and then using image dataset enhancement to increase the number of image samples; identifying the food name of each food in the image and counting the number of each food; generating a dietary description of the image based on the name and number of the food obtained in the step; creating at least one positive model sample in which the food shown in the image and the description of the food match; creating at least one negative model sample in which the food shown in the image and the description of the food do not match; and storing the image, its corresponding description, and the positive and negative samples as multimodal model samples in a database. This is a preferred approach for creating and providing the required digital images and dietary descriptions for the multimodal model samples. The precise point in time when those multimodal model samples need to be created is irrelevant. It is even unnecessary to include all possible meals allowed according to the dietary plan of the corresponding user. In case the corresponding meal cannot be understood by the computer.
[0011] Another of those preferred further developments of the disclosed model training method includes: using digital image processing operations like cropping, rotation, color conversion, blurring and multi-image synthesis to enhance the sample images to expand more images. These are just a few examples of possible digital image processing operations. Each operation that may enhance the quality and / or quantity of the corresponding sample images is applicable.
[0012] Another of those preferred further developments of the disclosed model training method includes: verification of the consistent model predictions includes checking whether a given threshold is exceeded for the model's output predictions, resulting in consistent images and description text, or whether a given threshold is not exceeded, resulting in inconsistent images and description text. This verification allows the trained model and the app using it to be subsequently applied on real-life dietary images taken by the user to make actual predictions about the eligibility of the intended meal according to the dietary constraints.
[0013] Another of those preferred further developments of the disclosed method comprises that the output prediction of the model comprises values between 0 and 1. Technically, other value ranges may also be used, but the range between 0 (disagreement) and 1 (agreement) has become widely distributed.
[0014] A further solution to the given task is a method for controlling a human diet plan via a computer using a deep learning model trained according to the previously described method, wherein the user takes at least one image of the diet he intends to consume, provides the image to the computer, in particular by using a mobile phone with a digital camera, and / or enters into the computer a description of what and how much he intends to eat, and the computer uses the trained deep learning model to check and verify whether the diet in the picture and / or description matches the set of allowed diets previously entered into the computer by the user. By using the present invention for training a deep learning model, the final trained model can be used to perform the following practical method: it checks whether the diet description of the diet that the user wants to consume and the image taken are consistent. The described diet includes two categories, one is a preset recipe description and the other is each diet entered by the patient. Before each diet, the user takes a picture of the diet record and uploads it to the application or enters what and how much to eat. The algorithm automatically verifies that the preset recipe description matches the picture taken; or verifies that the photo of the diet taken by the user matches the photo previously entered for the corresponding diet or whether the description now entered by the user about what to eat and how much to eat matches the preset recipe description. If it does not match, a reminder message is returned to the user to confirm whether the description is correct.
[0015] Another preferred further development of the disclosed method comprises that the set of allowed diets consists of a preset recipe description and at least one image for each respective diet. According to the training method for the used deep learning model requiring multimodal model samples with digital food images and diet descriptions, the available data related to the allowed diet meals should also provide recipe descriptions and images.
[0016] Another solution to the given task comprises: a computer device comprising a computer readable memory, a processor and a computer program, the computer program being stored on the memory and being executable on the processor, wherein when executing the computer program, the processor implements the method as described above. It further comprises a non-transitory computer readable storage medium storing the computer program for execution by the computing device having the processor, wherein the computer program, when executed by the processor, causes the computing device to perform the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The method and system according to the invention and its advantageous developments in terms of function are described in more detail below using at least one preferred exemplary embodiment with reference to the associated drawings. In the drawings, mutually corresponding elements are provided with the same reference numerals.
[0018] The accompanying drawings show: Figure 1aA sample picture from a collection database showing fried eggs with tomatoes is illustrated.
[0019] Figure 1b Another sample picture in the collection database showing a bowl of Kung Pao Chicken, a fish, a bowl of vegetables, and a bowl of rice is illustrated.
[0020] Figure 2 The preferred method for training the deep learning model used is illustrated.
[0021] Figure 3 Systems for performing corresponding methods for training and applying deep learning models are illustrated. DETAILED DESCRIPTION
[0022] In order to apply the method of the present invention, it is necessary to Figure 3 A suitable system 1 is disclosed in more detail, comprising a computer 6, software 8 executed on the computer 6, a digital camera 4 connected to the computer 6, the digital camera 4 being configured to take digital images 5 of a diet / meal 3. The computer 6 (preferably a suitable mobile phone with an integrated digital camera 4) further provides a user interface 7, which a user 2 can use to input settings or control the system 1 in any suitable manner. In a preferred embodiment, the computer 6 also hosts an application software 8 ("App"), which runs a deep learning model 9 for analyzing the images 5 and their descriptions 10.
[0023] Figure 2 A schematic overview of a preferred embodiment of the method of the invention is shown on how to train the used deep learning model 9. This embodiment including the necessary method steps will be further explained in the following sections.
[0024] Step 1. Construct a multimodal model sample.
[0025] a) Collect food seed images 5. Use cropping, rotation, color conversion, blurring and multi-image synthesis to enhance sample images to expand more images. There can be one food 3, multiple foods 3 or multiple types of food 3 in the image 5.
[0026] b) Using an unsupervised object detection algorithm or a suitable open source image segmentation algorithm to extract the food name of each food 3 in the image 5 and count the amount of each food 3 .
[0027] c) Construct a dietary description 10 based on the name and quantity of the food 3 in the image 5. Use pre-collected names of common foods 3 and corresponding eating units (e.g., bowls, cups, bars, pieces, etc.) from a suitable dictionary. The dictionary can be found based on the food names in the image 5. Get one or more food units, randomly select eating units, mark the format of edible quantity + food unit + food name, and generate edible information phrases, such as: 3 eggs, 1 cup of milk, 2 bowls of milk, etc. Repeat the operation for each food name in the image 5 to obtain multiple consumption information phrases. Put multiple consumption information phrases together in a random order to produce a dietary description of the image 5. (Preferably) manually check the generated dietary description 10 to ensure that the description 10 is correct.
[0028] d) Positive sample construction - for each image 5 in the image library, follow the sub-step c) method, and the food image 5 generates a corresponding diet description 10 as a positive sample of the model, which is labeled 1. Considering that the samples constructed by method step c) are automatically constructed, in order to ensure the generalization effect of model 9, for the images 5 in the image library, some food images 5 are selected to manually label the corresponding diet description 10 as another part of the model's positive samples, and the label is 1.
[0029] e) Negative sample construction. For each positive sample, if the dietary description of the food image 5 consists of multiple consumption information phrases, one or more food information phrases are randomly removed or randomly added, and the resulting dietary description 10 and the original food image 5 are negative samples and the label is 0. For each food image 5, a meal description 10 is made using a randomly generated consumption information phrase as another negative sample with label 0. Moreover, some non-diet images and random dietary descriptions 10 are used as the third part of the negative samples. The ratio of positive samples to negative samples should be less than 0.3.
[0030] Step 2. Use the deep learning model 9 to model the multimodal ITM task (image-text matching) for diet descriptions 10 and food images 5.
[0031] A deep learning model 9, such as ImageBERT, Unicoder-VL, UNIMO, UNITER, and other multimodal models, is used. The food description text is input into the model 9 through an embedding layer, and the food image 5 is input into the model 9 after CNN feature extraction. The training targets of the model 9 are 0 and 1. 1 means that the food description 10 and the food image 5 match, and 0 means that the food image 5 does not match. After N iterations of model convergence, the model training is completed and a model F is generated.
[0032] Step 3. Determine whether the input image 5 and the description text are consistent and the model prediction process works.
[0033] a) Get input, the input includes text or voice, and capture an image. Text refers to the typed input diet description X1. If voice input is used, it means that the voice recording of the description of the diet is provided. The captured image refers to the captured diet image X2.
[0034] b) X1 and X2 are input into model F to obtain the classification score. If it exceeds a given specific threshold, the image and text are consistent. If the score is lower than the threshold, the image and text are inconsistent.
[0035] Step 4. Feedback from User 2 The model predictions are validated by User 2, where as feedback to the model training process User 2 confirms the predicted outcome or rejects it.
[0036] When user 2 confirms the predicted consequence, the corresponding food image 5 and its corresponding description text 10 used for model prediction are kept within the training process of the model; and when user 2 rejects the predicted consequence, the image 5 and description text 10 are removed from the training process via computer 6, in which case the process of step 3 must be repeated.
[0037] After the deep learning model 9 is trained in the described manner, it is ready for use by the user 2 within the app 8. Therefore, before each diet, the user 2 takes at least one digital food image 5 of the diet record and uploads it to the app 8 or enters what and how much to eat. The algorithm of the model 9 automatically verifies that the preset recipe description 10 matches the image (5) taken; or verifies that the image (5) of the diet taken by the user matches the image previously entered for the corresponding diet or whether the description now entered by the user about what and how much to eat matches the preset recipe description (10), and if it does not match, a reminder message is returned to the user to confirm whether the description is correct.
[0038] Reference Mark List 1Human Diet Control System 2 users 3(s) meals / food 4. Digital Camera 5 Digital Food Images 6Computer / Mobile Phone 7. User Interface 8 Prediction Software / App 9 Deep Learning Models 10 Diet Description
Claims
1. A method for training a deep learning model (9) for controlling a human diet plan provided by software (8) on a computer (6), the steps comprising: Providing a multimodal model sample, comprising at least one digital food image (5) and a description (10) of the food (3) in the image (5); performing multimodal image-text matching with the deep learning model (9) executed by software executed on the computer, wherein food images (5) and their corresponding dietary description texts (10) from the multimodal model samples are input to the model (9), and the model (9) outputs a prediction of whether the image (5) and its text description (10) match; Validation of model predictions by a user (2), where the user (2) confirms the predicted outcome or rejects it as feedback to the model training process; Redo the multimodal image-text matching and model validation until the consistency predicted by the model reaches a given success rate.
2. The model of claim 1, wherein For positive feedback from the user (2) in which the user (2) confirms the predicted consequence, the corresponding food image (5) and its corresponding description text (10) used for model prediction are kept within the training process of the model, while for negative feedback from the user (2) in which the user (2) rejects the predicted consequence, the image (5) and the description text (10) are removed from the training process via the computer (6).
3. The method of claim 1, wherein The multimodal model samples are provided by retrieving them from a database.
4. The method of claim 3, wherein The multimodal model sample is created by the following method steps: Creating a food picture library by collecting digital sample food images (5), wherein the digital sample food images (5) may be of one food, multiple foods or multiple food categories; Use image dataset augmentation to increase the number of image samples; Identify the food name of each food (3) in the image (5) and count the number of each food (3); Generate dietary descriptions (10) for these food images (5) based on the names and quantities of the food (3) obtained in the previous step; Creating at least one positive model example where the food (3) shown in the image (5) and the description (10) of the food (3) match; Creating at least one negative model example where the food (3) shown in the image (5) and the dietary description (10) of the food (3) do not match; • Images (5), their corresponding dietary descriptions (10), and positive and negative samples are stored in the database as multimodal model samples.
5. The method according to any one of the preceding claims, wherein The sample images (5) are enhanced using digital image processing operations such as cropping, rotation, color conversion, blurring, and multi-image synthesis to provide more images (5).
6. The method according to any one of the preceding claims, wherein Verification of consistent model predictions includes: Check whether the output prediction of the model (9) exceeds a given threshold, resulting in the image (5) and the description text (10) being consistent, or whether the given threshold is not exceeded, resulting in the image (5) and the description text (10) being inconsistent.
7. The method according to any one of the preceding claims, wherein The output predictions of model (9) include values between 0 and 1.
8. A method for controlling a human diet plan via a computer (6) having software (8) using a deep learning model (9) trained according to a method as claimed in one of claims 1 to 7, wherein a user takes at least one image (5) of a food (3) that he intends to consume, provides the at least one image (5) to the computer (6), in particular by using a mobile phone (6) having a digital camera (4), and / or enters a description (10) of what and how much he intends to eat into the computer (6), and the computer (6) uses the trained deep learning model (9) via the software (8) to check and verify whether the food (3) in the at least one image (5) and / or description (10) matches a set of allowed diets previously entered into the computer (6) by the user (2).
9. The method of claim 8, wherein The set of allowed diets consists of preset recipe descriptions and images for each respective diet.
10. A computer device (6) comprising a computer readable memory, a processor and a computer program (8), the computer program (8) being stored on the memory and being executable on the processor, wherein when executing the computer program (8), the processor implements the method according to any one of claims 1 to 9.
11. A non-transitory computer-readable storage medium storing a computer program (8) for execution on a computing device (6) having a processor according to claim 9, wherein the computer program (8) when executed by the processor causes the computing device (6) to perform the method according to any one of claims 1 to 9.
