Food image recipe generation method based on retrieval enhancement large-scale multi-modal model

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to a food image recipe generation method based on a retrieval enhancement large-scale multi-modal model. According to the method, a retrieval model from images to recipes is combined with a large-scale multi-modal language model; and the language model adopts LLAVA of an enhanced LoRA patch. The food image recipe generation method comprises the steps of construction of a language model training data set, training of a language model and reasoning of the language model. Random diversified retrieval enhancement is introduced, recipes related to image semantics are retrieved from an existing data storage library to serve as supplements, the recipes are integrated into prompts, and therefore the user experience is improved. Various and rich contexts are added to the input image. In addition, a self-consistency integrated voting mechanism is introduced and used for determining the most confident predicted recipe as final output. A large number of experiments verify the effectiveness of the method, which shows that the method achieves the most advanced performance in the recipe generation task of the Recipe1M data set at present.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method for generating food image recipes. Background Art

[0002] As recipe generation technology continues to gain importance in computational culinary arts, a wide range of datasets such as Recipe1M, VireoFood-172, and Food2K have provided important support for the development of food-related computational tasks. The widespread use of these datasets has promoted the progress in this field [1]. However, since recipe generation involves multiple sentences of cooking instructions, this task faces significant challenges. This task requires the model to have a deep understanding of the ingredients, food images, and cooking processes in order to accurately perform the corresponding tasks [2,3,4].

[0003] Early attempts at recipe generation mainly adopted a two-stage approach, first generating ingredients from images, and then generating instructions using the embedded representations of ingredients and images. Among these methods, [5] utilized transformer models to generate detailed recipes, but they neglected to build a comprehensive structure for the recipes before generation, resulting in some steps being missed in the generated output. These early models did not exploit the potential of large multimodal models (LMMs). Recently, [6] proposed a unified food computation model based on LMMs, which includes the recipe generation task, by fine-tuning a multimodal large language model LISA [7] using multiple datasets simultaneously.

[0004] With the growing demand for versatile deep learning models, large pre-trained models such as BERT, ViT, and GPT have emerged. However, the generalization ability of these models is constrained by the limitations of single-modal models, which has prompted the development of large-scale visual-language multimodal models. The practice of using autoregressive language models as decoders for visual-language tasks has become increasingly popular. For example, the LLaVA[8] model combines the output of the visual encoder with LLaMA[9] and uses synthetic data to enhance the training of the alignment network and LMM. Similarly, Vicuna

[10] uses LLaMA[9] to facilitate conversational interaction with humans, designing a model with a wide range of capabilities, enabling it to effectively respond and provide comprehensive responses to a range of complex and diverse queries. The rapid development of LMMs has also promoted the application of multimodal models in various vertical fields. However, the hallucination problem introduced by LMMs and its solutions continue to emerge. Despite some progress, the FoodLMM model still faces the hallucination problem, and this paper aims to solve this problem through retrieval enhancement. Summary of the invention

[0005] The object of the present invention is to provide a method for food image recipe generation based on a large multimodal language model (LMM) with enhanced retrieval to improve the accuracy and richness of recipe generation.

[0006] The method for generating food image recipes based on a large multimodal language model with enhanced retrieval provided by the present invention combines the image-to-recipe retrieval model R

[11] and a large multimodal language model (referred to as a language model, also called a generative model, denoted as G); the generative model here uses LLAVA with enhanced LoRA

[16] patches. The method for generating food image recipes includes the following specific steps: constructing a language model training dataset, training a language model, and reasoning about the language model:

[0007] Step 1: Construction of language model training data set;

[0008] The Recipe1M dataset is used as the basis of the language model training dataset, denoted as data storage M. Let the food image x be the query, encode the image x into features through the encoder, and then input the features into the word segmenter to obtain the tag sequence x1,...,x corresponding to the image. n , and search the data store M for the query sequence x1,...,x n The first K most similar documents are denoted as M'=(m1,...,m K ), each document format includes food category, ingredient composition and food preparation process.

[0009] The training data is organized into dialogs: image-recipe pairs, where one image corresponds to three question-answer pairs, the three questions and answers are title (i.e. food category), ingredients (i.e. ingredients), and instructions (i.e. food preparation process); Recipe1M dataset, which contains each image and its corresponding title, ingredients, and instructions. To better align with the conventional practice of retrieval enhancement, questions about titles, ingredients, and instructions are considered queries. The training data is formatted as:

[0010] “[ Conversations: Q titles ,A titles ,Q ingredienst ,A ingredienst ,Q instructions ,A instructions ]”, (3)

[0011] Among them, Q titles Indicates the title of the image, A titles Indicates the answer to the title, Q ingredienst Indicates the composition problem of the image, A ingredienst The answer to the composition, Q instructionsA represents the problem of the production process of the image. instructions Represents the answer to the production process. These conversations are defined as training data.

[0012] Specifically, the task template of the dialogue in the original training data is as follows:

[0013] titles ,A titles :The food is...,Q ingredienst ,A ingredienst :The ingredients are:...,Q instructions ,A instructions :Here are the instructions:...>;

[0014] After adding the retrieved information M'=(m1,...,m K ), the dialogue template becomes:

[0015] titles ,A titles :The food is...,Search results...[searched ingredient 1, searched ingredient 2]Q ingredienst ,

[0016] A ingredienst :The ingredients are:...,Search results...[Ingredients retrieved 3, Production process retrieved 1]Q instructions ,A instructions :Here are the instructions:...>.

[0017] Through this method, a supplement of retrieved information is formed, and training data is obtained to train the generation model G.

[0018] In the present invention, the food category, ingredient composition and food preparation process are determined by a provided random diversified retrieval enhancement method. Specifically, the following steps are included:

[0019] Different from the existing retrieval enhancement methods [12, 13, 14] that directly use retrieval results as context, in order to ensure the diversity of retrieval information, three groups of components and one group of descriptions are randomly selected from the first K retrieval results as the final retrieval information and concatenated in Q as follows: ingredienst and Q instructions Before, if Figure 2 shown.

[0020] Among them, two of the three groups of components retrieved are in Q​​ingredienst Before, they are connected in series, and the remaining set of components is in Q instructions The previous ones are concatenated in sequence, and the model is told that this is a reference result: "The reference search result is 'retrieved information'". Finally, the final training data is obtained, such as Figure 1 Here, the prompts “Food is”, “Ingredients are:”, and “Description here:” are similar to the simple, fixed prompts used in other studies

[15] , but here they are customized to describe the ingredients and provide descriptions relevant to the input image.

[0021] Step 2: Language model training;

[0022] After getting the training data, we train the generative model G using the retrieval-augmented recipes. The generative model is built on top of the Large Scale Multimodal Model (LLaVA), which accepts images and textual prompts as input.

[0023] The text prompt (i.e., question) and retrieval-augmented recipe are concatenated and processed through a tokenizer and then fed into a text encoder to obtain text features, while the image is processed through an image encoder to obtain image features.

[0024] The image features are mapped to the same embedding space as the text features through the MLP. After receiving the text and image embeddings, the decoder proceeds to generate tags based on the image features X and the training data questions.

[0025] The loss function used in training is to maximize the conditional probability of the next word given the previous word sequence, that is, to maximize it by optimizing the parameter θ:

[0026]

[0027] Among them, x ＜i is x i The previous tag sequence, p θ refers to conditional probability; the generative model makes predictions conditionally based on the retrieved document M'. Specifically, it predicts the continuation of the input by simply incorporating the retrieved document into the input query:

[0028]

[0029] Among them, [a; b] represents the concatenation of strings a and b;

[0030] In order to reduce the computational requirements of training and maintain its generalization ability, the generative model LLaVA model is frozen and only its patch LoRA is trained, which allows the focus to shift to specific, smaller aspects of the model trained to adapt to recipe generation without requiring a lot of computation. The model is trained by minimizing the cross entropy loss of the next token prediction as follows:

[0031]

[0032] Where n is the index of the current marker, y ＜i Represents the token in the sequence before position i.

[0033] Step 3: Reasoning about Multimodal Language Models

[0034] After training, the final prediction recipe is obtained through the inference process, which is implemented by the self-consistent ensemble voting mechanism. The steps are as follows:

[0035] Given a food image at test time, the generated recipes using different retrieved recipes as context may be inconsistent. Therefore, a score-based ensemble voting method, namely the self-consistency ensemble voting mechanism, is introduced to maintain self-consistency and improve the quality of generated recipes.

[0036] Specifically, first, use the image x in the test set to search in the training set (data storage M) through the retrieval model to obtain the first s matching food production processes. In the same data organization form as in training, they are sequentially concatenated in front of the production process questions and input into the model together with the image x to obtain the corresponding s prediction results, expressed as P = {P1, ..., P s}, s is the number of recipes used for inference. Calculate the cosine similarity, BLEU, SacreBLEU score or ROUGE L score between these s recipes to generate an s×s matrix, where each row represents the consistency of all other predictions with the current prediction (excluding diagonal elements). The confidence score of each recipe is obtained by averaging the sum of the consistency in each row. Taking the calculation of confidence score by cosine similarity as an example, the recipe with the highest confidence score is selected as the final output, as shown below:

[0037]

[0038] in, refers to the cosine similarity between two predictions. The whole process is as follows Figure 2 (b) as shown.

[0039] The innovation of the present invention lies in:

[0040] (1) We propose the first large-scale multimodal retrieval-augmented model tailored for recipe generation. This model introduces a randomized retrieval-augmentation technique to enhance the diversity and richness of retrieval.

[0041] Furthermore, in the reasoning stage, a self-consistent ensemble voting strategy is adopted to ensure the consistency and agreement of the generated recipes using different retrieved recipes.

[0042] (2) The proposed model achieves state-of-the-art recipe generation performance on the Recipe1M dataset. A comprehensive ablation study is performed to validate the effectiveness of the design choices and demonstrate the contribution of each component to the overall performance.

[0043] (3) The model exhibits excellent adaptability and surpasses the current state-of-the-art (SOTA) results on the ingredient recognition task on the Recipe1M dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a diagram of the recipe dialogue task template of the training process proposed by the present invention.

[0045] Figure 2 It is a flowchart of the food image recipe generation method based on retrieval-enhanced large multimodal model (LMM) proposed in the present invention.

[0046] Figure 3 This is a prediction example of the self-consistent integrated voting strategy proposed in the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described below through specific examples.

[0048] Experiments were conducted using recipes with images from the Recipe1M dataset, which contains 571,587 training samples and 51,304 test samples. During the training process, each image corresponds to three question-answer pairs, as described in the invention content. In addition, the ingredients in Recipe1M are clustered into 1,488 types, and quantifiers are removed before the ingredients. During the testing process, the first image corresponding to each recipe in the test set is selected as the test sample, and it is processed into a format in which each image corresponds to a question-answer pair, so that the title, ingredients, and steps are tested separately. In addition, the training set of Recipe1M is stored as an external data store to retrieve information, because Recipe1M is the most comprehensive food image recipe dataset, so that external data that may affect the consistency and fairness of the experimental results is not introduced.

[0049] The F1 score and intersection over union (IOU) are used to evaluate the quality of ingredients, and document-level evaluation metrics, specifically BLEU, SacreBLEU, and RougeL, are used to evaluate the quality of instructions in generated recipes.

[0050] template:

[0051] Question title: Can you predict the food category of this image?

[0052] Answer title: This dish is...

[0053] Question Ingredients: Can you list the ingredients needed for this recipe?

[0054] Answer: These ingredients are...

[0055] Question steps: Can you provide the cooking steps corresponding to this picture?

[0056] Answer Steps: Here are the steps…

[0057] accomplish:

[0058] In the retrieval module, a transformer-based cross-modal recipe retrieval model is used. The generation model utilizes LLAVA augmented with LoRA patches. For the baseline model, we call it “LLaVA-FT”, fine-tuning LLAVA with LoRA without retrieval augmentation, using the same model architecture, training data, and computational resources to ensure a fair comparison.

[0059] During training, the first 50 ingredients and steps are retrieved for each image, and then 1 of each category is randomly added to the front of the ingredient and step questions, as described in the Summary of the Invention. The maximum sequence length of the generative model is 4096, which can contain all the above information. Then, the label prediction loss of the generative model over the entire concatenated sequence is optimized (Equation 4).

[0060] During the inference process, we first use the images in the test set as queries to retrieve from 571,587 instances in the training set, using the retrieval model and method described in the content of the invention. Then, we use the retrieved steps and ingredients and connect them in front of the question in the same way as in the training process. For self-consistent ensemble voting, we use the TopK retrieval information for an image, including ingredient information and production process, and add it to the question Q about the production process. instructions In front, then the recipe example is followed by "[ Conversations:Search results...[searched ingredients, searched production process],:Here are the instructions:...Q instructions ]" is input into the generative model to obtain K prediction results. Then, the cosine similarity, BLEU, SacreBLEU score or ROUGE L score is calculated between these K prediction results to generate an s×s matrix, where each row represents the consistency of all other predictions with the current prediction. The confidence score of each recipe is obtained by averaging the sum of the consistency in each row. Then, taking the calculation of the confidence score by cosine similarity as an example, the recipe with the highest confidence score is selected as the final output, which is considered to be the most reliable recipe prediction. Figure 3The figure shows the four predictions output when K=4, among which prediction 1 has the highest confidence, which means that it has the highest cosine similarity with the other three predictions, i.e., the most reliable output.

[0061] References

[0062] [1] Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. 2019. A survey on food computing. ACM Computing Surveys (CSUR) 52, 5 (2019), 1–36.

[0063] [2] Helena H. Lee, Ke Shu, Palakorn Achananuparp, Philips Kokoh Prasetyo, Yue Liu, Ee-Peng Lim, and Lav R Varshney. 2020. RecipeGPT: Generative pre-trainingbased cooking recipe generation and evaluation system. In Companion Proceedings of the Web Conference 2020.181–184.

[0064] [3]Hao Wang, Guosheng Lin, Steven CH Hoi, and ChunyanMiao. 2020. Structureaware generation network for recipe generation from images. In Computer Vision–ECCV 2020:16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16. Springer, 359–374.

[0065] [4]Hao Wang, Guosheng Lin, Steven CH Hoi, and Chunyan Miao. 2022. Learning

[0066] structural representations for recipe generation and foodretrieval.IEEE Transactions on Pattern Analysis and Machine Intelligence 45,3(2022),3363–3377.[5]Amaia Salvador,Michal Drozdzal,Xavier Giró-i Nieto,andAdriana Romero. images.InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.10453–10462.

[0067] [6]Yuehao Yin,Huiyan Qi,Bin Zhu,Jingjing Chen,Yu-Gang Jiang,andChong-Wah Ngo.2023.FoodLMM:A Versatile Food Assistant using Large Multi-modalModel.arXiv preprint arXiv:2312.149219(2023).

[0068] [7]Xin Lai,Zhuotao Tian,Yukang Chen,Yanwei Li,Yuhui Yuan,Shu Liu,andJiaya Jia.2023.Lisa:Reasoning segmentation via large language model.arXivpreprint arXiv:2308.00692(2023).

[0069] [8]Haotian Liu,Chunyuan Li,Qingyang Wu,and Yong Jae Lee.2024.Visualinstruction tuning.Advances in neural information processing systems 36.

[0070] [9]Hugo Touvron,Thibaut Lavril,Gautier Izacard,Xavier Martinet,Marie-Anne Lachaux,Timothée Lacroix,Baptiste Rozière,Naman Goyal,Eric Hambro,FaisalAzhar,et al.2023.Llama:Open and efffcient foundation language models.arXivpreprint arXiv:2302.13971(2023).

[0071]

[10] Lianmin Zheng,Wei-Lin Chiang,Ying Sheng,Siyuan Zhuang,ZhanghaoWu,Yonghao Zhuang,Zi Lin,Zhuohan Li,Dacheng Li,Eric Xing,etal.2024.Judgingllm-as-a-judge with mt-bench and chatbot arena.Advances inNeural InformationProcessing Systems 36.

[0072]

[11] Amaia Salvador,Erhan Gundogdu,Loris Bazzani,and MichaelDonoser.2021.Revamping cross-modal recipe retrieval with hierarchicaltransformers andselfsupervised learning.In Proceedings of the IEEE / CVFConference on Computer Visionand Pattern Recognition.15475–15484.

[0073]

[12] Zhengbao Jiang,Frank F Xu,Luyu Gao,Zhiqing Sun,Qian Liu,JaneDwivediYu,Yiming Yang,Jamie Callan,and Graham Neubig.2023.Active retrievalaugmentedgeneration.arXiv preprint arXiv:2305.06983.

[0074]

[13] Patrick Lewis,Ethan Perez,Aleksandra Piktus,Fabio Petroni,VladimirKarpukhin,Naman Goyal,Heinrich Küttler,Mike Lewis,Wen-tau Yih,Tim etal.2020.Retrieval-augmented generation for knowledge-intensivenlp tasks.Advancesin Neural Information Processing Systems 33,9459–9474.

[0075]

[14] Rita Ramos,Bruno Martins,Desmond Elliott,and YovaKementchedjhieva.2023.Smallcap:lightweight image captioning prompted withretrieval augmentation.InProceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2840–2849.

[0076]

[15] Junnan Li,Dongxu Li,Caiming Xiong,and Steven Hoi.2022.Blip:Bootstrappinglanguage-image pre-training for uniffed vision-languageunderstanding andgeneration.In International conference on machinelearning.PMLR,12888–12900.

[16] Hu E J,Shen Y,Wallis P,et al.Lora:Low-rankadaptation of large languagemodels[J].arXiv preprint arXiv:2106.09685,2021。

Claims

1. A method for generating food image recipes based on a large multimodal language model with enhanced retrieval, characterized in that: Combine the image-to-recipe retrieval model R with a large multimodal language model G); the language model here uses LLAVA with enhanced LoRA patches; the specific steps of food image recipe generation are: Step 1: Construction of language model training data set; The Recipe1M dataset is used as the basis of the language model training dataset, denoted as data storage M. Let the food image x be the query, encode the image x into features through the encoder, and then input the features into the word segmenter to obtain the tag sequence x1,...,x corresponding to the image. n , and search the data store M for the query sequence x1,...,x n The first K most similar documents, denoted as M'=(m1,...,m K ), each document format includes food category, ingredient composition and food preparation process; The training data is organized into a dialog format: image-recipe pairs, where one image corresponds to three question-answer pairs, the three questions and answers are title (i.e., food category), ingredients (i.e., ingredients), and instructions (i.e., food preparation process); the Recipe1M dataset, which contains each image and its corresponding title, ingredients, and instructions; to better align with the conventional practice of retrieval enhancement, questions about titles, ingredients, and instructions are considered queries; the training data is formatted as: "[ Conversations:Q titles ,A titles ,Q ingredienst ,A ingredienst ,Q instructions ,A instructions ]", where Q titles Indicates the title of the image, A titles Indicates the answer to the title, Q ingredienst Indicates the composition problem of the image, A ingredienst The answer to the composition, Q instructions A represents the problem of the production process of the image. instructions Represent the answer of the production process; use these conversations as training data; Specifically, the task template of the dialogue in the original training data is as follows: <Q titles ,A titles :The food is...,Q ingredienst ,A ingredienst :The ingredients are:...,Q instructions ,A instructions :Here are the instructions:...>; After adding the retrieved information M'=(m1,...,m K ), the dialogue template becomes: titles ,A titles :The food is...,Search results...[searched ingredient 1, searched ingredient 2]Q ingredienst ,​ A ingredienst :The ingredients are:...,Search results...[Ingredients retrieved 3, Production process retrieved 1]Q instructions ,A instructions :Here are the instructions:...>; Thus, a supplement of retrieved information is formed, and training data is obtained, which is used to train the generation model G; Among them, the food category, ingredient composition and food preparation process are determined by a provided random diversified retrieval enhancement method, the specific steps are: Randomly sample the first K search results, select three groups of components and one group of descriptions as the final search information, and splice them in Q in the following way ingredienst and Q instructions Before; Two of the three groups of components are retrieved in Q ingredienst Before, they are connected in series, and the remaining set of components is in Q instructions The previous steps are connected in series, and the model is informed that this is a reference result: "The reference search result is 'retrieved information'"; the final training data is obtained; Step 2: Language model training; After obtaining the training data, the language model G is trained using the retrieval-augmented recipes. The language model is built on top of the large multimodal model LLaVA, which accepts images and textual prompts as input. The textual prompts i.e., questions and retrieval-augmented recipes are concatenated and processed through a tokenizer and then fed into a text encoder to obtain text features, while the images are processed through an image encoder to obtain image features; The image features are mapped to the same embedding space as the text features through the MLP. After receiving the text and image embeddings, the decoder proceeds to generate tags based on the image features X and the training data questions. The loss function used in training is to maximize the conditional probability of the next word given the previous word sequence, that is, to maximize it by optimizing the parameter θ: Among them, x ＜i is x i The previous tag sequence, p θ refers to conditional probability; the language model makes predictions conditionally based on the retrieved document M'; specifically, by incorporating the retrieved document into the input query to predict the continuation of the input: Among them, [a; b] represents the concatenation of strings a and b; In order to reduce the computational requirements of training and maintain its generalization ability, the generative model LLaVA model is frozen and only its patch LoRA is trained. The model is trained by minimizing the cross entropy loss of the next token prediction as follows: Where n is the index of the current marker, y ＜i represents the token in the sequence before position i; Step 3: Reasoning about Multimodal Language Models After training, the final prediction recipe is obtained through the inference process, which is implemented by the self-consistent ensemble voting mechanism. The steps are as follows: First, use the image x in the test set to retrieve the first s matching food production processes in the data storage M through the retrieval model. The data organization form is the same as that of the training data. They are sequentially concatenated in front of the production process questions and input into the model together with the image x to obtain the corresponding s prediction results, expressed as P = {P1, ..., P s }, s is the number of recipes used for inference; calculate the cosine similarity between these s recipes to generate an s×s matrix, where each row, except for the diagonal elements, represents the consistency of all other predictions with the current prediction; obtain the confidence score of each recipe by averaging the sum of the consistency in each row; select the recipe with the highest confidence score as the final output, as shown below: in, Refers to the cosine similarity between two predictions.

Images