Diffusion model food generation method and system based on collaborative relationship perception

Abstract

The invention discloses a diffusion model food generation method and system based on collaborative relation perception, and relates to the technical field of text generation images, and the method comprises the steps: receiving food image data, inputting the food image data into a pre-established labeling model BLIP for text labeling, and obtaining labeled food image data; inputting the labeled food image data into a pre-established question and answer model GPT4 to perform relation extraction on the menu and the image, and extracting each food material and the corresponding cooking action as a two-tuple; inputting the two-tuple into a text encoder of a pre-trained large model CLIP for encoding, inputting an encoded relationship into a cross attention layer of a pre-established trunk diffusion model through a relationship injector, and outputting to obtain a generated food image; and inputting the generated food image into a pre-established image scoring feedback model to obtain an image scoring score, and optimizing the generated food image according to the image scoring score.

Description

Technical Field

[0001] The present invention relates to the technical field of text-generated image technology, and in particular to a diffusion model food generation method and system based on collaborative relationship perception. Background Art

[0002] Generating food images from recipes aims to synthesize tempting food images from input recipes or food names, which is a typical application of text-to-image generation technology. The generation of food images plays an important role in many practical applications and brings economic value, such as restaurant advertising, nutrition management, dish innovation, food packaging design, etc.

[0003] Food images, ingredients, and cooking actions form a complex collaborative relationship that directly affects the appearance of dishes. In some existing works, food image generation research is mainly based on generative adversarial networks, which sequentially simulate the cooking process and directly generate food images. However, the insufficient feature interaction between recipes and food limits the representation ability of generative adversarial networks, resulting in blurred and distorted generated food images. Although the latest text-to-image diffusion models can synthesize higher quality food images through continuous denoising steps, these models can only generate images based on simple text, such as food names or food categories, due to the limited ability to encode recipe semantics. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a diffusion model food generation method and system based on collaborative relationship perception, which can solve the technical problem that the model does not fully understand the cooking relationship, resulting in incorrect generation of the appearance details of the ingredients.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a diffusion model food generation method based on collaborative relationship perception, the method comprising the following steps:

[0006] Receiving food image data, inputting the food image data into a pre-established annotation model BLIP for text annotation, and obtaining annotated food image data;

[0007] The labeled food image data is input into the pre-established question-answering model GPT4 to extract the relationship between the recipe and the image, and extract each ingredient and the corresponding cooking action as a binary;

[0008] The bigrams are input into the text encoder of the pre-trained large model CLIP for encoding, and the encoded relations are input into the cross attention layer of the pre-established backbone diffusion model through the relation injector, and the generated food images are output;

[0009] The generated food image is input into a pre-established image scoring feedback model to obtain an image scoring score, and the generated food image is optimized according to the image scoring score.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of acquiring the food image data is as follows:

[0011] Use a crawler to crawl multiple food-related images from cooking websites and food photography websites; remove duplicate images, and use the CLIP model to calculate the semantic consistency between the original image and the recipe to filter out irrelevant images; after filtering out duplicate food images, keep high-resolution images with a resolution greater than the preset resolution; remove the watermarks of high-resolution images, and filter out aesthetically pleasing images as food image data.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the text annotation process and the process of screening out aesthetically pleasing images are as follows:

[0013] Use the OCR model text detection model to detect text in the image and output the location box of the text; use the logo detection API to detect trademarks and patterns in the image and output the location box; use the image restoration model Stable Diffusion-Inpainting to remove the information of the masked area generated from these location boxes; screen aesthetic food images with exquisite plating, texture and gloss.

[0014] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the food image data into a pre-established annotation model BLIP for text annotation includes:

[0015] Use the pre-trained text annotation model BLIP to generate descriptions of food images, then encode the food images and calculate a set of most similar prompts, which are then concatenated to obtain a set of prompts. The encoded food images are input into the BLIP model to output the text corresponding to each food image.

[0016] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the bigram into the text encoder of the pre-trained large model CLIP for encoding, and inputting the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector includes:

[0017] The relation injector fuses the encoded relations through the multi-layer perceptron MLP and the gated recurrent unit GRU. First, the raw materials are connected with the cooking actions and input into the MLP to obtain the i-th relation feature vector g i, a linear growth parameter λ is used in the feature injector to control each relation r i :

[0018] g i =MLP(λI i ,A i )

[0019] Among them, λ follows the uniform distribution of (0, 1), I i represents the i-th ingredient, A i Represents the cooking action corresponding to the i-th ingredient, and then uses GRU to fuse the synergistic relationship, expressed as:

[0020] g out =GRU(g i )

[0021] In the U-Net denoiser, the relation g at the tth time step is transformed through a linear layer. out The feature map is K t ' and V t ', integrate the relational features with the visual features, and transform K t ' and V t 'Add to the original K in the U-Net cross attention layer t and V t Above, it is expressed as follows:

[0022]

[0023] Among them, β1 and β2 are parameters that control the fusion ratio. By adjusting β1 and β2, the injection ratio of relational features can be controlled. The calculation process of cross attention is expressed as:

[0024]

[0025] Among them, φ out represents the output of the cross attention in the diffusion decoder block, and d is the dimension of the relation feature.

[0026] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the cross attention layer of the pre-established backbone diffusion model fine-tunes the backbone diffusion model by inserting two parameter-updatable low-rank matrices, and the process is as follows:

[0027] The backbone diffusion model is a Stable Diffusion 1.5 model, which generates images from conditional information and Gaussian noise through a denoising process. The backbone diffusion model consists of an autoencoder and a U-Net denoiser. The framework is built on two complementary random processes: the forward process and the reverse process.

[0028] In the cross-attention layer of the backbone diffusion model, two parameter-updatable low-rank matrices A and B are inserted. For the weight matrix W in the U-Net cross-attention q and W k , the introduction of low-rank matrices involves converting W q Update to W q ', W k Update to W k ', defined as:

[0029] W q '=W q +B1A1

[0030] W k '=W k +B2A2

[0031] Among them, r represents the rank of the low-rank matrix.

[0032] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the forward process and the reverse process include:

[0033] In the forward process, the diffusion model is trained using an autoencoder to convert the image into a latent space and reconstruct it. The input image is represented as x0. As the time step t increases, Gaussian noise z is gradually added to x0. The noisy image at time step t is defined as the following formula:

[0034]

[0035] Among them, a t is the noise schedule at time step t, z~N(0,1);

[0036] Reverse process, U-Net denoiser D θ For an image x in the latent space t Gradually denoising, getting a clearer image at each step t, the goal of the backbone diffusion model is to generate an image by combining the noise map ∈ ~N(0,1) and the conditional embedding c, which is obtained from the text prompt P using the CLIP text encoder T, given the noise feature z t and conditional embedding c, using D θ To predict the noise ∈ at the current time step t, the loss function is expressed as:

[0037]

[0038] in c is the conditional embedding encoded by the CLIP text encoder T, at time step t=0,1,…,T;

[0039] Given input conditions c and zt , D θ The noise estimate at each step t is predicted, and after T iterations, the final prediction result z0 is input into the autoencoder for image generation.

[0040] In combination with the first aspect, in some implementations of the first aspect, the method further includes: inputting the generated food image into a pre-established image scoring feedback model to obtain a calculation process of the image scoring score as follows:

[0041] Randomly select a time step t, and then directly use the latent features in the selected time step to generate an image x'0. Secondly, use the image reward model τ to evaluate the image x0 generated in step T and x'0. Finally, feed the high score back to step T-1 in the denoising process. The loss function is as follows:

[0042]

[0043] Among them, γ is the reward score weighting coefficient, φ represents the reward-loss mapping function, and y i Indicates input recipe, r i is the relationship set, θ is the parameter of the diffusion model, D θ (y i ) is the text y i Corresponding generated food images;

[0044] Introducing reward loss L reward After that, the final loss function L is expressed as follows:

[0045] L=L dm +L reward

[0047] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a diffusion model food generation system based on collaborative relationship perception, comprising:

[0048] A data processing module, used for receiving food image data, inputting the food image data into a pre-established annotation model BLIP for text annotation, and obtaining annotated food image data;

[0049] The relationship extraction module is used to input the annotated food image data into the pre-established question-answering model GPT4 to extract the relationship between the recipe and the image, and extract each ingredient and the corresponding cooking action as a binary;

[0050] The image generation module is used to input the binary into the text encoder of the pre-trained large model CLIP for encoding, input the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector, and output the generated food image;

[0051] The image scoring module is used to input the generated food image into a pre-established image scoring feedback model to obtain an image scoring score, and optimize the generated food image according to the image scoring score.

[0052] In conjunction with the second aspect, in certain implementations of the second aspect, the system further includes: the process of acquiring the food image data of the data processing module is as follows:

[0053] Use a crawler to crawl multiple food-related images from cooking websites and food photography websites; remove duplicate images and use the CLIP model to calculate the similarity between the original image and the recipe to filter out irrelevant images; after filtering out duplicate food images, keep high-resolution images with a resolution greater than the preset resolution; remove the watermarks of high-resolution images and filter out aesthetically pleasing images as food image data;

[0054] The process of the data processing module to filter out aesthetically pleasing images is as follows:

[0055] Use the OCR model text detection model to detect text in the image and output the location box of the text; use the logo detection API to detect trademarks and patterns in the image and output the location box; use the image restoration model Stable Diffusion-Inpainting to remove the information of the masked area generated from these location boxes; screen the aesthetic food images with exquisite plating, texture and gloss;

[0056] Alternatively, the process of the data processing module inputting the food image data into the pre-established annotation model BLIP for text annotation includes:

[0057] Use the pre-trained text annotation model BLIP to generate descriptions of food images, then encode the food images and calculate a set of most similar prompts, then concatenate them to obtain a set of prompts, input the encoded food images into the BLIP model, and output the text corresponding to each food image;

[0058] The image generation module inputs the binary into the text encoder of the pre-trained large model CLIP for encoding, and inputs the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector. The process includes:

[0059] The relation injector fuses the encoded relations through the multi-layer perceptron MLP and the gated recurrent unit GRU. First, the raw materials are connected with the cooking actions and input into the MLP to obtain the i-th relation feature vector g i , a linear growth parameter λ is used in the feature injector to control each relation r i Importance:

[0060] g i =MLP(λI i ,A i )

[0061] Among them, λ follows the uniform distribution of (0, 1), I i represents the i-th ingredient, A i Represents the cooking action corresponding to the i-th ingredient, and then uses GRU to fuse the synergistic relationship, expressed as:

[0062] g out =GRU(g i )

[0063] In the U-Net denoiser, the relation g at the tth time step is transformed through a linear layer. out The feature map is K t ' and V t ', integrate the relational features with the visual features, and transform K t ' and V t 'Add to the original K in the U-Net cross attention layer t and V t Above, it is expressed as follows:

[0064]

[0065] Among them, β1 and β2 are parameters that control the fusion ratio. By adjusting β1 and β2, the injection ratio of relational features can be controlled. The calculation process of cross attention is expressed as:

[0066]

[0067] Among them, φ out represents the output of the cross attention in the diffusion decoder block, and d is the dimension of the relation feature;

[0068] The cross-attention layer of the pre-established backbone diffusion model of the image generation module fine-tunes the backbone diffusion model by inserting two parameter-updatable low-rank matrices as follows:

[0069] The backbone diffusion model is a Stable Diffusion 1.5 model, which generates images from conditional information and Gaussian noise through a denoising process. The backbone diffusion model consists of an autoencoder and a U-Net denoiser. The framework is built on two complementary random processes: the forward process and the reverse process.

[0070] In the cross-attention layer of the backbone diffusion model, two parameter-updatable low-rank matrices A and B are inserted. For the weight matrix W in the U-Net cross-attention q and W k , the introduction of low-rank matrices involves converting Wq Update to W q ', W k Update to W k ', defined as:

[0071] W q '=W q +B1A1

[0072] W k '=W k +B2A2

[0073] The forward and reverse processes of the image generation module include:

[0074] In the forward process, the diffusion model is trained using an autoencoder to convert the image into a latent space and reconstruct it. The input image is represented as x0. As the time step t increases, Gaussian noise z is gradually added to x0. The noisy image at time step t is defined as the following formula:

[0075]

[0076] Among them, a t is the noise schedule at time step t, z~N(0,1);

[0077] Reverse process, U-Net denoiser D θ For an image x in the latent space t Gradually denoising, getting a clearer image at each step t, the goal of the backbone diffusion model is to generate an image by combining the noise map ∈ ~N(0,1) and the conditional embedding c, which is obtained from the text prompt P using the CLIP text encoder T, given the noise feature z t and conditional embedding c, using D θ To predict the noise ∈ at the current time step t, the loss function is expressed as:

[0078]

[0079] in c is the conditional embedding encoded by the CLIP text encoder T, at time step t=0,1,…,T;

[0080] Given input conditions c and z t , D θ The noise estimate at each step t is predicted, and after T iterations, the final prediction result z0 is input into the autoencoder for image generation;

[0081] The image scoring module inputs the generated food images into the pre-established image scoring feedback model, and the calculation process of the image scoring score is as follows:

[0082] Randomly select a time step t, and then directly use the latent features in the selected time step to generate an image x'0. Secondly, use the image reward model τ to evaluate the image x0 generated in step T and x'0. Finally, feed the high score back to step T-1 in the denoising process. The loss function is as follows:

[0083]

[0084] Among them, γ is the reward score weighting coefficient, φ represents the reward-loss mapping function, and y i Indicates input recipe, r i is the relationship set, θ is the parameter of the diffusion model, D θ (y i ) is the text y i Corresponding generated food images;

[0085] Introducing reward loss L reward After that, the final loss function L is expressed as follows:

[0086] L=L dm +L reward

[0088] Beneficial effects of the present invention:

[0089] The present invention effectively integrates the complex relationship between the injected ingredients and the cooking actions, and enhances the model's perception of the appearance and details of food. The generated images are more realistic and detailed, and compared with the existing GAN model, the blur and distortion are significantly reduced; the full interaction between the recipe semantics and the visual features of the food is achieved. When understanding complex recipes, the model more accurately maps semantic information to visual features, thereby improving the semantic consistency of the generated images; the generated food images are not only more realistic in shape and detail, but also more in line with the public's aesthetic preferences in texture, gloss and presentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0091] Figure 1 It is a schematic flow chart of the method of the present invention;

[0092] Figure 2 A data processing schematic diagram of a diffusion model food generation method based on collaborative relationship perception provided in the first embodiment of the present invention;

[0093] Figure 3A collaborative relationship extraction and injection diagram of a diffusion model food generation method based on collaborative relationship perception provided in the first embodiment of the present invention;

[0094] Figure 4 A diagram of a recipe fusion fine-tuning method for a diffusion model food generation method based on collaborative relationship perception provided in the first embodiment of the present invention;

[0095] Figure 5 An image feedback mechanism diagram of a diffusion model food generation method based on collaborative relationship perception provided in the first embodiment of the present invention;

[0096] Figure 6 A comparison diagram of image generation effects of the current model and other models of a diffusion model food generation method based on collaborative relationship perception provided in the first embodiment of the present invention;

[0097] Figure 7 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0099] Embodiment 1:

[0100] like Figure 1 As shown, a diffusion model food generation method based on collaborative relationship perception includes the following steps:

[0101] S101: receiving food image data, inputting the food image data into a pre-established annotation model BLIP for text annotation, and obtaining annotated food image data;

[0102] First, we collected about 400,000 food-related images from major cooking websites and food photography websites using crawler technology. Then, we deduplicated these images to ensure the uniqueness of the data. Then, we used the CLIP model to calculate the similarity between each original image P and the corresponding recipe text T, thereby filtering out images that are not related to the recipe content and improving the relevance of the dataset. The calculation formula for similarity sim is as follows:

[0103]

[0104] Next, high-quality images with a resolution greater than 480×480 are selected to ensure the clarity of the image. After that, watermarks on these high-resolution images are removed to ensure the purity of the image. Specifically, for the removal of text in the image, the OCR model text detection model is used to output the location box of the text; for the removal of patterns and trademarks in the image, the logo detection API provided by Baidu is used to output the location box; finally, the image restoration model Stable Diffusion-Inpainting is used to remove the information of the masked area generated from these location boxes. Aesthetic food images with exquisite plating, texture and gloss are screened. Works that meet aesthetic standards are selected to ensure that the final image set is not only practical but also visually attractive. The number of images retained is about 130,000.

[0105] like Figure 2 As shown in the figure, the description of the image is generated using the pre-trained text annotation model BLIP. First, the image is encoded, and then the encoded image is input into the BLIP model. The model outputs the corresponding text description for each image. In addition, in order to evaluate the accuracy of the generated annotations, 60 samples were manually selected for evaluation, and about 80% of the text annotations were accurate. This process helps ensure that the generated description is highly consistent with the image in terms of content and semantics.

[0106] S102: Input the annotated food image data into the pre-established question-answering model GPT4 to extract the relationship between the recipe and the image, and extract each ingredient and the corresponding cooking action as a binary;

[0107] like Figure 3 As shown in the figure, first, design questions to ask GPT-4 to decompose the menu into different ingredient and action pairs. The GPT-4 model is a pre-trained large-scale question-answering model based on the Transformer structure. The input is a food image, the corresponding recipe and a designed question. It extracts relations from the Recipe1M dataset, and then compares the extracted relations to select the question with the highest accuracy. The recipe set is represented as R = {y1, y2, ..., y n}, for the current i-th input recipe y i , which includes the cooking action relationship set corresponding to each ingredient {r 0 ,r 1 ,…,r m}, m represents the number of relations. Each relation r i From ingredients I i and the corresponding cooking action A i Composition, record this tuple as:

[0108] r i =(I i,A i )

[0109] S103: Input the binary group into the text encoder of the pre-trained large model CLIP for encoding, input the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector, and output the generated food image;

[0110] like Figure 3 As shown, the text encoder of the pre-trained large model CLIP is used to encode the extracted relations, and the encoded relations are input into the cross-attention layer of the diffusion model through the relation injector.

[0111] Attached Figure 3 The relation injector in fused the extracted relations through a multi-layer perceptron MLP and a gated recurrent unit GRU. First, the raw materials were connected with the cooking actions and input into the MLP to obtain the i-th relation feature vector g i By analyzing a large number of recipes and food pictures, we found that the later the cooking steps are, the greater the impact on the appearance of the food. Therefore, a linear growth parameter λ is used in the feature injector to control the relationship between each relationship r i Importance:

[0112] g i =MLP(λI i ,A i )

[0113] Among them, λ follows the uniform distribution of (0, 1), I i represents the i-th ingredient, A i represents the cooking action corresponding to the i-th ingredient. Then GRU is used to fuse the synergistic relationship, which can be expressed as:

[0114] g out =GRU(g i )

[0115] In the U-Net denoiser, the relation g at the tth time step is first transformed through a linear layer. out The feature map is K t ' and V t ', in order to integrate relational features with visual features, K t ' and V t 'Add to the original K in the U-Net cross attention layer t and V t Above, it is expressed as follows:

[0116]

[0117]

[0118] Among them, β1 and β2 are parameters that control the fusion ratio, ranging from [0,1]. By adjusting β1 and β2, the injection ratio of the relationship features can be controlled. The lower the value of β, the greater the influence of the relationship between the ingredients and the cooking action. In addition, the calculation process of cross attention can be expressed as:

[0119]

[0120] Among them, φ out represents the output of the cross attention in the diffusion decoder block, and d is the dimension of the relation feature.

[0121] like Figure 4 As shown, in the cross-attention layer of the backbone diffusion model, two parameter-updatable low-rank matrices are inserted to fine-tune the diffusion model.

[0122] The parameter efficiency is improved by freezing the pre-trained weight matrix and integrating additional trainable low-rank matrices in the model. In particular, in the cross-attention layer of the backbone diffusion model, two parameter-updatable low-rank matrices A and B are inserted. Then for the weight matrix W in the U-Net cross-attention q and W k , the introduction of low-rank matrices involves converting W q Update to W q ', W k Update to W k ', can be defined as:

[0123] W q '=W q +B1A1

[0124] W k '=W k +B2A2

[0125] In the training of the model, there are two processes: the forward process and the reverse process. Specifically:

[0126] In the forward process, the diffusion model is trained using an autoencoder to convert the image into a latent space and reconstruct it. The input image is represented as x0, and Gaussian noise z is gradually added to x0 as the time step t increases. The noisy image at time step t is defined as the following formula:

[0127]

[0128] Among them, a t is the noise schedule at time step t, z~N(0,1);

[0129] Reverse process, U-Net denoiser D θ For an image x in the latent spacet Gradually denoising, getting a clearer image at each step t, the goal of the backbone diffusion model is to generate an image by combining the noise map ∈ ~N(0,1) and the conditional embedding c, which is obtained from the text prompt P using the CLIP text encoder T, given the noise feature z t and conditional embedding c, using D θ To predict the noise ∈ at the current time step t, the loss function is expressed as:

[0130]

[0131] in c is the conditional embedding encoded by the CLIP text encoder T at time steps t = 0, 1, ..., T.

[0132] Given input conditions c and z t , D θ The noise estimate is predicted for each step t. After T iterations, the final prediction result z0 is input into the autoencoder for image generation.

[0133] S104: Input the generated food image into a pre-established image scoring feedback model to obtain an image scoring score, and optimize the generated food image according to the image scoring score.

[0134] like Figure 5 As shown in the figure, the image scoring feedback model scores the generated food images, and then feeds the scores back into the training of the denoising process to continuously optimize the image's food details and aesthetics.

[0135] Existing models are trained on a wide range of image categories, mainly scoring in terms of texture consistency, image quality, and harmfulness, making it difficult to apply them directly to the evaluation of food images. In order to obtain a reward model to evaluate the aesthetics of food images, it is fine-tuned. 150 food images are evaluated based on texture, glossiness, plate placement, and food integrity, and then ranked according to human preferences and used to fine-tune the image feedback model. This image feedback model is then applied to the training of food image generation. The specific process is:

[0136] First, we randomly select a time step t, and then directly use the latent features in the selected time step to generate an image x'0. Second, we use the image reward model τ to evaluate the image x0 generated in step T and x'0. Finally, we feed the higher score back to step t-1 in the denoising process. The loss function is as follows:

[0137]

[0138] Among them, γ is the reward score weighting coefficient, φ represents the reward-loss mapping function, and y i Indicates input recipe, r i is the relationship set, θ is the parameter of the diffusion model, D θ (y i ) is the text y i Corresponding generated food images.

[0139] After introducing the reward loss, the final loss function is expressed as follows:

[0140] L=L dm +L reward

[0141] Specifically, the scheme of the present invention is further described below through an embodiment: In order to evaluate the quality of the image, the frequency distance between the real image and the synthetic image is calculated by FID. The lower the FID value, the higher the authenticity of the generated image; the diversity and authenticity of the generated image are evaluated by IS. The higher the score, the higher the image quality; CLIPSIM is used to evaluate whether the generated food image and the input recipe are semantically consistent. The specific calculation formula of the indicator is as follows:

[0142]

[0143] Among them, T r represents the trace of the image feature matrix, x and g represent the real image and the synthetic image, R represents the input recipe, μ represents the mean, represents the distribution of images generated from the image generator, p(y|x) represents the probability distribution of the images belonging to each category, p(y) represents the marginal distribution of the synthesized images on all categories, and D KL Represents the calculation function of KL divergence.

[0144] As shown in Table 1, the three indicators mentioned above are used to judge the quality of generated food images in the Recipe1M dataset. Compared with the existing models, the three indicators of FID, IS and CLIPSIM have achieved significant improvements.

[0145] Table 1

[0146]

[0147] Furthermore, to evaluate the quality of food images, Figure 6 The comparison results of the images generated by the existing model are shown. Compared with the existing model, the method of the first embodiment achieves better results in terms of image details and aesthetics, and also has better authenticity.

[0148] Embodiment 2: In the second aspect, as Figure 7As shown, in order to achieve the above-mentioned purpose, the present invention discloses a diffusion model food generation system based on collaborative relationship perception, comprising:

[0149] The data processing module 11 is used to receive food image data, input the food image data into a pre-established annotation model BLIP for text annotation, and obtain annotated food image data;

[0150] The relationship extraction module 12 is used to input the annotated food image data into the pre-established question-answering model GPT4 to extract the relationship between the recipe and the image, and extract each ingredient and the corresponding cooking action as a binary group;

[0151] An image generation module 13 is used to input the binary group into the text encoder of the pre-trained large model CLIP for encoding, input the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector, fine-tune the backbone diffusion model with a parameter-updatable low-rank matrix, and output the generated food image;

[0152] The image scoring module 14 is used to input the generated food image into a pre-established image scoring feedback model to obtain an image scoring score, and optimize the generated food image according to the image scoring score.

[0153] In combination with the second aspect, in some implementations of the second aspect, the system further includes: the process of acquiring the food image data by the data processing module 11 is as follows:

[0154] Use a crawler to crawl multiple food-related images from cooking websites and food photography websites; remove duplicate images and use the CLIP model to calculate the similarity between the original image and the recipe to filter out irrelevant images; after filtering out duplicate food images, keep high-resolution images with a resolution greater than the preset resolution; remove the watermarks of high-resolution images and filter out aesthetically pleasing images as food image data;

[0155] The process of the data processing module 11 to select the images that meet the aesthetic requirements is as follows:

[0156] Use the OCR model text detection model to detect text in the image and output the location box of the text; use the logo detection API to detect trademarks and patterns in the image and output the location box; use the image restoration model Stable Diffusion-Inpainting to remove the information of the masked area generated from these location boxes; screen the aesthetic food images with exquisite plating, texture and gloss;

[0157] Alternatively, the process of the data processing module 11 inputting the food image data into the pre-established annotation model BLIP for text annotation includes:

[0158] Use the pre-trained text annotation model BLIP to generate descriptions of food images, then encode the food images and calculate a set of most similar prompts, then concatenate them to obtain a set of prompts, input the encoded food images into the BLIP model, and output the text corresponding to each food image;

[0159] The process of the image generation module 13 inputting the binary into the text encoder of the pre-trained large model CLIP for encoding, and inputting the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector includes:

[0160] The relation injector fuses the encoded relations through the multi-layer perceptron MLP and the gated recurrent unit GRU. First, the raw materials are connected with the cooking actions and input into the MLP to obtain the i-th relation feature vector g i , a linear growth parameter λ is used in the feature injector to control each relation r i :

[0161] g i =MLP(λI i ,A i )

[0162] Among them, λ follows the uniform distribution of (0, 1), I i represents the i-th ingredient, A i Represents the cooking action corresponding to the i-th ingredient, and then uses GRU to fuse the synergistic relationship, expressed as:

[0163] g out =GRU(g i )

[0164] In the U-Net denoiser, the relation g at the tth time step is transformed through a linear layer. out The feature map is K t ' and V t ', integrate the relational features with the visual features, and transform K t ' and V t 'Add to the original K in the U-Net cross attention layer t and V t Above, it is expressed as follows:

[0165]

[0166] Among them, β1 and β2 are parameters that control the fusion ratio. By adjusting β1 and β2, the injection ratio of relational features can be controlled. The calculation process of cross attention is expressed as:

[0167]

[0168] Among them, φ out represents the output of the cross attention in the diffusion decoder block, and d is the dimension of the relation feature;

[0169] The cross attention layer of the pre-established backbone diffusion model of the image generation module 13 fine-tunes the backbone diffusion model by inserting two parameter-updatable low-rank matrices, and the process is as follows:

[0170] The backbone diffusion model is a Stable Diffusion 1.5 model, which generates images from conditional information and Gaussian noise through a denoising process. The backbone diffusion model consists of an autoencoder and a U-Net denoiser. The framework is built on two complementary random processes: the forward process and the reverse process.

[0171] In the cross-attention layer of the backbone diffusion model, two parameter-updatable low-rank matrices A and B are inserted. For the weight matrix W in the U-Net cross-attention q and W k , the introduction of low-rank matrices involves converting W q Update to W q ', W k Update to W k ', can be defined as:

[0172] W q '=W q +B1A1

[0173] W k '=W k +B2A2

[0174] The forward process and the reverse process of the image generation module 13 include:

[0175] In the forward process, the diffusion model is trained using an autoencoder to convert the image into a latent space and reconstruct it. The input image is represented as x0. As the time step t increases, Gaussian noise z is gradually added to x0. The noisy image at time step t is defined as the following formula:

[0176]

[0177] Among them, a t is the noise schedule at time step t, z~N(0,1);

[0178] Reverse process, U-Net denoiser D θ For an image x in the latent space tGradually denoising, getting a clearer image at each step t, the goal of the backbone diffusion model is to generate an image by combining the noise map ∈ ~N(0,1) and the conditional embedding c, which is obtained from the text prompt P using the CLIP text encoder T, given the noise feature z t and conditional embedding c, using D θ To predict the noise ∈ at the current time step t, the loss function is expressed as:

[0179]

[0180] in c is the conditional embedding encoded by the CLIP text encoder T, at time step t=0,1,…,T;

[0181] Given input conditions c and z t , D θ The noise estimate at each step t is predicted, and after T iterations, the final prediction result z0 is input into the autoencoder for image generation;

[0182] The image scoring module 14 inputs the generated food image into a pre-established image scoring feedback model, and the calculation process of obtaining the image scoring score is as follows:

[0183] A time step t is randomly selected, and then the latent features in the selected time step are directly used to generate an image x'0. Secondly, the image reward model τ is used to evaluate the image x0 generated in step T and x'0. Finally, the higher score is fed back to step t-1 in the denoising process. The loss function is as follows:

[0184]

[0185] Among them, γ is the reward score weighting coefficient, φ represents the reward-loss mapping function, and y i Indicates the input formula, r i is the relationship set, θ is the parameter of the diffusion model, D θ (y i ) is the text y i Corresponding generated food images;

[0186] Introducing reward loss L reward After that, the final loss function L is expressed as follows:

[0187] L=L dm +L reward

[0188] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0189] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0190] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0191] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A diffusion model food generation method based on collaborative relationship perception, characterized in that: The method comprises the following steps: Receiving food image data, inputting the food image data into a pre-established annotation model BLIP for text annotation, and obtaining annotated food image data; The labeled food image data is input into the pre-established question-answering model GPT4 to extract the relationship between the recipe and the image, and extract each ingredient and the corresponding cooking action as a binary; The bigrams are input into the text encoder of the pre-trained large model CLIP for encoding, and the encoded relations are input into the cross attention layer of the pre-established backbone diffusion model through the relation injector, and the generated food images are output; The generated food image is input into a pre-established image scoring feedback model to obtain an image scoring score, and the generated food image is optimized according to the image scoring score.

2. The method for generating food using a diffusion model based on collaborative relationship perception according to claim 1, characterized in that: The process of obtaining the food image data is as follows: Use a crawler to crawl multiple food-related images from cooking websites and food photography websites; remove duplicate images, and use the CLIP model to calculate the semantic consistency between the original image and the recipe to filter out irrelevant images; after filtering out duplicate food images, keep high-resolution images with a resolution greater than the preset resolution; remove the watermarks of high-resolution images, and filter out aesthetically pleasing images as food image data.

3. The method for generating food using a diffusion model based on collaborative relationship perception according to claim 2, characterized in that: The process of selecting aesthetically pleasing images is as follows: Use the OCR model text detection model to detect the text in the image and output the location box of the text; use the logo detection API to detect the trademark and pattern in the image and output the location box; use the diffusion model-based image restoration model StableDiffusion-Inpainting to remove the information of the masked area generated from these location boxes; screen the aesthetic food images with exquisite plating, texture and gloss.

4. The method for generating food using a diffusion model based on collaborative relationship perception according to claim 1, characterized in that: The process of inputting food image data into a pre-established annotation model BLIP for text annotation includes: Use the pre-trained text annotation model BLIP to generate descriptions of food images, then encode the food images and calculate a set of most similar prompts, which are then concatenated to obtain a set of prompts. The encoded food images are input into the BLIP model to output the text corresponding to each food image.

5. The method for generating food using a diffusion model based on collaborative relationship perception according to claim 1, characterized in that: The process of inputting the big-tuple into the text encoder of the pre-trained large model CLIP for encoding, and inputting the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector includes: The relation injector fuses the encoded relations through the multi-layer perceptron MLP and the gated recurrent unit GRU. First, the raw materials are connected with the cooking actions and input into the MLP to obtain the i-th relation feature vector g i , a linear growth parameter λ is used in the feature injector to control each relation r i Importance: g i MLP(λI i ,A i ) Among them, λ follows the uniform distribution of (0, 1), I i represents the i-th ingredient, A i Represents the cooking action corresponding to the i-th ingredient, and then uses GRU to fuse the synergistic relationship, expressed as: g out =GRU(g i ) In the U-Net denoiser, the relation g at the tth time step is transformed through a linear layer. out The feature map is K t ' and V t ', integrate the relational features with the visual features, and transform K t ' and V t 'Add to the original K in the U-Net cross attention layer t and V t Above, it is expressed as follows: Among them, β1 and β2 are parameters that control the fusion ratio. By adjusting β1 and β2, the injection ratio of relational features can be controlled. The calculation process of cross attention is expressed as: Among them, φ out represents the output of the cross attention in the diffusion decoder block, and d is the dimension of the relation feature.

6. The method for generating food using a diffusion model based on collaborative relationship perception according to claim 5, characterized in that: The cross attention layer of the pre-established backbone diffusion model fine-tunes the backbone diffusion model by inserting two parameter-updatable low-rank matrices, and the process is as follows: The backbone diffusion model is a Stable Diffusion 1.5 model, which generates images from conditional information and Gaussian noise through a denoising process. The backbone diffusion model consists of an autoencoder and a U-Net denoiser. The framework is built on two complementary random processes: the forward process and the reverse process. In the cross-attention layer of the backbone diffusion model, two parameter-updatable low-rank matrices A and B are inserted. For the weight matrix W in the U-Net cross-attention q and W k , the introduction of low-rank matrices involves converting W q Update to W q ', W k Update to W k ', defined as: IN q '=W q +B1A1 <h2 style=";text-align:left;direction:ltr">W<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> '=W<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> +B2A2。 7. The method for generating food using a diffusion model based on collaborative relationship perception according to claim 6, characterized in that: The forward process and the reverse process include: In the forward process, the variational autoencoder VAE model is used to convert the image into a latent space and reconstruct it. The input image is represented as x0. As the time step t increases, Gaussian noise z is gradually added to x0. The noisy image at time step t is defined as the following formula: Among them, a t is the noise schedule at time step t, z~N(0,1); Reverse process, U-Net denoiser D θ For an image x in the latent space t Gradually denoising, getting a clearer image at each step t, the goal of the backbone diffusion model is to generate an image by combining the noise map ∈ ~N(0,1) and the conditional embedding c, which is obtained from the text prompt P using the CLIP text encoder T, given the noise feature z t and conditional embedding c, using D θ To predict the noise ∈ at the current time step t, the loss function is expressed as: in c is the conditional embedding encoded by the CLIP text encoder T, at time step t=0,1,…,T; Given input conditions c and z t , D θ The noise estimate at each step t is predicted, and after T iterations, the final prediction result z0 is input into the autoencoder for image generation.

8. The method for generating food using a diffusion model based on collaborative relationship perception according to claim 1, characterized in that: The generated food image is input into the pre-established image scoring feedback model to obtain the image scoring score calculation process as follows: Randomly select a time step t, and then directly use the latent features in the selected time step to generate an image x'0. Secondly, use the image reward model τ to evaluate the image x0 generated in step T and x'0. Finally, feed the high score back to step T-1 in the denoising process. The loss function is as follows: Among them, γ is the reward score weighting coefficient, φ represents the reward-loss mapping function, and y i Indicates input recipe, r i is the relationship set, θ is the parameter of the diffusion model, D θ (y i ) is the text y i Corresponding generated food images; Introducing reward loss L reward After that, the final loss function L is expressed as follows: L=L dm +L reward 9. A diffusion model food generation system based on collaborative relationship perception, characterized in that: include: A data processing module, used for receiving food image data, inputting the food image data into a pre-established annotation model BLIP for text annotation, and obtaining annotated food image data; The relationship extraction module is used to input the annotated food image data into the pre-established question-answering model GPT4 to extract the relationship between the recipe and the image, and extract each ingredient and the corresponding cooking action as a binary; The image generation module is used to input the binary into the text encoder of the pre-trained large model CLIP for encoding, input the encoded relationship into the cross attention layer of the pre-established backbone diffusion model through the relationship injector, and output the generated food image; The image scoring module is used to input the generated food image into a pre-established image scoring feedback model to obtain an image scoring score, and optimize the generated food image according to the image scoring score.

10. The food generation system based on the diffusion model of collaborative relationship perception according to claim 9, characterized in that: The process of obtaining food image data of the data processing module is as follows: Use a crawler to crawl multiple food-related images from cooking websites and food photography websites; remove duplicate images and use the CLIP model to calculate the semantic consistency between the original image and the recipe to filter out irrelevant images; after filtering out duplicate food images, keep high-resolution images with a resolution greater than the preset resolution; remove the watermarks of high-resolution images and filter out aesthetically pleasing images as food image data; The process of the data processing module to filter out aesthetically pleasing images is as follows: Use the OCR model text detection model to detect text in the image and output the location box of the text; use the logo detection API to detect trademarks and patterns in the image and output the location box; use the image restoration model Stable Diffusion-Inpainting to remove the information of the masked area generated from these location boxes; screen the aesthetic food images with exquisite plating, texture and gloss; Alternatively, the process of the data processing module inputting the food image data into the pre-established annotation model BLIP for text annotation includes: Use the pre-trained text annotation model BLIP to generate descriptions of food images, then encode the food images and calculate a set of most similar prompts, then concatenate them to obtain a set of prompts, input the encoded food images into the BLIP model, and output the text corresponding to each food image; The image generation module inputs the binary into the text encoder of the pre-trained large model CLIP for encoding, and the encoded relationship is input into the cross attention layer of the pre-established backbone diffusion model through the relationship injector. The process includes: The relation injector fuses the encoded relations through the multi-layer perceptron MLP and the gated recurrent unit GRU. First, the raw materials are connected with the cooking actions and input into the MLP to obtain the i-th relation feature vector g i , a linear growth parameter λ is used in the feature injector to control each relation r i Importance: g i MLP(λI i ,A i ) Among them, λ follows the uniform distribution of (0, 1), I i represents the i-th ingredient, A i Represents the cooking action corresponding to the i-th ingredient, and then uses GRU to fuse the synergistic relationship, expressed as: g out =GRU(g i ) In the U-Net denoiser, the relation g at the tth time step is transformed through a linear layer. out The feature map is K t ' and V t ', integrate the relational features with the visual features, and transform K t ' and V t 'Add to the original K in the U-Net cross attention layer t and V t Above, it is expressed as follows: Among them, β1 and β2 are parameters that control the fusion ratio. By adjusting β1 and β2, the injection ratio of relational features can be controlled. The calculation process of cross attention is expressed as: Among them, φ out represents the output of the cross attention in the diffusion decoder block, and d is the dimension of the relation feature; The image generation module fine-tunes the backbone diffusion model by inserting two parameter-updatable low-rank matrices into the cross-attention layer of the pre-established backbone diffusion model. The process is as follows: The backbone diffusion model is a Stable Diffusion 1.5 model, which generates images from conditional information and Gaussian noise through a denoising process. The backbone diffusion model consists of an autoencoder and a U-Net denoiser. The framework is built on two complementary random processes: the forward process and the reverse process. In the cross-attention layer of the backbone diffusion model, two parameter-updatable low-rank matrices A and B are inserted. For the weight matrix W in the U-Net cross-attention q and W k , the introduction of low-rank matrices involves converting W q Update to W q ', W k Update to W k ', defined as: IN q '=W q +B1A1 <h2 style=";text-align:left;direction:ltr">W<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> '=W<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> +B2A2 The forward and reverse processes of the image generation module include: In the forward process, the diffusion model is trained using an autoencoder to convert the image into a latent space and reconstruct it. The input image is represented as x0. As the time step t increases, Gaussian noise z is gradually added to x0. The noisy image at time step t is defined as the following formula: Among them, a t is the noise schedule at time step t, z~N(0,1); Reverse process, U-Net denoiser D θ For an image x in the latent space t Gradually denoising, getting a clearer image at each step t, the goal of the backbone diffusion model is to generate an image by combining the noise map ∈ ~N(0,1) and the conditional embedding c, which is obtained from the text prompt P using the CLIP text encoder T, given the noise feature z t and conditional embedding c, using D θ To predict the noise ∈ at the current time step t, the loss function is expressed as: in c is the conditional embedding encoded by the CLIP text encoder T, at time step t=0,1,…,T; Given input conditions c and z t , D θ The noise estimate at each step t is predicted, and after T iterations, the final prediction result z0 is input into the autoencoder for image generation; The image scoring module inputs the generated food images into the pre-established image scoring feedback model, and the calculation process of the image scoring score is as follows: Randomly select a time step t, and then directly use the latent features in the selected time step to generate an image x'0. Secondly, use the image reward model τ to evaluate the image x0 generated in step T and x'0. Finally, feed the high score back to step T-1 in the denoising process. The loss function is as follows: Among them, γ is the reward score weighting coefficient, φ represents the reward-loss mapping function, and y i Indicates input recipe, r i is the relationship set, θ is the parameter of the diffusion model, D θ (y i ) is the text y i Corresponding generated food images; Introducing reward loss L reward After that, the final loss function L is expressed as follows: L=L dm +L reward .

Images