Global and component fine-grained recipe retrieval method using segmentation and negative matching

Abstract

The invention discloses a global and component fine-grained recipe retrieval method using segmentation and negative matching, and belongs to the technical field of food cross-modal retrieval in food calculation. When the global features of the food image are extracted, the SAM model is combined to extract the main food features, and when the global features of the recipe are extracted, the component features of the title, the ingredient and the instruction are reserved at the same time; then designing a dynamic negative matching-triple joint loss function: optimizing a cross-modal mismatching sample spacing based on Monte Carlo sampling by utilizing global features of food images and recipes on a global level, and constructing fine-grained negative matching loss of titles / ingredients / instructions by utilizing component layer features to strengthen semantic distinguishing; and an alignment error is constrained through a similarity matrix of segmented main food features and recipe ingredient features. By means of the method, the problems of text redundancy interference, image background noise, component semantic confusion and the like existing in a traditional method are effectively solved, and the retrieval recall rate is further increased.

Description

Technical Field

[0001] The present invention relates to a global and component fine-grained recipe retrieval method using segmentation and negative matching, and belongs to the technical field of food cross-modal retrieval in food computing. Background Art

[0002] With the development of visual-language models and people's increasing attention to diet, cross-modal recipe retrieval has attracted widespread attention from industry and academia. However, since recipes usually contain multi-level structures and rich contextual information, cross-modal recipe retrieval faces huge challenges. Cross-modal recipe retrieval is a task that combines image and text information to identify and match recipes and their related images. Cross-modal retrieval can promote the realization of a series of applications, including food exploration, accurate search, catering marketing, personalized recipe creation, etc. The key to this task is to accurately identify information such as ingredients and cooking methods in food images and understand the corresponding descriptions in recipe texts. At the same time, a semantic association model between images and texts should be constructed so that the two can perform effective similarity calculations in the same semantic space to achieve accurate retrieval.

[0003] Most of the current cross-modal recipe retrieval methods use the method of global alignment of triples, which has significant disadvantages. First, the extracted global features of the image are mixed with a large amount of background noise, which dilutes the features of the ingredients. Secondly, the three components of the recipe are forcibly merged into an overall vector, which causes semantic confusion between recipes and the loss of unique information of each component. Finally, when aligning the triples globally, it is too sensitive to redundant information, and the use of forced alignment introduces semantic bias. It is precisely because of these reasons that the global feature alignment method is not satisfactory. Summary of the invention

[0004] The object of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a global and component fine-grained recipe retrieval method using segmentation and negative matching.

[0005] The technical solution provided by the present invention is as follows: a global and component fine-grained recipe retrieval method using segmentation and negative matching, characterized in that it comprises the following steps: Step S1, using the Recipe1M dataset, establish a dataset of two modalities: food images and recipes, where the food image includes two parts: food and background, and the recipe includes three components: title, ingredients, and instructions. The dataset of food images and recipes is divided into a training set, a validation set, and a test set; Step S2, respectively using the ViT visual model and the cascaded Transformer model to extract features from the food images and recipes in the training set, validation set, and test set, specifically including: Step S21, extracting global features of food images; Step S22, extracting the image features of the main food after image segmentation; Step S23, extracting text features of the recipe; Step S3, constructing a loss function on the training set, specifically including: Step S31, calculating the matching probability between food images and recipes and between recipes and food images, and using the calculated matching probabilities to construct an overall negative matching loss function to increase the distance of unmatched samples; Step S32, calculating the bidirectional matching probability between each component of the three recipes and the food image, and using the calculated matching probability to calculate the negative matching loss function at the three component levels, further increasing the distance between unmatched samples; Step S33, using the segmentation model to segment the main food, calculating the similarity matrix of the segmented main food and the ingredient components in the recipe, and using triplet loss to reduce the matching error between the main food and the ingredient components of the recipe, thereby enhancing the robustness of cross-modal alignment; Step S34, constructing a final dynamic negative match-triplet joint loss function; Step S4, performing cross-modal recipe retrieval, using the recall rate as an evaluation indicator.

[0006] Furthermore, in step S2: Step S21, extracting global features of food images: Food image collection Input to the ViT visual model to extract the global feature vector of food images , Indicates the use of ViT visual model, Indicates Food images, is the visual modality identifier, Indicates the number of food images; Step S22, extracting the image features of the main food after image segmentation: The SAM segmentation model is used to segment food images. First, a large number of masks are generated through the "everything" mode to cover various areas in the food image. Then, the area threshold is used to filter out small or meaningless areas, and the CLIP large model is used for semantic matching to retain masks related to food. Finally, the Top-N areas with the highest semantic scores are selected as enhanced data to extract the main food in the food image and remove background interference. The food image fragments formed after segmentation are Input to ViT visual model , average pooling is performed on the embeddings of all food image segments to obtain the average food image segment embedding vector , represents the number of food image segments, Indicates Food image segments; Step S23, extracting text features of the recipe: A cascaded Transformer model is used to process at the word level and sentence level. Given a recipe collection ,in, Represents the title component, Represents an ingredient component, Represents a command component, is a text identifier, Indicates the number of recipes; the title component , ingredient components and the directive component The first-layer Transformer model receives each word in the sentences of the three components and encodes them to capture the relationship between each word. Finally, the embedding representations of all words in each sentence are averaged through average pooling to obtain the sentence-level embeddings of the title component, ingredient component, and instruction component. = , Represents the first-layer Transformer model; the obtained sentence-level embeddings of the title component, ingredient component, and instruction component are sent to the second-layer Transformer to capture the relationship between each sentence in the title component, ingredient component, and instruction component. The sentence-level embeddings of the title component, ingredient component, and instruction component are averaged through average pooling to obtain the final overall fusion feature vector of the title component, ingredient component, and instruction component. = ,in It is the second-layer Transformer model; the overall fusion feature vectors of the title component, ingredient component, and instruction component are concatenated to obtain the final global feature vector of the recipe. , represents a fully connected layer.

[0007] Furthermore, in step S3: Step S31: The global feature vector of the food image , the global feature vector of the recipe , using Monte Carlo methods to calculate the global cross-modal matching probability of food images to recipes , and the global cross-modal matching probability of recipes to food images The formula is as follows: ; ; in, Representing the global feature vector of food images to the global feature vector of the recipe The similarity between A global feature vector representing a recipe to the global feature vector of the food image The similarity between is the temperature parameter, which is used to control the smoothness of the probability distribution. is a subset randomly sampled from the training set, representing the randomly sampled recipe samples and food image samples, represents the normalization factor of the recipe sample, represents the normalization factor of the food image sample, which is used to adjust the bias of random sampling, and N is the batch size; Utilizes the global cross-modal matching probability of calculating recipes to food images The formula constructs the matching probability set of negative samples from recipes to food images = N is the batch size, For the Recipes to The matching probability of food images; then construct the overall negative matching loss function from recipe to food image , the formula is as follows: ; Constructing an overall negative matching loss function from food images to recipes , construction method and The loss function is consistent, using the global cross-modal matching probability of calculating food images to recipes The formula constructs the matching probability set of food images to recipe negative samples = N is the batch size, Indicates Food images to The matching probability of recipes; then construct the overall negative matching loss function from food images to recipes , the formula is as follows: ; And add the two together to get the overall negative matching loss function formula as follows: ; Step S32, calculating the overall fusion feature vector of the title component to the global feature vector of the food image The matching probability , the overall fusion feature vector of the batching components to the global feature vector of the food image The matching probability The overall fusion feature vector of the instruction component to the global feature vector of the food image The matching probability ; The formulas are as follows: ; ; ; in, Represents the overall fused feature vector of the title components to the global feature vector of the food image The similarity between Represents the overall fusion feature vector of the batching components to the global feature vector of the food image The similarity between Represents the overall fusion feature vector of the batching components to the global feature vector of the food image The similarity between is the temperature parameter, which is used to control the smoothness of the probability distribution. is a subset randomly sampled from the training set, representing the Kth randomly sampled food image sample, It is represented as the normalization factor of the food image sample, which is used to adjust the bias of random sampling, and N is the batch size; Compute global eigenvectors of food images to the overall fused feature vector of the title component The matching probability , the global feature vector of food images The overall fusion feature vector of the batching component The matching probability , the global feature vector of food images to the overall fused feature vector of the title component The matching probability The formula is as follows: ; ; ; in, Represents the global feature vector of the image to the overall fused feature vector of the title component The similarity of Represents the global feature vector of the image The overall fusion feature vector of the batching component The similarity of Represents the global feature vector of the image To the overall fused feature vector of the instruction component The similarity of is the temperature parameter, which is used to control the smoothness of the probability distribution. They are respectively represented as a subset randomly sampled from the training set, representing the The overall fusion feature vector samples of the randomly sampled title component, the overall fusion feature vector samples of the ingredient component, and the overall fusion feature vector samples of the instruction component. is the normalization factor of the recipe sample, which is used to adjust the bias of random sampling, and N is the batch size; Construct title negative matching loss function respectively , negative matching loss function And instruction negative matching loss function , respectively using the overall fusion feature vector of the title component to the global feature vector of the food image The matching probability , the overall fusion feature vector of the batching components to the global feature vector of the food image The matching probability The overall fusion feature vector of the instruction component to the global feature vector of the food image The matching probability Construct a set of matching probabilities from the title component to the negative samples of the food image , the set of negative sample matching probabilities from ingredient component samples to food images , the set of negative sample matching probabilities from instruction component samples to food images , N is the batch size, Indicates Title components to The negative sample matching probability of food images, Indicates Ingredients to The negative sample matching probability of food images, Indicates The instruction component to The negative sample matching probability of food images; then construct the negative matching loss function from the title component to the food image , negative matching loss function from ingredient components to food images , negative matching loss function from ingredient components to food images The formula is as follows: ; ; ; Constructing negative matching loss function for food image to title component , negative matching loss function from food image to ingredient component , food image to instruction component negative matching loss function , respectively using the global feature vector of the food image to the overall fused feature vector of the title component The matching probability , the global feature vector of food images The overall fusion feature vector of the batching component The matching probability and the global feature vector of the food image To the overall fused feature vector of the instruction component The matching probability Construct a set of negative sample matching probabilities from food images to title components , the set of negative sample matching probabilities from food images to ingredient components , the set of negative sample matching probabilities of food images to instruction components , N is the batch size, Indicates Food images to The negative sample matching probability of title components, Indicates Food images to The negative sample matching probability of the ingredient components, Indicates Food images to The negative sample matching probability of the instruction component recipe image; then construct the negative matching loss function from the title component to the food image , negative matching loss function from ingredient components to food images , negative matching loss function from ingredient components to food images The formula is as follows: ; ; ; Add the loss functions in both directions to get the negative matching function of the title component , ingredient component negative matching function , ingredient component negative matching function The formula is as follows: ; ; ; Negatively match the title component to the function , ingredient component negative matching function , ingredient component negative matching function Add together to get the component negative matching loss function The formula is as follows: ; Step S33, embed each average food image segment after image segmentation into a vector Composition vector set and the overall fusion feature vector of each ingredient component Composition vector set , respectively calculate The similarity matrix and The similarity matrix , the specific calculation formula is as follows: ; ; Using triplet loss To minimize The difference between matrices is calculated as: ; in is the cosine similarity vector, is the triplet loss boundary parameter, To draw a point, is a positive sample, is a negative sample; for Transpose the matrix, for Transpose a matrix; Step S34, add the three loss functions to obtain the final dynamic negative match-triplet joint loss function : + + .

[0008] The beneficial effects of the present invention are as follows: the present invention first uses cascaded Transformers to extract recipe text features in layers. In the process of extracting features, it is necessary to construct a global feature vector and save the vectors of the three components, overcoming the semantic confusion problem caused by component information loss in traditional methods. Then, the SAM model is introduced to segment the food image and select the main food area. The ViT model is used to extract visual features to remove interference, effectively reducing noise interference. Finally, a dynamic negative matching-triplet joint loss function is constructed. The global negative matching loss dynamically expands the mismatched sample spacing through Monte Carlo sampling, and the component-level negative matching loss optimizes the fine-grained alignment of the title, ingredients, and instructions respectively. The local triplet loss strengthens the semantic association between the segmented main food features and the text ingredient items.

[0009] The present invention innovatively combines component fine-grained feature extraction with cross-modal contrast learning mechanism, uses SAM model to accurately locate the main food area in food image samples, removes the influence of background noise, maintains component semantic independence in recipe samples, and avoids text redundancy caused by forced alignment and semantic confusion caused by component information loss through negative matching-triplet joint optimization strategy, significantly improving the accuracy and robustness of cross-modal retrieval. Compared with traditional global alignment methods, the present invention has significant effects. DETAILED DESCRIPTION

[0010] The specific implementation modes of the present invention are described in detail below: A global and component fine-grained recipe retrieval method using segmentation and negative matching, comprising the following steps: Step S1, using the Recipe1M dataset, establish a dataset of two modalities: food images and recipes, where food images include two parts: food and background, and recipes include three components: title, ingredients, and instructions. The dataset of food images and recipes is divided into a training set, a validation set, and a test set.

[0011] Step S2, respectively using the ViT visual model and the cascaded Transformer model to extract features from the food images and recipes in the training set, the validation set, and the test set, specifically includes the following steps: Step S21, extracting global features of food images: Food image collection Input to the ViT visual model to extract the global feature vector of food images , Indicates the use of ViT visual model, Indicates Food images, is the visual modality identifier, Represents the number of food images.

[0012] Step S22, extracting the image features of the main food after image segmentation: Use the SAM segmentation model to segment food images. First, use the "everything" mode to generate a large number of masks to cover various areas in the food image. Then, use the area threshold to filter out areas that are too small or meaningless, and use the CLIP large model for semantic matching to retain masks related to food. Finally, select the Top-N areas with the highest semantic scores as enhanced data to extract the main food in the food image and remove background interference. It has been verified that the Top-4 effect is the best. The food image fragments formed after segmentation Feed into ViT visual model , average pooling is performed on the embeddings of all food image segments to obtain the average food image segment embedding vector , represents the number of food image segments, Indicates Food image clips.

[0013] Step S23, extracting text features of the recipe: A cascaded Transformer model is used to process at the word level and sentence level. Given a recipe collection ,in, Represents the title component, Represents an ingredient component, Represents a command component, is a text identifier, Indicates the number of recipes; the title component , ingredient components and the directive component The first-layer Transformer model receives each word in the sentences of the three components and encodes them to capture the relationship between each word. Finally, the embedding representations of all words in each sentence are averaged through average pooling to obtain the sentence-level embeddings of the title component, ingredient component, and instruction component. = , Represents the first-layer Transformer model; the obtained sentence-level embeddings of the title component, ingredient component, and instruction component are sent to the second-layer Transformer to capture the relationship between each sentence in the title component, ingredient component, and instruction component. The sentence-level embeddings of the title component, ingredient component, and instruction component are averaged through average pooling to obtain the final overall fusion feature vector of the title component, ingredient component, and instruction component. = ,in It is the second-layer Transformer model; the overall fusion feature vectors of the title component, ingredient component, and instruction component are concatenated to obtain the final global feature vector of the recipe. , represents a fully connected layer.

[0014] Step S3, constructing a loss function on the training set, specifically includes the following steps: Step S31, the core of constructing the negative matching loss is to find a common feature subspace so that the distance between the negative sample image-recipe pairs is as large as possible.

[0015] The global feature vector of the food image , the global feature vector of the recipe , using Monte Carlo methods to calculate the global cross-modal matching probability of food images to recipes , and the global cross-modal matching probability of recipes to food images The formula is as follows: ; ; in, Representing the global feature vector of food images to the global feature vector of the recipe The similarity between A global feature vector representing a recipe to the global feature vector of the food image The similarity between is the temperature parameter, which is used to control the smoothness of the probability distribution. is a subset randomly sampled from the training set, representing the randomly sampled recipe samples and food image samples, represents the normalization factor of the recipe sample, represents the normalization factor of the food image samples, which is used to adjust the bias of random sampling, and N is the batch size.

[0016] Utilizes the global cross-modal matching probability of calculating recipes to food images The formula constructs the matching probability set of negative samples from recipes to food images = N is the batch size, For the Recipes to The matching probability of food images; then construct the overall negative matching loss function from recipe to food image , the formula is as follows: ; Since we have two tasks: retrieving recipes from food images and retrieving food images from recipes, we need to construct an overall negative matching loss function from food images to recipes. , construction method and The loss function is consistent, using the global cross-modal matching probability of calculating food images to recipes The formula constructs the matching probability set of food images to recipe negative samples = N is the batch size, Indicates Food images to Then construct the overall negative matching loss function from food image to recipe. , the formula is as follows: ; And add the two together to get the overall negative matching loss function formula as follows: ; Through this overall negative matching loss function It can effectively solve the problem that it is difficult for images to fully express the content described by the text, that is, the problem of text information redundancy.

[0017] Step S32: Each component in the recipe contains more detailed information. By performing negative matching on each component, more detailed mismatching information can be captured to further increase the mismatching distance and calculate the overall fusion feature vector of the title component. to the global feature vector of the food image The matching probability , the overall fusion feature vector of the batching components to the global feature vector of the food image The matching probability The overall fusion feature vector of the instruction component to the global feature vector of the food image The matching probability The formulas are as follows: ; ; ; in, Represents the overall fused feature vector of the title components to the global feature vector of the food image The similarity between Represents the overall fusion feature vector of the batching components to the global feature vector of the food image The similarity between Represents the overall fusion feature vector of the batching components to the global feature vector of the food image The similarity between is the temperature parameter, which is used to control the smoothness of the probability distribution. is a subset randomly sampled from the training set, representing the Kth randomly sampled food image sample, It is represented as the normalization factor of the food image samples, which is used to adjust the bias of random sampling, and N is the batch size.

[0018] Since it is a bidirectional retrieval, it is necessary to calculate the global feature vector of the food image to the overall fused feature vector of the title component The matching probability , the global feature vector of food images The overall fusion feature vector of the batching component The matching probability , the global feature vector of food images to the overall fused feature vector of the title component The matching probability The formula is as follows: ; ; ; in, Represents the global feature vector of the image to the overall fused feature vector of the title component The similarity of Represents the global feature vector of the image The overall fusion feature vector of the batching component The similarity of Represents the global feature vector of the image To the overall fused feature vector of the instruction component The similarity of is the temperature parameter, which is used to control the smoothness of the probability distribution. They are respectively represented as a subset randomly sampled from the training set, representing the The overall fusion feature vector samples of the randomly sampled title component, the overall fusion feature vector samples of the ingredient component, and the overall fusion feature vector samples of the instruction component. is the normalization factor of the recipe sample, which is used to adjust the bias of random sampling, and N is the batch size.

[0019] Next, we construct the negative matching loss function for each component, namely the title negative matching loss function , negative matching loss function And instruction negative matching loss function , respectively using the overall fusion feature vector of the title component to the global feature vector of the food image The matching probability , the overall fusion feature vector of the batching components to the global feature vector of the food image The matching probability The overall fusion feature vector of the instruction component to the global feature vector of the food image The matching probability Construct a set of matching probabilities from the title component to the negative samples of the food image , the set of negative sample matching probabilities from ingredient component samples to food images , the set of negative sample matching probabilities from instruction component samples to food images , N is the batch size, Indicates Title components to The negative sample matching probability of food images, Indicates Ingredients to The negative sample matching probability of food images, Indicates The instruction component to Then construct the negative matching loss function from the title component to the food image. , negative matching loss function from ingredient components to food images , negative matching loss function from ingredient components to food images The formula is as follows: ; ; ; Since it is a bidirectional retrieval, it is necessary to construct a negative matching loss function from the food image to the title component. , negative matching loss function from food image to ingredient component , food image to instruction component negative matching loss function , respectively using the global feature vector of the food image to the overall fused feature vector of the title component The matching probability , the global feature vector of food images The overall fusion feature vector of the batching component The matching probability and the global feature vector of the food image To the overall fused feature vector of the instruction component The matching probability Construct a set of negative sample matching probabilities from food images to title components , the set of negative sample matching probabilities from food images to ingredient components , the set of negative sample matching probabilities of food images to instruction components , N is the batch size, Indicates Food images to The negative sample matching probability of title components, Indicates Food images to The negative sample matching probability of the ingredient components, Indicates Food images to The negative sample matching probability of the instruction component recipe image is constructed. Then the negative matching loss function of the title component to the food image is constructed. , negative matching loss function from ingredient components to food images , negative matching loss function from ingredient components to food images The formula is as follows: ; ; ; Add the loss functions in both directions to get the negative matching function of the title component , ingredient component negative matching function , ingredient component negative matching function The formula is as follows: ; ; ; Negatively match the title component to the function , ingredient component negative matching function , ingredient component negative matching function Add together to get the component negative matching loss function The formula is as follows: ; Step S33, embed each average food image segment after image segmentation into a vector Composition vector set and the overall fusion feature vector of each ingredient component Composition vector set , respectively calculate The similarity matrix and The similarity matrix , the specific calculation formula is as follows: ; ; Using triplet loss To minimize The difference between matrices is calculated as: ; in is the cosine similarity vector, is the triplet loss boundary parameter, To draw a point, is a positive sample, is a negative sample, for Transpose the matrix, for Transpose a matrix; Step S34, add the three loss functions to obtain the final dynamic negative match-triplet joint loss function : + + .

[0020] Step S4, performing cross-modal recipe retrieval, using the recall rate as an evaluation indicator.

[0021] This example is mainly trained and validated on the Recipe1M dataset, which contains pairs of images and recipes. The official training set, validation set, and test set contain 238,999, 51,119, and 51,303 pairs of samples, respectively. In order to evaluate the retrieval performance of the model, we use the median rank (medR) and recall (Recall@1, Recall@5, Recall@10, referred to as R1, R5, and R10, respectively) as indicators. These indicators are calculated on a ranking list of size N = {1,000, 10,000}, and the average results are reported on 10 groups of randomly selected samples. This case is compared with the classic model HT, and the experimental results are shown in Tables 1 and 2.

[0022] Table 1 N = 1000

[0023] Table 2 N = 10000

[0024] It should be understood that the parts not elaborated in detail in this specification belong to the prior art. The above embodiments are only descriptions of the preferred implementation methods of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made by ordinary engineers and technicians in this field to the technical solution of the present invention should fall within the protection scope determined by the claims of the present invention.

Claims

1. A global and component fine-grained recipe retrieval method using segmentation and negative matching, characterized in that It includes the following steps: Step S1, using the Recipe1M dataset, establish a dataset of two modalities: food images and recipes, where the food image includes two parts: food and background, and the recipe includes three components: title, ingredients, and instructions. The dataset of food images and recipes is divided into a training set, a validation set, and a test set; Step S2, respectively using the ViT visual model and the cascaded Transformer model to extract features from the food images and recipes in the training set, validation set, and test set, specifically including: Step S21, extracting global features of food images; Step S22, extracting the image features of the main food after image segmentation; Step S23, extracting text features of the recipe; Step S3, constructing a loss function on the training set, specifically including: Step S31, calculating the matching probability between food images and recipes and between recipes and food images, and using the calculated matching probabilities to construct an overall negative matching loss function to increase the distance of unmatched samples; Step S32, calculating the bidirectional matching probability between each component of the three recipes and the food image, and using the calculated matching probability to calculate the negative matching loss function at the three component levels, further increasing the distance between unmatched samples; Step S33, using the segmentation model to segment the main food, calculating the similarity matrix of the segmented main food and the ingredient components in the recipe, and using triplet loss to reduce the matching error between the main food and the ingredient components of the recipe, thereby enhancing the robustness of cross-modal alignment; Step S34, constructing a final dynamic negative match-triplet joint loss function; Step S4, performing cross-modal recipe retrieval, using the recall rate as an evaluation indicator.

2. A global and component fine-grained recipe retrieval method using segmentation and negative matching as claimed in claim 1, characterized in that In the step S2: Step S21, extracting global features of food images: Food image collection Input to the ViT visual model to extract the global feature vector of food images , Indicates the use of ViT visual model, Indicates Food images, is the visual modality identifier, Indicates the number of food images; Step S22, extracting the image features of the main food after image segmentation: The SAM segmentation model is used to segment food images. First, a large number of masks are generated through the "everything" mode to cover various areas in the food image. Then, the area threshold is used to filter out small or meaningless areas, and the CLIP large model is used for semantic matching to retain masks related to food. Finally, the Top-N areas with the highest semantic scores are selected as enhanced data to extract the main food in the food image and remove background interference. The food image fragments formed after segmentation are Input to ViT visual model , average pooling is performed on the embeddings of all food image segments to obtain the average food image segment embedding vector , represents the number of food image segments, Indicates food image segments; Step S23, extracting text features of the recipe: A cascaded Transformer model is used to process at the word level and sentence level. Given a recipe collection ,in, Represents the title component, Represents an ingredient component, Represents a command component, is a text identifier, Indicates the number of recipes; the title component , ingredient components and the directive component The first-layer Transformer model receives each word in the sentences of the three components and encodes them to capture the relationship between each word. Finally, the embedding representations of all words in each sentence are averaged through average pooling to obtain the sentence-level embeddings of the title component, ingredient component, and instruction component. = , Represents the first-layer Transformer model; the obtained sentence-level embeddings of the title component, ingredient component, and instruction component are sent to the second-layer Transformer to capture the relationship between each sentence in the title component, ingredient component, and instruction component. The sentence-level embeddings of the title component, ingredient component, and instruction component are averaged through average pooling to obtain the final overall fusion feature vector of the title component, ingredient component, and instruction component. = ,in It is the second-layer Transformer model; the overall fusion feature vectors of the title component, ingredient component, and instruction component are concatenated to obtain the final global feature vector of the recipe. , represents a fully connected layer.

3. A global and component fine-grained recipe retrieval method using segmentation and negative matching as claimed in claim 1, characterized in that In the step S3: Step S31: The global feature vector of the food image , the global feature vector of the recipe , using Monte Carlo methods to calculate the global cross-modal matching probability of food images to recipes , and the global cross-modal matching probability of recipes to food images The formula is as follows: ； ； in, Representing the global feature vector of food images to the global feature vector of the recipe The similarity between A global feature vector representing a recipe to the global feature vector of the food image The similarity between is the temperature parameter, which is used to control the smoothness of the probability distribution. is a subset randomly sampled from the training set, representing the randomly sampled recipe samples and food image samples, represents the normalization factor of the recipe sample, represents the normalization factor of the food image sample, which is used to adjust the bias of random sampling, and N is the batch size; Utilizes the global cross-modal matching probability of calculating recipes to food images The formula constructs the matching probability set of negative samples from recipes to food images = N is the batch size, For the Recipes to The matching probability of food images; then construct the overall negative matching loss function from recipe to food image , the formula is as follows: ； Constructing an overall negative matching loss function from food images to recipes , construction method and The loss function is consistent, using the global cross-modal matching probability of calculating food images to recipes The formula constructs the matching probability set of food images to recipe negative samples = N is the batch size, Indicates Food images to The matching probability of recipes; then construct the overall negative matching loss function from food images to recipes , the formula is as follows: ； And add the two together to get the overall negative matching loss function formula as follows: ； Step S32, calculating the overall fusion feature vector of the title component to the global feature vector of the food image The matching probability , the overall fusion feature vector of the ingredient components to the global feature vector of the food image The matching probability The overall fusion feature vector of the instruction component to the global feature vector of the food image The matching probability ; The formulas are as follows: ； ； ； in, Represents the overall fused feature vector of the title components to the global feature vector of the food image The similarity between Represents the overall fusion feature vector of the batching components to the global feature vector of the food image The similarity between Represents the overall fusion feature vector of the batching components to the global feature vector of the food image The similarity between is the temperature parameter, which is used to control the smoothness of the probability distribution. is a subset randomly sampled from the training set, representing the Kth randomly sampled food image sample, It is represented as the normalization factor of the food image sample, which is used to adjust the bias of random sampling, and N is the batch size; Compute global eigenvectors of food images to the overall fused feature vector of the title component The matching probability , the global feature vector of food images The overall fusion feature vector of the batching component The matching probability , the global feature vector of food images to the overall fused feature vector of the title component The matching probability The formula is as follows: ； ； ； in, Represents the global feature vector of the image to the overall fused feature vector of the title component The similarity of Represents the global feature vector of the image The overall fusion feature vector of the batching component The similarity of Represents the global feature vector of the image To the overall fused feature vector of the instruction component The similarity of is the temperature parameter, which is used to control the smoothness of the probability distribution. They are respectively represented as a subset randomly sampled from the training set, representing the The overall fusion feature vector samples of the randomly sampled title component, the overall fusion feature vector samples of the ingredient component, and the overall fusion feature vector samples of the instruction component. is the normalization factor of the recipe sample, which is used to adjust the bias of random sampling, and N is the batch size; Construct title negative matching loss function respectively , negative matching loss function And instruction negative matching loss function , respectively using the overall fusion feature vector of the title component to the global feature vector of the food image The matching probability , the overall fusion feature vector of the batching components to the global feature vector of the food image The matching probability The overall fusion feature vector of the instruction component to the global feature vector of the food image The matching probability Construct a set of matching probabilities from the title component to the negative samples of the food image , the set of negative sample matching probabilities from ingredient component samples to food images , the set of negative sample matching probabilities from instruction component samples to food images , N is the batch size, Indicates Title components to The negative sample matching probability of food images, Indicates Ingredients to The negative sample matching probability of food images, Indicates The instruction component to The negative sample matching probability of food images; then construct the negative matching loss function from the title component to the food image , negative matching loss function from ingredient components to food images , negative matching loss function from ingredient components to food images The formula is as follows: ； ； ； Constructing negative matching loss function for food image to title component , negative matching loss function from food image to ingredient component , food image to instruction component negative matching loss function , respectively using the global feature vector of the food image to the overall fused feature vector of the title component The matching probability , the global feature vector of food images The overall fusion feature vector of the batching component The matching probability and the global feature vector of the food image To the overall fused feature vector of the instruction component The matching probability Construct a set of negative sample matching probabilities from food images to title components , the set of negative sample matching probabilities from food images to ingredient components , the set of negative sample matching probabilities from food images to instruction components , N is the batch size, Indicates Food images to The negative sample matching probability of title components, Indicates Food images to The negative sample matching probability of the ingredient components, Indicates Food images to The instruction component recipe image negative sample matching probability; then construct the title component to food image negative matching loss function , negative matching loss function from ingredient components to food images , negative matching loss function from ingredient components to food images The formula is as follows: ； ； ； Add the loss functions in both directions to get the negative matching function of the title component , ingredient component negative matching function , negative matching function of ingredient components The formula is as follows: ； ； ； Negatively match the title component to the function , negative matching function of ingredient components , negative matching function of ingredient components Add together to get the component negative matching loss function The formula is as follows: ； Step S33, embed each average food image segment after image segmentation into a vector Composition vector set and the overall fusion feature vector of each ingredient component Composition vector set , respectively calculate The similarity matrix and The similarity matrix , the specific calculation formula is as follows: ； ； Using triplet loss To minimize The difference between matrices is calculated as: ； in is the cosine similarity vector, is the triplet loss boundary parameter, To draw a point, is a positive sample, is a negative sample; for Transpose the matrix, for Transpose a matrix; Step S34, add the three loss functions to obtain the final dynamic negative match-triplet joint loss function : + + 。

Images