Hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement

Abstract

The invention discloses a hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement, and belongs to the technical field of food cross-modal retrieval in food calculation. According to the method, three component features of a title, ingredients and an instruction are reserved while global features of a recipe text are extracted, then dynamic semantic enhancement operation is performed on the three component features, and three core semantic units of entity words, attribute words and relational words are extracted. The extracted recipe text global features and food images are used for triple loss to achieve coarse-grained retrieval of the recipes, then three core semantic units including entity words, attribute words and relation words are used for triple loss with the food images to achieve fine-grained retrieval of the recipes, and finally a triple alignment strategy is adopted to achieve fine-grained retrieval of the recipes. And finally realizing hierarchical text multi-granularity recipe retrieval. Through the dynamic semantic enhancement, three core semantic units are extracted to construct a fine-grained matching method, so that the retrieval precision in a complex cooking scene is remarkably improved.

Description

Technical Field

[0001] The invention relates to a hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement, and belongs to the technical field of food cross-modal retrieval in food computing. Background Art

[0002] With the rapid development of information technology, cross-modal retrieval technology integrates multimodal data such as text, images, and voice to achieve a deep understanding of complex information. In the field of cooking, recipes are typical multimodal documents that integrate information such as ingredient descriptions, operation steps, and finished product displays. With the process of digitalization, people's demand for their retrieval has increased significantly. They expect to quickly and accurately find recipes that meet their needs by entering text descriptions and uploading pictures of dishes. Most of the current cross-modal recipe retrieval methods use a simple alignment method between recipe text and food images. This method has significant disadvantages. First, there is a representation difference between the two modalities of recipe text and food image. The extracted recipe text contains a large amount of redundant information, which deepens the representation difference between the modalities. The noise generated causes the features to be diluted, causing alignment interference of unnecessary information. Secondly, the recipe text has complex semantic features, including nested ingredient entities "French red wine stewed beef", time-related action instructions "fry first and then bake", and state description attribute words "frosting covered". Complex text information is easy to lose, making semantic alignment difficult, thereby affecting the accuracy of retrieval. It is precisely because of these reasons that the retrieval effect of food images and recipe text is not satisfactory. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and to provide a hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement.

[0004] The technical solution provided by the present invention is as follows: a hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement, characterized in that it comprises the following steps: Step S1, using the Recipe1M dataset, establish a dataset of two modalities: food images and recipe texts, where the recipe texts contain three components: title, ingredients, and instructions. The datasets of food images and recipe texts are divided into a training set, a validation set, and a test set; Step S2, respectively using the ViT visual model and the cascaded Transformer model to extract features from the food images and recipe texts in the training set, validation set, and test set, specifically including: Step S21, extracting global features of the food image; Step S22, extracting global features of the recipe text; Step S23, extracting three core semantic unit features of the recipe text; Step S3, constructing a loss function on the training set, specifically including: Step S31, constructing a global triplet hinge loss function; Step S32, constructing a triple hinge loss function of the three core semantic unit features of the recipe text respectively; Step S33, constructing the final dynamic triplet joint loss function; Step S4, performing cross-modal recipe retrieval, using the recall rate as an evaluation indicator.

[0005] Furthermore, the step S2 specifically includes the following steps: Furthermore, the step S2 specifically includes the following steps: Step S21, extracting global features of food images: Food image collection Input into the ViT visual model encoder, and then through the ViT visual model encoder mapping function Generate corresponding global embedding features of food images ,in, represents the fth food image, v represents the visual modality identifier, Indicates the number of food images; Step S22, extracting global features of the recipe text: A two-layer Transformer network is used to process the words and sentences of the recipe text. Given a set of recipe texts ,in, Indicates Title components, Indicates Ingredient components, Indicates command components, Represents a recipe text identifier, Indicates the amount of recipe text; the title component , ingredient components and the directive component The first-layer Transformer network receives each word in the sentences of the three components and encodes them, captures the relationship between each word, and averages the embedding representations of all words in each sentence through average pooling, thereby obtaining sentence-level embedding representations of the title component, ingredient component, and instruction component. = ,in, Represents the first-layer Transformer network; the obtained sentence-level embeddings of the title component, ingredient component, and instruction component are respectively sent to the second-layer Transformer network, and the sentence-level embeddings of the title component, ingredient component, and instruction component are averaged through average pooling to obtain the final overall fusion feature vector of the title component, ingredient component, and instruction component = ,in The second-layer Transformer network concatenates the overall fusion feature vectors of the title component, ingredient component, and instruction component to obtain the final recipe text global feature vector ,in, represents a fully connected layer; Step S23, extracting three core semantic unit features of the recipe text: First, the sentence-level embedding representation of the title component, ingredient component, and instruction component is Perform word segmentation to obtain the initial feature representation of each word; then use the Transformer encoder to model the global context of the word sequence and generate enhanced features with semantic associations , the modeling formula is , Indicates The initial features of the words are then used to predict the word category. This process goes through two layers of classifiers: First, the first layer uses the ReLU activation function to enhance the features. Perform nonlinear transformation ,in, represents the nonlinear transformation function, is the nonlinear transformation weight matrix, is the nonlinear transformation bias vector; then, the second layer outputs the ternary probability vector through linear mapping and Softmax normalization , the calculation formula of the ternary probability vector is: ,in, is the linear transformation weight matrix, is the linear transformation bias vector, the ternary probability vector The representation form is , Represents the probability of entity words, represents the probability of attribute words, Represents the probability of relation words; Finally, based on the three-element probability vector Weighted fusion of recipe text features to generate three core semantic unit features of recipe text , used for cross-modal alignment tasks, where Represents the semantic unit features of entity words, Represents the semantic unit characteristics of attribute words, Represents the semantic unit feature of the relation word; the formula for calculating the semantic unit feature of the entity word is: ; The formula for calculating the semantic unit features of attribute words is: ; The formula for calculating the semantic unit features of relational words is: .

[0006] Furthermore, the step S3 specifically includes the following steps: Step S31, constructing a global triplet hinge loss function to optimize the global embedding features of food images and the recipe text global feature vector similarity; Using the triplet loss formula Construct a double-constrained triplet loss function for a single food image and a single recipe text , the formula is as follows: ; In the triplet loss formula middle represents the anchor point sample, represents a positive sample, represents negative samples, Expressed as cosine similarity calculation function, is the boundary threshold, and the double-constrained triplet loss function for a single food image and a single recipe text middle Represent the positive and negative samples of a single food image, respectively. Represent the positive and negative samples of a single recipe text respectively; Utilizing a dual-constrained triplet loss function for a single food image and a single recipe text , construct a double-constrained triplet loss function based on batch food images and batch recipe text The formula is as follows: ; in is the batch size, ; Using a dual-constrained triplet loss function based on batch food images and batch recipe text And global embedding features of food images , the recipe global feature vector Construct a global triplet hinge loss function The formula is as follows: = ; in, Represents the global embedding features of food images The positive sample of Represents the global feature vector of recipe text The positive sample of Represents the global feature vector of recipe text Negative samples of Represents the global embedding features of food images Negative samples of Step S32, constructing a triple hinge loss function of the three core semantic unit features of the recipe text respectively; Using a dual-constrained triplet loss function based on batch images and batch text , global embedding features of food images , and entity word semantic unit features , semantic unit characteristics of attribute words and the semantic unit features of relational words Construct the hinge loss function of entity word triples respectively , hinge loss function for attribute word triples , hinge loss function for relation word triples , the formula is as follows: = ; = ; = ; in, Represents the global embedding features of food images The positive sample of Represents the semantic unit features of entity words The positive sample of Represents the semantic unit features of entity words Negative samples of Represents the semantic unit characteristics of attribute words The positive sample of Represents the semantic unit characteristics of attribute words Negative samples of Representing the semantic unit characteristics of relational words The positive sample of Representing the semantic unit characteristics of relational words Negative samples of Represents the global embedding features of food images Negative samples of Step S33, constructing a final dynamic triplet joint loss function to implement a triple alignment strategy of food images and recipe texts; The global triplet hinge loss function is obtained , entity word triple hinge loss function , hinge loss function for attribute word triples , hinge loss function for relation word triples The final dynamic triplet joint loss function L is obtained by adding and summing, and the formula is as follows: + + .

[0007] The beneficial effects of the present invention are as follows: the present invention firstly extracts the global features of food images and the global features of recipe texts respectively through a feature extraction algorithm, and constructs a global triple hinge loss function to optimize the similarity between the global embedding features of food images and the global vectors of recipe texts. Secondly, the three core semantic unit features of the recipe text are extracted, and the triple hinge loss functions of the three semantic unit features are respectively constructed. The three core semantics of entity words, attribute words, and relation words are used to further enhance the features of the recipe text, and a fine-grained matching basis is constructed, which effectively solves the problem that complex recipe text information is easily lost during the retrieval process. Finally, a dynamic triple joint loss function is constructed to implement a triple alignment strategy of food images and recipe texts, which alleviates the difficulty of semantic alignment and improves the accuracy of retrieval.

[0008] The present invention combines the coarse-grained image-text retrieval and the fine-grained recipe retrieval method with dynamic semantic enhancement. On the one hand, it compensates for the cross-modal representation differences by enhancing text features and eliminates the noise interference of redundant information in the recipe text. On the other hand, it uses the three extracted core semantic units to construct fine-grained matching basis, which effectively solves the problem that complex text information is easily lost during the retrieval process, significantly improves the accuracy and robustness of cross-modal retrieval, and has remarkable results. DETAILED DESCRIPTION

[0009] The specific implementation modes of the present invention are described in detail below: A hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement includes the following steps: Step S1, using the Recipe1M dataset to establish a dataset of two modalities: food images and recipe texts, wherein the recipe text includes three components: title, ingredients, and instructions, and the dataset of food images and recipe texts is divided into a training set, a validation set, and a test set.

[0010] Step S2, respectively using the ViT visual model and the cascaded Transformer model to extract features from the food images and recipe texts in the training set, validation set, and test set, specifically including: Step S21, extracting global features of food images: Food image collection Input into the ViT visual model encoder, and then through the ViT visual model encoder mapping function Generate corresponding global embedding features of food images ,in, represents the fth food image, v represents the visual modality identifier, Indicates the number of food images; Step S22, extracting global features of the recipe text: A two-layer Transformer network is used to process the words and sentences of the recipe text. Given a set of recipe texts ,in, Indicates Title components, Indicates Ingredient components, Indicates command components, Represents a recipe text identifier, Indicates the amount of recipe text; the title component , ingredient components and the directive component The first-layer Transformer network receives each word in the sentences of the three components and encodes them, captures the relationship between each word, and averages the embedding representations of all words in each sentence through average pooling, thereby obtaining sentence-level embedding representations of the title component, ingredient component, and instruction component. = ,in, Represents the first-layer Transformer network; in order to further capture the relationship between each sentence in the title component, ingredient component, and instruction component, the obtained sentence-level embeddings of the title component, ingredient component, and instruction component are respectively sent to the second-layer Transformer network, and the sentence-level embeddings of the title component, ingredient component, and instruction component are averaged by average pooling to obtain the final overall fusion feature vector of the title component, ingredient component, and instruction component = ,in The second-layer Transformer network concatenates the overall fusion feature vectors of the title component, ingredient component, and instruction component to obtain the final recipe global feature vector ,in, represents a fully connected layer; Step S23, extracting three core semantic unit features of the recipe text: First, the sentence-level embedding representation of the title component, ingredient component, and instruction component is Perform word segmentation to obtain the initial feature representation of each word. Then use the Transformer encoder to model the global context of the word sequence and generate enhanced features with semantic associations. , the modeling formula is , Indicates The initial features of the words are then used to predict the word category. This process goes through two layers of classifiers: First, the first layer uses the ReLU activation function to enhance the features. Perform nonlinear transformation ,in, represents the nonlinear transformation function, is the nonlinear transformation weight matrix, is the nonlinear transformation bias vector; then, the second layer outputs the ternary probability vector through linear mapping and Softmax normalization , the calculation formula of the ternary probability vector is: ,in, is the linear transformation weight matrix, is the linear transformation bias vector, the ternary probability vector The representation form is , Represents the probability of entity words, represents the probability of attribute words, Represents the probability of relation words.

[0011] Finally, based on the three-element probability vector Weighted fusion of recipe text features to generate three core semantic unit features of recipe text , used for cross-modal alignment tasks, where Represents the semantic unit features of entity words, Represents the semantic unit characteristics of attribute words, Represents the semantic unit feature of the relation word. The formula for calculating the semantic unit feature of the entity word is: ; The formula for calculating the semantic unit features of attribute words is: ; The formula for calculating the semantic unit features of relational words is: .

[0012] This method significantly improves the efficiency and robustness of relationship modeling through dynamic weight allocation, end-to-end joint training and feature dimensionality reduction design.

[0013] Step S3, constructing a loss function on the training set, specifically includes the following steps: Step S31, constructing a global triplet hinge loss function to optimize the global embedding features of food images and the recipe text global feature vector The similarity.

[0014] Using the triplet loss formula Construct a double-constrained triplet loss function for a single food image and a single recipe text , the formula is as follows: ; In the triplet loss formula middle represents the anchor point sample, represents a positive sample, represents negative samples, Expressed as cosine similarity calculation function, is the boundary threshold (set to 0.3 in this example), and the double-constrained triplet loss function for a single food image and a single recipe text middle Represent the positive and negative samples of a single food image, respectively. Represent the positive and negative samples of a single recipe text respectively.

[0015] Utilizing a dual-constrained triplet loss function for a single food image and a single recipe text , construct a double-constrained triplet loss function based on batch food images and batch recipe text The formula is as follows: ; in is the batch size, .

[0016] Using a dual-constrained triplet loss function based on batch food images and batch recipe text And global embedding features of food images , the recipe global feature vector Construct a global triplet hinge loss function The formula is as follows: = ; in, Represents the global embedding features of food images The positive sample of Represents the global feature vector of recipe text The positive sample of Represents the global feature vector of recipe text Negative samples of Represents the global embedding features of food images Negative samples.

[0017] Step S32, for the three core semantic unit features of the recipe text, construct triple hinge loss functions of the three semantic unit features respectively, to further enhance the semantic features and improve the recipe retrieval performance.

[0018] Using a dual-constrained triplet loss function based on batch images and batch text , global embedding features of food images , and entity word semantic unit features , semantic unit characteristics of attribute words and the semantic unit features of relational words Construct the hinge loss function of entity word triples respectively , hinge loss function for attribute word triples , hinge loss function for relation word triples , the formula is as follows: = ; = ; = ; in, Represents the global embedding features of food images The positive sample of Represents the semantic unit features of entity words The positive sample of Represents the semantic unit features of entity words Negative samples of Represents the semantic unit characteristics of attribute words The positive sample of Represents the semantic unit characteristics of attribute words Negative samples of Representing the semantic unit characteristics of relational words The positive sample of Representing the semantic unit characteristics of relational words Negative samples of Represents the global embedding features of food images Negative samples.

[0019] Step S33, constructing the final dynamic triplet joint loss function to implement the triple alignment strategy of food images and recipe texts.

[0020] The global triplet hinge loss function is obtained , entity word triple hinge loss function , hinge loss function for attribute word triples , hinge loss function for relation word triples The final dynamic triplet joint loss function L is obtained by adding and summing, and the formula is as follows: + + .

[0021] Step S4, performing cross-modal recipe retrieval, using the recall rate as an evaluation indicator.

[0022] This example is mainly trained and validated on the Recipe1M dataset, which contains pairs of food images and recipe texts. The divided training set, validation set, and test set contain 238,999, 51,119, and 51,303 pairs of samples, respectively. In order to evaluate the retrieval performance of the model, median ranking (medR) and recall (Recall@1, Recall@5, Recall@10, referred to as R1, R5, and R10, respectively) are used as indicators. These indicators are calculated on a ranking list of size N = {1,000, 10,000}, and the average results are reported on 10 groups of randomly selected samples. This case is compared with the classic model HT, and the experimental results are shown in Tables 1 and 2.

[0023] Table 1 N = 1000

[0024] Table 2 N = 10000

[0025] It should be understood that the parts not elaborated in detail in this specification belong to the prior art. The above embodiments are only descriptions of the preferred implementation methods of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made by ordinary engineers and technicians in this field to the technical solution of the present invention should fall within the protection scope determined by the claims of the present invention.

Claims

1. A hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement, characterized in that: It includes the following steps: Step S1, using the Recipe1M dataset, establish a dataset of two modalities: food images and recipe texts, where the recipe texts contain three components: title, ingredients, and instructions. The datasets of food images and recipe texts are divided into a training set, a validation set, and a test set; Step S2, respectively using the ViT visual model and the cascaded Transformer model to extract features from the food images and recipe texts in the training set, validation set, and test set, specifically including: Step S21, extracting global features of the food image; Step S22, extracting global features of the recipe text; Step S23, extracting three core semantic unit features of the recipe text; Step S3, constructing a loss function on the training set, specifically including: Step S31, constructing a global triplet hinge loss function; Step S32, constructing a triple hinge loss function of the three core semantic unit features of the recipe text respectively; Step S33, constructing the final dynamic triplet joint loss function; Step S4, performing cross-modal recipe retrieval, using the recall rate as an evaluation indicator.

2. The hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement as claimed in claim 1, characterized in that: The step S2 specifically includes the following steps: Step S21, extracting global features of food images: Food image collection Input into the ViT visual model encoder, and then through the ViT visual model encoder mapping function Generate corresponding global embedding features of food images ,in, represents the fth food image, v represents the visual modality identifier, Indicates the number of food images; Step S22, extracting global features of the recipe text: A two-layer Transformer network is used to process the words and sentences of the recipe text. Given a set of recipe texts ,in, Indicates Title components, Indicates Ingredient components, Indicates command components, Represents a recipe text identifier, Indicates the amount of recipe text; the title component , ingredient components and the directive component The first-layer Transformer network receives each word in the sentences of the three components and encodes them, captures the relationship between each word, and averages the embedding representations of all words in each sentence through average pooling, thereby obtaining sentence-level embedding representations of the title component, ingredient component, and instruction component. = ,in, Represents the first-layer Transformer network; the obtained sentence-level embeddings of the title component, ingredient component, and instruction component are respectively sent to the second-layer Transformer network, and the sentence-level embeddings of the title component, ingredient component, and instruction component are averaged through average pooling to obtain the final overall fusion feature vector of the title component, ingredient component, and instruction component = ,in The second-layer Transformer network concatenates the overall fusion feature vectors of the title component, ingredient component, and instruction component to obtain the final recipe text global feature vector ,in, represents a fully connected layer; Step S23, extracting three core semantic unit features of the recipe text: First, the sentence-level embedding representation of the title component, ingredient component, and instruction component is Perform word segmentation to obtain the initial feature representation of each word; then use the Transformer encoder to model the global context of the word sequence and generate enhanced features with semantic associations , the modeling formula is , Indicates The initial features of the words are then used to predict the word category. This process goes through two layers of classifiers: First, the first layer uses the ReLU activation function to enhance the features. Perform nonlinear transformation ,in, represents the nonlinear transformation function, is the nonlinear transformation weight matrix, is the nonlinear transformation bias vector; then, the second layer outputs the ternary probability vector through linear mapping and Softmax normalization , the calculation formula of the ternary probability vector is: ,in, is the linear transformation weight matrix, is the linear transformation bias vector, the ternary probability vector The representation form is , Represents the probability of entity words, represents the probability of attribute words, Represents the probability of relation words; Finally, based on the three-element probability vector Weighted fusion of recipe text features to generate three core semantic unit features of recipe text , used for cross-modal alignment tasks, where Represents the semantic unit features of entity words, Represents the semantic unit characteristics of attribute words, Represents the semantic unit feature of the relation word; the formula for calculating the semantic unit feature of the entity word is: ; The formula for calculating the semantic unit features of attribute words is: ; The formula for calculating the semantic unit features of relational words is: .

3. The hierarchical text multi-granularity recipe retrieval method based on dynamic semantic enhancement as claimed in claim 1, characterized in that: The step S3 specifically comprises the following steps: Step S31, constructing a global triplet hinge loss function to optimize the global embedding features of food images and the recipe text global feature vector similarity; Using the triplet loss formula Construct a double-constrained triplet loss function for a single food image and a single recipe text , the formula is as follows: ； In the triplet loss formula middle represents the anchor point sample, represents a positive sample, represents negative samples, Expressed as cosine similarity calculation function, is the boundary threshold, and the double-constrained triplet loss function for a single food image and a single recipe text middle Represent the positive and negative samples of a single food image, respectively. Represent the positive and negative samples of a single recipe text respectively; Utilizing a dual-constrained triplet loss function for a single food image and a single recipe text , construct a double-constrained triplet loss function based on batch food images and batch recipe text The formula is as follows: ； in is the batch size, ; Using a dual-constrained triplet loss function based on batch food images and batch recipe text And global embedding features of food images , the recipe global feature vector Construct a global triplet hinge loss function The formula is as follows: = ； in, Represents the global embedding features of food images The positive sample of Represents the global feature vector of recipe text The positive sample of Represents the global feature vector of recipe text Negative samples of Represents the global embedding features of food images Negative samples of Step S32, constructing a triple hinge loss function of the three core semantic unit features of the recipe text respectively; Using a dual-constrained triplet loss function based on batch images and batch text , global embedding features of food images , and entity word semantic unit features , semantic unit characteristics of attribute words and the semantic unit features of relational words Construct the hinge loss function of entity word triples respectively , hinge loss function for attribute word triples , hinge loss function for relation word triples , the formula is as follows: = ； = ； = ； in, Represents the global embedding features of food images The positive sample of Represents the semantic unit features of entity words The positive sample of Represents the semantic unit features of entity words Negative samples of Represents the semantic unit characteristics of attribute words The positive sample of Represents the semantic unit characteristics of attribute words Negative samples of Representing the semantic unit characteristics of relational words The positive sample of Representing the semantic unit characteristics of relational words Negative samples of Represents the global embedding features of food images Negative samples of Step S33, constructing a final dynamic triplet joint loss function to implement a triple alignment strategy of food images and recipe texts; The global triplet hinge loss function is obtained , entity word triple hinge loss function , hinge loss function for attribute word triples , hinge loss function for relation word triples The final dynamic triplet joint loss function L is obtained by adding and summing, and the formula is as follows: + + 。

Images