Recipe retrieval method based on modal interaction

Abstract

The invention discloses a recipe retrieval method based on modal interaction, and the method is specifically implemented according to the following steps: 1, obtaining a cross-modal recipe retrieval data set, and dividing the cross-modal recipe retrieval data set into a training set and a test set; step 2, constructing a cross-modal recipe retrieval network model based on interaction between fine-grained modals; 3, using a training set to train the cross-modal recipe retrieval network model based on the interaction between the fine-grained modals constructed in the step 2; 4, testing the network model trained in the step 3 by using a test set to obtain a recipe text retrieval image result, and evaluating the retrieval result; and 5, testing the network model trained in the step 3 on the test set to obtain a recipe image retrieval text result, and evaluating the retrieval result. The problem that in the prior art, due to the fact that semantic gaps between different modes exist in a cross-modal recipe retrieval task, accuracy of cross-modal recipe retrieval is insufficient is solved.

Description

Technical Field

[0001] This invention belongs to the technical field of recipe retrieval methods, and relates to a recipe retrieval method based on modal interaction. Background Art

[0002] As food becomes increasingly important to people's health and lives, the internet and social networks offer a wealth of food images and recipes. However, these resources often lack meaningful connections, making the development of effective recipe retrieval systems crucial.

[0003] Against this backdrop, cross-modal recipe retrieval technology has rapidly emerged, enabling the retrieval of images corresponding to given recipes. The release of the Recipe1M dataset provides a rich, large-scale cross-modal recipe dataset, offering strong support for algorithm training and evaluation. Most existing cross-modal recipe retrieval work relies on recurrent networks, such as LSTMs or Transformers, to encode recipe data; basic neural network architectures, such as ResNet-50 or ViT, are widely used for processing and encoding image data. However, relying solely on global similarity while ignoring fine-grained modal interactions limits performance to some extent. Furthermore, due to the semantic gaps between different modalities in cross-modal recipe retrieval tasks, accuracy remains insufficient; therefore, researching accurate cross-modal recipe retrieval is of great significance. Summary of the Invention

[0004] The purpose of this invention is to provide a recipe retrieval method based on modal interaction, which solves the problem of insufficient accuracy in cross-modal recipe retrieval due to the semantic gap between different modalities in the existing technology.

[0005] The technical solution adopted in this invention is a recipe retrieval method based on modal interaction, which is implemented according to the following steps:

[0006] Step 1: Obtain the cross-modal recipe retrieval dataset and divide it into training and test sets;

[0007] Step 2: Construct a cross-modal recipe retrieval network model based on fine-grained intermodal interactions;

[0008] Step 3: Train the cross-modal recipe retrieval network model based on fine-grained intermodal interactions constructed in Step 2 using the training set;

[0009] Step 4: Test the network model trained in Step 3 using the test set to obtain the image retrieval results for recipe text and evaluate the retrieval results.

[0010] Step 5: Test the network model trained in Step 3 on the test set to obtain the recipe image retrieval text results, and evaluate the retrieval results.

[0011] The invention is further characterized in that,

[0012] Step 1 is as follows:

[0013] Download the recipe dataset D = {I, T} from the official website, where I = {I1, ..., I...} m ,…,I M}, T={T1,…,T n ,…,T N}, I m Let T represent the m-th image in D. n Let I represent the nth recipe in D, where 1 ≤ m ≤ M, 1 ≤ n ≤ N, M represents the total number of images, and N represents the total number of recipes. m ∈R W×H×D W, H, and D correspond to I respectively. m The width, height, and number of channels of the image. T represents respectively n The title, components, and instructions are represented in the text. Where N1 and L1 represent the number of sentences and sentence length of the recipe title, respectively. The recipe title has 1 sentence, so N1 = 1. N2 and L2 represent the number of sentences and sentence length of the recipe components, respectively. N3 and L3 represent the number of sentences and sentence length of the recipe instructions, respectively. D is divided into a training set TrD and a test set TeD according to the ratio of A:B, where TrD = {i TrD ,t TrD}, X + Y = A;

[0014] X′+Y′=B, This represents the x-th image in TrD. This represents the y-th recipe in TrD. This represents the x'th image in TeD. This represents the y'th recipe in TeD.

[0015] The cross-modal recipe retrieval network model based on fine-grained intermodal interaction in step 2 includes a recipe hierarchical encoding module, a text context-based visual enhancement image encoding module, and a pairing module;

[0016] The working process of the cross-modal recipe retrieval network model based on fine-grained intermodal interactions is as follows:

[0017] Step 2.1, the recipe grading coding module extracts t from TRD. TrD Extract the y-th recipe Among them, recipes Will The data is fed into the hierarchical coding module and outputs TTL. y ,ing y ,ins y The encoding features are as follows: Will After passing through a vector concatenation layer and a fully connected dimensionality reduction network layer, the final deep feature representation of the recipe is obtained.

[0018] Step 2.2, the image encoding module based on text context visual enhancement extracts t from the TrD TrD Extract the x-th image The data is fed into the ResNet-50 backbone network to extract shallow features from the k-th layer of the ResNet-50 network. Will and the title and component coding features obtained in step 2.1 Feature enhancement is performed to obtain Then The data is then fed into the (k+1)th layer of the ResNet-50 backbone network, the layer preceding the classification layer, to extract deep features. The depth features of the image are then obtained through a fully connected layer.

[0019] Step 2.3: The pairing module calculates the image depth features. and recipe depth features The pairing loss.

[0020] The hierarchical coding module in step 2.1 consists of an embedding submodule, a first-level attention mechanism submodule, and a second-level attention mechanism submodule;

[0021] The working process of the hierarchical coding module is as follows:

[0022] First, E is obtained by sending it to the output of the embedded submodule. rec , E rec ={E ttl E ing E ins},in Secondly, put E rec Feed into the first-level attention mechanism submodule and output in Next, put the first and The data is fed into the secondary attention mechanism submodule, and the output is... and in Finally The data is concatenated along the D1 dimension, and the concatenated features are input into a fully connected layer, outputting the dimensionality-reduced data. in

[0023] The process of embedding a submodule is as follows: As input, setting the output vector dimension D1, the open-source library PyTorch's text word data mapping function nn.embedding is called to output E. rec , E rec ={E ttl E ing E ins},in

[0024] The first-level attention mechanism submodule consists of an attention layer and a global average pooling layer;

[0025] The working process of the first-level attention mechanism submodule is as follows:

[0026] The output E of the embedded submodule rec ={E ttl E ing E ins The attention layer defines the query vector as input to the attention layer. value vector and key vector Assign values ​​to the query vector, value vector, and key vector: beg and Relevance and Relevance and Relevance The solution formula is as follows:

[0027]

[0028] Where softmax() is the probability-normalized activation function;

[0029] Then calculate the output of the attention mechanism. in

[0030]

[0031] Finally, The data are fed into a global average pooling layer, and the average value is used as the overall representation of the recipe title feature, recipe ingredient feature, and recipe instruction feature. and in

[0032] The second-level attention mechanism submodule consists of an attention layer and a global average pooling layer;

[0033] The working process of the secondary attention mechanism submodule is as follows:

[0034] The output data of the first-level attention mechanism submodule As input to the secondary attention mechanism submodule, the query vector is fed into the attention layer of the secondary attention mechanism submodule. value vector and key vector Assign values ​​to the query vector, value vector, and key vector: beg and Relevance and Relevance

[0035]

[0036] Where softmax() is the probability-normalized activation function;

[0037] Then calculate the output of the attention mechanism.

[0038]

[0039]

[0040] Finally, The ingredients are fed into a global average pooling layer, and their average value is used as the overall representation of the recipe component features and recipe instruction features. and in

[0041] The hierarchical encoding module outputs the first-level attention mechanism submodule. and the output of the secondary attention mechanism submodule As TTL respectively y ,ingy ,ins y The encoded feature output.

[0042] The image coding module based on text context visual enhancement in step 2.2 consists of a shallow image feature extraction module, a text-image patch similarity measurement submodule, a residual connection submodule, and a deep image feature extraction submodule.

[0043] The working process of the image shallow feature extraction module is as follows:

[0044] t from TRD TrD Extract the x-th image in Will As input to the shallow image feature extraction module, Shallow features are extracted from the ResNet-50 network. Specifically:

[0045]

[0046] Among them, ResNet-50 sl () indicates that the parentheses are used to represent the parentheses. The data is fed into a ResNet-50 network, passes through sub-networks from layer 1 to layer k, and outputs the features of layer k.

[0047] The text-image patch similarity measurement submodule works as follows:

[0048] Output data from the first-level attention mechanism submodule in step 2.1 Output data of the image shallow feature extraction module As input to the text-image patch similarity measurement submodule, and By splicing them together, we get Where cat() is a concatenation function:

[0049]

[0050] Where cat() is a concatenation function;

[0051] And, calling the array shape reshaping function in the open-source library NumPy. Will Change to W k ×H k D k Dimensional features

[0052] Then calculate and The dot product between them is then called with the probability-normalized activation function softmax() to obtain the text. and image blocks Similarity score between in,

[0053]

[0054] The text-image patch similarity measurement submodule will As output;

[0055] The working process of the residual connection submodule is as follows:

[0056] The output of the text-image patch similarity measurement submodule is used as input, and the text and image residuals are concatenated based on the obtained similarity scores to obtain...

[0057]

[0058] The working process of the deep feature extraction submodule is as follows:

[0059] The output of the residual connection submodule As input to the deep image feature extraction submodule, it is then used by the ResNet-50 network to extract... deep features

[0060]

[0061] Among them, ResNet-50 dl () indicates that the parentheses are used to represent the parentheses. The data is fed into the ResNet-50 network, passing through layer k+1 to layer K, and the features of layer K are output.

[0062] Finally, The final image features are fed into the fully connected layer.

[0063]

[0064] Where Fc() represents the dimension of D K Features It is fed into a fully connected layer, and the output dimension is D2.

[0065] Step 3 specifically involves:

[0066] Step 3.1: Set the network model training parameters; that is, set the learning rate variable lr, the maximum number of training iterations variable Max_iter, the batch size of paired data variable batch_size_paired, and the number of training iterations variable step, and initialize step = 1;

[0067] Step 3.2: Input the training set TRD into the cross-modal recipe retrieval network model based on fine-grained intermodal interaction constructed in Step 2, and train the network according to the set training parameters. When the change of the target loss function is less than MinLoss or the number of training iterations step ≥ Max_iter, the network training ends and the network model Model is saved; otherwise, step = step + 1, and the Adam optimizer is used to back-correct the weight coefficients of each network layer in the network model, and training continues.

[0068] The target loss function in step 3.2 is the paired loss function L. pair :

[0069]

[0070] L cos (x a ,x p ,x n )=max(0,c(x a ,x n )-c(x a ,x p )+m) (20)

[0071]

[0072] Among them, the pairing loss L pair A mean bidirectional triple loss L bi Composition, in which L pair The input data to the loss function are the output data from the recipe grading coding module. Output data of the image encoding module based on text context visual enhancement Average bidirectional triple loss L bi This refers to calculating the bidirectional triple loss L' between each recipe image and text pair in each batch. bi The average value of the bidirectional triple loss function L' for each recipe image and text. bi The input data consists of the i-th and j-th recipe image-text pairs in a batch. The bidirectional triple loss is composed of two triple losses L. cos Composed of, where when i = j, δ(i,j) = 0, and when i ≠ j, δ(i,j) = 1; triple loss function L cos (xa ,x p ,x n The input data for ) is a triple, where x a Let x represent the training sample. p Let x represent a positive sample. n Let m represent a negative sample, c() represent the cosine distance, and the function aims to reduce the number of training samples x. a and positive sample x p The distance between them increases the size of the training samples x. a and negative sample x n The distance between them; the bidirectional triple loss function L' bi It consists of two parts. and in Input data This represents the output data of the i-th recipe image text in the training samples after passing through a text context-based visual enhancement image encoding module. The output data represents the i-th recipe image text in the training samples after passing through the recipe hierarchical encoding module, and represents the positive sample. The output data of the j-th recipe image text in the training samples after passing through the recipe hierarchical encoding module represents the negative sample. The function aims to reduce the cosine distance between recipe image data and text data in the same recipe image text pair, and to increase the cosine distance between recipe image data and text data in different recipe image text pairs. Input data This represents the output data of the i-th recipe image text in the training samples after passing through the recipe hierarchical encoding module. The output data of the i-th recipe image text in the training samples after passing through the image encoding module based on text context visual enhancement represents a positive sample. The output data of the j-th recipe image text in the training samples after passing through the image encoding module based on text context visual enhancement represents the negative sample. The function aims to reduce the cosine distance between recipe text data and image data in the same recipe image text pair, and to increase the cosine distance between recipe text data and image data in different recipe image text pairs.

[0073] Step 4 is as follows:

[0074] Step 4.1: Define text feature index y', initialize it to 1; define image feature index x', initialize it to 1; define the text feature V output by the recipe hierarchical coding module. rec Initialized to empty, V rec =NULL, defines the output image features V of the text-contextualized visual enhancement module.img Initialized to empty, V img =NULL, defines the cosine distance result D of the text retrieval image, initializes D to null, i.e., D = NULL, defines the text retrieval image result R2i TeD Initialize R2i TeD It is empty, i.e., R2i TeD =NULL;

[0075] Step 4.2, from the test set TeD={i TeD ,t TeD} of i TeD Selecting the x'th image feature The image is fed into a pre-trained image encoding module with text context-based visual enhancement, and the output is... And Add to V img In step 4, determine whether x' is less than or equal to X'. If x' ≤ X', then x' = x' + 1, and continue repeating step 4.2.

[0076] Step 4.3, from the test set TeD={i TeD ,t TeD} of t TeD Select the y'th text feature The input is fed into the trained embedding submodule, the first-level attention mechanism submodule, and the second-level attention submodule, and the output is...

[0077] Step 4.4: Use the torch.mm() function from the open-source library PyTorch to calculate. and cosine similarity cosine distance

[0078]

[0079] Step 4.5, use Python's built-in sorting function sprt() to sort... Sort the data; the smaller the cosine distance, the higher the similarity. Return the sorted result D. y' and sort the results The image corresponding to index number x' is appended to

[0080]

[0081] Step 4.6: Determine if y' is greater than Y'. If y' ≤ Y', then y' = y' + 1, and repeat step 4.3.

[0082] Step 4.7, retrieve image results according to the finally output recipe text Evaluate the network model, calculate the median rank MedR of the correct results in the retrieval results output by the network model, and calculate the recall rate Recall@k of the network model R2i :

[0083]

[0084] where N represents the number of selected samples, and n represents the number of correct results among the selected samples that are among the top k

[0085] Step 5 is specifically as follows:

[0086] Step 5.1, define the text feature index y', initialize it to 1, define the image feature index x', initialize it to 1, and define the text feature V output by the recipe encoding module rec , initialize it to be empty, V rec = NULL, define the image feature V output by the vision enhancement module based on text context img , initialize it to be empty, V img = NULL, define the cosine distance result D of image retrieval text, initialize D to be empty, that is, D = NULL, and define the image retrieval text result I2r TeD , initialize I2r TeD to be empty, that is, I2r TeD = NULL;[[ID=3ģ]]

[0087] Step 5.2, select the y'-th text feature from t in the test set TeD = {i TeD ,t TeD} TeD and send it to the trained embedding sub-module, the first-level attention mechanism sub-module and the second-level attention sub-module, and the output result is append it to V ​​​​​​​​​​​​​​​​​​​​ cosine similarity cosine distance

[0090]

[0091] Step 5.5, use Python's built-in sorting function sort() to sort... Sort the data; the smaller the cosine distance, the higher the similarity. Return the sorted result D. x' and sort the results The text corresponding to the index number y' is appended to

[0092]

[0093] Step 5.6: Determine if x' is greater than X'. If x' ≤ X', then x' = x' + 1, and repeat step 5.3.

[0094] Step 5.7: Result of the text retrieval task based on the final output recipe image. To evaluate the network model, calculate the median rank (MedR) of the correct results output by the model and calculate the recall (Recall@k) of the model. I2r :

[0095]

[0096] Where N represents the number of selected samples, and n represents the number of correct results among the first k selected samples.

[0097] The beneficial effects of the present invention are:

[0098] This invention is based on a text-contextualized visual enhancement mechanism that calculates the correlation between image features and text features to enhance image features. This invention uses shallow image features, which pay more attention to the local features of the image and preserves the position and detail information of the original image, so that the method of this invention achieves high accuracy in the field of cross-modal recipe retrieval. Attached Figure Description

[0099] Figure 1 This is a flowchart of the recipe retrieval method based on modal interaction of the present invention;

[0100] Figure 2 This is a network structure diagram of the recipe retrieval method based on modal interaction of the present invention;

[0101] Figure 3This is a schematic diagram of the text-contextualized visual enhancement module structure in the recipe retrieval method based on modal interaction of the present invention;

[0102] Figure 4 This is a subjective effect diagram of applying the modal interaction-based recipe retrieval method of the present invention to the Recipe1M dataset in Embodiment 3 of the present invention for image recipe retrieval;

[0103] Figure 5 This is a subjective effect diagram of applying the modal interaction-based recipe retrieval method of the present invention to the recipe retrieval image on the Recipe1M dataset in Embodiment 3 of the present invention;

[0104] Figure 6 This is a visualization diagram of the text-based contextualized visual enhancement module in the modal interaction-based recipe retrieval method of the present invention, as shown in Embodiment 3 of the present invention. Detailed Implementation

[0105] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0106] Example 1

[0107] This invention relates to a recipe retrieval method based on modal interaction, the overall process of which is as follows: Figure 1 As shown, the specific steps are as follows:

[0108] Step 1: Obtain the cross-modal recipe retrieval dataset and divide it into a training set and a test set, specifically:

[0109] Download the recipe dataset D = {I, T} from the official website, where I = {I1, ..., I...} m ,…,I M}, T={T1,…,T n ,…,T N}, I m Let T represent the m-th image in D. n Let I represent the nth recipe in D, where 1 ≤ m ≤ M, 1 ≤ n ≤ N, M represents the total number of images, and N represents the total number of recipes. m ∈R W×H×D W, H, and D correspond to I respectively. m The width, height, and number of channels of the image are: W = H = 224, D = 3. T represents respectively n The title, components, and instructions are represented in the text. Where N1 and L1 represent the number of sentences and sentence length of the recipe title, respectively. The recipe title has 1 sentence, so N1 = 1. N2 and L2 represent the number of sentences and sentence length of the recipe components, respectively. N3 and L3 represent the number of sentences and sentence length of the recipe instructions, respectively. D is divided into a training set TrD and a test set TeD according to the ratio of A:B, where A = 238408, B = 51304, and TrD = {i TrD ,t TrD}, X+Y=A;TeD={i TeD ,t TeD}, X'+Y'=B, This represents the x-th image in TrD. This represents the y-th recipe in TrD. This represents the x'th image in TeD. This represents the y'-th recipe in TeD;

[0110] Step 2: Construct a cross-modal recipe retrieval network model based on fine-grained intermodal interactions, the structure of which is as follows: Figure 2 As shown, it includes a recipe hierarchical encoding module, a text-context-based visual enhancement image encoding module, and a pairing module; the working process of the cross-modal recipe retrieval network model based on fine-grained intermodal interaction is as follows:

[0111] Step 2.1, the recipe grading coding module extracts t from TRD. TrD Extract the y-th recipe Among them, recipes Will The data is fed into the hierarchical coding module and outputs TTL. y ,ing y ,ins y The coding features are as follows: Will After passing through a vector concatenation layer and a fully connected dimensionality reduction network layer, the final deep feature representation of the recipe is obtained.

[0112] The hierarchical coding module consists of an embedding submodule, a first-level attention mechanism submodule, and a second-level attention mechanism submodule.

[0113] The working process of the hierarchical coding module is as follows:

[0114] First, E is obtained by sending it to the output of the embedded submodule. rec E rec ={E ttl,E ing ,E ins},in Secondly, put E rec Feed into the first-level attention mechanism submodule and output in Next, put the first and The data is fed into the secondary attention mechanism submodule, and the output is... and in Finally The data is concatenated along the D1 dimension, and the concatenated features are input into a fully connected layer, outputting the dimensionality-reduced data. in

[0115] The process of embedding a submodule is as follows: As input, setting the output vector dimension D1, the open-source library PyTorch's text word data mapping function nn.embedding is called to output E. rec E rec ={E ttl ,E ing ,E ins},in

[0116] The first-level attention mechanism submodule consists of an attention layer and a global average pooling layer;

[0117] The working process of the first-level attention mechanism submodule is as follows:

[0118] The output E of the embedded submodule rec ={E ttl ,E ing ,E ins The attention layer defines the query vector as input to the attention layer. value vector and key vector Assign values ​​to the query vector, value vector, and key vector: beg and Relevance and Relevance and Relevance The solution formula is as follows:

[0119]

[0120] Where softmax() is the probability-normalized activation function;

[0121] Then calculate the output of the attention mechanism. in

[0122]

[0123] Finally, The data are fed into a global average pooling layer, and the average value is used as the overall representation of the recipe title feature, recipe ingredient feature, and recipe instruction feature. and in

[0124] The second-level attention mechanism submodule consists of an attention layer and a global average pooling layer;

[0125] The working process of the secondary attention mechanism submodule is as follows:

[0126] The output data of the first-level attention mechanism submodule As input to the secondary attention mechanism submodule, the query vector is fed into the attention layer of the secondary attention mechanism submodule. value vector and key vector Assign values ​​to the query vector, value vector, and key vector: beg and Relevance and Relevance

[0127]

[0128] Where softmax() is the probability-normalized activation function;

[0129] Then calculate the output of the attention mechanism.

[0130]

[0131] Finally, The ingredients are fed into a global average pooling layer, and their average value is used as the overall representation of the recipe component features and recipe instruction features. and in

[0132] The hierarchical encoding module outputs the first-level attention mechanism submodule. and the output of the secondary attention mechanism submodule As TTL respectively y ,ing y ,ins y The encoded features are output.

[0133] Step 2.2, the image encoding module based on text context visual enhancement extracts t from the TrD TrD Extract the x-th image The data is fed into the ResNet-50 backbone network to extract shallow features from the k-th layer of the ResNet-50 network. k = 23, W k ×H k =28,D k =512, will and the title and component coding features obtained in step 2.1 Feature enhancement is performed to obtain Then The data is then fed into the (k+1)th layer of the ResNet-50 backbone network, the layer preceding the classification layer, to extract deep features. The depth features of the image are then obtained through a fully connected layer.

[0134] like Figure 3 As shown, the image coding module based on text context visual enhancement consists of a shallow image feature extraction module, a text-image patch similarity measurement submodule, a residual connection submodule, and a deep image feature extraction submodule.

[0135] The working process of the image shallow feature extraction module is as follows:

[0136] t from TRD TrD Extract the x-th image in Will As input to the shallow image feature extraction module, Shallow features are extracted from the ResNet-50 network. Specifically:

[0137]

[0138] Among them, ResNet-50 sl () indicates that the parentheses are used to represent the parentheses. The data is fed into a ResNet-50 network, passes through the first to the kth sub-network, and outputs the features of the kth layer, where k = 23.

[0139] The text-image patch similarity measurement submodule works as follows:

[0140] Output data from the first-level attention mechanism submodule in step 2.1 Output data of the image shallow feature extraction module As input to the text-image patch similarity measurement submodule, and By splicing them together, we can obtain Where cat() is a concatenation function:

[0141]

[0142] Where cat() is a concatenation function;

[0143] And, calling the array shape reshaping function in the open-source library NumPy. Will Change to W k ×H k D k Dimensional features

[0144] Then calculate and The dot product between them is then called with the probability-normalized activation function softmax() to obtain the text. and image blocks Similarity score between in,

[0145]

[0146] The text-image patch similarity measurement submodule will As output;

[0147] The working process of the residual connection submodule is as follows:

[0148] The output of the text-image patch similarity measurement submodule is used as input, and the text and image residuals are concatenated based on the obtained similarity scores to obtain...

[0149]

[0150] The working process of the deep feature extraction submodule is as follows:

[0151] The output of the residual connection submodule As input to the deep image feature extraction submodule, it is then used by the ResNet-50 network to extract... deep features

[0152]

[0153] Among them, ResNet-50 dl () indicates that the parentheses are used to represent the parentheses. The data is fed into a ResNet-50 network, passing through the (k+1)th layer to the Kth layer, and the features of the Kth layer are output, where K = 50.

[0154] Finally, The final image features are fed into the fully connected layer.

[0155]

[0156] Where Fc() represents the dimension of D K D K =2048 characteristics It is fed into a fully connected layer, and the output dimension is D2, D2 = 1024.

[0157] Step 2.3: The pairing module calculates the image depth features. and recipe depth features The pairing loss.

[0158] Step 3: Train the cross-modal recipe retrieval network model based on fine-grained intermodal interactions constructed in Step 2 using the training set; specifically:

[0159] Step 3.1: Set the network model training parameters as shown in Table 1, namely, set the learning rate variable lr, the maximum number of training iterations variable Max_iter, the batch size of paired data per batch variable batch_size_paired, and the number of training iterations variable step, and initialize step = 1;

[0160] Step 3.2: Input the training set TRD into the cross-modal recipe retrieval network model based on fine-grained intermodal interaction constructed in Step 2, and train the network according to the set training parameters. When the change of the target loss function is less than MinLoss (the minimum range of change of the loss function, that is, when the change of the loss function is less than this value, training stops) or the number of training iterations step ≥ Max_iter, the network training ends and the network model Model is saved; otherwise, step = step + 1, and the Adam optimizer is used to back-correct the weight coefficients of each network layer in the network model, and training continues.

[0161] Table 1 Network model training parameters

[0162] parameter Parameter Description Value lr Learning rate 0.0001 MinLoss Minimum change of the target loss function 0.005 Max_iter Maximum number of training iterations 100 batch_size Batch data size 64 step Number of training iterations 1

[0163] The target loss function in step 3.2 is the paired loss function L. pair :

[0164]

[0165] L cos (x a ,x p ,x n )=max(0,c(x a ,x n )-c(x a ,x p )+m) (20)

[0166]

[0167] Among them, the pairing loss L pair A mean bidirectional triple loss L bi Composition, in which L pair The input data to the loss function are the output data from the recipe grading coding module. Output data of the image encoding module based on text context visual enhancement Average bidirectional triple loss L bi This refers to calculating the bidirectional triple loss L' between each recipe image and text pair in each batch. bi The average value of the bidirectional triple loss function L' for each recipe image and text. bi The input data consists of the i-th and j-th recipe image-text pairs in a batch. The bidirectional triple loss is composed of two triple losses L. cos Composed of, where when i = j, δ(i,j) = 0, and when i ≠ j, δ(i,j) = 1; triple loss function L cos (x a ,x p ,x n The input data for ) is a triple, where x a Let x represent the training sample. p Let x represent a positive sample. n Let m represent a negative sample, c() represent the cosine distance, and the function aims to reduce the number of training samples x. a and positive sample x p The distance between them increases the size of the training samples x. a and negative sample x n The distance between them; the bidirectional triple loss function L' bi It consists of two parts. and in Input data This represents the output data of the i-th recipe image text in the training samples after passing through a text context-based visual enhancement image encoding module. The output data represents the i-th recipe image text in the training samples after passing through the recipe hierarchical encoding module, and represents the positive sample. The output data of the j-th recipe image text in the training samples after passing through the recipe hierarchical encoding module represents the negative sample. The function aims to reduce the cosine distance between recipe image data and text data in the same recipe image text pair, and to increase the cosine distance between recipe image data and text data in different recipe image text pairs. Input data This represents the output data of the i-th recipe image text in the training samples after passing through the recipe hierarchical encoding module. The output data of the i-th recipe image text in the training samples after passing through the image encoding module based on text context visual enhancement represents a positive sample. The output data of the j-th recipe image text in the training samples after passing through the image encoding module based on text context visual enhancement represents the negative sample. The function aims to reduce the cosine distance between recipe text data and image data in the same recipe image text pair, and to increase the cosine distance between recipe text data and image data in different recipe image text pairs.

[0168] Step 4: Test the network model trained in Step 3 using the test set to obtain the image retrieval results for the recipe text, and evaluate the retrieval results; specifically:

[0169] Step 4.1: Define text feature index y', initialize it to 1; define image feature index x', initialize it to 1; define the text feature V output by the recipe hierarchical coding module. rec Initialized to empty, V rec =NULL, defines the output image features V of the text-contextualized visual enhancement module. img Initialized to empty, V img =NULL, defines the cosine distance result D of the text retrieval image, initializes D to null, i.e., D = NULL, defines the text retrieval image result R2i TeD Initialize R2i TeD It is empty, i.e., R2i TeD =NULL;

[0170] Step 4.2, from the test set TeD={i TeD ,t TeD} of i TeD Selecting the x'th image feature The image is fed into a pre-trained image encoding module with text context-based visual enhancement, and the output is... And Add to V img In step 4, determine whether x' is less than or equal to X'. If x' ≤ X', then x' = x' + 1, and continue repeating step 4.2.

[0171] Step 4.3, from the test set TeD={i TeD ,t TeD} of t TeD Select the y'th text feature The input is fed into the trained embedding submodule, the first-level attention mechanism submodule, and the second-level attention submodule, and the output is...

[0172] Step 4.4: Use the torch.mm() function from the open-source library PyTorch to calculate. and cosine similarity cosine distance

[0173]

[0174] Step 4.5, use Python's built-in sorting function sort() to sort... Sort the data; the smaller the cosine distance, the higher the similarity. Return the sorted result D. y' and sort the results The image corresponding to index number x' is appended to

[0175]

[0176] Step 4.6: Determine if y' is greater than Y'. If y' ≤ Y', then y' = y' + 1, and repeat step 4.3.

[0177] Step 4.7: Retrieve image results based on the final output recipe text. To evaluate the network model, calculate the median rank (MedR) of the correct results in the network model's output search results, and calculate the network model's recall (Recall@k). R2i :

[0178]

[0179] Where N represents the number of selected samples N = {1000, 10000}, and n represents the number of correct results among the first k selected samples, k = {1, 5, 10}.

[0180] Step 5. Test the network model trained in Step 3 on the test set to obtain the recipe image retrieval text results, and evaluate the retrieval results. Specifically:

[0181] Step 5.1. Define the text feature index y', initialize it to 1, define the image feature index x', initialize it to 1, and define the text feature V output by the recipe encoding module rec , initialize it to empty, V rec = NULL, define the image feature V output by the text-contextualized visual enhancement module img , initialize it to empty, V img = NULL, define the cosine distance result D of the image retrieval text, initialize D to empty, i.e., D = NULL, and define the image retrieval text result I2r TeD , initialize I2r TeD to be empty, i.e., I2r TeD = NULL;

[0182] Step 5.2. Select the y'-th text feature from t in the test set TeD = {i TeD ,t TeD} TeD and send it to the trained embedding sub-module, the first-level attention mechanism sub-module, and the second-level attention sub-module. The output result is Append it to V in. Judge whether y' is less than or equal to Y'. If y' ≤ Y', then y' = y' + 1, and continue to repeat Step 5.2; rec

[0183] Step 5.3. Select the x'-th image feature from i in the test set TeD = {i TeD ,t TeD} TeD and send it to the trained image encoding module for text-contextualized visual enhancement. The output result is

[0184] Step 5.4. Use the torch.mm() function in the open-source library pytorch to calculate and the cosine similarity cosine distance

[0185]

[0186] Step 5.5. Use the built-in sorting function sort() in Python to sort ​​Sort the data; the smaller the cosine distance, the higher the similarity. Return the sorted result D. x' and sort the results The text corresponding to the index number y' is appended to

[0187]

[0188] Step 5.6: Determine if x' is greater than X'. If x' ≤ X', then x' = x' + 1, and repeat step 5.3.

[0189] Step 5.7: Result of the text retrieval task based on the final output recipe image. To evaluate the network model, calculate the median rank (MedR) of the correct results output by the model and calculate the recall (Recall@k) of the model. I2r :

[0190]

[0191] Where N represents the number of selected samples N = {1000, 10000}, and n represents the number of correct results among the first k selected samples, k = {1, 5, 10}.

[0192] Example 2

[0193] Based on Example 1, N2 and L2 represent the number of sentences and sentence length of recipe components, respectively, and N3 and L3 represent the sentence length and word count of recipe instructions, respectively. N2, N3 ≤ 20, L2, L3 ≤ 15, the training set size A = 720,639, the test set size B = 51,304, the dimension D1 output by the hierarchical encoding module = 512, and the dimension D2 output by the recipe encoding module = 1024.

[0194] Example 3

[0195] Based on Example 2, this example uses a standard dataset (Recipe1M dataset) to test the present invention, as detailed below:

[0196] Recipe1M dataset description:

[0197] Recipe1M, released by Salvador et al. in 2017, is a commonly used dataset for cross-modal recipe retrieval. During local training, it uses data containing 238,408 image-recipe pairs, plus an additional 482,231 data points containing only the recipes. Each recipe in the dataset contains three parts: title, ingredients, and steps. This invention employs the same training and testing scale as other classic methods, using both the 238,408 image-recipe pairs and the test data containing the same 238,408 image-recipe pairs for evaluation. The Recipe1M dataset comprehensively covers common recipes and is the most widely used dataset for cross-modal recipe retrieval tasks.

[0198] Algorithm Evaluation:

[0199] The invention was evaluated using two objective metrics: Median Ranking (MedR) and Recall@k.

[0200] The objective evaluation results are shown in Tables 2 and 3, which respectively demonstrate the MedR and Recall@k results of the present invention and previous methods HT and T-Food on the Recipe1M dataset with different sample numbers. The method proposed in this invention exhibits excellent performance in all performance indicators, outperforming HT (ResNet-50) and T-Food. For the image recipe retrieval task with 1k and 10k sample numbers, the present invention achieved a MedR of 1.0, with R@1 reaching 76.2% and 48.3%, R@5 reaching 95.7% and 76.1%, and R@10 reaching 97.5% and 81.9%. For the recipe image retrieval task with 1k and 10k sample numbers, the present invention achieved a MedR of 1.0, with R@1 reaching 76.9% and 49.5%, R@5 reaching 95.6% and 79.2%, and R@10 reaching 97.7% and 81.1%, demonstrating the effectiveness of the method proposed in this invention.

[0201] Table 2 shows the results with a sample size of 1k.

[0202]

[0203] Table 3 shows the results with a sample size of 10k.

[0204]

[0205] The subjective evaluation results are as follows:

[0206] Subjective results of applying the modal interaction-based recipe retrieval method of this invention to the Recipe1M dataset are shown in the image. Figure 4 and Figure 5 As shown, Figure 4This represents an image-based recipe retrieval task, where images are fed into a modal interaction-based recipe retrieval model to obtain the top three recipes. Figure 5 This represents a recipe retrieval image task, where recipes are fed into a modal interaction-based recipe retrieval model to obtain the top five images. Figure 6 This paper presents a visualization of the input and output of the text-contextualized visual enhancement module in the recipe retrieval method based on modal interaction proposed in this invention. Figure 6 (a) is the input to the model of a recipe retrieval method based on modal interaction. Figure 6 (b) is a visualization of the input features of the text-contextualized visual enhancement module. Figure 6 (c) is a visualization of the output features of the text-contextualized visual enhancement module; from Figure 6 (b) and Figure 6 The comparison in (c) shows that the features after the text contextualization visual enhancement module are significantly enhanced in the feature regions corresponding to the parts in the recipe, which proves the effectiveness and accuracy of the recipe retrieval method based on modal interaction proposed in this invention.

Claims

1. A recipe retrieval method based on modal interaction, characterized in that: The specific steps are as follows: Step 1: Obtain a cross-modal recipe retrieval dataset and divide it into a training set and a test set; Step 2: Build a cross-modal recipe retrieval network model based on fine-grained inter-modal interaction; Step 3: Use the training set to train the cross-modal recipe retrieval network model based on fine-grained inter-modal interaction constructed in step 2; Step 4, using the test set to test the network model trained in step 3, obtain recipe text retrieval image results, and evaluate the retrieval results; Step 5: Test the network model trained in step 3 on the test set to obtain the recipe image retrieval text results and evaluate the retrieval results.

2. The recipe retrieval method based on modal interaction according to claim 1 is characterized in that: The step 1 is specifically as follows: Download the recipe dataset D = {I, T} from the official website, where I = {I1, ..., I m ,…,I M }, T = {T1, ..., T n ,…,T N }, I m represents the mth image in D, T n represents the nth recipe in D, 1≤m≤M, 1≤n≤N, M represents the total number of images, N represents the total number of recipes, I m ∈R W×H×D , W, H, D correspond to I respectively m The width, height and number of channels of the image in , Respectively represent T n The title, ingredients and instructions are three parts. Where N1 and L1 represent the number and length of sentences in the recipe title, respectively. The number of sentences in the recipe title is 1, so N1 = 1. N2 and L2 represent the number and length of sentences in the recipe ingredients, respectively. N3 and L3 represent the number and length of sentences in the recipe instructions, respectively. D is divided into a training set TrD and a test set TeD according to the ratio of A:B. TrD = {i TrD , t TrD }, X+Y=A; X+Y=B, represents the xth image in TrD, represents the yth recipe in TrD, represents the x′th image in TeD, represents the y′th recipe in TeD.

3. The recipe retrieval method based on modal interaction according to claim 2 is characterized in that: The cross-modal recipe retrieval network model based on fine-grained inter-modal interaction in step 2 includes a recipe hierarchical encoding module, an image encoding module based on text context visual enhancement, and a pairing module; The working process of the cross-modal recipe retrieval network model based on fine-grained inter-modal interaction is as follows: Step 2.1, the recipe hierarchical encoding module is from t in TrD T r D Extract the yth recipe 1≤y≤Y, where recipe Will Send to the hierarchical encoding module and output ttly, ing y , ins y The encoding features are Will After a vector concatenation and a fully connected dimension reduction network layer, the final recipe deep feature representation is obtained Step 2.2, the image encoding module based on text context visual enhancement is obtained from t in TrD TrD Extract the xth image Send it to the backbone network ResNet-50 to extract the shallow features of the kth layer of the ResNet-50 network Will and the title and component encoding features obtained in step 2.1 Perform feature enhancement to obtain Then Then send it to the k+1th layer of the backbone network ResNet-50 to extract deep features from the network before the classification Then pass the fully connected layer to obtain the deep features of the image Step 2.3, the pairing module calculates image depth features and recipe depth features pairing loss.

4. The recipe retrieval method based on modal interaction according to claim 3 is characterized in that: The hierarchical encoding module in step 2.1 is composed of an embedding submodule, a first-level attention mechanism submodule and a second-level attention mechanism submodule; The working process of the hierarchical coding module is as follows: First, Send it to the embedding submodule output to get E rec , E rec ={E ttl , E ing , E ins },in Secondly, put E rec Send it to the first-level attention mechanism submodule and output in Next, put the and They are sent to the secondary attention mechanism submodules respectively, and the output and in Finally Splice in the D1 dimension, input the spliced ​​features into a fully connected layer, and output the reduced-dimensional data in The working process of the embedding submodule is: As input, set the output vector dimension D1, call the open source library PyTorch text word data mapping function nn.embedding function, and output E rec , E rec ={E ttl , E ing , E ins },in 5. The recipe retrieval method based on modal interaction according to claim 4 is characterized in that: The first-level attention mechanism submodule of the hierarchical encoding module consists of an attention layer and a global average pooling layer; The working process of the first-level attention mechanism submodule is as follows: The output E of the embedded submodule rec ={E ttl , E ing , E ins } as input to the attention layer, which defines the query vector Value vector and key vector Assign values ​​to the query vector, value vector, and key vector: beg and The correlation and The correlation and The correlation The solution formula is as follows: Among them, softmax() is the probability normalization activation function; Then calculate the output of the attention mechanism in Finally, They are sent to the global average pooling layer respectively to obtain their average values ​​as the overall representation of recipe title features, recipe ingredient features, and recipe instruction features and in The secondary attention mechanism submodule consists of an attention layer and a global average pooling layer; The working process of the secondary attention mechanism submodule is as follows: The output data of the first-level attention mechanism submodule As the input of the secondary attention mechanism submodule, it is input to the attention layer of the secondary attention mechanism submodule, and the attention layer defines the query vector Value vector and key vector Assign values ​​to the query vector, value vector, and key vector: beg and The correlation and The correlation Among them, softmax() is the probability normalization activation function; Then calculate the output of the attention mechanism Finally, They are sent to the global average pooling layer respectively to obtain their average values ​​as the overall representation of the recipe ingredient features and recipe instruction features and in The hierarchical encoding module converts the output of the first-level attention mechanism submodule into And the output of the secondary attention mechanism submodule As ttl y ,ing y , ins y The encoded feature output.

6. The recipe retrieval method based on modal interaction according to claim 5 is characterized in that: The image encoding module based on text context visual enhancement in step 2.2 is composed of an image shallow feature extraction module, a text-image block similarity measurement submodule, a residual connection submodule and an image deep feature extraction submodule; The working process of the image shallow feature extraction module is as follows: From t in TrD TrD Extract the xth image in Will As the input of the image shallow feature extraction module, Send it to the ResNet-50 network to extract shallow features Specifically: Among them, ResNet-50 sl () means to Send it to the ResNet-50 network, pass through the 1st layer to the kth layer sub-network, and output the features of the kth layer; The working process of the text-image block similarity measurement submodule is as follows: Output data of the first-level attention mechanism submodule in step 2.1 And the output data of the image shallow feature extraction module As the input of the text-image patch similarity measurement submodule, and Splice and get Among them, cat() is a concatenation function: Among them, cat() is a concatenation function; And, call the array shape change function in the open source library Numpy Will Change to W k ×H k D k Characteristics of dimensions Then calculate and The point product between them is used to call the probability normalization activation function softmax() to get the text and image blocks The similarity score between in, The text-image block similarity measurement submodule will As output; The working process of the residual connection submodule is as follows: The output of the text-image block similarity measurement submodule is used as input, and the text and image residuals are concatenated according to the obtained similarity score. The working process of the image deep feature extraction submodule is as follows: Connect the output of the residual submodule As the input of the image deep feature extraction submodule, the ResNet-50 network is then used to extract Deep features of Among them, ResNet-50 dl () means to Send it to the ResNet-50 network, pass through the k+1th layer to the Kth layer, and output the features of the Kth layer; Finally, The final image features are sent to the fully connected layer Among them, Fc() means that the dimension is D K Features It is sent to the fully connected layer and the output dimension is D2.

7. The recipe retrieval method based on modal interaction according to claim 6 is characterized in that: The step 3 is specifically as follows: Step 3.1, set the network model training parameters; that is, set the learning rate variable lr, the maximum number of training iterations variable Max_iter, the size of each batch of paired data variable batch_size_paired, the number of training iterations variable step, and initialize step = 1; Step 3.2, input the training set TrD into the cross-modal recipe retrieval network model based on fine-grained modal interaction constructed in step 2, and perform network training according to the set training parameters. When the target loss function changes less than MinLoss or the training iteration number variable step≥Max_iter, the network training ends and the network model Model is saved; otherwise step=step+1, use the Adam optimizer to reversely correct the weight coefficients of each network layer in the network model, and continue training.

8. The recipe retrieval method based on modal interaction according to claim 7 is characterized in that: The target loss function in step 3.2 is the pairing loss function L pair : L cos (x a ,x p ,x n )=max(0,c(x a ,x n )-c(x a ,x p )+m) (20) Among them, the pairing loss L pair By an average two-way triplet loss L bi Composition, where L pair The input data of the loss function are the output data of the recipe classification encoding module Output data of image encoding module based on textual contextual visual enhancement Average two-way triplet loss L bi Refers to calculating the bidirectional triplet loss L′ between each recipe image and text pair in each batch bi The average value of the bidirectional triplet loss function L′ between each recipe image and text bi The input data is the i-th and j-th recipe image-text pairs in a batch, and the bidirectional triple loss is composed of two triple losses L cos Composition, where when i = j, δ(i, j) = 0, when i ≠ j, δ(i, j) = 1; triple loss function L cos (x a , x p , x n ) is a triple, where x a represents the training sample, x p represents a positive sample, x n represents negative samples, m is a hyperparameter, and c() represents the cosine distance, which aims to reduce the number of training samples x a and positive sample x p The distance between them, expand the training sample x a and negative samples x n The distance between them; the bidirectional triple loss function L′ bi It consists of two parts, and in Input data represents the output data of the i-th recipe image text in the training sample after the image encoding module based on text context visual enhancement, represents the output data of the i-th recipe image text in the training sample after the recipe hierarchical encoding module, representing the positive sample, represents the output data of the j-th recipe image text in the training sample after the recipe hierarchical encoding module, representing the negative sample, The function aims to reduce the cosine distance between the recipe image data and the text data in the same recipe image text pair, and to expand the cosine distance between the recipe image data and the text data in different recipe image text pairs; Input data represents the output data of the i-th recipe image text in the training sample after the recipe hierarchical encoding module, represents the output data of the i-th recipe image text in the training sample after the image encoding module based on text context visual enhancement, representing the positive sample, represents the output data of the j-th recipe image text in the training sample after the image encoding module based on text context visual enhancement, and represents the negative sample, The function aims to reduce the cosine distance between the recipe text data and the image data in the same recipe image text pair, and to expand the cosine distance between the recipe text data and the image data in different recipe image text pairs.

9. The recipe retrieval method based on modal interaction according to claim 8, characterized in that: The step 4 is specifically as follows: Step 4.1, define the text feature index y′, initialized to 1, define the image feature index x′, initialized to 1, define the output text feature V of the recipe hierarchical encoding module rec , initialized to empty, V rec = NULL, define the output image feature V of the visual enhancement module based on text context img , initialized to empty, V img = NULL, define the cosine distance result D of the text retrieval image, initialize D to empty, that is, D = NULL, define the text retrieval image result R2i TeD , initialize R2i TeD Empty, that is, R2i TeD = NULL; Step 4.2, from the test set TeD = {i TeD , t TeD } TeD Select the x′th image feature It is sent to the trained image encoding module based on text context visual enhancement, and the output result is and will Append to V img In , determine whether x′ is less than or equal to X′. If x′≤X′, then x′=x′+1, and continue to repeat step 4.2; Step 4.3, from the test set TeD = {i TeD , t TeD } T e D Select the y′th text feature It is sent to the trained embedding submodule, the first-level attention mechanism submodule and the second-level attention submodule, and the output result is Step 4.4, use the torch.mm() function in the open source library pytorch to calculate and V img = Cosine similarity of Cosine distance Step 4.5, use Python's built-in sorting function sort() to sort Sorting is performed. The smaller the cosine distance, the higher the similarity. The sorting result D is returned. y , and the sorted result y The image corresponding to the index number x′ is appended to Step 4.6, determine whether y′ is greater than Y′, if y′≤Y′, then y′=y′+1, and repeat step 4.3; Step 4.7, retrieve image results based on the final output recipe text Evaluate the network model, calculate the median ranking MedR of the correct results in the retrieval results output by the network model, and calculate the recall rate Recall@k of the network model R2i : Among them, N represents the number of selected samples, and n represents the number of correct results in the top k among the selected samples.

10. The recipe retrieval method based on modal interaction according to claim 9, characterized in that: The step 5 is specifically as follows: Step 5.1, define the text feature index y′, initialized to 1, define the image feature index x′, initialized to 1, define the output text feature V of the recipe encoding module rec , initialized to empty, V rec = NULL, define the output image feature V of the visual enhancement module based on text context img , initialized to empty, V img = NULL, define the cosine distance result D of the image retrieval text, initialize D to empty, that is, D = NULL, define the image retrieval text result I2r TeD , initialize I2r TeD Empty, that is, I2r TeD = NULL; Step 5.2, from the test set TeD = {i TeD , t TeD } TeD Select the y′th text feature It is sent to the trained embedding submodule, the first-level attention mechanism submodule and the second-level attention submodule, and the output result is Append to Vrec, determine whether y′ is less than or equal to Y′, if y′≤Y′, then y′=y′+1, and continue to repeat step 5.2; Step 5.3, select the x'-th image feature from the test set TeD = {i TeD , t TeD}, where i TeD is selected and sent to the trained image coding module based on text-contextualized visual enhancement, and the output result is Step 5.4, use the torch.mm() function in the open source library pytorch to calculate and Cosine similarity of Cosine distance Step 5.5, use Python's built-in sorting function sort() to sort Sorting is performed. The smaller the cosine distance, the higher the similarity. The sorting result D is returned. x′ , and sort the results The text corresponding to the corresponding index number y′ is appended to Step 5.6, determine whether x′ is greater than X′. If x′≤X′, then x′=x′+1, and repeat step 5.3; Step 5.7: Results of the task of retrieving text from the final output recipe image Evaluate the network model Model, calculate the median ranking MedR of the correct results in the retrieval results output by the model Model, and calculate the recall rate Recall@k of the model Model I2r : Among them, N represents the number of selected samples, and n represents the number of correct results in the top k among the selected samples.

Images