Cross-modal food retrieval method

Abstract

The invention relates to the field of interdisciplinary research of artificial intelligence and machine learning, in particular to a method. The invention provides a cross-modal food retrieval method, which is characterized in that a multi-granularity interactive encoder is added on the basis of an image encoder and a recipe encoder to construct a TE I framework, given image-text pairing data are respectively input into the image encoder and the recipe encoder for training and learning, the image encoder extracts global visual features, and the overall visual features are extracted by the recipe encoder; the recipe encoder extracts hierarchical features and obtains global recipe features by using a linear layer, the multi-granularity interaction encoder extracts multi-granularity features, the hierarchical features and the multi-granularity features calculate paired self-attention, and an average value is taken to obtain multi-granularity interaction features; and aligning the correlation between the two modalities based on a hierarchical semantic alignment loss function. The objective of the invention is to explore multi-granularity interaction correlation between image and text modals so as to improve cross-modal retrieval performance.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary research field of artificial intelligence and machine learning, and in particular to a cross-modal food retrieval method. Background Art

[0002] Food is a basic and essential component of human daily diet. The recent rise of social media in acquiring a large amount of food data has attracted increasing attention from researchers to food-related research. Analyzing food data using joint vision-language (VL) learning has become a hot research area, especially in the field of cross-modal food retrieval. The cross-modal food retrieval task consists of food data in two modalities, image and text. The goal is to retrieve positive food images from a candidate list using text recipes (i.e., title, ingredients, and recipe) as queries, or to retrieve positive recipes using food images as queries. Previous works have focused on bridging the gap between the two modalities by mapping visual and textual representations into latent spaces and minimizing the cosine distance similarity between them. For example, stacked attention networks (SANs) have been proposed to learn to maximize the similarity of image-recipe pairs by extracting a joint space of visual and textual features. Some researchers have extracted the attention of text recipes to project into a shared space, which reduces the gap between visual and textual modalities. Although these works have achieved competitive performance, the cross-modal food retrieval task still faces several challenges, which are mainly reflected in two aspects.

[0003] The main challenge is that the image feature extractor in the existing dual encoder architecture has difficulty in fully extracting multi-granularity information from food images. Existing methods mainly focus on global feature extraction of food images, without considering capturing fine-grained visual information in certain regions of the food. Global visual features alone are insensitive to the highly structured nature of recipe features, which makes it difficult to enhance the multi-granularity semantic interaction between the two modalities. The above phenomenon motivates us to explore a new encoder architecture that can not only extract multi-granularity visual features but also preserve global features.

[0004] A secondary challenge is that directly computing similarity measures for image-recipe pairs is not particularly effective in aligning the potential between the multi-granularity of food images and the multi-level of text recipes. Recent methods use text-image pairing samples to train a joint embedding retrieval model through some metric loss function to update the retrieval model. Due to the wide distribution of the structure of image-level visual information and three-level text information, there will be a mismatch in similarity measures between the two modalities. The mismatched retrieval performance mainly exists in two points: (1) due to the same ingredients and food containers, different categories of food show slight visual differences; (2) the same category of food shows large visual appearance differences after different cooking methods. Therefore, the purpose of this paper is to explore the multi-granularity interactive correlation between image and text modalities to improve cross-modal retrieval performance.

[0005] To alleviate the above challenges, this paper provides a cross-modal food retrieval method called TEI to explore the correspondence between a single image and multi-level recipes. Summary of the invention

[0006] The purpose of the present invention is to explore the multi-granularity interactive correlation between image and text modalities to improve cross-modal retrieval performance and to improve the problems of the prior art in multi-granularity semantic interaction difficulties and unsatisfactory potential alignment.

[0007] A cross-modal food retrieval method provided by the present invention comprises the following steps:

[0008] Based on the image encoder and recipe encoder, a multi-granularity interactive encoder is added to build the TEI framework;

[0009] Input the given image-text pairing data into the image encoder and recipe encoder for training and learning respectively;

[0010] The image encoder extracts global visual features;

[0011] The recipe encoder extracts hierarchical features and uses linear layers to obtain global recipe features;

[0012] The multi-granularity interactive encoder extracts multi-granularity features;

[0013] The hierarchical features and the multi-granularity features are used to calculate paired self-attention and average the multi-granularity interaction features;

[0014] Align the correlation between the two modalities based on the hierarchical semantic alignment loss function;

[0015] The trained TEI framework is used for cross-modal food retrieval.

[0016] The cross-modal food retrieval method provided by the present invention has the technical effects of accuracy, strong robustness and strong adaptability.

[0017] Optionally, the image encoder uses a visual transformer with pre-trained weights of CLIP-B / 16 as a backbone to extract the global visual features.

[0018] Optionally, the recipe encoder uses a transformer encoder as a backbone to capture text context information through self-attention, and uses three independent transformer encoders to encode the title, ingredients and instruction information in the recipe to obtain the hierarchical features.

[0019] Optionally, the multi-granularity interaction encoder consists of an MRA module and an additional image encoder, which uses the same backbone as the image encoder and shares weight parameters throughout the training process.

[0020] Optionally, when the given image-text pairing data is respectively input into the image encoder and the recipe encoder for training and learning, the text data is divided into three entities of ingredients, title and instruction information and injected into the recipe encoder.

[0021] Optionally, when the image encoder extracts global visual features, the class labels in the final output of the image encoder are linearly projected through a fully connected layer and used as global image features.

[0022] Optionally, when the recipe encoder extracts hierarchical features and uses a linear layer to obtain a global recipe feature, the hierarchical features are connected and averaged in the channel dimension to obtain a global recipe embedding, which is linearly projected through a fully connected layer and used as a global recipe feature.

[0023] Optionally, when the multi-granularity interactive encoder extracts multi-granularity features, the food image is converted into a tensor of three granularity levels and interactively calculated with MRA to obtain three levels of fine-grained input and multi-granularity features through linear projection.

[0024] Optionally, when aligning the correlation between the two modalities based on the hierarchical semantic alignment loss function, a multi-granularity interaction similarity score is obtained by two-stage similarity calculation in a small batch of data, and the hierarchical semantic alignment loss function L is used. HMSA The mismatch problem of the image-text pairing data is reduced.

[0025] Optionally, when aligning the correlation between the two modalities based on the hierarchical semantic alignment loss function, in L HMSA Based on this, we introduce the general cross-modal triplet loss L tri and semantic triplet loss L sem And by L HMSA , L tri and L sem Joint supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is an overview diagram of the TEI framework of a cross-modal food retrieval method provided by the present invention;

[0027] Figure 2 It is a workflow diagram of an MRA module of a cross-modal food retrieval method provided by the present invention;

[0028] Figure 3 It is a multi-granularity interactive similarity score calculation flow chart of a cross-modal food retrieval method provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0030] An embodiment of the present invention provides a cross-modal food retrieval method, comprising the following steps:

[0031] S1. Add a multi-granularity interactive encoder to build the TEI framework based on the image encoder and recipe encoder;

[0032] S2, input the given image-text pairing data into the image encoder and recipe encoder for training and learning respectively;

[0033] S3, image encoder extracts global visual features;

[0034] S4, the recipe encoder extracts hierarchical features and uses a linear layer to obtain global recipe features;

[0035] S5, multi-granularity interactive encoder extracts multi-granularity features;

[0036] S6, the hierarchical features and the multi-granularity features are used to calculate paired self-attention and average the multi-granularity interaction features;

[0037] S7, align the correlation between the two modalities based on the hierarchical semantic alignment loss function;

[0038] S8. Use the trained TEI framework for cross-modal food retrieval.

[0039] When executing step S1, the TEI architecture consists of three elements: an image encoder E img 、Recipe Encoder E rec and multi-granularity interactive encoder E mul .

[0040] The TEI framework overview of a cross-modal food retrieval method provided by the present invention is shown in FIG. Figure 1 shown.

[0041] The image encoder is based on a visual transformer with pre-trained weights of CLIP-B / 16 to extract the global visual features.

[0042] Image encoder: In order to extract global visual features, the visual transformer (ViT) with pre-trained weights of CLIP-B / 16 is used as the image encoder E img The class labels in the final output of the image encoder are linearly projected through a fully connected layer and used as the global image feature f img .

[0043] The recipe encoder is based on a transformer encoder to capture textual context information through self-attention, and uses three independent transformer encoders to encode the title, ingredients and instruction information in the recipe to obtain the hierarchical features.

[0044] Recipe Encoder: Existing methods use recurrent neural networks (RNNs) as recipe encoders to process text input. Since RNNs are too expensive to train, our recipe encoder E rec A transformer encoder is used as the backbone, which can capture higher quality text context information through self-attention. In terms of feature extraction, the title, ingredients, and instruction information in the recipe are encoded using three independent transformer encoders to obtain hierarchical recipe features. Then, the hierarchical recipe features are concatenated and averaged in the channel dimension to obtain the global recipe embedding. Finally, similar to the image encoder, the global recipe embedding is linearly projected as the global recipe feature f rec .

[0045] The multi-granularity interaction encoder consists of an MRA module and an extra image encoder, which uses the same backbone as the image encoder and shares weight parameters throughout the training process.

[0046] Multi-granularity interaction encoder: Many dual encoder frameworks ignore the multi-granularity appearance information, which makes it difficult to align multi-granularity visual features and hierarchical recipe features in the latent space. To overcome this problem, a multi-granularity interaction encoder E is proposed. mul To extract multi-granular visual features The multi-granular interactive encoder consists of the MRA module and an additional image encoder based on CLIP. Among them, the image encoder uses the same img The same backbone and weight parameters are shared throughout the training process. In the process of extracting multi-granular visual features, the food image I will be converted into a tensor T at three granularity levels. m ∈{T 1st ; T 2nd ; T 3rd}, and then interact with MRA to generate three levels of fine-grained inputs. These inputs will be injected into And obtain multi-granularity visual features through linear projection So far, the interaction between multi-granular visual features and hierarchical recipe features has been extracted.

[0047] The MRA module workflow diagram of a cross-modal food retrieval method provided by the present invention is as follows: Figure 2 shown.

[0048] When executing step S2, when the given image-text pairing data is respectively input into the image encoder and the recipe encoder for training and learning, the text data is divided into three entities of ingredients, title and instruction information and injected into the recipe encoder.

[0049] The image-text pairing data is input to the encoder E img and E rec For text recipes, the entities of ingredients, titles, and instruction information are injected into the recipe encoder to extract the corresponding hierarchical features.

[0050] When executing step S3, when the image encoder extracts the global visual features, the class labels in the final output of the image encoder are linearly projected through a fully connected layer and used as global image features.

[0051] When executing step S4, the recipe encoder extracts hierarchical features and uses a linear layer to obtain a global recipe feature, connects and averages the hierarchical features in the channel dimension to obtain a global recipe embedding, and linearly projects it through a fully connected layer to be used as a global recipe feature.

[0052] When executing step S5, when the multi-granularity interactive encoder extracts multi-granularity features, the food image is converted into a tensor of three granularity levels and interactively calculated with MRA to obtain three levels of fine-grained input and multi-granular features through linear projection.

[0053] Existing dual encoder methods aim to align visual features and recipe features at a global level. That is, there is an urgent need to bring together the three levels of recipe information (title, ingredients, instructions) into a global recipe feature so that the image-text pairs can be directly brought closer in a shared space. This global alignment method has difficulty in preserving more fine-grained elements in food images and is insensitive to the size, shape, and other ingredient relationships of the dishes. For recipes, the three levels of valuable information of recipe entities appear simultaneously, and a single global recipe feature cannot effectively contain the corresponding relationships. In order to facilitate alignment with multi-level recipes. In the multi-granular interactive encoder E mul The MRA module is designed to extract multi-granularity visual features The proposed MRA workflow is as follows Figure 2 As shown, it consists of two parts: a patch-aware module and a channel-aware module.

[0054] Given an input image We use three different scales of convolution kernels to perform convolution operations and generate three granularity levels of tensors with the same size as I: As shown in formula (1):

[0055]

[0056] Where H and W represent the length and width of the image respectively. 1×1 (·),φ 5×5 (·), andφ 9×9 (·) is the convolution operation with different kernel sizes, so that T 1st ,T 2nd and T 3rd Contains visual information at different receptive field levels.

[0057] It is worth noting that the food images are input into the multi-granularity interactive encoder E mul The embedding layer is then segmented into a series of patches. Each patch is then treated as a token for subsequent visual feature extraction. 1st ; T 2nd ; T 3rd Each patch of each channel in} is downsampled to obtain the corresponding patch perception vector Where N = H × W / P 2is the resolution of each patch. The convolution kernel size is P×P and the stride is P. In order to explore the correlation between patch pairs, the inter-patch relationship vector is calculated by

[0058]

[0059] Where m∈{1st, 2nd, 3rd}, T represents the transpose operation of the matrix.

[0060] After concatenating each patch perception vector at the channel level, they are then averaged and combined with the Concatenate to obtain three multi-granularity patch relationship vectors It can be expressed as:

[0061]

[0062] Where avg(·) represents the average calculation at the channel level to reduce the impact of the channel dimension. concat(·) means concatenating the input in the channel dimension.

[0063] Based on the Tm and The patch relation-aware attention vector is extracted as shown in formula (5):

[0064]

[0065] Where Sigmoid means activation function andφ p (·)is a convolutionoperations.Emb(·)represents the embedding layer of the encoderE mul andconsiders each patch as a token.

[0066] Where Sigmoid represents the activation function, φ p (·) is the convolution operation. Emb(·) represents the encoder E mul The embedding layer considers each patch as a token.

[0067] Since the embedding layer increases the dimension of the output channels, Some noise may have been introduced. Therefore, we will Input into formula (2), (3) to obtain the corresponding output They are further fed into Equation (4) excluding the embedding layer and then used as mul To obtain the final multi-granular features It is worth noting that, different from the three formula operations in the previous patch-aware module, our goal is to pay more attention to multi-granularity information in the channel dimension than in the patch dimension.

[0068] When executing step S6, food-related retrieval tasks have the problem of large intra-class differences and small inter-class differences. Different food categories using the same ingredients and containers may only show slight visual differences in appearance. In addition, foods of the same category may also show significant differences in visual appearance after being cooked in different ways. Therefore, calculating the feature similarity between food images and recipes at a global level does not effectively achieve semantic alignment. To address the above problems, a hierarchical semantic alignment loss is proposed, which uses multi-granularity interaction similarity scores to establish multi-granularity of food images and multi-level of text recipes, thereby achieving high-quality cross-modal retrieval performance.

[0069] The multi-granularity interactive similarity score calculation flow chart of a cross-modal food retrieval method provided by the present invention is as follows: Figure 3 shown.

[0070] Considering the encoder E mul Output and Recipe Encoder E rec Output according to and The relationship between , the corresponding granularity and level are grouped into feature pairs. Three feature pairs are obtained through the self-attention mechanism of the transformer, thus obtaining three granularity interaction features It can be expressed as:

[0071]

[0072] Where cross(Q,K,V) represents the self-attention calculation. Q, K, and V represent query, key, and value, respectively. Then, these three granular features are averaged and linearly projected to obtain the final multi-granular interaction feature f c .

[0073] When executing step S7, when aligning the correlation between the two modalities based on the hierarchical semantic alignment loss function, a multi-granularity interaction similarity score is obtained by two-stage similarity calculation in a small batch of data, and a hierarchical semantic alignment loss function L is used to align the correlation between the two modalities. HMSA The mismatch problem of the image-text pairing data is reduced.

[0074] Based on three features {f img ;f rec ;f c}, the multi-granularity interaction similarity score S can be obtained by two-stage similarity calculation within a small batch of data cross , defined as formula (6):

[0075]

[0076] where s img Refers to the global visual feature f img With f c The cosine similarity matrix between s rec represents the global recipe feature f rec With f c Next, we calculate the cosine similarity matrix between s img and rec The cosine similarity of is designed to measure the gap between modalities by comparing the neighborhood similarities of image-text pairs in batch training data.

[0077] Based on the above calculation, a hierarchical multi-granularity semantic alignment loss L is designed HMSA , to reduce the mismatch problem between image and recipe samples, the loss includes two branch losses. The two branch losses are calculated using images or text patterns as anchor samples respectively. The comprehensive hierarchical multi-granularity semantic alignment loss L HMSA And its branch loss is calculated as follows:

[0078]

[0079] in and represents the two branch losses. are the features of the image anchor, recipe positive and negative examples, and vice versa. α is an interval value designed to increase adaptively in each cycle. The increment of α is calculated by calculating The average value of the maximum value less than 0 is obtained.

[0080] When aligning the correlation between the two modalities based on the hierarchical semantic alignment loss function, HMSA Based on this, we introduce the general cross-modal triplet loss L tri and semantic triplet loss L sem And by L HMSA , L tri and L sem Joint supervision.

[0081] In addition, this embodiment evaluates the performance of the proposed method in cross-modal food retrieval and food classification tasks on the RecipelM and Food101 datasets, respectively. For RecipelM, we use 238,999 image-recipe pairs as training sets, 51,119 as validation sets, and 51,303 as test sets. Each recipe contains three layers of text information, namely title, ingredients, and methods. Food101 includes 101 food categories, each with 1,000 food images. We also use 750 food images in each category, a total of 75,750 images as training sets, and the remaining 25,250 images as test sets to conduct experiments in subsequent research parts.

[0082] This embodiment uses the same evaluation metrics as the existing methods to evaluate all experiments. Specifically, in the testing phase, we randomly select 10 (or 5) subsets from the test set, each subset containing 1000 (or 10000) correctly matched image-recipe pairs (the numerical pairs of 10-1000 and 5-10000 correspond to the 1k and 10k settings in this article, respectively). Then, using the text or image modality as the query, the cosine distance with all samples in the current subset and the other modality is calculated. Subsequently, the calculated distances are arranged in order in the search result list. For the search result list, we calculate the median retrieval rank (MedR) and the recall percentage (R@K) of the top K as evaluation metrics for the cross-modal food retrieval task.

[0083] In this example, CLIP-B / 16 is selected as the pre-trained model for the image encoder and the multi-granularity interaction encoder, and the transformer is selected as the pre-trained model for the recipe encoder. The batch size and the total training cycle are both set to 80. The recipe encoder and the image encoder are trained using the Adam optimizer at constant learning rates of 1e-5 and 1e-6, respectively (the image encoder and the multi-granularity interaction encoder share parameters). During training, the image is resized to 224×224 pixels and used as input to the image encoder. All experiments are run in the Ubuntu 18.04 operating system and Pytorch environment equipped with 4 Tesla P40s.

[0084] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A cross-modal food retrieval method, characterized in that: The following steps are involved: Based on the image encoder and recipe encoder, a multi-granularity interactive encoder is added to build the TEI framework; Input the given image-text pairing data into the image encoder and recipe encoder for training and learning respectively; The image encoder extracts global visual features; The recipe encoder extracts hierarchical features and uses linear layers to obtain global recipe features; The multi-granularity interactive encoder extracts multi-granularity features; The hierarchical features and the multi-granularity features are used to calculate paired self-attention and average the multi-granularity interaction features; Align the correlation between the two modalities based on the hierarchical semantic alignment loss function; The trained TEI framework is used for cross-modal food retrieval.

2. A cross-modal food retrieval method according to claim 1, characterized in that: The image encoder is based on a visual transformer with pre-trained weights of CLIP-B / 16 to extract the global visual features.

3. A cross-modal food retrieval method according to claim 1, characterized in that: The recipe encoder is based on a transformer encoder to capture textual context information through self-attention, and uses three independent transformer encoders to encode the title, ingredients and instruction information in the recipe to obtain the hierarchical features.

4. A cross-modal food retrieval method according to claim 1, characterized in that: The multi-granularity interaction encoder consists of an MRA module and an extra image encoder, which uses the same backbone as the image encoder and shares weight parameters throughout the training process.

5. A cross-modal food retrieval method according to claim 1, characterized in that: When the given image-text pairing data is respectively input into the image encoder and the recipe encoder for training and learning, the text data is divided into three entities of ingredients, title and instruction information and injected into the recipe encoder.

6. A cross-modal food retrieval method according to claim 1, characterized in that: When the image encoder extracts global visual features, the class labels in the final output of the image encoder are linearly projected through a fully connected layer and used as global image features.

7. A cross-modal food retrieval method according to claim 1, characterized in that: When the recipe encoder extracts hierarchical features and uses a linear layer to obtain a global recipe feature, the hierarchical features are concatenated and averaged in the channel dimension to obtain a global recipe embedding, which is linearly projected through a fully connected layer and used as a global recipe feature.

8. A cross-modal food retrieval method according to claim 1, characterized in that: When the multi-granularity interactive encoder extracts multi-granularity features, the food image is converted into a tensor of three granularity levels and interactively calculated with MRA to obtain three levels of fine-grained input and multi-granularity features through linear projection.

9. A cross-modal food retrieval method according to claim 1, characterized in that: When aligning the correlation between the two modalities based on the hierarchical semantic alignment loss function, a multi-granularity interaction similarity score is obtained by two-stage similarity calculation in a small batch of data, and the hierarchical semantic alignment loss function L is used to align the correlation between the two modalities. HMSA The mismatch problem of the image-text pairing data is reduced.

10. A cross-modal food retrieval method according to claim 1, characterized in that: When aligning the correlation between the two modalities based on the hierarchical semantic alignment loss function, in L HMSA Based on this, we introduce the general cross-modal triplet loss L tri and semantic triplet loss L sem And by L HMSA , L tri and L sem Joint supervision.

Images