A cross-modal food retrieval method

Abstract

The present application relates to the interdisciplinary research field of artificial intelligence and machine learning, in particular to the technical field of cross-modal food retrieval.The present application provides a cross-modal food retrieval method, which focuses on adding a multi-granularity interactive encoder to build a TEI framework on the basis of an image encoder and a recipe encoder, inputting given image-text pair data into the image encoder and the recipe encoder respectively for training and learning, the image encoder extracting global visual features, the recipe encoder extracting hierarchical features and obtaining global recipe features using a linear layer, the multi-granularity interactive encoder extracting multi-granularity features, the hierarchical features and the multi-granularity features calculating paired self-attention and averaging to obtain multi-granularity interactive features, and aligning the correlation between the two modalities based on a hierarchical semantic alignment loss function.The purpose is to explore the multi-granularity interaction correlation between the image and text modalities to improve the cross-modal retrieval performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the interdisciplinary research field of artificial intelligence and machine learning, and particularly relates to a cross-modal food retrieval method. BACKGROUND

[0002] Food is a basic and essential part of human daily diet. The recent rise of social media in acquiring large amounts of food data has attracted increasing attention from researchers on food-related research. Utilizing visual-linguistic (VL) joint learning to analyze food data has become a popular research field, especially in the field of cross-modal food retrieval. The cross-modal food retrieval task consists of food data in two modalities of image and text, and the purpose is to use the text recipe (i.e. title, ingredients and methods) as a query to retrieve the positive food image from the candidate list, or use the food image as a query to retrieve the positive recipe. Previous work has focused on minimizing the cosine distance similarity between visual and textual representations by mapping them into a latent space, thereby narrowing the gap between the two modalities. For example, a stacked attention network (SAN) is proposed to learn a joint space by extracting visual and textual features, maximizing the similarity of image-recipe pairs. Some researchers extract text recipe attention to project into a shared space, which reduces the gap between visual and textual modalities. Although these works have achieved competitive performance, the cross-modal food retrieval task still faces several challenges, mainly in two aspects.

[0003] The main challenge is that the image feature extractor in the existing dual-encoder architecture is difficult to fully extract multi-granularity information from the food image. Existing methods mainly focus on the extraction of global features of food images, without considering the capture of fine-grained visual information in some regions of the food. The separate global visual features are not sensitive to the highly structured characteristics of the recipe features, which makes it difficult to improve the multi-granularity semantic interaction between the two modalities. The above phenomenon prompts us to explore a new type of encoder architecture that can not only extract multi-granularity visual features, but also retain global features.

[0004] The secondary challenge is that directly calculating the similarity measure of the image-recipe pair is not particularly effective in the potential alignment between the multi-granularity of food images and the multi-level of text recipes. Recent methods train the joint embedding retrieval model using text-image paired samples through some metric loss functions to update the retrieval model. Due to the structural distribution of image-level visual information and three-level text information, the similarity measure between the two modalities will be mismatched. The mismatched retrieval performance mainly exists in two points: (1) different categories of food show slight visual differences due to the same food materials and food containers; (2) the same category of food shows a large visual appearance difference after different cooking methods. Therefore, the purpose of the present application is to explore the multi-granularity interaction correlation between the image and text modalities to improve the cross-modal retrieval performance.

[0005] In order to alleviate the above challenges, the present application provides a TEI cross-modal food retrieval method to explore the correspondence between a single image and a multi-level recipe. SUMMARY

[0006] The purpose of the present application is to explore the multi-granularity interaction correlation between the image and text modalities to improve the cross-modal retrieval performance, and to improve the problems of difficult multi-granularity semantic interaction and unsatisfactory potential alignment in the prior art.

[0007] The cross-modal food retrieval method provided by the present application comprises the following steps:

[0008] A multi-granularity interaction encoder is added to the image encoder and the recipe encoder to construct a TEI framework;

[0009] The given image-text paired data are input into the image encoder and the recipe encoder for training and learning;

[0010] The image encoder extracts global visual features;

[0011] The recipe encoder extracts hierarchical features and obtains global recipe features using a linear layer;

[0012] The multi-granularity interaction encoder extracts multi-granularity features;

[0013] The hierarchical features and the multi-granularity features are calculated into paired self-attention and averaged to obtain multi-granularity interaction features;

[0014] The hierarchical semantic alignment loss function is used to align the correlation between the two modalities;

[0015] The trained TEI framework is used for cross-modal food retrieval.

[0016] The cross-modal food retrieval method provided by the present application has the technical effects of accuracy, strong robustness and strong adaptability.

[0017] Optionally, the image encoder takes a visual transformer with pre-trained weights of CLIP-B / 16 as the backbone to extract the global visual features.

[0018] Optionally, the recipe encoder takes a transformer encoder as the backbone to capture the text context information through self-attention, and uses three independent transformer encoders to encode the title, ingredient, and instruction information in the recipe to obtain the hierarchical features.

[0019] Optionally, the multi-granularity interaction encoder is composed of an MRA module and an additional image encoder, and the additional image encoder uses the same backbone as the image encoder and shares the weight parameters throughout the training process.

[0020] Optionally, when the given image-text pair data is input into the image encoder and the recipe encoder for training and learning, the text data is divided into ingredient, title, and instruction information and injected into the recipe encoder.

[0021] Optionally, when the image encoder extracts global visual features, the class labels in the final output of the image encoder are linearly projected through a fully connected layer to serve as global image features.

[0022] Optionally, when the recipe encoder extracts hierarchical features and uses a linear layer to obtain global recipe features, the hierarchical features are connected and averaged in the channel dimension to obtain global recipe embeddings, and are linearly projected through a fully connected layer to serve as global recipe features.

[0023] Optionally, when the multi-granularity interaction encoder extracts multi-granularity features, the food image is converted into three levels of granularity tensors and interacted with the MRA to obtain three levels of fine-grained input and linearly projected to obtain multi-granularity features.

[0024] Optionally, when the hierarchical semantic alignment loss function aligns the correlation between the two modalities, the multi-granularity interaction similarity score is obtained through two-stage similarity calculation within a small batch of data, and the hierarchical semantic alignment loss function L HMSA Reduces the mismatch problem of the image-text pair data.

[0025] Optionally, when the hierarchical semantic alignment loss function aligns the correlation between the two modalities, the general cross-modal triplet loss L HMSA , the semantic triplet loss L tri and the hierarchical semantic alignment loss function L sem are introduced on the basis of L HMSA , L tri and L sem are jointly supervised. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a TEI framework overview diagram of a cross-modal food retrieval method provided by the present application;

[0027] Figure 2 is a MRA module workflow diagram of a cross-modal food retrieval method provided by the present application;

[0028] Figure 3 is a multi-granularity interaction similarity score calculation flow diagram of a cross-modal food retrieval method provided by the present application. DETAILED DESCRIPTION

[0029] In order to make the objects, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should be understood as the general meaning understood by those of ordinary skill in the art to which the present application belongs. The words such as “comprise” and similar words used herein mean that the elements or objects before the words cover the elements or objects listed after the words and their equivalents, and do not exclude other elements or objects.

[0030] Embodiments of the present application provide a cross-modal food retrieval method, comprising the following steps:

[0031] S1, a multi-granularity interaction encoder is added on the basis of an image encoder and a recipe encoder to construct a TEI framework;

[0032] S2, given image-text pair data are respectively input to the image encoder and the recipe encoder for training and learning;

[0033] S3, the image encoder extracts global visual features;

[0034] S4, the recipe encoder extracts hierarchical features and obtains global recipe features using a linear layer;

[0035] S5, the multi-granularity interaction encoder extracts multi-granularity features;

[0036] S6, the hierarchical features and the multi-granularity features calculate paired self-attention and take average values to obtain multi-granularity interaction features;

[0037] S7, the relevance between the two modalities is aligned based on a hierarchical semantic alignment loss function;

[0038] S8, using the trained TEI framework for cross-modal food retrieval.

[0039] When performing step S1, the TEI architecture consists of three elements: an image encoder E img , a recipe encoder E rec , and a multi-granularity interaction encoder E mul .

[0040] The TEI framework overview of a cross-modal food retrieval method provided by the present application is shown in Figure 1 .

[0041] The image encoder takes the pre-trained weight of the visual transformer of CLIP-B / 16 as the backbone, and is used to extract the global visual features.

[0042] Image encoder: In order to extract global visual features, the visual transformer (ViT) with pre-trained weight of CLIP-B / 16 is used as the backbone of the image encoder E img . The class label in the final output of the image encoder is linearly projected through a fully connected layer and used as the global image feature f img .

[0043] The recipe encoder takes a transformer encoder as the backbone, which is used to capture text context information through self-attention, and uses three independent transformer encoders to encode the title, ingredient and instruction information in the recipe to obtain the hierarchical features.

[0044] Recipe encoder: existing methods use a recurrent neural network (RNN) as a recipe encoder to process text input. Since the training of RNN consumes too much, our recipe encoder E rec takes a transformer encoder as the backbone, which can capture higher quality text context information through self-attention. In terms of feature extraction, the title, ingredient and instruction information in the recipe are encoded using three independent transformer encoders to obtain hierarchical recipe features Then, the hierarchical recipe features are connected and averaged in the channel dimension to obtain the global recipe embedding. Finally, similar to the image encoder, the global recipe embedding is linearly projected as the global recipe feature f rec .

[0045] The multi-granularity interaction encoder consists of an MRA module and an additional image encoder, which uses the same backbone as the image encoder and shares weight parameters throughout the training process.

[0046] Multi-granularity interaction encoder: Many dual-encoder frameworks ignore the multi-granularity appearance information, which makes it difficult to align multi-granularity visual features and hierarchical recipe features in the latent space. To overcome this problem, a multi-granularity interaction encoder E mul is proposed to extract multi-granularity visual features The multi-granularity interaction encoder is composed of an MRA module and an additional image encoder, i.e., a CLIP-based image encoder E img , which uses the same backbone as E m and shares weight parameters throughout the training process. In the process of extracting multi-granularity visual features, the food image I will be converted into three levels of tensors T 1st ∈{T 2nd ; T 3rd}, and then interacted with the MRA to generate three levels of fine-grained inputs. These inputs will be injected into and obtain multi-granularity visual features By this time, the interaction between multi-granularity visual features and hierarchical recipe features has been extracted.

[0047] The MRA module workflow of the cross-modal food retrieval method provided by the application is as shown in Figure 2 .

[0048] When performing step S2, when the given image-text pair data is input into the image encoder and the recipe encoder for training and learning, the text data is divided into ingredient, title and instruction information three entities and injected into the recipe encoder.

[0049] The image-text pair data is input into the encoder E img and E rec . For the text recipe, the entities of ingredient, title and instruction information are injected into the recipe encoder to extract the corresponding hierarchical features

[0050] When performing step S3, when the image encoder extracts the global visual features, the class labels in the final output of the image encoder are linearly projected through a fully connected layer and used as global image features.

[0051] When performing step S4, when the recipe encoder extracts hierarchical features and obtains global recipe features using a linear layer, the hierarchical features are connected and averaged in the channel dimension to obtain global recipe embeddings, and are linearly projected through a fully connected layer and used as global recipe features.

[0052] In step S5, the multi-granularity interaction encoder extracts multi-granularity features, the food image is converted into three levels of granularity tensors and interacts with MRA to obtain three levels of fine-grained input and multi-granularity features through linear projection.

[0053] The existing dual-encoder method aims to align visual features and recipe features at a global level. That is, it is urgent to concentrate the three levels of recipe information (title, ingredients, instructions) into a global recipe feature in order to directly narrow the distance between image-text pairs in a shared space. This global alignment method is difficult to preserve more fine-grained elements in food images, and is not sensitive to the size, shape and other ingredient relationships of cooking. For recipes, the valuable information of the three levels of recipe entities appears at the same time, and a single global recipe feature cannot effectively contain the corresponding relationship. In order to promote alignment with multi-level recipes, a multi-granularity interaction encoder E mul is designed to extract multi-granularity visual features The workflow of the proposed MRA is shown in Figure 2 , which includes two parts: patch-aware module and channel-aware module.

[0054] Given an input image We use three different size convolution kernels for convolution operation to generate three levels of granularity tensors with the same size as I As shown in equation (1):

[0055]

[0056] Where H and W represent the length and width of the image, respectively. φ 1×1 (·), φ 5×5 (·), and φ 9×9 (·) are convolution operations of different size convolution kernels, so that T 1st , T 2nd and T 3rd contain visual information of different receptive field levels.

[0057] It is worth noting that the food image is input into the embedding layer of the multi-granularity interaction encoder E mul , and then divided into a series of patches. Then, each patch is regarded as a token for subsequent visual feature extraction. Each patch in each channel of the tensor {T 1st ; T 2nd ; T 3rd} is down-sampled to obtain the corresponding patch-aware vector where N = H x W / P 2is the resolution of each patch. The convolution kernel size is P x P with a stride of P. To explore the correlation between patch pairs, the inter-patch relation vector is calculated by

[0058]

[0059] where m e {lst, 2nd, 3rd}, T represents the transpose operation of the matrix.

[0060] After concatenating each patch perception vector at the channel level, it is then averaged and concatenated with each granularity of to obtain three multi-granularity patch relation vectors which can be represented as:

[0061]

[0062] where avg(·) represents the average calculation at the channel level to reduce the influence of the channel dimension. concat(·) means concatenating the input in the channel dimension.

[0063] Based on the Tm and obtained above, the patch relation perception attention vector is extracted as shown in equation (5)

[0064]

[0065] Where Sigmoid means activation function andφ p (·)is a convolutionoperations.Emb(·)represents the embedding layer ofthe encoderE mul andconsiders each patch as a token.

[0066] where Sigmoid represents the activation function, andφ p (·)is a convolution operation. Emb(·) represents the embedding layer of the encoder E mul and considers each patch as a token.

[0067] Since the embedding layer increases the dimension of the output channel, some noise may be introduced. Therefore, we input into equations (2), (3) to obtain the corresponding outputs They are further input into equation (4) but do not include the embedding layer, and then use E mul to obtain the final multi-granularity features It is worth noting that, unlike the three formula operations in the previous patch awareness module, our goal is to focus more on multi-granular information at the channel dimension than at the patch dimension.

[0068] In step S6, food-related retrieval tasks suffer from the problem of large intra-class differences and small inter-class differences. Different food categories using the same ingredients and containers may only exhibit minor visual differences in appearance. Furthermore, foods of the same category may show significant visual differences after different cooking methods. Therefore, calculating feature similarity between food images and recipes at the global level is not effective for semantic alignment. To address these issues, a hierarchical semantic alignment loss is proposed, using multi-granularity interaction similarity scores to establish multi-granularity for food images and multi-layered structure for text recipes, thereby achieving high-quality cross-modal retrieval performance.

[0069] The flowchart of the multi-granularity interactive similarity score calculation method for cross-modal food retrieval provided by this invention is as follows: Figure 3 As shown.

[0070] Considering encoder E mul Output and recipe encoder E rec Output according to and The relationships between features are grouped according to feature pairs, with corresponding granularities and levels. Three feature pairs are obtained through the self-attention mechanism of the transformer, thus yielding three granularity interaction features. This can be expressed as:

[0071]

[0072] Here, cross(Q,K,V) represents self-attention computation. Q, K, and V represent the query, key, and value, respectively. Then, these three granular features are averaged and linearly projected to obtain the final multi-granularity interaction feature f. c .

[0073] When performing step S7, in aligning the relevance between the two modalities based on the hierarchical semantic alignment loss function, a multi-granularity interaction similarity score is obtained through two-stage similarity calculation within a small batch of data, and then the hierarchical semantic alignment loss function L is applied. HMSA Reduce the mismatch problem of the image-text pairing data.

[0074] Based on three features {f img ;f rec ;f cThe multi-granularity interaction similarity score S can be obtained by two-stage similarity calculation within small batch data cross defined as formula (6):

[0075]

[0076] where s img represents the cosine similarity matrix between global visual features f img and f c . s rec represents the cosine similarity matrix between global recipe features f rec and f c . Next, we calculate the cosine similarity of s img and s rec , aiming to measure the distance between the inter-modal by comparing the neighborhood similarity of image-text pairs in batch training data.

[0077] Based on the above calculation, a hierarchical multi-granularity semantic alignment loss L HMSA is designed to reduce the mismatch problem of image-recipe samples, which includes two branch losses. Two branch losses are calculated using image or text mode as anchor sample respectively. The calculation of the comprehensive hierarchical multi-granularity semantic alignment loss L HMSA and its branch loss is as follows:

[0078]

[0079] where and represent two branch losses. is the feature of the image anchor, the positive and negative examples of the recipe, and vice versa. Alpha is an interval value designed to adaptively increase the value in each cycle. The increment of alpha is obtained by calculating the average value of the maximum value of all batches less than 0.

[0080] When the correlation between the two modalities is aligned based on the hierarchical semantic alignment loss function, the general cross-modal triple loss L HMSA and the semantic triple loss L sem are introduced on the basis of L tri , and are jointly supervised by L HMSA , L tri and L sem .

[0081] In addition, the embodiments are evaluated on Recipe1M and Food101 datasets for cross-modal food retrieval and food classification tasks, respectively. For Recipe1M, we use 238999 image-recipe pairs as the training set, 51119 as the validation set, and 51303 as the test set. Each recipe contains three layers of text information, namely title, ingredients, and procedures. Food101 includes 101 food categories, with 1000 food images in each category. We also use 750 food images in each category, totaling 75750 images, as the training set, and the remaining 25250 images as the test set for experiments in the subsequent research section.

[0082] The embodiments use the same evaluation metrics as existing methods to evaluate all experiments. Specifically, in the test phase, we randomly select 10 (or 5) subsets from the test set, each containing 1000 (or 10000) correctly matched image-recipe pairs (the numerical pair of 10-1000 and 5-10000 corresponds to 1k and 10k settings in this paper, respectively). Then, using the text or image modality as a query, the cosine distance with all samples in the current subset and the other modality is calculated. Subsequently, the calculated distances are arranged in order in the search result list. For the search result list, we calculate the median retrieval rank (MedR) and the recall percentage at K (R@K) as evaluation metrics for the cross-modal food retrieval task.

[0083] The embodiments select CLIP-B / 16 as the pre-trained model for the image encoder and the multi-granularity interaction encoder, and select the transformer as the pre-trained model for the recipe encoder. The batch size and total training period are both set to 80. The recipe encoder and the image encoder (the image encoder and the multi-granularity interaction encoder share parameters) are trained using the Adam optimizer with constant learning rates of 1e-5 and 1e-6, respectively. During training, the image is resized to 224x224 pixels and used as input for the image encoder. All experiments are run in an Ubuntu 18.04 operating system and Pytorch environment equipped with 4 Tesla P40.

[0084] While the embodiments of the application have been illustrated and described in detail, it will be apparent to those skilled in the art that various modifications and changes can be made therein without departing from the scope and spirit of the application as described in the claims. Moreover, the application described herein can have other embodiments and be practiced or implemented in various ways.

Claims

1. A cross-modal food retrieval method, characterized in that, The method comprises the following steps: adding a multi-granularity interaction encoder to the image encoder and the recipe encoder to build a TEI framework; inputting given image-text pair data into the image encoder and the recipe encoder for training and learning; the image encoder extracts global visual features; the recipe encoder extracts hierarchical features and obtains global recipe features using a linear layer; when the multi-granularity interaction encoder extracts multi-granularity features, the food image is converted into three levels of granularity tensors and interacts with the MRA to obtain three levels of fine-grained input and multi-granularity features through linear projection; the hierarchical features and the multi-granularity features calculate paired self-attention and take the average to obtain multi-granularity interaction features; When aligning the correlation between the two modalities based on the hierarchical semantic alignment loss function, a multi-granularity interaction similarity score is obtained through two-stage similarity calculation within a small batch of data, and the hierarchical semantic alignment loss function is used to align the correlation between the two modalities Reduce the mismatch problem of the image-text pair data; on the basis of Introducing general cross-modal triple loss And semantic triple loss And jointly supervised by , And ​ the trained TEI framework is used for cross-modal food retrieval.

2. The cross-modal food retrieval method of claim 1, wherein, The image encoder takes the visual transformer of CLIP-B / 16 with pre-training weights as the backbone to extract the global visual features.

3. The cross-modal food retrieval method of claim 1, wherein, The recipe encoder takes a transformer encoder as the backbone to capture the text context information through self-attention, and uses three independent transformer encoders to encode the title, ingredients and instruction information in the recipe to obtain the hierarchical features.

4. The cross-modal food retrieval method of claim 1, wherein, The multi-granularity interaction encoder is composed of an MRA module and an additional image encoder, and the additional image encoder uses the same backbone as the image encoder and shares weight parameters during the entire training process.

5. The cross-modal food retrieval method of claim 1, wherein, When the given image-text pair data is input into the image encoder and the recipe encoder for training and learning, the text data is divided into ingredient, title and instruction information three entities and injected into the recipe encoder.

6. The cross-modal food retrieval method of claim 1, wherein, When the image encoder extracts global visual features, the class label in the final output of the image encoder is linearly projected through a fully connected layer and used as the global image feature.

7. The cross-modal food retrieval method of claim 1, wherein, When the recipe encoder extracts hierarchical features and obtains global recipe features using a linear layer, the hierarchical features are connected in the channel dimension and averaged to obtain global recipe embedding, and are linearly projected through a fully connected layer and used as global recipe features.

Images