A training method of a model for food image classification and an image retrieval method

Abstract

A training method of a model for food image classification, the model comprising a feature extractor and a classifier, the method comprising: S1, obtaining an initial training set containing a plurality of samples and corresponding labels and dividing the initial training set into a plurality of batches, and constructing a plurality of triplets containing anchor samples, positive samples and negative samples based on the samples in each batch, wherein each sample is a food image, and the label corresponding to the sample indicates the category of the food image; S2, optimizing selected triplets of each batch of the initial training set, wherein in the optimization, the negative sample of the selected triplet is replaced by its positive sample, and the positive sample of the selected triplet is replaced by its anchor sample to obtain an optimized triplet, and the optimized training set is formed based on the optimized triplets and the unselected triplets; and S3, training the feature extractor for extracting sample features and the classifier for classifying samples according to sample features in batches using the optimized training set.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to the field of image retrieval, and more specifically to a training method for a model for food image classification and an image retrieval method. Background Technology

[0002] Food is closely related to human health and well-being, and in recent years, food-related research has become a hot topic. With the development of technologies such as computer vision, food analysis tasks, such as food image recognition, plate detection, and nutritional analysis, have been extensively studied. At the same time, the development of big data and other technologies in the internet environment has generated a large amount of multimodal data, which often comes with information redundancy. Therefore, extracting the information users want from this massive amount of data has become an urgent problem. Under these conditions, the task of food image retrieval has emerged. Food image retrieval aims to find all relevant images from a raw food database using query terms.

[0003] Based on the methods for extracting food image features, existing food image retrieval methods can be categorized as follows:

[0004] Methods for manually extracting features based on scale-invariant feature transformation and other techniques, such as method [1], which extracts manual features and retrieves bag-of-words features for each image in imitation of traditional text retrieval methods.

[0005] Deep learning-based methods, such as method [2], directly obtain the image features corresponding to the image through a neural network model (e.g., a convolutional neural network model), calculate the similarity between the image features corresponding to the image to be retrieved and the image features corresponding to the images in the image database, sort the images in the image database according to the similarity, and return the final retrieval result.

[0006] Numerous studies have demonstrated that deep learning-based methods outperform manual feature extraction methods. Among deep learning-based methods, current mainstream research focuses on optimizing convolutional neural networks (CNNs) to output image features with semantic meaning. Specifically, deep learning-based methods enhance the expressive power of CNNs by constructing appropriate samples, applying suitable feature extractors, and designing clever loss functions.

[0007] Existing food image retrieval methods focus on performance in "visible" categories. Specifically, in existing food image retrieval methods, the training and test sets contain the same categories on publicly available food image datasets, such as ETH Food-101 and Vireo Food-172. However, the food industry is constantly evolving, with new food categories emerging daily. Users, when searching, not only need to focus on existing categories but also often hope to find similar foods in new "unseen" categories. Therefore, the model needs to maintain good generalization ability in "unseen" categories. This is precisely the same problem that the industry needs to solve in general image retrieval tasks. Specifically, the categories used in the training and test sets should be different. To address this, the first problem to solve is how to bridge the gap between categories and transfer the knowledge learned on the training set to the "unseen" categories to achieve model generalization in "unseen" categories. Existing methods primarily focus on performance in the "visible" category, neglecting the performance of knowledge transfer learned on the training set in the "unseen" category, thus limiting their performance in this category.

[0008] Existing methods [3] point out that excessive compression of the distance between samples in the feature space will impair the generalization of models for tasks such as image retrieval where the training and test sets have large differences in distribution. In terms of food image retrieval, due to factors such as geography and culture, there are two different domains in food, such as Eastern cuisine and Western cuisine. Specifically, Western cuisine is often visually represented as a simple combination of various cooked ingredients, while Eastern cuisine is usually represented as a further processing of various ingredients mixed together. For example, the visual difference between Guo Bao Rou (sweet and sour pork), which belongs to Eastern cuisine, and steak, which belongs to Western cuisine, is very obvious. In addition, the differences between different food categories within the same domain may also be very obvious. Taking the ETH Food-101 dataset as an example, steak, ice cream, and chicken wings, which are all Western cuisines, have large visual differences due to differences in cooking methods, raw materials, and other attributes. This is also a specific manifestation of the complex classification system that food tasks have, unlike other general fine-grained tasks. When these large-difference categories belong to the training set and the test set, the difference between the training set and the test set is further amplified, further increasing the difficulty of the problem being studied.

[0009] In general, existing deep metric learning methods often improve performance by designing training sample construction methods and loss functions suitable for specific tasks. Among them, the existing training sample construction methods optimize the network by bringing the anchor image closer to the positive sample pair and pushing it further away from the negative sample. However, this method forces the same optimization intensity to be applied to all positive samples, resulting in the sample features corresponding to the same type of sample being over-compressed in the feature space. The sample features of the same type of sample are close to each other in the feature space, which affects the generalization of the model.

[0010] References:

[0011] [1]Spyromitros-Xioufis E,Papadopoulos S,Kompatsiaris IY,et al.AComprehensive Study Over VLAD and Product Quantization in Large-Scale ImageRetrieval[J].IEEE Transactions on Multimedia,2014,16(6):1713-1728.

[0012] [2]G.Ciocca,P.Napoletano,and R.Schettini.2017.Learning CNN-basedFeatures for Retrieval of Food Images.In InternationalConference on ImageAnalysis and Processing.426–434.

[0013] [3]K.Roth,T.Milbich,S.Sinha,P.Gupta,B.Ommer,andJ.Cohen.2020.Revisiting Training Strategies and Generalization Performance inDeep Metric Learning.In International Conference on Machine Learning.8242–8252. Summary of the Invention

[0014] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a training method and an image retrieval method for a model used for food image classification.

[0015] According to a first aspect of the present invention, a training method for a model for food image classification is provided, the model including a feature extractor and a classifier, the method comprising: S1, acquiring an initial training set containing multiple samples and corresponding labels and dividing it into multiple batches, constructing multiple triples containing anchor samples, positive samples and negative samples based on the samples in the current batch in each batch, wherein each sample is a food image and the label corresponding to the sample indicates the category of the food image; S2, optimizing a portion of the triples selected in each batch of the initial training set, wherein during optimization, the negative samples of the selected triples are replaced with their positive samples and the positive samples of the selected triples are replaced with their anchor samples to obtain optimized triples, and an optimized training set is formed based on the optimized triples and the unselected triples; S3, training the feature extractor for extracting sample features and the classifier for classifying samples according to sample features using the optimized training set in batches, wherein during training, the parameters of the feature extractor and the classifier are updated according to the total loss determined by the classification cross-entropy loss and the distance loss corresponding to the triples.

[0016] In some embodiments of the present invention, in step S3, the feature extractor for extracting sample features and the classifier for classifying samples based on sample features are trained in batches using samples from each batch of the initial training set and samples from the corresponding batches in the optimized training set. During training, the parameters of the feature extractor and the classifier are updated based on the total loss determined by the classification cross-entropy loss corresponding to samples in the initial training set and the distance loss corresponding to triples in the optimized training set.

[0017] Preferably, the anchor samples of all triples in the same batch are from the same image.

[0018] In some embodiments of the present invention, in step S2, some triples are selected for optimization according to the selection probability corresponding to each triple, wherein the selection probability corresponding to each triple is negatively correlated with the distance between the anchor sample and the positive sample in the triple.

[0019] In some embodiments of the invention, during training in each batch, the distance loss is configured to be positively correlated with the logarithm of the distance between the anchor sample and the positive sample of the current batch of samples in the optimized training set.

[0020] Preferably, the distance loss is determined in the following manner:

[0021]

[0022] Where K represents the number of triples in the training set optimized in the current batch, and k represents the triple number of the sample in the training set optimized in the current batch. This represents the distance between the anchor sample and the positive sample in the k-th triplet of the optimized training set in the current batch, where m represents the preset margin. This represents the distance between the anchor sample and the negative sample in the k-th triplet of the optimized training set samples in the current batch.

[0023] In some embodiments of the present invention, in each batch of training, the total loss is configured to be positively correlated with the classification cross-entropy loss corresponding to the samples of the initial training set of the current batch and with the distance loss corresponding to the optimized training set of the current batch.

[0024] Preferably, the total loss is determined as follows:

[0025] L=αL c +(1-α)L m

[0026] Among them, L c L represents the classification cross-entropy loss corresponding to the initial training set samples in the current batch. m Let α represent the distance loss corresponding to the samples in the optimized training set for the current batch. c Preset weights.

[0027] According to a second aspect of the present invention, a food image classification method is provided, the method comprising: acquiring a food image to be classified; and classifying the food image to be classified using a model for food image classification trained by the method described in the first aspect of the present invention.

[0028] According to a third aspect of the present invention, a food image retrieval method is provided, the method comprising: acquiring a food image to be retrieved; classifying the food image to be retrieved using a model for food image classification trained by the method described in the first aspect of the present invention to obtain a category of the food image; and retrieving food images under the category of the food image from a predetermined food image database.

[0029] According to a fourth aspect of the present invention, a food image retrieval method is provided, the method comprising: acquiring a food image to be retrieved; extracting image features corresponding to the food image to be retrieved using a feature extractor in a food image classification model trained by the method described in the first aspect of the present invention; and determining a retrieval result based on the similarity between the image features corresponding to the food image to be retrieved and the image features corresponding to each image in a predetermined food image database.

[0030] Compared with the prior art, the advantages of the present invention are as follows:

[0031] This invention proposes a training method for a food image classification model. By optimizing the training set, it avoids excessive distance compression of samples in the feature space, thereby improving the model's generalization ability. Furthermore, based on this, this invention improves the contrastive loss function in the traditional metric learning loss function to further avoid excessive distance compression of samples in the feature space and enhance the model's generalization ability. Attached Figure Description

[0032] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0033] Figure 1 This is a schematic diagram of a training method for a food image classification model according to an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram illustrating the optimization of selected triples according to an embodiment of the present invention;

[0035] Figure 3 a is a schematic diagram of a traditional optimization method for training set samples according to an embodiment of the present invention;

[0036] Figure 3 b is a schematic diagram of the improved optimization method for the training set according to an embodiment of the present invention;

[0037] Figure 4 a is a schematic diagram of a local training process of a model for food image classification according to an embodiment of the present invention;

[0038] Figure 4 b is a schematic diagram of the overall training process of a model for food image classification according to an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] As mentioned in the background section, existing training methods for neural network models lead to excessive compression of sample features corresponding to similar samples in the feature space. The close proximity of sample features among similar samples in the feature space negatively impacts the model's generalization ability. Therefore, the inventors propose a training method for a food image classification model. First, based on an initial training set, this invention constructs multiple traditional triples from food images, including anchor samples, positive samples (of the same class as the anchor samples), and negative samples (of different classes than the anchor samples). Then, a subset of these triples is selected for optimization. During optimization, the negative samples of the selected triples are replaced with their positive samples to obtain optimized triples. The optimized triples, along with the unselected triples, constitute an optimized training set. At this point, the optimized training set contains optimized triples. For these optimized triples, the distances between their corresponding features are calculated... Distance loss, nominally, aims to reduce the distance between the features corresponding to the anchor sample and positive samples in the feature space, and increase the distance between the features corresponding to the anchor sample and negative samples. However, since the positive samples in the optimized triples become samples completely identical to the anchor samples, and the negative samples are actually samples of the same category as their anchor samples, when these samples are used for training and distance loss calculation, it effectively penalizes images of the same category that are too close together. To a certain extent, it widens the distance between their corresponding features in the feature space, preventing the features of samples of the same category from being over-compressed, thus allowing the model to achieve better generalization. Furthermore, the accuracy of image retrieval based on this model can also be improved.

[0041] To better understand this invention, the following detailed description, in conjunction with the accompanying drawings and embodiments, covers four aspects: model structure, training samples, model training, and application scenarios.

[0042] I. Model Structure

[0043] According to one embodiment of the present invention, a model for food image classification is provided. The model includes a feature extractor and a classifier. The feature extractor extracts features from samples to generate sample features, and the classifier classifies samples based on these features. The feature extractor can employ a backbone network from existing neural networks, such as the ResNet, AlexNet, VGG, or Transformer networks, or a user-defined feature extractor. In this invention, since Transformer networks are better able to establish correlations between targets, and multi-objective tasks in food image processing are suitable for this network structure, the present invention preferably uses a Transformer network (there are many available versions of Transformer networks, such as the DeiT-small model). The classifier includes a linear layer (e.g., one or more fully connected layers) for linearly transforming sample features to obtain classification features, and a classification layer for classifying based on the classification features. For binary classification, a logit layer may be used; for multi-class classification, a Softmax layer may be used.

[0044] II. Training Samples

[0045] According to one embodiment of the present invention, the dataset of the present invention can adopt an existing food image dataset, wherein the samples are food images, and the labels corresponding to the samples indicate the category of the food images. The category of food images can have various definitions, and the present invention does not impose any limitations on this; however, for ease of understanding, the following illustrative embodiments are given.

[0046] According to one embodiment of the present invention, the dataset includes multiple food image samples and labels indicating the category of the food image for each sample. The food image categories include all categories or combinations thereof from the ETH Food-101 dataset, the Vireo Food-172 dataset, and the ISLA Food-500 dataset. Before training, the dataset is typically divided into training and test sets in a certain ratio, such as a 7:3 ratio. (The ETH...) Taking the Food-101 dataset as an example, it contains images of 101 food categories, with 1000 images for each category. The food categories include: apple pie, American-style roasted pork ribs, baklava, Italian raw beef, beef tartare, Georgian salad, bennet, bibimbap, bread pudding, breakfast tortilla rolls, Italian toast, Caesar salad, cream pancake rolls, Italian tomato salad, carrot cake, ceviche, cheesecake, cheese platter, chicken curry, chicken patties, chicken wings, chocolate cake, chocolate mousse, churros, clam chowder, chafing, crab cakes, caramel cream, French sandwiches, cupcakes, deviled eggs, donuts, dumplings, Japanese edamame, eggs Benedict, escargot, fried bean balls, filet mignon, fish and chips, chips, French onion soup, French toast, and fried calamari rings. Fried rice, frozen yogurt, garlic bread, Italian dumplings, Greek salad, grilled cheese sandwiches, grilled salmon, Mexican guacamole, Japanese dumplings, hamburgers, hot and sour soup, hot dogs, tortillas, hummus, ice cream, lasagna, lobster bisque, lobster sandwiches, macarons, miso soup, oysters, nachos, onion rings, oysters, Thai stir-fried noodles, Spanish risotto, pizza, panna cotta, Peking duck, pizza, Vietnamese pho, pork chops, gravy fries, grilled beef short ribs, American BBQ shredded pork sandwiches, ramen, ravioli, red velvet cake, risotto, samosas, sashimi, scallops, seaweed salad, shrimp and corn porridge, spaghetti bolognese, spring rolls, steak, strawberry cake, sushi, tacos, takoyaki, tiramisu, tuna tartare, and waffles.

[0047] It should be understood that the food image categories in the above embodiments are merely illustrative, and categories from other existing food image datasets or user-defined categories can also be used; those skilled in the art can adjust and set them as needed, and the present invention does not impose any restrictions on this.

[0048] III. Model Training

[0049] According to an embodiment of the present invention, the present invention provides a training method for a model for food image classification, the model including a feature extractor and a classifier, the method including three steps S1, S2 and S3, as shown in the appendix. Figure 1 As shown, the following sections will introduce them separately.

[0050] In step S1, an initial training set containing multiple samples and corresponding labels is obtained and divided into multiple batches. In each batch, multiple triples containing anchor samples, positive samples and negative samples are constructed based on the samples in the current batch. Each sample is a food image, and the label corresponding to the sample indicates the category of the food image. The anchor sample of all triples in the same batch is the same image.

[0051] According to one embodiment of the present invention, after dividing the initial training set into multiple batches, multiple triples are constructed in each batch. Each triple contains an anchor sample, a positive sample, and a negative sample, in the form of {anchor sample, positive sample, negative sample}. It should be understood that the definitions of anchor sample, positive sample, and negative sample in traditional triples are well-known, where a positive sample is a sample belonging to the same class as the anchor sample, and a negative sample is a sample belonging to a different class than the anchor sample. Preferably, to avoid interference from inter-class distance loss that may be caused by multiple different anchor samples in a batch, and to improve model performance, the anchor sample of all triples in the same batch can be set to the same food image.

[0052] In step S2, a portion of the selected triples in each batch of the initial training set are optimized. During optimization, the negative samples of the selected triples are replaced with their positive samples, and the positive samples of the selected triples are replaced with their anchor samples. The optimized training set is formed based on the selected triples and the remaining unselected triples.

[0053] In existing technologies, conventional sample construction methods construct triples in which all positive samples belong to the same category as the anchor sample, and all negative samples belong to different categories. During training, this causes the model to continuously update its parameters by narrowing the distance between the anchor sample and positive samples while simultaneously widening the distance between the anchor sample and negative samples. This causes all positive samples to continuously converge towards the features of their anchor samples, leading to excessive compression of corresponding features in the feature space and thus affecting the model's generalization ability. To address this, according to an embodiment of the present invention, optimization is achieved by selecting a subset of triples from each batch of triples in the initial training set, i.e., ρ-sampling. For example, suppose a triplet is {anchor sample (image A), positive sample (image B), negative sample (image C)}. If this triplet is selected, its corresponding optimized triplet will be {anchor sample (image A), positive sample (image A), negative sample (image B)}. This involves replacing the negative sample (corresponding food image) of the selected triplet with its positive sample (corresponding food image) and replacing the positive sample (corresponding food image) of the selected triplet with its anchor sample (corresponding food image) to obtain the optimized triplet. See the appendix for details. Figure 2As shown. This is equivalent to optimizing a selected subset of triples, where the anchor sample, positive sample, and negative sample in the optimized triples all have the same label, the anchor sample and positive sample are the same image, and the negative sample is an image different from the anchor sample. Ultimately, the set of the selected optimized triples and the set of unselected triples constitutes the optimized training set. In other words, the inventors have proposed a new method for constructing triples, which avoids excessive compression of samples in the feature space by maximizing partial intra-class distances during training, thereby improving the model's generalization ability. Considering that excessive distance compression of samples in the feature space leads to reduced model generalization, this invention provides a method for optimizing partial triples during training, or a new sampling method (hereinafter referred to as ρ-sampling). When the training set obtained in this way is used to train the model, because the negative samples that were originally pushed away are replaced with positive samples, it can, to some extent, avoid excessive compression of the features of anchor samples and positive samples that actually belong to the same class in the feature space. For easier understanding, see [link to relevant documentation]. Figure 3 a, among which Figure 3 Figure 'a' illustrates the change in sample distribution under traditional optimization methods (only positive and anchor samples are shown in the figure), where black and white dots represent different categories, and arrows indicate optimization directions. c The anchor sample represents the sample features corresponding to the anchor sample. x1, x2, and x3 represent the sample features corresponding to different positive samples in the same category. As shown on the left, after the samples are optimized using the traditional optimization method, the distribution changes as shown on the right. At this time, the boundaries between different categories are clearer. At the same time, all positive samples move closer to the central anchor sample, but this can lead to over-optimization, making the distance between different samples too small.

[0054] When performing ρ sampling, the triples to be optimized can be selected indiscriminately. According to one embodiment of the present invention, the triples are selected by assigning each triple the same probability ρ of being selected. The value of ρ can be, for example, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5 or other values. The present invention preferably uses a value of 0.2 for ρ.

[0055] However, indiscriminately selecting triples to optimize may further push away samples whose features are already far from the anchor sample's features in the feature space, affecting inter-class discrimination. To address this, according to one embodiment of the present invention, a subset of triples are selected for optimization based on the selection probability corresponding to each triple. The selection probability of each triple is negatively correlated with the distance between the anchor sample and the positive sample in that triple. See the specific illustration. Figure 2As shown, during ρ-sampling, this invention assigns a relatively higher probability of selection to triples in which the anchor sample is relatively closer to the positive sample. Specifically, the probability of a triple being selected decreases with respect to the distance between the positive sample it contains and the anchor sample, ensuring that samples with higher confidence are selected with a greater probability.

[0056] For ease of understanding, see Figure 3 As shown in b, where... Figure 3 b illustrates the change in sample distribution after ρ sampling. ρ sampling can appropriately increase the distance between the anchor sample and the positive sample. After ρ sampling, the original x2 and the anchor sample x... c The distance between the anchor sample and the other samples x1 and x3 is increased, while the distance between the anchor sample and the other samples x1 and x3 is maintained at a relatively appropriate size. By maintaining a certain intra-class variance, excessive intra-class compression is avoided, thus improving the generalization of the model. The technical solution of this embodiment can achieve at least the following beneficial technical effects: Since the distribution of the distance between samples in the high-dimensional feature space tends to be a Gaussian distribution centered at the square root of 2, if all triples are assigned the same probability for selection, a large number of positive samples with low confidence will be selected, which will limit the performance improvement of the model in subsequent training. Therefore, when selecting, it is preferable to select more triples containing positive samples that are closer to the class center for optimization. These triple samples have higher confidence, and the negative impact of back-optimizing these samples is also smaller. In subsequent training, the distance loss based on the triple can maintain the intra-class distance and avoid excessive distance compression of samples in the feature space by widening the intra-class distance of these samples through optimized triples, thereby improving the generalization of the model.

[0057] In step S3, the feature extractor for extracting sample features and the classifier for classifying samples based on sample features are trained in batches using the optimized training set. During training, the parameters of the feature extractor and the classifier are updated based on the total loss determined by the classification cross-entropy loss and the distance loss corresponding to the triples.

[0058] Existing methods also update the parameters of the feature extractor and classifier based on the total loss calculated from the classification cross-entropy loss and the distance loss corresponding to the triples. However, in each batch, the optimization intensity of positive samples with different distances from the anchor sample is the same when calculating the distance loss. Relatively speaking, this leads to positive samples with lower confidence being given higher weights, which will optimize the model parameters more quickly and bring these positive samples closer to the anchor sample. However, this is not conducive to the purpose of this invention to avoid intra-class over-compression. Therefore, according to an embodiment of the present invention, in the training of each batch, the total loss (which can be called the gradient adaptive loss function) is calculated as follows:

[0059] L=αL c +(1-α)L m

[0060] Among them, L c L represents the classification cross-entropy loss calculated based on all samples in the current batch. m Let α represent the distance loss calculated for each triple in the training set optimized based on the current batch, denoted as L. c Preset weights; wherein the distance loss is configured to be positively correlated with the logarithm of the distance between the anchor sample and the positive sample in the current batch of samples in the optimized training set. For example, the distance loss in the total loss is calculated as follows:

[0061]

[0062] Where K represents the number of triples in the current batch of optimized training set, and k represents the triplet number of the sample in the current batch of optimized training set. This represents the distance between the anchor sample and the positive sample in the k-th triplet of the optimized training set in the current batch, where m represents the preset margin. This represents the distance between the anchor sample and the negative sample in the k-th triplet of the optimized training set samples in the current batch. Preferably, the weight α is set to 0.8, 0.85, or 0.9. The margin value m used in this invention is set to 0.1, 0.15, or 0.2. For the calculation of the corresponding distance loss, see, for example, the appendix. Figure 4 As shown in Figure a, for each triplet sampled by ρ, a Transformer network is used to obtain the sample features φ corresponding to each sample in that triplet. a (Sample characteristics of anchor samples), φ p (Sample characteristics of positive samples), φ n (Sample features of negative samples). Based on these features, calculate the sample distance within each triplet to obtain D. ap (Distance between positive samples and anchor samples) and D an (Distance between negative samples and anchor samples), wherein the sample distance can be calculated using methods including but not limited to cosine similarity, Euclidean distance, etc., and the present invention preferably uses cosine similarity to calculate the sample distance. It should be noted that since the calculation of cosine similarity between sample features is well known in the art, the specific calculation process involving cosine similarity will not be described here.

[0063] According to one embodiment of the present invention, the classification cross-entropy loss in the total loss is well known in the art, i.e., it is calculated by inputting all samples in the current training batch into the model and using the corresponding labels. See again the appendix. Figure 4As shown in Figure a, the Transformer network is used during computation to obtain the sample features φ corresponding to each sample in the current batch. cls Then, the sample features are input into the fully connected layer and the classification layer in the classifier to obtain the classification result (during training, if it is a binary classification, it corresponds to the logit; if it is a multi-class classification, it corresponds to the confidence probability of each class). The classification result is then compared with the benchmark (i.e., the label ground truth) to calculate the classification cross-entropy loss.

[0064] It should be understood that the above embodiments correspond to the optimal implementation of the present invention, but are not the only implementation. Implementers may make adjustments as needed to obtain other feasible implementations. For calculating the distance loss and classification crossover loss in the total loss, some other feasible implementations are described below. It should be noted that the implementations described below can achieve the technical effects of the present invention to a certain extent compared to existing technologies, namely, avoiding excessive distance compression of sample features in the feature space.

[0065] Without considering the number of samples seen by the model, the model can be trained entirely using the optimized training set. According to another embodiment of the invention, when training the model using the optimized training set, the parameters of the feature extractor and classifier are updated based on the classification cross-entropy loss corresponding to all samples in each batch of the optimized training set and the distance loss corresponding to all triples in each batch. In this embodiment, since the classification cross-entropy loss is calculated based on the optimized training set, and the negative samples in the selected triples for optimization are replaced with positive samples during optimization, the number of samples seen by the model may decrease. Therefore, the aforementioned embodiment's method of using the initial training set (i.e., the unselected triples for optimization) to calculate the classification cross-entropy loss for each batch of samples is more effective. However, it should be understood that this embodiment is also feasible in principle.

[0066] Without considering the optimization strength of positive samples at different distances from the anchor sample, according to one embodiment of the present invention, this embodiment differs from the previous embodiment in that the distance loss of the previous embodiment is replaced by calculation in the following manner:

[0067]

[0068] The meanings of the parameters are the same as those in the distance loss section above, and will not be explained again here. Essentially, the distance loss function proposed in this embodiment, compared to the previous embodiment, has a different effect on D. apThe logarithmic constraint of ln has been removed. For ease of comparison, the contrastive loss function in traditional deep metric learning methods will be referred to as the unimproved distance loss function (i.e., the total loss function in the aforementioned embodiment is replaced with the distance loss function of this embodiment), and the distance loss function proposed in the aforementioned embodiment will be referred to as the improved distance loss function. See Appendix again. Figure 4 As shown on the right side of b, compared with the original distance loss function, the improved loss function has D... ap The corresponding gradient varies with D ap As the gradient of the loss function increases, the optimization intensity decreases accordingly. Since the gradient of the loss function corresponds to the optimization strength for a given sample, the optimization strength also decreases when the gradient decreases. The distance loss function before the improvement... ap The corresponding gradient remains fixed and unaffected by its value. The improved distance loss function assigns smaller weights to positive samples with lower confidence, thus slowly optimizing them and further avoiding excessive compression of sample features in the feature space. However, the original distance loss function remains feasible in principle after adopting the optimized triplet.

[0069] It should be understood that in practical applications, the two distance loss calculation methods mentioned in the above embodiments can be combined pairwise with the two classification cross-entropy loss calculation methods to construct a total loss function, and the feature extractor and classifier can be trained to update the parameters of the feature extractor and classifier; thus generating more implementation methods.

[0070] IV. Application Scenarios

[0071] In summary, this invention improves upon existing deep metric learning frameworks, resulting in a model for food image classification with better generalization capabilities. Furthermore, the accuracy of image retrieval based on this model can also be enhanced.

[0072] According to one embodiment of the present invention, the present invention provides a food image classification method, the method comprising: acquiring a food image to be classified; and classifying the food image to be classified using a model for food image classification trained by the aforementioned method.

[0073] According to one embodiment of the present invention, the present invention also provides a food image retrieval method, the method comprising: acquiring a food image to be retrieved; classifying the food image to be retrieved using a model for food image classification trained by the aforementioned method to obtain a category of the food image; and retrieving food images under the category of the food image from a predetermined food image database.

[0074] The following describes a method for retrieval using image classification labels. Taking the ETH Food-101 dataset as the original dataset, 70% of the food images (i.e., 700 images) from each of the 101 categories (each category contains 1000 food images) can be used for training, and the remaining 30% (i.e., the remaining 300 images) can be used for testing. After training the model using the aforementioned method, any one of the three food image datasets can be selected from the test set as the input query, and the model will output all food images under its corresponding label. In practical applications, any one of the three food image datasets mentioned above can be used to construct the training and test sets, or a combination thereof. In addition to using the ETH Food-101, Vireo Food-172, and ISIA Food-500 datasets listed above, a user-defined dataset can also be used for model training and testing. Besides the method of retrieval using classification labels, similarity can also be used for food image retrieval. According to one embodiment of the present invention, the present invention also provides a food image retrieval method, the method comprising: acquiring a food image to be retrieved; extracting image features corresponding to the food image to be retrieved using a feature extractor in a food image classification model trained by the aforementioned method; and determining a retrieval result based on the similarity between the image features corresponding to the food image to be retrieved and the image features corresponding to each image in a predetermined food image library. It should be understood that the trained samples are input into the feature extractor to obtain sample features, and the food image to be retrieved is input into the feature extractor to obtain image features; however, for the sake of distinction, both sample features and image features are outputs of the feature extractor, and their size and data format are the same. Furthermore, to improve retrieval efficiency, the image features corresponding to each image in the food image library can be obtained by pre-inputting each image in the food image library into the feature extractor in the food image classification model trained by the aforementioned method, and then storing each image and its corresponding image features in association in the food image library for later retrieval. This improves retrieval efficiency.

[0075] To verify the effectiveness of the present invention, the inventors also conducted experiments on the above-mentioned method of food image retrieval using similarity, and obtained the recall results corresponding to the image retrieval accuracy shown in the table below. Table 1 shows the comparison of the final retrieval accuracy using the ETHFood-101, Vireo Food-172, and ISIA Food-500 datasets as test sets (70% of all categories in the three datasets were selected for training, i.e., 70, 120, and 350 categories respectively, and the remaining 30% of categories were used as "unseen" categories for testing), based on the Transformer network as the backbone network, using existing sampling methods for food image retrieval, using the ρ-sampling proposed in this invention, and using a combination of ρ-sampling and an improved loss function for food image retrieval; Table 2 shows the comparison of the final retrieval accuracy using the ETH Food-101 dataset for training and the Vireo Food-172 dataset for testing, and using the Vireo Food-172 dataset for training and the ETH Food-101 dataset for testing. In the table, R@1, R@2, and R@4 represent recall (%), and NMI represents normalized mutual information (%). It should be noted that the methods of this invention in Tables 1 and 2 refer to the optimal implementation method corresponding to the aforementioned gradient adaptive loss function.

[0076] Table 1

[0077]

[0078] Table 2

[0079]

[0080] It can be seen that, for image feature-based retrieval methods, the retrieval accuracy obtained by the method provided by this invention is higher than that of existing methods.

[0081] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0082] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0083] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0084] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a model used for food image classification, the model comprising a feature extractor and a classifier, characterized in that, The method includes: S1. Obtain an initial training set containing multiple samples and corresponding labels and divide it into multiple batches. In each batch, construct multiple triplets containing anchor samples, positive samples, and negative samples based on the samples in the current batch. Each sample is a food image, and the label corresponding to the sample indicates the category of the food image. S2. For each batch of the initial training set, select a portion of triples for optimization. During optimization, replace the negative samples of the selected triples with their positive samples and replace the positive samples of the selected triples with their anchor samples to obtain optimized triples. The optimized triples and the unselected triples constitute an optimized training set. S3. Train the feature extractor for extracting sample features and the classifier for classifying samples based on sample features in batches using the optimized training set. In S3, the feature extractor used to extract sample features and the classifier used to classify samples based on sample features are trained in batches using samples from each batch of the initial training set and samples from the corresponding batches in the optimized training set. During training, the parameters of the feature extractor and the classifier are updated based on the total loss determined by the classification cross-entropy loss corresponding to samples in the initial training set and the distance loss corresponding to triples in the optimized training set. The distance loss is determined as follows: in, This indicates the number of triples in the training set being optimized in the current batch. This indicates the triplet number of the sample in the optimized training set for the current batch. This represents the first sample in the optimized training set of the current batch. The distance between the anchor sample and the positive sample in each triplet This indicates the preset margin. This represents the first sample in the optimized training set of the current batch. The distance between the anchor sample and the negative sample in each triplet.

2. The method according to claim 1, characterized in that, Anchor samples for all triples in the same batch are from the same image.

3. The method according to claim 2, characterized in that, In S2, some triples are selected for optimization according to the selection probability corresponding to each triple, wherein the selection probability corresponding to each triple is negatively correlated with the distance between the anchor sample and the positive sample in the triple.

4. The method according to claim 3, characterized in that, In each batch of training, the distance loss is configured to be positively correlated with the logarithm of the distance between the anchor sample and the positive sample of the current batch of samples in the optimized training set.

5. The method according to any one of claims 1-4, characterized in that, In each batch of training, the total loss is configured to be positively correlated with the classification cross-entropy loss corresponding to the samples of the initial training set of the current batch and with the distance loss corresponding to the optimized training set of the current batch.

6. The method according to claim 5, characterized in that, The total loss shall be determined as follows: in, This represents the classification cross-entropy loss corresponding to the samples in the initial training set of the current batch. This represents the distance loss corresponding to the samples in the optimized training set for the current batch. Represented as Preset weights.

7. A food image classification method, characterized in that, The method includes: Obtain images of the food to be classified; The food images to be classified are classified using a model for food image classification trained by the method described in any one of claims 1-6.

8. A food image retrieval method, characterized in that, The method includes: Obtain the image of the food to be searched; The food image to be retrieved is classified using a model for food image classification trained by any one of claims 1-6, thereby obtaining the category of the food image. Retrieve food images of the specified food category from a predefined food image library.

9. A food image retrieval method, characterized in that, The method includes: Obtain the image of the food to be searched; The feature extractor in the food image classification model trained by any one of claims 1-6 is used to extract the image features corresponding to the food image to be retrieved. The search results are determined based on the similarity between the image features corresponding to the food image to be searched and the image features corresponding to each image in a pre-defined food image database.

10. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-6, 7, 8 and 9.

11. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1-6, 7, 8, and 9.

Images