A method for retaining samples of dishes

Abstract

A method for leaving samples of dishes disclosed by the present invention determines whether a target dish is in the original dish total library; if so, calculates a similarity value with the pictures of dishes of this category in the original total library; forms a first sample-keeping picture group with the pictures of this category in the original total library that are greater than or equal to the preset similarity threshold, otherwise forms a second sample-keeping picture group; determines whether the number of pictures in the first sample-keeping picture group is greater than zero, if so, extracts the first recognition feature through the self-attention mechanism method, otherwise extracts the first recognition feature through the self-attention mechanism method after expanding the number of pictures; determines whether the number of pictures in the second sample-keeping picture group is greater than zero, if so, extracts the second recognition feature through the mutual information method, otherwise extracts the second recognition feature through the mutual information method after expanding the number of pictures. If not, after expanding the number of pictures, the first recognition feature is extracted through the self-attention mechanism method and the second recognition feature is extracted through the mutual information method respectively. Finally, the two features are retained as the recognition features of dishes of this category for storage.

Description

Technical Field

[0001] The present invention relates to the technical field of dish sample retention, and particularly to a method for retaining dish samples. Background Art

[0002] In order to identify dishes during settlement, restaurants or canteens usually have special personnel responsible for sample retention. After sampling each portion of food according to relevant regulations, it is immediately placed in a disinfected sample retention tableware, and a label is attached externally, indicating the sample retention date and time, dish name, meal number, and sample retention person in charge; the labeled sample retention food is stored in a dedicated constant temperature refrigerator for refrigeration according to regulations, and finally, the relevant information of each sample retention food for each meal is registered and recorded.

[0003] Currently, the sample retention feature used in the intelligent sample retention method is a single feature after weighted average. Since the cooking methods of the same dish are different in different meal periods and there are differences, it is bound to result in the loss of a large amount of important feature information, resulting in a low dish recognition rate during settlement. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for retaining dish samples that can improve the dish recognition rate in view of the defects of the prior art.

[0005] The method for retaining dish samples according to the present invention includes:

[0006] Obtain a picture of the dish to be sampled, and obtain a target dish area picture based on the picture of the dish to be sampled; determine whether the dish category of the target dish area picture is in the original total dish library category;

[0007] If it is in the original total dish library category, calculate the similarity value between the target dish area picture and the pictures of the dishes in this category in the original total library;

[0008] Form a first sample retention picture group with the pictures in the original total library whose similarity values are greater than or equal to the preset similarity threshold, and determine whether the number of pictures in the first sample retention picture group is greater than zero. If it is greater than zero, add the target dish area picture to the first sample retention picture group and extract the first recognition feature through the self-attention mechanism method. Otherwise, add the target dish area pictures of the expanded preset number of samples to the first sample retention picture group, and extract the first recognition feature through the self-attention mechanism method. After completion, clear the expanded data;

[0009] Form a second sample retention picture group with the pictures in the original total library whose similarity values are less than the preset similarity threshold, and determine whether the number of pictures in the second sample retention picture group is greater than zero. If it is greater than zero, add the target dish area picture to the second sample retention picture group and extract the second recognition feature through the mutual information method. Otherwise, add the target dish area pictures of the expanded preset number of samples to the first sample retention picture group, and extract the second recognition feature through the mutual information method. After completion, clear the expanded data;

[0010] The first identification feature and the second identification feature extracted are both stored in the identification feature library as the sample identification features of the dishes of this category;

[0011] If it is not in the original dish library category, then add the target dish area map with a preset number of retained samples to form a retained sample picture group;

[0012] The first and second identification features corresponding to the dishes of this category are extracted from the retained sample image group through the attention mechanism method and the mutual information method respectively, and stored. After completion, the expanded data is cleared.

[0013] As a preferred solution of the present invention, the self-attention mechanism method extracts the first recognition feature as follows:

[0014] Assume that there are N sample images in the first sample image group, and use the recognition network to extract the features of each image. i (1≤i≤N), the feature dimension of each image is 1×1536, that is, features i (1×1536);

[0015] The features of N images are stacked to form a tensor X with a dimension of N×1×1536, and X is transformed into a tensor with a dimension of N×1536;

[0016] Using X as the input of query Q, key K and value V, the self-attention mechanism formula is as follows:

[0017]

[0018] Among them, the dimensions of Q, K, and V are all N×1536, dk represents the dimension of the key, which is 1536, and the scalar factor used to scale the similarity score is Through the above formula, a first recognition feature featuresA with a dimension of 1×1536 can be obtained.

[0019] As a preferred solution of the present invention, the mutual information method is used to extract the second identification feature as follows:

[0020] Assume that there are N retained sample images in the second retained sample image group, and use the recognition network to extract the features featuresi (1≤i≤N) of each image. The feature dimension of each image is 1×1536, that is, featuresi (1×1536);

[0021] The feature vector featuresi of each image extracted above is considered as a random variable xi with a dimension of (1×1536), where i represents the i-th image;

[0022] Calculate the mutual information value I(xi,xj) between each pair of images using the mutual information formula, where i and j represent the indices of the corresponding feature values of the images respectively, and the formula is:

[0023]

[0024] Among them, p(xi,xj) is the joint probability that the image features Xi and Xj take one of their values simultaneously, and p(xi) and p(xj) are the marginal probability distributions of the image features xi and xj respectively;

[0025] According to the calculated mutual information value I(xi,xj), a correlation matrix M between N×N image features can be obtained, where Mij represents the mutual information between images i and j;

[0026] Normalize the values in the correlation matrix M, and then the weights between each pair of images can be obtained, and their range is between 0 and 1;

[0027] Apply the weights to the feature vectors of the images, and fuse the image features with high correlation in a weighted average manner. Specifically, multiply each feature dimension by the weight of the corresponding image, and sum the weighted feature vectors of all images. The formula is:

[0028]

[0029] Among them, weighti is the weight of image i, and featuresi dimension is the feature vector of image i;

[0030] Through the above formula, a second recognition feature featuresB with a dimension of 1×1536 can be obtained.

[0031] Advantages of the present invention:

[0032] First, determine whether the dish category in the target dish area map is in the original total dish library category. If it is, calculate the similarity value between the target dish area map and the dish pictures of this category in the original total library. Form the first retained sample picture group with the pictures of this category in the original total library whose similarity values are greater than or equal to the preset similarity threshold, and form the second retained sample picture group with the pictures of this category in the original total library whose similarity values are less than the preset similarity threshold. Then, determine whether the number of pictures in the first retained sample picture group is greater than zero. If it is greater than zero, add the target dish area map to the first retained sample picture group and extract the first recognition feature through the self-attention mechanism method. Otherwise, add the target dish area map with the preset number of retained samples to the first retained sample picture group and extract the first recognition feature through the self-attention mechanism method. After completion, clear the augmented data. Determine whether the number of pictures in the second retained sample picture group is greater than zero. If it is greater than zero, add the target dish area map to the second retained sample picture group and extract the second recognition feature through the self-attention mechanism method. Otherwise, add the target dish area map with the preset number of retained samples to the first retained sample picture group and extract the second recognition feature through the self-attention mechanism method. After completion, clear the augmented data. And store the first recognition feature and the second recognition feature extracted as the retained sample recognition features of this category of dishes in the recognition feature library. It completes an overall similarity threshold judgment, data augmentation and retention of two different features as the retained sample recognition features of this category of dishes for storage, which can greatly improve the recognition rate of various categories of dishes during settlement, thus solving the problem of low dish recognition rate caused by the different cooking methods of the same dish in different meal periods, resulting in differences, and obtaining its single feature through weighted average, resulting in the loss of many important feature information. In addition, if it is not in the original total dish library map, form a retained sample picture group by adding the target dish area map with the preset number of retained samples. Extract the corresponding first recognition feature and second recognition feature of this category of dishes from the retained sample picture group through the attention mechanism method and the mutual information method respectively. After completion, clear the augmented data, and perform data augmentation and retention of two different features as the retained sample recognition features of this category of dishes for storage, further improving the recognition rate of various categories of dishes during settlement. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 FIG. is a schematic flowchart of a method for retaining samples of dishes according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0035] The present invention provides a method for retaining samples of dishes, as Figure 1As shown in the figure, it includes: obtaining the dish picture of the retained sample, and obtaining the target dish area picture according to the dish picture of the retained sample.

[0036] Among them, a camera can be used to take the original picture (original_image) of the dish that needs to be retained as a sample, and an internal algorithm is used for automatic detection and recognition. The detection algorithm frames the target position of the dish retained as a sample. If the target box (target box) cannot be correctly detected, the corresponding dish retained as a sample can be manually framed, and then the framed target dish area picture is provided to the recognition algorithm for recognition.

[0037] Judge whether the dish category of the target dish area picture is in the original dish total library category, and the top N (default set to 10) dish categories (classes) of the original dish total library picture can be given for selection and confirmation.

[0038] If it is in the original dish total library category, calculate the similarity value between the target dish area picture and the pictures of the dishes in this category in the original total library.

[0039] The pictures of the original total library in this category with similarity values greater than or equal to the preset similarity threshold are grouped into the first retained sample picture group, and judge whether the number of pictures in the first retained sample picture group is greater than zero. If it is greater than zero, add the target dish area picture to the first retained sample picture group and extract the first recognition feature through the self-attention mechanism method; otherwise, add the target dish area pictures of the retained samples with the preset number of expansions to the first retained sample picture group, and extract the first recognition feature through the self-attention mechanism method. After completion, clear the expanded data.

[0040] Among them, the self-attention mechanism method for extracting the first recognition feature is as follows: Suppose there are N retained sample pictures in the first retained sample picture group, and the recognition network is used to extract the features features i (1 ≤ i ≤ N) of each picture. The feature dimension of each picture is 1×1536, that is, features i (1×1536);

[0041] Stack the features of the N pictures to form a tensor X with a dimension of N×1×1536, and perform a dimension transformation on X to make it transform into a tensor with a dimension of N×1536;

[0042] Use X as the input of the query Q, key K, and value V, and use the self-attention mechanism formula as follows:

[0043]

[0044] Among them, the dimensions of Q, K, and V are all N×1536, and d k represents the dimension of the key, with a size of 1536, and the scalar factor used to scale the similarity score is

[0045] A first recognition feature featuresA with a dimension of 1×1536 can be obtained through the above formula.

[0046] The original category pictures in the original general library with similarity values less than the preset similarity threshold are grouped into a second retained sample picture group, and it is judged whether the number of pictures in the second retained sample picture group is greater than zero. If it is greater than zero, the target dish area map is added to the second retained sample picture group and the second recognition feature is extracted by the mutual information method. Otherwise, the target dish area map with the preset number of retained samples is added to the first retained sample picture group, and the second recognition feature is extracted by the mutual information method. After completion, the extended data is cleared; the first recognition feature and the second recognition feature extracted are both stored in the recognition feature library as the retained sample recognition features of this category of dishes.

[0047] Among them, the specific process of extracting the second recognition feature by the mutual information method is as follows: Suppose there are N retained sample pictures in the second retained sample picture group, and the recognition network is used to extract the features features i (1≤i≤N) of each picture. The feature dimension of each picture is 1×1536, that is, features i (1×1536);

[0048] The feature vector features i with a dimension of (1×1536) extracted from each picture above is regarded as a random variable x i , where i represents the i-th picture;

[0049] The mutual information value I(x i ,x j ) between each picture is calculated using the mutual information formula, where i and j respectively represent the indexes of the corresponding feature values of the pictures. The formula is:

[0050]

[0051] Among them, p(x i ,x j ) is the joint probability that the picture features X i , X j take one of the values at the same time. p(x i ) and p(x j ) are the marginal probability distributions of the picture features x i and x j respectively;

[0052] According to the calculated mutual information value I(x i ,x j ), a correlation matrix M between N×N picture features can be obtained, and M ij represents the mutual information between pictures i and j;

[0053] By normalizing the values in the correlation matrix M, the weights between each image can be obtained, ranging from 0 to 1.

[0054] The weights are applied to the feature vectors of the images, and the highly correlated image features are fused in a weighted average manner. Specifically, each feature dimension is multiplied by the weight of the corresponding image, and the weighted feature vectors of all images are summed. The formula is:

[0055]

[0056] where weight i is the weight of image i, features i dimension is the feature vector of image i;

[0057] Through the above formula, a second recognition feature featuresB with a dimension of 1×1536 can be obtained.

[0058] It can greatly improve the recognition rate of dishes of various categories at the time of settlement through similarity threshold judgment, data expansion and retention of two different features as sample identification features of dishes of this category, thereby solving the problem of low dish recognition rate caused by the loss of more important feature information due to the different cooking methods of the same dish in different meal stages and the loss of a single feature by weighted average.

[0059] In addition, if the dish is not in the original dish library category, it is expanded by adding a preset number of target dish area maps to form a retained sample picture group. The first identification feature and the second identification feature corresponding to the dish of this category are extracted from the retained sample picture group through the attention mechanism method and the mutual information method respectively. After completion, the expanded data is cleared, and the data expansion and the retained two different features are stored as the retained sample identification features of the dish of this category, which further improves the recognition rate of each category of dishes at the time of settlement.

[0060] As a preferred embodiment of the present invention, in the description of this specification, the description with reference to the term "preferred" or the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0061] The above embodiments are only used to illustrate the detailed solutions of the present invention. The present invention is not limited to the above detailed solutions, that is, it does not mean that the present invention must rely on the above detailed solutions to be implemented. Those skilled in the art should understand that any improvement to the present invention, the equivalent substitution of each raw material of the product of the present invention, the addition of auxiliary components, the selection of specific methods, etc., all fall within the protection scope and the disclosure scope of the present invention.

Claims

1. A method for retaining samples of dishes, characterized in that: Including: Obtain the picture of the sampled dish, and obtain the target dish area picture according to the picture of the sampled dish; Judge whether the dish category of the target dish area picture is in the original total dish library category; If it is in the original total dish library category, calculate the similarity value between the target dish area picture and the pictures of the dishes of this category in the original total library; Form the first sampled picture group with the pictures of this category in the original total library whose similarity values are greater than or equal to the preset similarity threshold, and judge whether the number of pictures in the first sampled picture group is greater than zero. If it is greater than zero, add the target dish area picture to the first sampled picture group and extract the first recognition feature through the self-attention mechanism method. Otherwise, add the target dish area pictures of the preset number of samples to be expanded to the first sampled picture group, and extract the first recognition feature through the self-attention mechanism method. After completion, clear the expanded data; Form the second sampled picture group with the pictures of this category in the original total library whose similarity values are less than the preset similarity threshold, and judge whether the number of pictures in the second sampled picture group is greater than zero. If it is greater than zero, add the target dish area picture to the second sampled picture group and extract the second recognition feature through the mutual information method. Otherwise, add the target dish area pictures of the preset number of samples to be expanded to the first sampled picture group, and extract the second recognition feature through the mutual information method. After completion, clear the expanded data; Both the first recognition feature and the second recognition feature extracted are stored as the sampled recognition features of the dishes of this category in the recognition feature library; If it is not in the original total dish library category, add the target dish area pictures of the preset number of samples to be expanded to form a sampled picture group; Extract the corresponding first recognition feature and second recognition feature of the dishes of this category through the attention mechanism method and the mutual information method for the sampled picture group respectively, and store them. After completion, clear the expanded data.

2. The method for leaving samples of dishes according to claim 1, wherein, The specific process of extracting the first recognition feature by the self-attention mechanism method is as follows: Assume that there are N sample images in the first sample image group, and use the recognition network to extract the features of each image (1≤i≤N), the feature dimension of each image is 1×1536, that is (1×1536); Stack the features of N pictures to form a tensor X with a dimension of N×1×1536, and perform a dimension transformation on X to make it a tensor with a dimension of N×1536; Use X as the input of the query Q, key K, and value V, and use the self-attention mechanism formula as follows: Attention(Q, K, V) = softmax( ).V Among them, the dimensions of Q, K, and V are all N×1536, and d k represents the dimension of the key, which is 1536, and the scalar factor used to scale the similarity score is; Through the above formula, a first recognition feature featuresA with a dimension of 1×1536 can be obtained.

3. The method for leaving samples of dishes according to claim 1, characterized in that, The specific process of extracting the second recognition feature by the mutual information method is as follows: Assume that there are N retained images in the second retained image group, and use the recognition network to extract the features of each image. i (1≤i≤N), the feature dimension of each image is 1×1536, that is, features i (1×1536); The feature vectors features of each image extracted above i with a dimension of (1×1536) are regarded as random variable x i , where i represents the i-th image; Calculate the mutual information value I(x i , x j ) between each pair of images using the mutual information formula, where i and j represent the indices of the corresponding feature values of the images respectively, and the formula is: I( , ) = p( , ) log( ) where p(x i , x j ) is the joint probability of taking one of the values of the image feature X i , X j at the same time, and p(x i ) and p(x j ) are the marginal probability distributions of the image features x i and x j respectively; According to the calculated mutual information value I(x i ,x j ), a correlation matrix M between N×N picture features can be obtained, where M ij represents the mutual information between pictures i and j; Normalize the values in the correlation matrix M, and the weights between each picture can be obtained, and the range is between 0 and 1; Apply the weights to the feature vectors of the pictures to fuse the picture features with high correlation in a weighted average manner. Specifically, multiply each feature dimension by the weight of the corresponding picture, and sum the weighted feature vectors of all pictures. The formula is: featuresB= where weight i is the weight of image i, and features i is the feature vector of image i; Through the above formula, a second recognition feature featuresB with a dimension of 1×1536 can be obtained.

Images