Method and device for assisting dish identification based on tableware information

Abstract

The invention discloses a method and device for assisting dish identification based on tableware information, and the method comprises the steps: carrying out the positioning of a dish in a target dish picture through employing a target detection algorithm, carrying out the feature extraction of the located dish through employing a dish identification algorithm, obtaining a to-be-inputted dish type, and carrying out the recognition of the to-be-inputted dish type, calculating a similarity value between the to-be-input dish category and dishes in the dish set, and comparing whether the similarity value is smaller than a similarity threshold value or not; if yes, storing the to-be-input dish category in a warehouse; if not, the to-be-input dish category and the bound tableware ID are put in storage; when the dishes are identified, whether the category bound with the tableware ID exists in the categories of the identification result information is judged; when the dish category bound with the tableware ID exists in the dish set, judging whether the dish category bound with the tableware ID exists in the identification result information or not; if yes, modifying the category of the dish information; if not, not modifying; according to the invention, through a tableware identification algorithm, secondary identification and correction of dishes can be assisted.

Description

Technical Field

[0001] The present invention relates to the technical field of dish recognition, and more specifically to a method and device for assisting dish recognition based on tableware information. Background Art

[0002] With the development of modern science and technology, the settlement methods in group dining scenarios such as restaurants and canteens have been eliminated, mainly because the manual settlement method has the problem of high labor costs and low efficiency, resulting in poor dining experience for users; and modern vision-related technology development has launched a dish recognition settlement system based on vision and deep learning. In the current field of science and technology, dish recognition technology has made significant progress. These technologies are mainly divided into two categories: traditional vision-based methods and deep learning-based methods. First, in the traditional vision method, dish recognition technology relies on image processing technology. First, by analyzing the color, shape, texture and other feature information of the dish as dish information, and then calculating the similarity between feature classes to determine the name of the dish category; the dish recognition technology based on deep learning network is to collect data to train the model. The model has a good feature expression ability for different dishes. The similarity between the features obtained by the model can be used to calculate the category name of the dish.

[0003] The dish recognition algorithm based on traditional image processing will be affected by various objective factors such as the lighting of the on-site environment and the actual color of the dishes, and its generalization ability for the complex and diverse dishes in different canteens is poor. Moreover, due to different seasons and meal periods, the change in the number of dish categories will also lead to a decrease in its adaptability. In addition, the increase in the similarity between dishes caused by factors such as the replacement of chefs will also make the recognition effect worse. The dish recognition based on deep learning can effectively reduce the impact of on-site environmental factors, and has a certain generalization for the complex and diverse dishes in the canteen. However, it still cannot avoid the existence of different dishes with the same cooking method. Due to their high similarity, it is difficult to distinguish them from the dish images. Although the deep neural network recognition returns the correct category in the TopN (N≥5), it can be corrected by manually correcting the category name, but it will undoubtedly increase the labor cost. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and device for assisting dish identification based on tableware information. When the similarity between dishes is extremely high, the tableware recognition algorithm is used to assist in identifying the shape and color of the tableware bound to the dish to improve the accuracy of dish identification, thereby solving the deficiencies in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for assisting dish recognition based on tableware information comprises the following steps:

[0007] S1. When entering dishes, obtain the target dish image to be stored;

[0008] Use the target detection algorithm to locate the position of the dish in the target dish image to obtain the coordinate area of ​​the dish, and use the dish recognition algorithm to extract the features of the dish in the coordinate area to obtain the dish category to be entered;

[0009] S2. Read the dish set in the dish database, and calculate the similarity value between the dish category to be entered and each dish category in the dish set;

[0010] Compare each of the similarity values ​​with each similarity threshold in the preset threshold set to obtain a corresponding comparison result;

[0011] When the similarity value is less than the similarity threshold, the dish category to be entered is stored, otherwise step S3 is executed;

[0012] S3. When the similarity value is greater than the similarity threshold, the dish category to be entered is bound to a tableware ID, and the features of the tableware ID bound to the dish category to be entered are extracted using a tableware recognition algorithm, and the dish category to be entered and the bound tableware ID are stored;

[0013] S4. When identifying dishes, use a dish recognition algorithm to match the dish category to be identified with the features of each dish category in the dish set, and output recognition result information TopN;

[0014] Determine whether there is a category bound to the tableware ID in the TopN categories of the recognition result information; if not, directly output the recognition result; if yes, use the tableware recognition algorithm to identify the tableware ID, and determine whether there is a dish category bound to the tableware ID in the dish set;

[0015] When the dish category bound to the tableware ID does not exist in the dish set, the dish recognition result is output;

[0016] When there is a dish category bound to the tableware ID in the dish set, it is determined whether the dish category bound to the tableware ID exists in the recognition result information TopN. If so, the recognition result information Top1 is modified to the dish information corresponding to the tableware ID. Otherwise, the dish recognition result information TopN is not modified.

[0017] Furthermore, in step S1, the target detection algorithm is used to locate the position of the dish in the target dish image to obtain the coordinate area of ​​the dish, and the dish recognition algorithm is used to extract features of the dish in the coordinate area to obtain the dish category to be entered, specifically:

[0018] Based on the target detection model, the location information of the dish in the target dish image is located to obtain the coordinate area of ​​the dish, and the image within the coordinate area is cut into sub-images;

[0019] Based on the dish recognition model, features are extracted for each sub-graph, and the category of the dish to be entered is determined based on the extracted features.

[0020] Furthermore, in step S2, the dish set of the dish library is read, and the similarity value between the dish category to be entered and each dish category in the dish set is calculated. The similarity value is calculated by:

[0021] Read the feature information of N categories in the dish set of the total dish database, and the calculation formula of the similarity value is:

[0022]

[0023] In the above formula, features j Refers to the characteristics of the jth dish category in the total library; refers to the i-th feature value of the j-th category dish in the total database; featuresA refers to the category features of the dish to be entered, featuresA i Refers to the i-th feature value of the dish category to be entered; It refers to the similarity values ​​between all the dish category features in the total dish library and the dish features to be entered; similarityVector is the storage vector of all similarity values, with a dimension of N.

[0024] Furthermore, the target detection model is a trained YoloV8 target detection network model, and the specific training process of the YoloV8 target detection network model includes:

[0025] Collect a large number of food datasets with different lighting conditions and backgrounds for target frame annotation, and divide the annotated food datasets into training sets and validation sets in proportion;

[0026] Take the YoloV8 network as the basic network, use the training set to train the basic network, and get the target detection model;

[0027] The target detection model is verified using the validation set, and the optimal target detection model is selected.

[0028] Furthermore, the basic network is trained using the training set, specifically:

[0029] Initialize the basic network and perform data augmentation on the food images in the training set;

[0030] The SGD optimization algorithm is used to optimize the comprehensive loss function of the food images after data enhancement, which includes classification and bounding box regression losses.

[0031] The data enhancement includes: rotation, scaling and color jittering.

[0032] Furthermore, the dish recognition model is a trained student model, and the specific training process of the student model includes:

[0033] Based on the optimal object detection model, the regional positions of the dishes in the dish dataset are located and the pictures of the dishes in the regional positions are cut into sub-images. The dish category names of the sub-images are labeled, and the dish category names of the labeled sub-images are divided into training sets and validation sets according to the proportion.

[0034] Construct the Vision Transformer model as the teacher model, train the teacher model using the training set, and obtain the output results of the teacher model;

[0035] Build a ResNet backbone network as the student model, use the pre-trained weights of the teacher model to train the student model, and use the training set to train the student model to obtain the output results of the student model;

[0036] The student model is optimized through multiple iterations of classification loss and distillation loss. When the difference between the teacher model and the student model is minimized, the optimal student model is saved. Specifically:

[0037] The classification loss is used to evaluate the difference between the output of the student model and the dish category names of the labeled sub-graphs;

[0038] Use distillation loss to evaluate the difference between the output of the student model and the output of the teacher model;

[0039] Based on the optimal student model, feature extraction is performed on the sub-images in the regional position to obtain the dish information after feature extraction;

[0040] Calculate the similarity between the dish information after feature extraction and the dish features in the dish dataset, and select the category with the highest similarity value as the recognition result.

[0041] Furthermore, in step S3, the tableware recognition algorithm is executed using the trained tableware positioning and segmentation model, and the training process of the tableware positioning and segmentation model is specifically as follows:

[0042] Obtain a number of tableware images under different lighting conditions and backgrounds, and divide the tableware images into a training set and a validation set in proportion;

[0043] The YOLACT algorithm model is trained using the training set to obtain a tableware location and segmentation model;

[0044] The tableware location and segmentation model is verified based on the validation set, and the optimal tableware location and segmentation model is selected.

[0045] Furthermore, in step S3, the tableware recognition algorithm is used to extract the features of the tableware ID bound to the dish category to be entered, specifically including:

[0046] Use the optimal tableware positioning and segmentation model to locate the tableware image in the target dish image and extract the outline of the tableware image;

[0047] The contour of the tableware image extracted by the above segmentation model is optimized by using the Canny algorithm and the expansion and corrosion method in the traditional image algorithm to obtain the optimized contour map of the tableware image; and the edge feature information of the optimized contour map of the tableware image is extracted;

[0048] Matching the edge feature information of the contour map of the tableware picture with the edge feature information of the stored tableware, calculating the gradient cosine similarity value of the two, and storing the maximum matching score value of the edge feature information of the contour map of the tableware picture at the current angle in the gradient cosine similarity value;

[0049] The maximum matching score value at the current angle is compared with the global maximum matching score value. If the maximum matching score value at the current angle is greater than the global maximum matching score value, the ID corresponding to the tableware image is used as the recognition result.

[0050] Furthermore, the contour of the tableware picture is extracted specifically by: the shape and color of the tableware picture.

[0051] The present invention provides a device for assisting dish identification based on tableware information, comprising:

[0052] The acquisition module scans the input dish through a camera device to obtain a target dish image;

[0053] The target detection algorithm module is used to locate the position of the dish in the target dish image and obtain the coordinate area of ​​the dish;

[0054] The dish recognition algorithm module is used to extract the features of the dishes in the coordinate area and obtain the category of the dishes to be entered;

[0055] A tableware recognition algorithm module is used to extract the features of the tableware ID bound to the dish category to be entered;

[0056] A classification processing module, used to calculate the similarity value between the dish category to be entered and each dish category in the dish set;

[0057] Each of the similarity values ​​is compared with each similarity threshold in the preset threshold set to obtain a corresponding comparison result.

[0058] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the computer program is configured to be executed by the processor. When the computer program is executed by the processor, a method for assisting dish identification based on tableware information is implemented.

[0059] According to the specific embodiments provided by the present invention, the present invention has the following technical effects:

[0060] In the present application, when dishes are entered, the position of the dishes in the target dish image is located by the target detection algorithm, and the characteristics of the dishes are extracted by the dish recognition algorithm to obtain the category of the dishes to be entered, and the categories are compared with the categories in the dish set. When the category is not in the dish set, the tableware ID is bound to the dish to be identified by the tableware recognition algorithm and then put into storage; when identifying, if there is no dish to be identified in the dish set, the tableware recognition algorithm is used to identify the tableware ID, and then the category of the dish to be identified is obtained; the tableware-assisted dish identification method can adapt to the dish identification in different scenarios, and can effectively improve the recognition accuracy of the dishes; and the tableware recognition algorithm can effectively assist the dishes in secondary recognition and correction; and avoid the addition of other hardware and other costs, it can effectively improve the dish recognition rate and its generalization ability, and can also ensure that it will not be affected in terms of recognition speed, so that the recognition ability in different scenarios and dish diversity is further improved, and the robustness and accuracy of the algorithm are effectively improved, and its recognition accuracy can also guarantee the user's dining experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0062] The method and device for assisting dish identification based on tableware information of the present invention will be further described below in conjunction with the accompanying drawings;

[0063] Figure 1 It is an overall flow chart of the method for assisting dish identification based on tableware information provided by the present invention. DETAILED DESCRIPTION

[0064] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0065] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0066] The present invention provides a method for assisting dish identification based on tableware information, comprising the following steps:

[0067] S1. When recording a dish, use a camera device to scan the dish to be recorded to obtain a target dish image; use a target detection algorithm to locate the position of the dish in the target dish image to obtain a coordinate area of ​​the dish, and use a dish recognition algorithm to extract features of the dish in the coordinate area to obtain a category of the dish to be recorded;

[0068] It should be noted that the target position coordinates of the dish are located as ROI (x, y, w, h) through the trained YoloV8 target detection network algorithm. If the target frame cannot be detected correctly, the corresponding dish is manually selected.

[0069] S2. Read the dish set in the dish database, and calculate the similarity value between the dish category to be entered and each dish category in the dish set;

[0070] Compare each of the similarity values ​​with each similarity threshold in the preset threshold set to obtain a corresponding comparison result;

[0071] When the similarity value is less than the similarity threshold, the dish category to be entered is stored, otherwise step S3 is executed;

[0072] S3. When the similarity value is greater than the similarity threshold, the dish category to be entered is bound to a tableware ID, and the features of the tableware ID bound to the dish category to be entered are extracted using a tableware recognition algorithm, and the dish category to be entered and the bound tableware ID are stored;

[0073] S4. When identifying dishes, use a dish recognition algorithm to match and identify the dish category to be identified with the features of each dish category in the dish set, and output recognition result information TopN;

[0074] Determine whether there is a category bound to the tableware ID in the TopN categories of the recognition result information; if not, directly output the recognition result; if yes, use the tableware recognition algorithm to identify the tableware ID, and determine whether there is a dish category bound to the tableware ID in the dish set;

[0075] It should be noted that in the integrated settlement, the recognition results of all dishes are integrated and the corresponding dish categories are output for settlement.

[0076] When the dish category bound to the tableware ID does not exist in the dish set, the dish recognition result is output;

[0077] When there is a dish category bound to the tableware ID in the dish set, it is determined whether the dish category bound to the tableware ID exists in the recognition result information TopN. If so, the recognition result information Top1 is modified to the dish information corresponding to the tableware ID. Otherwise, the dish recognition result information TopN is not modified.

[0078] The target detection model is a trained YoloV8 target detection network model. The specific training process of the YoloV8 target detection network model includes:

[0079] Collect a large number of food datasets with different lighting conditions and backgrounds for target frame annotation, and divide the annotated food datasets into training sets and validation sets in proportion;

[0080] Take the YoloV8 network as the basic network, use the training set to train the basic network, and get the target detection model;

[0081] The target detection model is verified using the validation set, and the optimal target detection model is selected.

[0082] The training of the basic network using the training set specifically includes: initializing the basic network and performing data enhancement on the target dish images in the training set;

[0083] The SGD optimization algorithm is used to optimize the comprehensive loss function of the target dish image after data enhancement, which includes classification and bounding box regression losses.

[0084] The data enhancement includes: rotation, scaling and color jittering.

[0085] It should be noted that the dish detection model is trained based on the YoloV8 network architecture. First, the pre-trained model is loaded for initialization to speed up the convergence, and the images in the training data set are enhanced (such as rotation, scaling, color jitter, etc.) to improve the robustness of the model. The SGD optimization algorithm is used to optimize the comprehensive loss function including classification and bounding box regression losses.

[0086] The dish recognition model is a trained student model, and the specific training process of the student model includes:

[0087] Based on the optimal object detection model, the regional positions of the dishes in the dish dataset are located and the pictures of the dishes in the regional positions are cut into sub-images. The dish category names of the sub-images are labeled, and the dish category names of the labeled sub-images are divided into training sets and validation sets according to the proportion.

[0088] Construct the Vision Transformer model as the teacher model, train the teacher model using the training set, and obtain the output results of the teacher model;

[0089] It should be noted that the selected dish target position is cut into sub-images as the target dish images to be stored. The ResNet backbone network based on Transformer network distillation is used to train the metric learning model DML that can learn and extract dish features. The features of the dishes in the sub-image are extracted through the dish recognition model (DML).

[0090] Build a ResNet backbone network as the student model, use the pre-trained weights of the teacher model to train the student model, and use the training set to train the student model to obtain the output results of the student model;

[0091] It should be noted that: the ResNet backbone network is selected as the student model, and the pre-trained weights are used for initialization to speed up the training process;

[0092] The student model is optimized through multiple iterations of classification loss and distillation loss. When the difference between the teacher model and the student model is minimized, the optimal student model is saved. Specifically:

[0093] The classification loss is used to evaluate the difference between the output of the student model and the dish category names of the labeled sub-graphs;

[0094] Use distillation loss to evaluate the difference between the output of the student model and the output of the teacher model;

[0095] Based on the optimal student model, feature extraction is performed on the sub-images in the regional position to obtain the dish information after feature extraction;

[0096] Calculate the similarity between the dish information after feature extraction and the dish features in the dish dataset, and select the category with the highest similarity value as the recognition result.

[0097] It should be noted that the performance of the student model is evaluated on the validation set and fine-tuned to ensure that its ability to extract dish features is optimized. The optimally trained feature extraction model is selected as the dish metric learning model DML. The dish feature dimension extracted by this model is 1X1536, i.e., featuresA (1×1536) .

[0098] The optimal student model is a dish recognition model. Based on the dish recognition model, features of dishes in the coordinate area are extracted to obtain dish information after feature extraction, and the similarity value between the dish information after feature extraction and the dish features in the data set of the total dish library is calculated, and the category with the highest similarity value is selected as the recognition result.

[0099] It should be noted that if there is no dish category in the original total library whose similarity value similarityVector is greater than the set similarity threshold similarThre (default similarThre=0.8), the feature corresponding to the dish category is directly stored in the library; otherwise, the following steps are executed.

[0100] In step S3, the tableware recognition algorithm is executed using the trained tableware positioning and segmentation model. The training process of the tableware positioning and segmentation model is specifically as follows:

[0101] Obtain a number of tableware images under different lighting conditions and backgrounds, and divide the tableware images into a training set and a validation set in proportion;

[0102] The YOLACT algorithm model is trained using the training set to obtain a tableware location and segmentation model;

[0103] The tableware location and segmentation model is verified based on the validation set, and the optimal tableware location and segmentation model is selected.

[0104] In step S3, the tableware recognition algorithm is used to extract the features of the tableware ID bound to the dish category to be entered, specifically including:

[0105] Use the optimal tableware positioning and segmentation model to locate the tableware image in the target dish image and extract the outline of the tableware image;

[0106] The Canny algorithm and the expansion and corrosion method in the traditional image algorithm are used to optimize the contour of the tableware image extracted by the tableware positioning and segmentation model to obtain the optimized contour map of the tableware image; and the edge feature information of the optimized contour map of the tableware image is extracted;

[0107] Matching the edge feature information of the contour map of the tableware picture with the edge feature information of the stored tableware, calculating the gradient cosine similarity value of the two, and storing the maximum matching score value of the edge feature information of the contour map of the tableware picture at the current angle in the gradient cosine similarity value;

[0108] The maximum matching score value at the current angle is compared with the global maximum matching score value. If the maximum matching score value at the current angle is greater than the global maximum matching score value, the ID corresponding to the tableware image is used as the recognition result.

[0109] The tableware image to be identified includes: the shape and color of the tableware.

[0110] In the step S2, the dish set in the dish library is read, and the similarity value between the dish category to be entered and each dish category in the dish set is calculated. The calculation method of the similarity threshold is:

[0111] The dish recognition algorithm extracts features of dishes within the coordinate area, and the dimension of the feature extraction is 1×1536;

[0112] Read the feature information of N categories in the dataset of the total dish library, and the calculation formula of the similarity threshold is:

[0113]

[0114] In the above formula, Teatures j Refers to the characteristics of the jth dish category in the total library; refers to the i-th feature value of the j-th category dish in the total database; featuresA refers to the category features of the dish to be entered, featuresA i refers to the i-th feature value of the dish category to be entered; It refers to the similarity values ​​between all the dish category features in the total dish library and the dish features to be entered; similarityVector is the storage vector of all similarity values, with a dimension of N.

[0115] The present invention also provides a device for assisting dish identification based on tableware information, comprising:

[0116] The acquisition module scans the input dish through a camera device to obtain a target dish image;

[0117] The target detection algorithm module is used to locate the position of the dish in the target dish image and obtain the coordinate area of ​​the dish;

[0118] The dish recognition algorithm module is used to extract the features of the dishes in the coordinate area and obtain the category of the dishes to be entered;

[0119] A tableware recognition algorithm module is used to extract the features of the tableware ID bound to the dish category to be entered;

[0120] A classification processing module, used to calculate the similarity value between the dish category to be entered and each dish category in the dish set;

[0121] Each of the similarity values ​​is compared with each similarity threshold in the preset threshold set to obtain a corresponding comparison result.

[0122] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the computer program is configured to be executed by the processor. When the computer program is executed by the processor, a method for assisting dish identification based on tableware information is implemented.

[0123] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for assisting dish recognition based on tableware information, characterized in that: The following steps are involved: S1. When entering dishes, obtain the target dish image to be stored; Use the target detection algorithm to locate the position of the dish in the target dish image to obtain the coordinate area of ​​the dish, and use the dish recognition algorithm to extract the features of the dish in the coordinate area to obtain the dish category to be entered; S2. Read the dish set in the dish database, and calculate the similarity value between the dish category to be entered and each dish category in the dish set; Compare each of the similarity values ​​with each similarity threshold in the preset threshold set to obtain a corresponding comparison result; When the similarity value is less than the similarity threshold, the dish category to be entered is stored, otherwise step S3 is executed; S3. When the similarity value is greater than the similarity threshold, the dish category to be entered is bound to a tableware ID, and the features of the tableware ID bound to the dish category to be entered are extracted using a tableware recognition algorithm, and the dish category to be entered and the bound tableware ID are stored; S4. When identifying dishes, use a dish recognition algorithm to match and identify the dish category to be identified with the features of each dish category in the dish set, and output recognition result information TopN; Determine whether there is a category bound to the tableware ID in the TopN categories of the recognition result information; if not, directly output the recognition result; If yes, the tableware identification algorithm is used to identify the tableware ID, and it is determined whether the dish category bound to the tableware ID exists in the dish set; When the dish category bound to the tableware ID does not exist in the dish set, the dish recognition result is output; When there is a dish category bound to the tableware ID in the dish set, it is determined whether the dish category bound to the tableware ID exists in the recognition result information TopN. If so, the recognition result information Top1 is modified to the dish information corresponding to the tableware ID. Otherwise, the dish recognition result information TopN is not modified.

2. The method for assisting dish identification based on tableware information according to claim 1, characterized in that: In step S1, the target detection algorithm is used to locate the position of the dish in the target dish image to obtain the coordinate area of ​​the dish, and the dish recognition algorithm is used to extract the features of the dish in the coordinate area to obtain the dish category to be entered, specifically: Based on the target detection model, the location information of the dish in the target dish image is located to obtain the coordinate area of ​​the dish, and the image within the coordinate area is cut into sub-images; Based on the dish recognition model, features are extracted for each sub-graph, and the category of the dish to be entered is determined based on the extracted features.

3. The method for assisting dish identification based on tableware information according to claim 1, characterized in that: In step S2, the dish set in the dish library is read, and the similarity value between the dish category to be entered and each dish category in the dish set is calculated. The similarity value is calculated by: Read the feature information of N categories in the dish set of the total dish database, and the calculation formula of the similarity value is: In the above formula, features j Refers to the characteristics of the jth dish category in the total library; refers to the i-th feature value of the j-th category dish in the total database; featuresA refers to the category features of the dish to be entered, featuresA i Refers to the i-th feature value of the dish category to be entered; It refers to the similarity values ​​between all the dish category features in the total dish library and the dish features to be entered; similarityVector is the storage vector of all similarity values, with a dimension of N.

4. The method for assisting dish identification based on tableware information according to claim 2, characterized in that: The target detection model is a trained YoloV8 target detection network model. The specific training process of the YoloV8 target detection network model includes: Collect a large number of food datasets with different lighting conditions and backgrounds for target frame annotation, and divide the annotated food datasets into training sets and validation sets in proportion; Take the YoloV8 network as the basic network, use the training set to train the basic network, and get the target detection model; The target detection model is verified using the validation set, and the optimal target detection model is selected.

5. The method for assisting dish identification based on tableware information according to claim 4, characterized in that: The training of the basic network using the training set is specifically as follows: Initialize the basic network and perform data augmentation on the food images in the training set; The SGD optimization algorithm is used to optimize the comprehensive loss function of the food images after data enhancement, which includes classification and bounding box regression losses. The data enhancement includes: rotation, scaling and color jittering.

6. The method for assisting dish identification based on tableware information according to claim 3, characterized in that: The dish recognition model is a trained student model, and the specific training process of the student model includes: Based on the optimal object detection model, the regional positions of the dishes in the dish dataset are located and the pictures of the dishes in the regional positions are cut into sub-images. The dish category names of the sub-images are labeled, and the dish category names of the labeled sub-images are divided into training sets and validation sets according to the proportion. Construct the Vision Transformer model as the teacher model, train the teacher model using the training set, and obtain the output results of the teacher model; Build a ResNet backbone network as the student model, use the pre-trained weights of the teacher model to train the student model, and use the training set to train the student model to obtain the output results of the student model; The student model is optimized through multiple iterations of classification loss and distillation loss. When the difference between the teacher model and the student model is minimized, the optimal student model is saved. Specifically: The classification loss is used to evaluate the difference between the output of the student model and the dish category names of the labeled sub-graphs; Use distillation loss to evaluate the difference between the output of the student model and the output of the teacher model; Based on the optimal student model, feature extraction is performed on the sub-images in the regional position to obtain the dish information after feature extraction; Calculate the similarity between the dish information after feature extraction and the dish features in the dish dataset, and select the category with the highest similarity value as the recognition result.

7. The method for assisting dish identification based on tableware information according to claim 1, characterized in that: In step S3, the tableware recognition algorithm is executed using the trained tableware positioning and segmentation model. The training process of the tableware positioning and segmentation model is specifically as follows: Obtain a number of tableware images under different lighting conditions and backgrounds, and divide the tableware images into a training set and a validation set in proportion; The YOLACT algorithm model is trained using the training set to obtain a tableware location and segmentation model; The tableware location and segmentation model is verified based on the validation set, and the optimal tableware location and segmentation model is selected.

8. The method for assisting dish identification based on tableware information according to claim 7, characterized in that: In step S3, the tableware recognition algorithm is used to extract the features of the tableware ID bound to the dish category to be entered, specifically including: Use the optimal tableware positioning and segmentation model to locate the tableware image in the target dish image and extract the outline of the tableware image; The Canny algorithm and the expansion and corrosion method in the traditional image algorithm are used to optimize the contour of the tableware image extracted by the tableware positioning and segmentation model to obtain the optimized contour map of the tableware image; and the edge feature information of the optimized contour map of the tableware image is extracted; Matching the edge feature information of the contour image of the tableware picture with the edge feature information of the stored tableware, calculating the gradient cosine similarity value of the two, and storing the maximum matching score value of the edge feature information of the contour image of the tableware picture at the current angle in the gradient cosine similarity value; The maximum matching score value at the current angle is compared with the global maximum matching score value. If the maximum score value at the current angle is greater than the global maximum matching score value, the ID corresponding to the tableware image is used as the recognition result.

9. The method for assisting dish identification based on tableware information according to claim 7, characterized in that: The contour of the tableware picture is extracted specifically including the shape and color of the tableware picture.

10. A device for assisting dish recognition based on tableware information, used to execute the method for assisting dish recognition based on tableware information according to any one of claims 1 to 9, characterized in that: include: The acquisition module scans the input dish through a camera device to obtain a target dish image; The target detection algorithm module is used to locate the position of the dish in the target dish image and obtain the coordinate area of ​​the dish; The dish recognition algorithm module is used to extract the features of the dishes in the coordinate area and obtain the category of the dishes to be entered; A tableware recognition algorithm module is used to extract the features of the tableware ID bound to the dish category to be entered; A classification processing module, used to calculate the similarity value between the dish category to be entered and each dish category in the dish set; Each of the similarity values ​​is compared with each similarity threshold in the preset threshold set to obtain a corresponding comparison result.

Images