A method for positioning and identifying dish information on a dinner plate based on deep learning

Abstract

The method for positioning and identifying dish information based on deep learning disclosed in the present invention reads the information of the plate templates stored in the library and extracts their template features and contour maps, as well as the order images. The deep learning model is used for image positioning and contour extraction to obtain the plate contour map through secondary contour extraction. Calculate the matching centroid positioning map of all plates in the order image. Circularly read the information of the plate templates stored in the library after initialization at different angles, and calculate the optimal TopN coincidence position coordinates between the contour of the plate to be recognized and the contour of the current template by calculating with the feature values of the plate to be recognized and the information of the plate templates stored in the library at different angles. Calculate the gradient cosine similarity value between the plate to be recognized and the current plate template, store the maximum matching score value of the id of the current plate template at the current angle, compare and assign the maximum matching score value of the id of the current plate template at the current angle with the global maximum matching score value, and calculate the price, dishes, and nutritional information corresponding to each plate in the order image.

Description

Technical Field

[0001] The present invention relates to the fields of dinner plates and dish recognition, and in particular to a method for positioning and recognizing dish information based on deep learning for dinner plates. Background Art

[0002] The main settlement methods in existing restaurants and canteens include traditional manual card machine settlement, RFID chip-based recognition settlement, image recognition-based, and depth image detection and recognition-based methods.

[0003] The traditional manual card machine settlement method requires arranging personnel to use the card machine for order settlement in real time. It requires investment in manpower, affects the settlement efficiency during peak dining hours, and is prone to price calculation errors.

[0004] The RFID chip-based recognition settlement uses customized dinner plates with RFID chips, and reads the bound dinner plate information through an RFID decoder for order settlement. It requires customizing tableware embedded with RFID chips, which has a high cost for such tableware and a high maintenance cost, and various types of dinner plates cannot be used.

[0005] The image recognition-based method associates the feature information such as the shape, size, texture pattern, and color of the dinner plate with the dinner plate price information, and collects order pictures through a vision module for the settlement device to identify and settle. This method has requirements for the shape, size, texture pattern, and color of the dinner plate, and also has relatively strict requirements for the deployment scenario environment, resulting in poor environmental adaptability, low recognition rate, and high false recognition rate in actual use.

[0006] The depth image detection and recognition method locates the dinner plates in the order pictures collected by the vision module through a deep learning detection model, and uses traditional images of the located dinner plates to extract their feature information such as shape, size, texture pattern, and color for recognition and pricing. This method still has relatively high requirements for the dinner plate and the scenario, and both the recognition accuracy and robustness are poor; in addition, it has relatively high requirements for the device performance, increasing the investment cost of the device. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for positioning and recognizing dish information based on deep learning for dinner plates with low input cost and high recognition accuracy.

[0008] The present invention provides a method for positioning and recognizing dish information based on deep learning for dinner plates, including the following steps:

[0009] S1. Initialize the deep learning model, read the stored dinner plate template information and extract its template features and contour maps, read the entered dish categories for the current meal and their corresponding nutritional information, and the dinner plate template information includes the ID number of the dinner plate template.

[0010] S2. Read the order image to be recognized and matched, and use a deep learning model for image localization and extraction of the plate contour;

[0011] S3. Perform secondary contour extraction on the extracted order image contour to obtain a plate contour map;

[0012] S4. Extract the feature information of the plate contour map and calculate the matching centroid localization map of all plates in the order image;

[0013] S5. Loop to read the plate information to be recognized in the order image, including the matching centroid localization map and feature values;

[0014] S6. Loop to read the information of the stored plate templates at different angles after initialization, including relative contour coordinates and feature values;

[0015] S7. Taking the centroid point of the plate contour to be recognized in the matching centroid localization map as the reference centroid, set the centroid offset of N×N to calculate the optimal coincidence position coordinates with the stored plate templates at different angles, and take its TopN as the reference matching centroid point;

[0016] S8. Loop through and extract the optimal coincidence position coordinates, calculate the gradient cosine similarity value between the plate to be recognized and the current plate template, and store the maximum matching score value of the current plate template's id at the current angle;

[0017] S9. Compare the maximum matching score value of the current plate template's id at the current angle with the global maximum matching score value. If the maximum matching score value of the current plate template's id at the current angle is greater than the global maximum matching score value, then assign the maximum matching score value of the current plate template's id at the current angle to the global maximum matching score value, and assign the id of the current plate template to the ID of the global optimal matching plate template;

[0018] S10. Calculate the dish prices, total order price, dish names, and their nutritional information corresponding to each plate in the order image according to the ID of the global optimal matching plate template corresponding to each plate and its corresponding maximum matching score value.

[0019] As a preferred solution of the present invention, the step S3 includes:

[0020] S3-1. Perform multi-channel contour extraction on the extracted order image contour, and fuse each channel contour map to obtain a channel map;

[0021] S3-2. Take the contour edge map obtained by performing dilation and erosion operations on the contour extracted by deep learning;

[0022] S3-3. Obtain an edge ring map through the intersection operation of the contour edge map and the channel map;

[0023] S3-4. Find the contours in the edge ring diagram whose contour areas are greater than the threshold, and store them as the dinner plate contour diagram.

[0024] As a preferred solution of the present invention, the step S7 includes:

[0025] S7-1. Perform coincidence voting calculation on the matching centroid positioning diagram of the dinner plate contour diagram in the order diagram and the current template diagram;

[0026] S7-2. Set the offset of N×N for matching, and search for the coincidence degree of each assumed centroid point;

[0027] S7-3. After sorting, take the top N as the optimal coincidence position coordinates, and take its top N as the reference matching centroid points.

[0028] Advantages of the present invention:

[0029] For the method for positioning and identifying dish information of dinner plates based on deep learning according to the present invention, a deep learning model is used to position the dinner plates and extract their contours. This method can reduce the requirements of the device for the environment and improve its robustness; in addition, it solves the problems of difficult extraction of the contours of dinner plates with the same or similar colors and the dinner plates under trays, and difficult extraction of contours of different materials; since the contours extracted by deep learning may not necessarily be complete or correct, secondary contour extraction is performed on the contours, so as to realize the extraction of the complete contours of the dinner plates and trays that need to be matched in different colors, materials and shapes, so as to ensure the accurate extraction of subsequent matching features; by circularly reading the dinner plate information to be recognized in the order diagram, including the matching centroid positioning diagram and feature values, and circularly reading the information of the dinner plate templates stored in the library after initialization at different angles, including the relative coordinates and feature values of the contours, taking the centroid point of the dinner plate contour to be recognized in the matching centroid positioning diagram as the reference centroid, setting the centroid offset of N×N to calculate the optimal coincidence position coordinates with the dinner plate templates stored in the library at different angles, and taking the top N after sorting as the reference matching centroid points to reduce the number of reference centroid point matches and improve the matching speed, achieving the purpose of accelerating the matching; by comparing the maximum matching score value of the current dinner plate template ID at the current angle with the global maximum matching score value, if the maximum matching score value of the current dinner plate template ID at the current angle is greater than the global maximum matching score value, then assign the maximum matching score value of the current dinner plate template ID at the current angle to the global maximum matching score value, and assign the current dinner plate template ID to the ID of the globally optimal matching dinner plate template. Performing assignment matching can reduce the storage and calculation consumption of the model while ensuring that the accuracy is hardly lost. Therefore, the model can be compatible with devices of different performances and configurations without affecting the detection speed and accuracy, can effectively reduce the input cost, and at the same time greatly improves the matching speed while ensuring the accuracy, solving the problem that the matching speed is affected by the angle and occlusion and requires centroid offset, and can meet the actual real-time online use. Brief Description of the Drawings

[0030] Figure 1 It is a schematic flow chart of a method for positioning a dinner plate and identifying dish information based on deep learning according to the present invention;

[0031] Figure 2 It is a schematic flow chart of the present invention for performing secondary contour extraction on the extracted order image contour to obtain a dinner plate contour map;

[0032] Figure 3 It is a schematic flow chart of step 7 of the present invention. Detailed Embodiment

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] The present invention uses a single camera installed directly above the settlement desk to take order pictures, and the settlement system processes the order pictures.

[0035] The present invention provides a method for positioning a dinner plate and identifying dish information based on deep learning, as shown in Figure 1As shown below, it includes the following steps: S1. Initialize the deep learning model, read the information of the plate templates stored in the library, extract their template features and contour maps, read the categories of the dishes for the current meal and their corresponding nutritional information. The plate template information includes the ID number of the plate template; S2. Read the order image that needs to be recognized and matched, and use the deep learning model for image positioning and plate contour extraction; S3. Perform secondary contour extraction on the extracted order image contour to obtain the plate contour map; S4. Extract the feature information of the plate contour map and calculate the matching centroid positioning map of all plates in the order image; S5. Loop to read the plate information to be recognized in the order image, including the matching centroid positioning map and feature values; S6. Loop to read the information of the plate templates stored in the library at different angles, including the relative coordinates of the contours and feature values; S7. Take the centroid point of the plate contour to be recognized in the matching centroid positioning map as the reference centroid, set the centroid offset of N×N to calculate the optimal coincidence position coordinates with the plate templates stored in the library at different angles, and take the TopN after sorting as the reference matching centroid point; S8. Loop through the extracted optimal coincidence position coordinates, calculate the gradient cosine similarity value between the plate to be recognized and the current plate template, and store the maximum matching score value of the ID of the current plate template at the current angle; S9. Compare the maximum matching score value of the ID of the current plate template at the current angle with the global maximum matching score value. If the maximum matching score value of the ID of the current plate template at the current angle is greater than the global maximum matching score value, then assign the maximum matching score value of the ID of the current plate template at the current angle to the global maximum matching score value, and assign the ID of the current plate template to the ID of the globally optimal matching plate template; S10. Calculate the dish prices, total order price, dish names and their nutritional information corresponding to each plate in the order image according to the ID of the globally optimal matching plate template corresponding to each plate and its corresponding maximum matching score value.

[0036] The deep learning model is used for the positioning of the dinner plate and the extraction of its contour. This method can reduce the requirements of the device for the environment and improve its robustness. In addition, it solves the problems of difficult contour extraction of dinner plates with the same or similar colors and dinner plates under trays, as well as difficult contour extraction of different materials. Since the contours extracted by deep learning may not be complete or correct, secondary contour extraction is performed on the contours, so as to realize the extraction of the complete contours of each dinner plate and tray that need to match the dinner plate in different colors, materials and shapes, so as to ensure the accurate extraction of subsequent matching features. By circularly reading the information of the dinner plates to be recognized in the order diagram, including matching the center-of-gravity positioning diagram and feature values, and circularly reading the information of the dinner plate templates stored in the database after initialization at different angles, including the relative coordinates and feature values of the contours, with the center of gravity of the contour of the dinner plate to be recognized in the center-of-gravity positioning diagram as the reference center of gravity, set the center-of-gravity offset calculation of N×N to calculate the optimal coincidence position coordinates of the stored dinner plate template at different angles, and take the TopN after sorting as the reference matching center-of-gravity points to reduce the number of reference center-of-gravity point matches and improve the matching speed, so as to achieve the purpose of accelerating the matching. By comparing the maximum matching score value of the current dinner plate template ID at the current angle with the global maximum matching score value, if the maximum matching score value of the current dinner plate template ID at the current angle is greater than the global maximum matching score value, then assign the maximum matching score value of the current dinner plate template ID at the current angle to the global maximum matching score value, and assign the current dinner plate template ID to the ID of the globally optimal matching dinner plate template. Performing assignment matching can reduce the storage and calculation consumption of the model while ensuring that the accuracy is hardly lost. Therefore, the model can be compatible with devices of different performances and configurations without affecting the detection speed and accuracy, effectively reducing the input cost. At the same time, while ensuring the accuracy, the matching speed is greatly improved, solving the problem that the matching speed is affected by the angle and occlusion and requires the center of gravity to be offset, and can meet the actual real-time online use.

[0037] In step 7, as Figure 3 shown, it includes S7-1. Calculate the coincidence vote by matching the center-of-gravity positioning diagram of the dinner plate contour diagram in the order diagram with the current template diagram; S7-2. Set the matching offset of N×N to search for the coincidence degree of each assumed center-of-gravity point; S7-3. After sorting, take the TopN as the optimal coincidence position coordinates and take the TopN as the reference matching center-of-gravity points.

[0038] In calculating the optimal coincidence position coordinates of the contour of the dinner plate to be recognized and the current template contour, by setting the center-of-gravity offset calculation of N×N to calculate the optimal coincidence position coordinates of the stored dinner plate template at different angles, and taking the TopN of them as the optimal center-of-gravity point position coordinates. Here, set The value is 16, and the TopN is set to 5. The method is to quickly locate the positions of the TopN reference center-of-gravity points where the outline of the dinner plate overlaps the template outline the most, and perform contour map feature matching based on the TopN reference center-of-gravity points. This method can greatly improve the matching speed while ensuring the accuracy, and also solves the problem that the matching speed is affected by angles and occlusions, and can meet the actual real-time use.

[0039] When calculating the dinner plate price corresponding to each dinner plate in the order diagram, the dish recognition model can be called to recognize the dish category and its nutritional information in the current dinner plate, and then calculate the dish price, the total order price, the dish name and its nutritional information corresponding to each dinner plate in the order diagram.

[0040] As Figure 2 shown, in step S3, the outline of the extracted order image is secondarily extracted to obtain the dinner plate outline map, which specifically includes: S3-1, performing channel-by-channel outline extraction on the extracted order image outline, and fusing the outline maps of each channel to obtain a channel map; S3-2, performing dilation and erosion operations on the outline extracted by deep learning to obtain an outline edge map; S3-3, obtaining an outline ring map through the intersection operation of the outline edge map and the channel map; S3-4, searching for the outlines in the outline ring map that satisfy the condition that their outline areas are greater than the threshold and storing them as the dinner plate outline map, and the extracted outline can be secondarily optimized to improve the accuracy and precision of the dinner plate outline.

[0041] As a preferred implementation of the present invention, in the description of this specification, the description with reference to terms such as "preferred" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expression of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0042] The above embodiments are only used to illustrate the detailed solutions of the present invention. The present invention is not limited to the above detailed solutions, that is, it does not mean that the present invention must rely on the above detailed solutions to be implemented. Those skilled in the art should understand that any improvement to the present invention, the equivalent replacement of each raw material of the products of the present invention, the addition of auxiliary components, and the selection of specific methods, etc., all fall within the protection scope and the disclosure scope of the present invention.

Claims

1. A method for positioning and identifying dish information based on deep learning, characterized in that, Including the following steps: S1. Initialize the deep learning model, read the information of the stored plate templates, extract their template features and contour maps, read the categories of dishes for the current meal and their corresponding nutritional information. The plate template information includes the ID number of the plate template; S2. Read the order images to be recognized and matched, and use the deep learning model for image positioning and plate contour extraction; S3. Perform secondary contour extraction on the extracted order image contour to obtain the plate contour map; S4. Extract the feature information of the plate contour map and calculate the matching centroid positioning map of all plates in the order image; S5. Loop to read the plate information to be recognized in the order image, including the matching centroid positioning map and eigenvalue; S6. Loop to read the information of the stored plate templates at different angles after initialization, including the relative coordinates of the contour and eigenvalue; S7. Taking the centroid point of the plate contour to be recognized in the matching centroid positioning map as the reference centroid, set the centroid offset of N×N to calculate the optimal coincidence position coordinates with the stored plate templates at different angles, and take its TopN as the reference matching centroid point; S8. Loop through and extract the optimal coincidence position coordinates, calculate the gradient cosine similarity value between the plate to be recognized and the current plate template, and store the maximum matching score value of the ID of the current plate template at the current angle; S9. Compare the maximum matching score value of the ID of the current plate template at the current angle with the global maximum matching score value. If the maximum matching score value of the ID of the current plate template at the current angle is greater than the global maximum matching score value, then assign the maximum matching score value of the ID of the current plate template at the current angle to the global maximum matching score value, and assign the ID of the current plate template to the ID of the globally optimal matching plate template; S10. Calculate the dish price, order total price, dish name and its nutritional information corresponding to each plate in the order image according to the ID of the globally optimal matching plate template corresponding to each plate and its corresponding maximum matching score value.

2. The method for positioning and identifying dish information based on deep learning according to claim 1, wherein The step S3 includes: S3-1. Perform channel-by-channel contour extraction on the extracted order image contour, and fuse the channel contour maps to obtain a channel map; S3-2. Take out the contour edge map by performing dilation and erosion operations on the contour extracted by deep learning; S3-3. Obtain an edge ring map through the intersection operation of the contour edge map and the channel map; S3-4. Search for the contours in the edge ring map that satisfy the condition that their contour area is greater than the threshold and store them as the plate contour map.

3. The method for positioning and identifying dish information based on deep learning according to claim 1, wherein The step S7 includes: S7-1. Perform coincidence voting calculation through the matching centroid positioning map of the plate contour map in the order image and the current template map; S7-2. Set the matching offset of N×N and search for the coincidence degree of each assumed centroid point; S7-3. After sorting, take the TopN as the optimal coincidence position coordinates and take its TopN as the reference matching centroid point.

Images