Dish recognition method based on deep learning

Abstract

The present invention discloses a dish recognition method based on deep learning, comprising the following steps: capturing an image to be recognized using a camera; detecting trays, drinks, and plates in the image using a deep learning algorithm; filtering the detected areas using the deep learning algorithm to obtain a complete and clear area; performing logical judgment based on the coordinates of the dish, filtering out interfering areas to obtain the area to be recognized; using the deep learning algorithm to identify the area to be recognized to obtain the corresponding dish name; and matching the dish information corresponding to the dish from a database based on the dish name. The dish recognition method based on deep learning provided by the present invention improves the efficiency and accuracy of dish recognition, significantly enhancing the intelligentization of restaurants.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and specifically relates to a dish recognition method based on deep learning. Background Art

[0002] Smart catering management is a cutting-edge research topic in the global catering management field. In recent years, its findings have been increasingly applied to various areas of social life, playing a positive role in improving work efficiency, facilitating catering processes, and maintaining order and safety. Smart catering management commonly uses traditional RFID and deep learning technologies to collect and identify objects in images and videos of food items in the catering industry. Currently, traditional RFID technology is widely used for food item identification in the catering industry. This relatively mature technology has been widely used in specific applications such as restaurant customer flow monitoring, payment monitoring, and food billing, achieving significant social and economic benefits. However, traditional RFID recognition technology relies on antennas pre-installed at the bottom of the checkout area to identify chips embedded in the bottom of the plates. This method is costly to implement and requires custom-made plates with embedded chips. Furthermore, it can only identify the plates, not the individual dishes. Therefore, when precise dish location is required, RFID solutions require the procurement of large quantities of plates to identify each dish individually, making implementation costly and challenging.

[0003] In deep learning technology, dish location is the first step in dish recognition, crucially impacting the efficiency and accuracy of subsequent dish recognition. Currently, dish location and recognition have a high accuracy rate under good lighting and shooting conditions. Secondly, identifying the dish's location area is a key step in dish recognition. Existing algorithms can only address dish recognition under specific conditions to a certain extent, and cannot guarantee recognition accuracy while restaurants continuously add new dishes. To improve the adaptability and accuracy of tableware recognition algorithms, it is crucial to study dish recognition methods in complex environments. Summary of the Invention

[0004] The present invention provides a dish recognition method based on deep learning, which adopts the following technical solutions:

[0005] A dish recognition method based on deep learning includes the following steps:

[0006] Step S1: Capture the image to be identified through the camera;

[0007] Step S2: Detect the tray, beverage, and plate in the image using a deep learning algorithm;

[0008] Step S3: Filter the detected area through the deep learning algorithm to obtain a complete and clear area;

[0009] Step S4: Perform logical judgment based on the coordinates of the dish, filter the interference area to obtain the area to be identified;

[0010] Step S5: Use a deep learning algorithm to identify the area to be identified and obtain the corresponding dish name;

[0011] Step S6: Match the dish information corresponding to the dish from the database according to the dish name.

[0012] Furthermore, the specific steps for detecting trays, drinks, and plates in images using a deep learning algorithm are as follows:

[0013] Step S21: scaling the image captured by the camera to a first size, filling the image with melanin to keep the scale of the scaled image unchanged, and recording the scaling ratio R;

[0014] Step S22: Using the food detection model trained based on the YOLO algorithm to predict the scaled image, a first prediction frame S is obtained;

[0015] Step S23: Use the NMS algorithm to filter the first prediction box S to obtain a second prediction box S'.

[0016] Furthermore, the specific steps of using the NMS algorithm to filter the first prediction box S to obtain the second prediction box S' are:

[0017] Step S231: sort all the boxes in the first prediction box S from large to small according to confidence, delete the box B with the highest score from the prediction box S and add it to the second prediction box S';

[0018] Step S232: traverse the remaining boxes in the first predicted box S, and delete the boxes whose IoU with the highest score box B is greater than a certain threshold from the first predicted box S;

[0019] Step S233: Repeat steps S231 and S232 until there are no remaining frames in the first prediction frame S, thereby obtaining the final second prediction frame S'.

[0020] Furthermore, the specific steps of filtering the detected areas through the deep learning algorithm to obtain complete and clear areas are as follows:

[0021] Step S31: cropping a small image Cr according to the prediction frame B in the second prediction frame S';

[0022] Step S32: scaling the small image Cr to a second size, and filling it with black pigment to keep the scale of the scaled small image Cr unchanged;

[0023] Step S33: input the scaled small image Cr into the pre-trained MobileNetv3-Tiny network;

[0024] Step S34: Filter incomplete or motion-blurred images through the MobileNetv3-Tiny network and output a complete and clear candidate box Can.

[0025] Furthermore, the specific steps of performing logical judgment based on the coordinates of the dish and filtering the interference area to obtain the area to be identified are as follows:

[0026] Step S41: Obtain the coordinates of the corresponding tray, plate, and beverage from the candidate frame Can;

[0027] Step S42: processing the number of trays to 1;

[0028] Step S43: Calculate the center coordinates of the coordinates corresponding to the plate and the beverage;

[0029] Step S44: Delete the candidate frames corresponding to the dinner plate and beverage whose center point coordinates are outside the coordinates of the tray to obtain the area to be identified Can.

[0030] Furthermore, in the process of processing the number of trays to 1, if the number of trays itself is 1, no processing is required;

[0031] If the number of pallets is greater than 1, first delete the pallets that are included in other pallets;

[0032] If the number of pallets is still greater than 1 at this time, only the pallet with the largest area is retained.

[0033] Furthermore, the specific steps of using the deep learning algorithm to identify the area to be identified and obtain the corresponding dish name are as follows:

[0034] Step S51: mapping all coordinates in the area to be identified Can back to the original image to obtain the original image coordinates;

[0035] Step S52: cropping the region to be recognized reInput without compression loss from the original image according to the original image coordinates, scaling it to a third size, and filling it with melanin to keep the image ratio unchanged;

[0036] Step S53: Use the pre-trained dish recognition model resNet50 to identify the area to be recognized reInput to obtain the dish name.

[0037] Furthermore, the first size is 640*640, the second size is 128*128, and the third size is 256*256.

[0038] Furthermore, the dish recognition method based on deep learning also includes:

[0039] Step S7: When a new dish is added, the new dish is trained to update the dish recognition model resNet50.

[0040] Furthermore, the specific steps for training new dishes to update the dish recognition model resNet50 are as follows:

[0041] Step S71: Take a photo of the new dish, and obtain the area to be identified Can after steps S2, S3, and S4;

[0042] Step S72: The uncompressed region to be recognized reInput is obtained by processing the region to be recognized Can through steps S51 and S52;

[0043] Step S73: Freeze the shallow parameters of the dish recognition model resNet50 so that it does not participate in training, and send the area to be recognized reInput to the dish recognition model resNet50 for training to obtain a new model that can recognize new dishes.

[0044] Furthermore, the dish information includes dish price and / or dish calorie content.

[0045] The benefit of the present invention lies in that the dish recognition method based on deep learning is provided, which improves the recognition efficiency and accuracy of dishes and greatly improves the intelligence level of the restaurant. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the dish recognition method based on deep learning of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 The present invention illustrates a dish recognition method based on deep learning, running on a custom-built all-in-one checkout machine for restaurants. The machine records video in real time and captures frames. It then uses the video frames to detect, locate, and identify dishes, obtaining relevant dish information and returning it to the machine, which is then called by other programs to implement intelligent checkout in the restaurant. Specifically, the dish recognition method based on deep learning includes the following steps: Step S1: Capture an image to be recognized using a camera. Step S2: Detect the tray, beverage, and plate in the image using a deep learning algorithm. Step S3: Filter the detected areas using the deep learning algorithm to obtain a complete and clear area. Step S4: Perform logical analysis based on the dish's coordinates, filtering out interfering areas to obtain the area to be recognized. Step S5: Use the deep learning algorithm to identify the area to be recognized and obtain the corresponding dish name. Step S6: Match the dish's name to the corresponding dish information from a database. The present invention's recognition method improves dish recognition efficiency and accuracy, significantly enhancing the restaurant's intelligent capabilities. The following describes these steps in detail.

[0049] In step S1: a picture to be identified is captured by a camera.

[0050] First, the camera collects the image to be identified, which includes a tray, a drink, and a plate.

[0051] For step S2: detect the tray, beverage and plate in the image through deep learning algorithm.

[0052] Preferably, the specific steps of detecting the tray, beverage, and plate in the image using a deep learning algorithm are:

[0053] Step S21: scaling the image captured by the camera to a first size, filling melanin to keep the scale of the scaled image unchanged, and recording the scaling ratio R. In this application, the first size is 640*640.

[0054] Step S22: The scaled image is predicted using a dish detection model trained based on the YOLO algorithm to obtain a first prediction frame S.

[0055] Step S23: Using the NMS (Non-Maximum Suppression) algorithm to filter the first prediction box S to obtain a second prediction box S'.

[0056] In step S23, the specific steps of using the NMS algorithm to filter the first prediction frame S to obtain the second prediction frame S' are:

[0057] Step S231: sort all the boxes in the first prediction box S from large to small according to confidence, delete the highest-scoring box B from the prediction box S and add it to the second prediction box S'.

[0058] Step S232: traverse the remaining boxes in the first predicted box S, and delete the boxes whose IoU (Intersection-over-Union) with the highest score box B is greater than a certain threshold from the first predicted box S.

[0059] Step S233: Repeat steps S231 and S232 until there are no remaining frames in the first prediction frame S, thereby obtaining the final second prediction frame S'.

[0060] For step S3: the detected area is screened by a deep learning algorithm to obtain a complete and clear area.

[0061] Specifically, the steps for obtaining a complete and clear area by filtering the detected area through the deep learning algorithm are as follows:

[0062] Step S31: cropping a small image Cr according to the prediction box B in the second prediction box S'.

[0063] Step S32: scaling the small image Cr to a second size and filling it with black pigment to keep the scale of the scaled small image Cr unchanged. In this application, the second size is 128*128.

[0064] Step S33: Input the scaled small image Cr into the pre-trained MobileNetv3-Tiny network.

[0065] Step S34: Filter incomplete or motion-blurred images through the MobileNetv3-Tiny network and output a complete and clear candidate box Can.

[0066] For step S4: perform logical judgment based on the coordinates of the dish, filter the interference area to obtain the area to be identified.

[0067] The specific steps for performing logical judgment based on the coordinates of the dish and filtering out the interference area to obtain the area to be identified are as follows:

[0068] Step S41: Get the coordinates of the corresponding tray, plate, and drink from the candidate frame Can. The coordinates of the tray, plate, and drink are Tray[(x1, y1), (x2, y2)], Dp[(x1, y1), (x2, y2)], and Dr[(x1, y1), (x2, y2)], respectively.

[0069] Step S42: Set the number of pallets to 1. Specifically, if the number of pallets is 1, no further processing is required. If the number of pallets is greater than 1, pallets that are contained by other pallets are first deleted. If the number of pallets is still greater than 1, the pallet with the largest area is retained.

[0070] Step S43: Calculate the center point coordinates CenPo of the coordinates corresponding to the plate and the beverage. The calculation formula is as follows:

[0071]

[0072] Step S44: Delete the candidate frames corresponding to the dinner plate and beverage whose center point coordinates are outside the coordinates of the tray to obtain the area to be identified Can.

[0073] For step S5: use a deep learning algorithm to identify the area to be identified and obtain the corresponding dish name.

[0074] Preferably, the specific steps of using a deep learning algorithm to identify the area to be identified and obtain the corresponding dish name are:

[0075] Step S51: Map all coordinates in the area to be identified Can back to the original image to obtain the original image coordinates oriCor[T] (T=Dp, Dr, Tray). The calculation formula is as follows:

[0076]

[0077] Step S52: Cut out the compression-free region reInput from the original image according to the original image coordinates oriCor and scale it to a third size, filling it with melanin to keep the image ratio unchanged. In this application, the third size is 256*256.

[0078] Step S53: Use the pre-trained dish recognition model resNet50 to identify the area to be recognized reInput to obtain the dish name.

[0079] For step S6: match the dish information corresponding to the dish from the database according to the dish name.

[0080] Specifically, the dish information includes dish price and / or dish calorie count.

[0081] like Figure 1 As shown, the dish recognition method based on deep learning further includes: Step S7: When a new dish is added, the new dish is trained to update the dish recognition model resNet50. It is understandable that when a new dish is added, the recognition model needs to be retrained with the new dish so that the recognition model can recognize the new dish.

[0082] Specifically, the specific steps for training new dishes to update the dish recognition model resNet50 are:

[0083] Step S71: Take a photo of the new dish, and obtain the area to be identified Can after steps S2, S3 and S4.

[0084] Step S72: The region to be recognized Can is processed through steps S51 and S52 to obtain the uncompressed region to be recognized reInput.

[0085] Step S73: Freeze the shallow parameters of the dish recognition model resNet50 so that it does not participate in training, and send the area to be recognized reInput to the dish recognition model resNet50 for training to obtain a new model that can recognize new dishes.

[0086] The deep learning-based dish recognition method of this application has been tested over a period of time on nearly 50,000 images collected from different time periods, lighting conditions, cuisines, cooking styles, and restaurants. The method demonstrates its adaptability to dish detection and recognition in various conditions, including camera overexposure and severe light reflections. The accuracy rate reached 99.5% under challenging conditions, while the positioning accuracy reached 99.8% under a combination of these conditions.

[0087] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the scope of protection of the present invention.

Claims

1. A dish recognition method based on deep learning, characterized in that: The following steps are involved: Step S1: Capture the image to be identified through the camera; Step S2: Detect the tray, beverage, and plate in the image using a deep learning algorithm; Step S3: Filter the detected areas through the deep learning algorithm to obtain complete and clear areas; Step S4: Perform logical judgment based on the coordinates of the dish, filter the interference area to obtain the area to be identified; Step S5: Use a deep learning algorithm to identify the area to be identified and obtain the corresponding dish name; Step S6: Matching the dish information corresponding to the dish from the database according to the dish name; The specific steps of detecting trays, drinks, and plates in the image using a deep learning algorithm are as follows: Step S21: scaling the image captured by the camera to a first size, filling the image with melanin to keep the scale of the scaled image unchanged, and recording the scaling ratio R; Step S22: Using the food detection model trained based on the YOLO algorithm to predict the scaled image, a first prediction frame S is obtained; Step S23: Use the NMS algorithm to filter the first prediction box S to obtain a second prediction box S'; The specific steps of using the NMS algorithm to filter the first prediction box S to obtain the second prediction box S' are: Step S231: sort all the boxes in the first prediction box S from large to small according to confidence, delete the box B with the highest score from the prediction box S and add it to the second prediction box S'; Step S232: traverse the remaining boxes in the first predicted box S, and delete the boxes whose IoU with the highest score box B is greater than a certain threshold from the first predicted box S; Step S233: repeating steps S231 and S232 until there are no remaining frames in the first prediction frame S, thereby obtaining the final second prediction frame S'; The specific steps of obtaining a complete and clear area by screening the detected area through the deep learning algorithm are: Step S31: cropping a small image Cr according to the prediction frame B in the second prediction frame S'; Step S32: scaling the small image Cr to a second size, and filling it with black pigment to keep the scale of the scaled small image Cr unchanged; Step S33: input the scaled small image Cr into the pre-trained MobileNetv3-Tiny network; Step S34: Filter incomplete or motion-blurred images through the MobileNetv3-Tiny network and output a complete and clear candidate box Can.

2. The dish recognition method based on deep learning according to claim 1 is characterized in that: The specific steps of performing logical judgment based on the coordinates of the dish and filtering the interference area to obtain the area to be identified are: Step S41: Obtain the coordinates of the corresponding tray, plate, and beverage from the candidate frame Can; Step S42: processing the number of trays to 1; Step S43: Calculate the center coordinates of the coordinates corresponding to the plate and the beverage; Step S44: Delete the candidate frames corresponding to the dinner plate and beverage whose center point coordinates are outside the coordinates of the tray to obtain the area to be identified Can.

3. The dish recognition method based on deep learning according to claim 2 is characterized in that: In the process of processing the number of pallets to 1, if the number of pallets itself is 1, no processing is required; If the number of pallets is greater than 1, first delete the pallets that are included in other pallets; If the number of pallets is still greater than 1 at this time, only the pallet with the largest area is retained.

4. The dish recognition method based on deep learning according to claim 2, characterized in that: The specific steps of using the deep learning algorithm to identify the area to be identified and obtain the corresponding dish name are: Step S51: mapping all coordinates in the area to be identified Can back to the original image to obtain the original image coordinates; Step S52: cropping the region to be recognized reInput without compression loss from the original image according to the original image coordinates, scaling it to a third size, and filling it with melanin to keep the image ratio unchanged; Step S53: Use the pre-trained dish recognition model resNet50 to identify the area to be recognized reInput to obtain the dish name.

5. The dish recognition method based on deep learning according to claim 4 is characterized in that: The first size is 640*640, the second size is 128*128, and the third size is 256*256.

6. The dish recognition method based on deep learning according to claim 4 is characterized in that: The dish recognition method based on deep learning also includes: Step S7: When a new dish is added, the new dish is trained to update the dish recognition model resNet50.

7. The dish recognition method based on deep learning according to claim 6, characterized in that: The specific steps of training new dishes to update the dish recognition model resNet50 are as follows: Step S71: Take a photo of the new dish, and obtain the area to be identified Can after steps S2, S3, and S4; Step S72: The uncompressed region to be recognized reInput is obtained by processing the region to be recognized Can through steps S51 and S52; Step S73: Freeze the shallow parameters of the dish recognition model resNet50 so that it does not participate in training, and send the area to be recognized reInput to the dish recognition model resNet50 for training to obtain a new model that can recognize new dishes.

Images