A dish recognition method and system based on Dish-YOLO

Abstract

The present invention discloses a dish recognition method and system based on Dish‑YOLO, which belongs to the field of image recognition. The present invention comprises three parts: a dish recognition module, an auxiliary settlement and data set update module, and a model training and cloud storage module. The dish recognition module first locates the dish area of ​​the plate picture and recognizes the dish of the single dish picture, and then manually reviews the dish recognition result and pricing result through the auxiliary settlement and data set update module and updates the cloud training data set. Finally, the dish recognition model is trained in the cloud through the model training and cloud storage module and synchronously updated to the local end of the restaurant. The present invention realizes the deep integration of the traditional manual cashier settlement mode and the automatic dish recognition settlement mode through the dish recognition method of first locating and then recognizing, combined with auxiliary settlement and cloud training, so that the intelligent dish recognition and settlement system has stronger practicality and fault tolerance.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition, and in particular relates to a dish recognition method and system based on Dish-YOLO. Background Art

[0002] Traditional cashier work in the catering industry has the problems of many types of dishes being difficult to remember and high repetitiveness of work, so reform is imperative. In recent years, with the rapid development of industrial intelligence, the catering industry has also gradually moved towards intelligence, and a large number of intelligent dish recognition solutions have gradually emerged. Intelligent dish recognition can greatly reduce the cashier's settlement time, improve work efficiency, reduce queuing and waiting time, and thus increase profitability. From the current point of view, the solutions mainly include two categories:

[0003] The first type is to add chips to the dishes and use radio frequency technology to identify the dishes and then distinguish the dishes. This solution mainly uses radio frequency technology to input the price information corresponding to different dishes into the chip of the plate, and distinguishes different dish prices by reading the information in the chip. For example, Chinese patent number CN 109345737 A, patent name: Intelligent catering settlement system, method and medium, this patent uses radio frequency technology to input the sub-classification coding information corresponding to dishes of different price categories into the chip of the dish, and restores the corresponding dish price information through the read sub-classifier coding information. This solution is a method of indirect identification of dishes, which requires additional chips to be added to the dishes, which is costly and inflexible, and the chip is easily damaged or corroded during high-temperature cleaning.

[0004] The second type is a method that directly uses the characteristics of the dishes themselves for identification. This method directly uses pictures of dishes, uses machine learning and deep learning methods to train the dishes, obtains the characteristics of different dishes, and then directly obtains the price information of the dishes through identification. For example, Chinese patent number CN 111128341 A, patent name: A dish recognition APP based on deep learning, which uses the training set and test set of dishes to train the model, calls the trained model to identify the dishes, and displays the recognition results. The training set required for this method to train the model is fixed, and it is impossible to generate exclusive recognition models according to different users, and factors such as light changes caused by environmental changes will also lead to inaccurate recognition results. Summary of the invention

[0005] In order to solve many problems existing in the above schemes and provide a dish intelligent recognition and settlement solution that can be actually used in restaurants of various environments, the present invention proposes a dish recognition method and system based on Dish-YOLO, which deeply integrates the traditional manual cashier settlement mode and the dish automatic recognition and settlement mode, so that the dish intelligent recognition and settlement system has stronger practicality and fault tolerance.

[0006] The specific technical solutions adopted by the present invention are as follows:

[0007] In a first aspect, the present invention provides a dish recognition method based on Dish-YOLO, which comprises:

[0008] The restaurant obtains a plate image to be identified at the local end, wherein the plate image contains one or more dishes selected by the target customer and placed on the plate in the form of bowls and plates, and uses the trained Dish-YOLO model to locate the bowl and plate area of ​​each dish in the plate image, and then extracts a single dish image of each dish in the plate image; then the single dish image of each dish in the plate image is respectively input into the trained dish recognition model to identify all the dishes contained in the plate image, and form a dish recognition result; according to the preset unit price of the dish, all the dishes contained in the plate image are priced to form a pricing result; the Dish-YOLO model adopts an improved YOLOv3 model, which is obtained by the YOLOv3 model by discarding the 52*52 scale layer but retaining the 13*13 and 26*26 scale layers and then reallocating the anchor point frame;

[0009] The restaurant local end obtains the dish recognition result and pricing result of the plate image, sends them to the local manual review end and receives the verification instruction issued by the manual review end. If the verification instruction is correct, it enters the payment link. If the verification instruction is wrong, it receives the change data from the manual review end and enters the payment link based on the dish recognition result and pricing result changed by the manual review end; the plate image and dish recognition result entering the payment link in each round are updated to the cloud training data set of the dish recognition model;

[0010] When the restaurant local end receives the plate image entered for the new dish, it sends it to the cloud, and the trained Dish-YOLO model is used in the cloud to locate the new dish in the plate image, extract the single dish image of the new dish, regenerate the labeled training sample and add it to the cloud training data set of the dish recognition model; when the cloud training data set of the dish recognition model is updated, the updated training data set is re-used in the cloud to train the dish recognition model and update the network parameters, and after the training is completed, the restaurant local end is notified to update and synchronize.

[0011] As a preferred embodiment of the first aspect above, the Dish-YOLO model is pre-trained using a picture data set uploaded by the local end of each restaurant before it is put into use, and the picture data set contains different plate pictures, each plate picture contains dishes served in bowls and placed on the plate, the dishes in all plate pictures should cover all dishes sold by the restaurant, and the picture data set needs to be data enhanced to expand the sample size.

[0012] As a preferred embodiment of the first aspect, in the Dish-YOLO model, a total of 16 anchor boxes are set, namely (12, 46), (16, 111), (22, 76), (39, 72), (22, 166), (30, 113), (75, 79), (33, 198), (49, 148), (40, 282), (57, 327), (291, 53), (64, 205), (94, 294), (135, 154) and (194, 310), and each scale predicts 8 target bounding boxes with the help of the anchor box.

[0013] As a preferred embodiment of the first aspect, the training of the Dish-YOLO model and the dish recognition model are both performed in a cloud server, and the trained models are stored in the cloud server for downloading by the restaurant local end.

[0014] As a preferred embodiment of the first aspect above, the dish recognition model is a VGG16 network model.

[0015] In a second aspect, the present invention provides a dish recognition system based on Dish-YOLO, which includes:

[0016] The dish recognition module is used to obtain a dish image to be recognized at the local end of the restaurant, wherein the dish image contains one or more dishes selected by the target customer and placed on the dish in the form of bowls and plates, and use the trained Dish-YOLO model to locate the dish area of ​​each dish in the dish image, and then extract a single dish image of each dish in the dish image; then input the single dish image of each dish in the dish image into the trained dish recognition model respectively, recognize all the dishes contained in the dish image, and form a dish recognition result; according to the preset dish unit price, all the dishes contained in the dish image are priced to form a pricing result; the Dish-YOLO model adopts an improved YOLOv3 model, which is obtained by the YOLOv3 model by discarding the 52*52 scale layer but retaining the 13*13 and 26*26 scale layers and then reallocating the anchor point frame;

[0017] The auxiliary settlement and data set update module is used to obtain the dish recognition result and pricing result of the plate image at the local end of the restaurant, send it to the local manual review end and receive the verification instruction issued by the manual review end, if the verification instruction is correct, enter the payment link, if the verification instruction is wrong, receive the change data of the manual review end, and enter the payment link based on the dish recognition result and pricing result changed by the manual review end; update the plate image and dish recognition result entering the payment link in each round to the cloud training data set of the dish recognition model;

[0018] The model training and cloud storage module is used to receive plate images uploaded by the restaurant local end for the entered new dishes, use the trained Dish-YOLO model in the cloud to locate the new dishes in the plate images, extract the single dish images of the new dishes, regenerate labeled training samples and add them to the cloud training data set of the dish recognition model; after the cloud training data set of the dish recognition model is updated, the updated training data set is reused in the cloud to train the dish recognition model and update the network parameters, and after the training is completed, the restaurant local end is notified to update and synchronize.

[0019] As a preferred embodiment of the above-mentioned second aspect, the Dish-YOLO model is pre-trained using a picture data set uploaded by the local end of each restaurant before it is put into use, and the picture data set contains different plate pictures, each plate picture contains dishes served in the form of bowls and placed on the plate, the dishes in all plate pictures should cover all dishes sold by the restaurant, and the picture data set needs to be data enhanced to expand the sample size.

[0020] As a preferred embodiment of the second aspect, in the Dish-YOLO model, a total of 16 anchor boxes are set, namely (12, 46), (16, 111), (22, 76), (39, 72), (22, 166), (30, 113), (75, 79), (33, 198), (49, 148), (40, 282), (57, 327), (291, 53), (64, 205), (94, 294), (135, 154) and (194, 310), and 8 target bounding boxes are predicted at each scale with the help of the anchor box.

[0021] As a preferred embodiment of the second aspect, the training of the Dish-YOLO model and the dish recognition model are both performed in a cloud server, and the trained models are stored in the cloud server for downloading by the local restaurant.

[0022] As a preferred embodiment of the second aspect, the dish recognition model is a VGG16 network model.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1) The dish recognition method proposed in the present invention has high accuracy and short training time. The present invention abandons the commonly used OneStage method. Although the One Stage method has the advantage of fast detection speed, it has a long training time and poor recognition accuracy, and is not suitable for dish recognition scenarios. The present invention adopts a method of first positioning and then recognition, in which the positioning model only needs to be trained once with the data set collected in the initial use, and it can be used for a long time; the classification recognition model can be trained in real time according to the daily menu, and its training time only takes 10 to 15 minutes, and the accuracy rate can reach about 94%.

[0025] 2) In the dish recognition method of the present invention, the data set of the model can be automatically updated and is less affected by environmental factors. The present invention can automatically save dish images while identifying dishes, and add them to the training data set by the auxiliary settlement and data set update module. When environmental factors (such as light) change, the latest environmental information of the dish can be added to the training set for training, thereby reducing the impact of environmental changes on the accuracy of dish recognition.

[0026] 3) The dish recognition method proposed in the present invention has higher applicability and practicality. The present invention adopts a combination of the traditional cashier method and dish image recognition, which makes up for the defect of the traditional cashier system's heavy dish code recognition and increases the fault tolerance of the dish recognition method. Therefore, the method and system of the present invention have stronger practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of a dish recognition method and system based on Dish-YOLO;

[0028] Figure 2 This is an example of the original image sample of the Dish-YOLO model;

[0029] Figure 3 Label examples for the Dish-YOLO model's image samples;

[0030] Figure 4 This is the Dish-YOLO network structure diagram;

[0031] Figure 5 This is a diagram of the UI interface when the recognition results are provided for manual review on the restaurant's local terminal. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0033] In a preferred embodiment of the present invention, a dish recognition method based on Dish-YOLO is provided, which is established on a restaurant local end and a cloud server, wherein the restaurant local end refers to a client installed in the restaurant that actually sells dishes, and the cloud server establishes a communication connection with the restaurant local client through a wired or wireless network to transmit data. The common carrier of the restaurant local end is an automatic settlement device with a camera and a host computer, the camera is used to take a picture of the plate after the customer takes the dish, and the host computer obtains the plate picture through the slave computer, and provides an interactive interface for restaurant managers to review, modify, control and other operations through the display screen.

[0034] like Figure 1 As shown, the dish recognition method based on Dish-YOLO includes the following processes 1) to 3), and each sub-process can realize a specific execution function through a corresponding functional module. Sub-processes 1) to 3) are described in detail below.

[0035] 1) Use the dish recognition module to locate the dish area of ​​the plate image and recognize the dish in the single dish image:

[0036] For the plate after the target customer takes the dishes, the local end of the restaurant obtains the image of the plate area through the camera as the plate picture to be identified, and the plate picture contains one or more dishes selected by the target customer and placed on the plate in the form of bowls and plates. The trained Dish-YOLO model is used to locate the bowl and plate area of ​​each dish in the plate picture, and the bowl and plate area positioning frame of each dish is obtained, and then the single dish picture of each dish in the plate picture is extracted according to the positioning frame. Then, the single dish picture of each dish in the plate picture is input into the trained dish recognition model respectively, and all the dishes contained in the plate picture are identified to form a dish recognition result. This dish recognition result contains the classification result of the dishes in each positioning frame in the plate picture. After all the dishes contained in the plate picture are identified, all the dishes contained in the plate picture can be priced according to the preset unit price of each dish to form a pricing result. In order to facilitate subsequent display and analysis, the pricing results should include several individual information such as dish type, unit price, quantity, total price of each dish, and the total price of all selected dishes of the target customers formed by summarizing the individual information.

[0037] It should be specially explained that the Dish-YOLO model in the present invention is improved based on the YOLOv3 model. Through the analysis of a large number of dinner plate images to be identified, it is found that the area of ​​the dish to be identified often accounts for more than one-tenth of the entire image, which belongs to the medium and large target scales of YOLOv3, and the scale of small targets is rarely used. Therefore, the YOLOv3 model is improved in the present invention, the 52*52 scale layer of YOLOv3 is discarded, and only the 13*13 and 26*26 scale layers are retained. In order to adapt to the changes in the number of scale layers, the present invention also assigns more accurate anchor boxes to large targets in smaller feature maps, and finally the improved model is named Dish-YOLO. The final Dish-YOLO model structure is as follows Figure 4 shown.

[0038] Since the three scales of YOLOv3 are modified to two scales, the anchor box needs to be re-determined, and it can be optimized based on the final positioning effect. In this embodiment, 16 anchor boxes are taken, namely (12,46), (16,111), (22,76), (39,72), (22,166), (30,113), (75,79), (33,198), (49,148), (40,282), (57,327), (291,53), (64,205), (94,294), (135,154), (194,310). Each cell at each scale predicts 8 target borders with the help of 8 anchor boxes.

[0039] Before the Dish-YOLO model is put into use at each local end of a restaurant, it needs to be trained in advance using a picture data set uploaded by the local end of the restaurant. In order to ensure the reliability of the positioning of the Dish-YOLO model, the picture data set contains different pictures of dinner plates, each of which contains dishes placed on the dinner plates in the form of bowls and plates. The dishes in all the pictures of dinner plates in the picture data set should cover all the dishes sold by the restaurant and the types of bowls and plates used to hold the dishes. In order to ensure the positioning robustness of the Dish-YOLO model, the picture data set needs to be enhanced to expand the sample size. The data enhancement methods can include image rotation, scaling, translation, brightness adjustment, contrast adjustment, etc. In actual tests, the Dish-YOLO model is better suited for the positioning of the dinner plate area in the dinner plate picture than the original YOLOv3 model, which improves the accuracy of positioning.

[0040] In addition, any feasible classification model can be used as a dish recognition model in the present invention, and a common one is a multi-classifier such as a convolutional neural network. In this embodiment, the dish recognition model adopts a VGG16 network model, and its specific network results belong to the prior art, which will not be described in detail.

[0041] The specific training methods of the above-mentioned Dish-YOLO model and VGG16 network model can be implemented by using existing technologies, and are not limited to this.

[0042] Actual tests found that by adopting the method of first positioning and then identifying, the VGG16 network model can be trained in real time according to daily recipes. The training time only takes 10 to 15 minutes, and the final dish recognition accuracy can reach about 94%.

[0043] 2) Manually review the dish recognition and pricing results and update the cloud training data set through the auxiliary settlement and data set update module:

[0044] The local end of the restaurant obtains the dish recognition result and pricing result of the plate image obtained in the above step 1) and sends it to the local manual review end. At the manual review end, the restaurant manager can directly view the dish recognition result and pricing result through the device screen, and quickly check whether it matches the actual meal pickup result of the customer, and then confirm whether to enter the payment link through the device screen. The confirmation operation will generate a verification instruction and send it to the auxiliary settlement and data set update module. If the verification instruction is correct, the payment link will be entered. If the verification instruction is wrong, the change data of the manual review end will be received, and the payment link will be entered based on the changed dish recognition result and pricing result of the manual review end. The above device screen confirmation operation can be implemented by providing corresponding buttons on the UI interface of the device screen, such as setting two types of buttons, confirmation payment and modification. Clicking the confirmation payment button can directly enter the payment link, and clicking the modification button can modify the dish recognition result in the plate image, and regenerate the pricing result according to the modified dish recognition result. As a result, the traditional manual cashier settlement mode and the automatic dish recognition settlement mode are deeply integrated, making the dish intelligent recognition settlement system more practical and fault-tolerant. As the accuracy of dish recognition results continues to increase, managers only need to do a simple check before making payments, which greatly saves the time required to manually enter dishes.

[0045] In addition, in order to maximize the accuracy of dish recognition results, the auxiliary settlement and data set update module also needs to update the dish plate images and dish recognition results that finally enter the payment link in each round to the cloud training data set of the dish recognition model. It should be noted that if the dish recognition results obtained by the dish recognition model have been modified by the manual review end, the dish recognition results uploaded to the cloud should be those that have been manually modified to ensure accuracy.

[0046] 3) The dish recognition model is trained in the cloud through the model training and cloud storage modules and updated to the restaurant’s local end synchronously:

[0047] When the local end of the restaurant receives the plate image entered for the new dish, it sends it to the cloud, where the trained Dish-YOLO model is used to locate the new dish in the plate image, extract the single dish image of the new dish, generate the labeled training sample, and add it to the cloud training data set of the dish recognition model. It should be noted that the restaurant's entry of new dishes will exist when the restaurant first uses the method of the present invention and when the restaurant updates the dishes. Therefore, every time the local end of the restaurant receives a picture of a new dish, it needs to retrain the VGG16 network model as a dish recognition model before putting it into use so that the model can recognize new dishes.

[0048] In addition, whether it is the entry of new dishes, or the upload and expansion of sample data of existing dishes by the auxiliary settlement and data set update module, the cloud training data set will be updated. When the cloud training data set of the dish recognition model is updated, it is necessary to re-use the updated training data set on the cloud to train the dish recognition model and update the network parameters, and notify the restaurant to update the local end after the training is completed. Of course, the retraining and update of this model is not necessarily real-time, unless new dishes need to be entered and put into use immediately, otherwise the retraining and update of the model can be performed once a certain amount of new samples have been accumulated to save computing resources and avoid too frequent updates on the local end of the restaurant.

[0049] Therefore, the training of the Dish-YOLO model and the dish recognition model are both carried out in the cloud server, and the trained models are saved in the cloud server for download by the local restaurant. The local restaurant does not need to train the model, only the parameters of the model need to be updated, which greatly reduces the performance requirements for the local restaurant equipment and saves hardware and operating costs. Generally speaking, unless there are major changes in the way the dishes are served or the form of the dishes in the restaurant, the Dish-YOLO model can be reused for a long time without updating, but the dish recognition model needs to be constantly updated to ensure its accuracy.

[0050] The present invention mainly adopts dish identification, and manually changes to assist when identification errors occur, which can greatly save the time required for the restaurant's cashier. Corresponding to the dish identification method based on Dish-YOLO in the above embodiment, the present invention can further provide a dish identification system based on Dish-YOLO, which includes a dish identification module, an auxiliary settlement and data set update module, and a model training and cloud storage module. The dish identification module is used to implement the above sub-process 1), the auxiliary settlement and data set update module is used to implement the above sub-process 2), and the model training and cloud storage module is used to implement the above sub-process 3).

[0051] In general, the three modules communicate with each other and are interdependent. They can be continuously updated during use, and the user experience can be gradually optimized. The dish recognition module is mainly responsible for identifying the dishes purchased by customers and completing the pricing work, and delivering the pricing results to the auxiliary settlement module; while identifying the dishes, it automatically collects new dish pictures and delivers them to the auxiliary settlement and data set update module. The auxiliary settlement and data set update module mainly verifies the recognition results and pricing results obtained by the image recognition module. If the recognition is successful and there is no error, it will directly enter the payment link. If the recognition error is found, it will be manually changed and the final result will be used in the payment link; at the same time, the recognition result picture will be automatically uploaded to the recognition model data set of the corresponding dish, and the dish recognition model data set will be automatically updated to further improve the accuracy of dish recognition. In addition, the model training and cloud storage module is responsible for entering the dish pictures and making the dish recognition model data set. The restaurant can choose the dishes participating in the training to update the dish classification model at any time. At the same time, this module integrates the newly obtained dish images from the auxiliary settlement and data set update module with the previous dish image training set, retrains the model, increases the robustness and robustness of the dish recognition model, and thereby continuously improves the accuracy of image recognition.

[0052] In order to enable those skilled in the art to better understand the specific implementation of the present invention, the dish recognition method of Dish-YOLO shown in 1) to 3) above and the construction and use process of the system are demonstrated through a specific example below.

[0053] Example

[0054] In this example, the dish recognition method of Dish-YOLO shown in 1) to 3) above and the construction and use process of the system are demonstrated, which specifically includes the following steps:

[0055] 1. Restaurant model training and initial construction of cloud storage module

[0056] Step 1.1 Preparation and data preprocessing of positioning model dataset

[0057] In this example, the newly opened restaurant needs to manually input the plate image data The i-th image of the store numbered s for positioning training is labeled using the LabelImg labeling tool to label the dish area in the image. Different from object detection, this method labels all dishes as the same category, namely, bowl. Figure 1 The following is one of the original dinner plate image samples. After annotation, the image sample can be formed as follows Figure 2 Annotated sample shown.

[0058] The original image data and annotation box information used for Dish-YOLO positioning model training are used as the input data set. First, the incoming image is padded to 416*416 size, then the image is translated by -0.2 to 0.2 times in the horizontal or vertical direction, the image is randomly reduced or enlarged by 0.2 times, and the image is randomly flipped to obtain a mirror image. The input data set is enhanced by the above methods.

[0059] Step 1.2: Build, train and save the Dish-YOLO network model;

[0060] The positioning model of this example is improved based on the YOLOv3 model, discarding the 52*52 scale layer of YOLOv3, retaining only the 13*13 and 26*26 scale layers, and allocating more accurate anchor boxes to large targets in smaller feature maps. The improved model is named Dish-YOLO. In addition, after the three scales of YOLOv3 are modified to two scales in this embodiment, the anchor boxes need to be re-determined, and 16 anchor boxes are taken, namely (12,46), (16,111), (22,76), (39,72), (22,166), (30,113), (75,79), (33,198), (49,148), (40,282), (57,327), (291,53), (64,205), (94,294), (135,154), (194,310). Each cell at each scale predicts 8 object bounding boxes with the help of 8 anchor boxes. The final Dish-YOLO model structure is as follows: Figure 4 shown.

[0061] Input the input data into Dish-YOLO to start training the model and calculate the network accuracy. When calculating the accuracy, use the test set form and input it into the trained neural network. The accuracy is calculated based on whether the predicted value and the true value of each sample in the test set are consistent. When the test set loss value does not change by more than 0.05 in 50 epochs, it is considered to meet the requirements of model training, and the model is saved and named Dish-YOLOModel(s), otherwise continue training.

[0062] The positioning model Dish-YOLOModel(s) is used to locate the position of a single dish in a dinner plate image. When the shape and size of the tableware in the restaurant are fixed, the positioning model can be used for a long time and does not need to be updated frequently.

[0063] Step 1.3 Preparation and data preprocessing of dish recognition model dataset

[0064] 1. Create a data set: The training data set for the dish recognition model in this example first comes from the plate images input by the restaurant. represents the jth image of the i-th dish. First, use the positioning model Dish-YOLOModel(s) obtained in step 1.2 to locate the plate image The position of the plate area of ​​a single dish in the image is obtained, and the position information of the dish is obtained. Top, bottom, left, and right represent the coordinate values ​​of the top, bottom, left, and right of the dish position respectively. The dish is cut out according to the coordinate information to obtain the image. To get Perform data augmentation to obtain the training set Dish(i) for the dish recognition model train , and from the obtained training set Dish(i) train Randomly select 20% of the images as the test set Dish(i) test .

[0065] In addition, for new restaurants, the training data set of their dish recognition model only contains the plate images input by the restaurant.

[0066] If the auxiliary settlement and data set update module uploads a new plate image after the order is settled, the two parts of dish data can be fused to obtain a dish recognition data set for training the dish recognition model.

[0067] 2. Dataset preprocessing: Dish(i) train The image is scaled to 224*224 in size and the edges are padded. Then the image is flipped, and the image contrast and brightness are changed for data enhancement.

[0068] Step 1.4 Training and saving the dish recognition model

[0069] 1. Load the training data, process the image data, and generate the input and output data formats required by the model. The input data is the training set Dish(i) train , the image data is uniformly divided by 255 for normalization processing to ensure that the data value is between 0 and 1, and then the image is converted into a 512*512 numerical matrix and input into the VGG16 network for training. According to the input image features, select appropriate hyperparameters to build a convolutional neural network model and train the neural network. The loss function uses categorical_crossentropy, and the formula is:

[0070]

[0071] Among them, assuming that there are n categories of prediction targets, the true label of the i-th sample is The predicted value is

[0072] 2. Start training the model and calculate the network accuracy. Use the test set as input to the trained neural network, and calculate the accuracy based on whether the predicted value of each sample in the test set is consistent with the true value. When the test set loss value does not change by more than 0.05 in 12 epochs, we consider it to meet the requirements and save the model, otherwise continue training. The obtained model is named VGGModel(s) according to the store identifier s and the training version identifier m m .

[0073] 2. Use of dish recognition module

[0074] The dish recognition module first needs to download the trained Dish-YOLOModel(s) from the model training and cloud storage module, and detect the VGGModel(s) at the same time m When the model is updated, the restaurant staff will be prompted to download it. Positioning model Dish-YOLOModel(s) and dish recognition model VGGModel(s) m It is the basis for the work of the dish recognition module.

[0075] The dish recognition module is used to interact with customers and collect image data of dishes on the plates in the current scene. After collecting the image data of the dishes on the plates, the dish recognition process will begin. The specific steps are as follows:

[0076] Step 2.1 Dish positioning

[0077] Use the trained Dish-YOLOModel(s) model to locate the image to be identified, obtain the top, bottom, left, and right position information of each dish, and crop the single dish image of a single dish according to the position information.

[0078] Step 2.2 Dish identification

[0079] For the single dish image obtained in step 2.1, use the corresponding dish recognition model VGGModel(s) m Perform identification and obtain the identification result of the dish.

[0080] 3. Auxiliary Settlement and Dataset Update Module Process

[0081] If there are uncertain factors such as unreasonable collection of recognition images due to sudden changes in the environment or incorrect placement of the dining staff's plates, which may lead to errors in the recognition results, this example provides a corresponding auxiliary settlement function. At the same time, after each settlement is completed, the collected recognition images can be added to the corresponding image recognition data set for fusion, so that the image recognition data set can contain data with a wider time span, greatly reducing the impact of environmental changes on the recognition accuracy. The specific process of this module is as follows:

[0082] 3.1 Result verification and settlement. The present invention deeply integrates the traditional cashier method and the plate image recognition cashier method. First, the dish recognition results in step 2.2 are displayed, and the cashier verifies the recognition results. If there is a recognition error, it is manually changed until the result is correct and payment can be made. In this example, Figure 2 As shown, a UI interface is provided on the device screen as a manual review end, which displays the positioning boxes of all single dishes in the plate image and the recognition results of the corresponding dish categories, and lists the pricing results according to the unit price and quantity. In this interface, a selection button for confirming to enter the payment and a modification button are provided. If the cashier confirms the dish recognition result and the pricing result and verifies that they are correct, click the payment method required by the customer to enter the payment process. If the verification result is wrong, use the modification function of the manual review end on the device screen to immediately change the recognition result, correct the dish classification corresponding to each single dish picture, and then enter the payment process.

[0083] 3.2 Image collection and automatic update of data set. At the time of settlement, the plate images collected from the order are automatically stored and added to the dish recognition data set for training to obtain the updated dish recognition model VGGModel(s) m+1 , so that the dataset can contain image data with a wider time span, which can effectively solve the problem of decreased recognition accuracy caused by changes in environmental factors.

[0084] In this example, the Dish-YOLO dish recognition method and system can accurately identify dishes and quickly calculate prices during actual operation, greatly reducing cashier settlement time, improving work efficiency, reducing queuing and waiting time, and thus improving the restaurant's intelligence level.

[0085] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A dish recognition method based on Dish-YOLO, characterized in that: include: The restaurant obtains a plate image to be identified at the local end, wherein the plate image contains one or more dishes selected by the target customer and placed on the plate in the form of bowls and plates, and uses the trained Dish-YOLO model to locate the bowl and plate area of ​​each dish in the plate image, and then extracts a single dish image of each dish in the plate image; then the single dish image of each dish in the plate image is respectively input into the trained dish recognition model to identify all the dishes contained in the plate image, and form a dish recognition result; according to the preset unit price of the dish, all the dishes contained in the plate image are priced to form a pricing result; the Dish-YOLO model adopts an improved YOLOv3 model, which is obtained by the YOLOv3 model by discarding the 52*52 scale layer but retaining the 13*13 and 26*26 scale layers and then reallocating the anchor point frame; The restaurant local end obtains the dish recognition result and pricing result of the plate image, sends them to the local manual review end and receives the verification instruction issued by the manual review end. If the verification instruction is correct, the payment link is entered. If the verification instruction is wrong, the change data of the manual review end is received, and the payment link is entered based on the dish recognition result and pricing result changed by the manual review end. The plate images and dish recognition results entering the payment phase in each round are updated into the cloud training data set of the dish recognition model; When the restaurant local end receives the plate image entered for the new dish, it sends it to the cloud, and the trained Dish-YOLO model is used in the cloud to locate the new dish in the plate image, extract the single dish image of the new dish, regenerate the labeled training sample and add it to the cloud training data set of the dish recognition model; when the cloud training data set of the dish recognition model is updated, the updated training data set is re-used in the cloud to train the dish recognition model and update the network parameters, and after the training is completed, the restaurant local end is notified to update and synchronize.

2. The dish recognition method based on Dish-YOLO according to claim 1, characterized in that: Before the Dish-YOLO model is put into use at each restaurant's local end, it is pre-trained using an image dataset uploaded by the restaurant's local end, and the image dataset contains different plate images, each of which contains dishes served in bowls and placed on the plate. The dishes in all plate images should cover all dishes sold by the restaurant, and the image dataset needs to undergo data enhancement to expand the sample size.

3. The dish recognition method based on Dish-YOLO according to claim 1, characterized in that: In the Dish-YOLO model, a total of 16 anchor boxes are set, namely (12,46), (16,111), (22,76), (39,72), (22,166), (30,113), (75,79), (33,198), (49,148), (40,282), (57,327), (291,53), (64,205), (94,294), (135,154) and (194,310). At each scale, 8 target bounding boxes are predicted with the help of the anchor boxes.

4. The dish recognition method based on Dish-YOLO according to claim 1, characterized in that: The training of the Dish-YOLO model and the dish recognition model are both performed in the cloud server, and the trained models are saved in the cloud server for downloading by the restaurant local end.

5. The dish recognition method based on Dish-YOLO according to claim 1, characterized in that: The dish recognition model is a VGG16 network model.

6. A dish recognition system based on Dish-YOLO, characterized in that: include: The dish recognition module is used to obtain a dish image to be recognized at the local end of the restaurant, wherein the dish image contains one or more dishes selected by the target customer and placed on the dish in the form of bowls and plates, and use the trained Dish-YOLO model to locate the dish area of ​​each dish in the dish image, and then extract a single dish image of each dish in the dish image; then input the single dish image of each dish in the dish image into the trained dish recognition model respectively, recognize all the dishes contained in the dish image, and form a dish recognition result; according to the preset dish unit price, all the dishes contained in the dish image are priced to form a pricing result; the Dish-YOLO model adopts an improved YOLOv3 model, which is obtained by the YOLOv3 model by discarding the 52*52 scale layer but retaining the 13*13 and 26*26 scale layers and then reallocating the anchor point frame; The auxiliary settlement and data set update module is used to obtain the dish recognition result and pricing result of the plate image at the local end of the restaurant, send them to the local manual review end and receive the verification instruction issued by the manual review end, if the verification instruction is correct, enter the payment link, if the verification instruction is wrong, receive the change data of the manual review end, and enter the payment link based on the dish recognition result and pricing result changed by the manual review end; The plate images and dish recognition results entering the payment phase in each round are updated into the cloud training data set of the dish recognition model; The model training and cloud storage module is used to receive plate images uploaded by the restaurant local end for the entered new dishes, use the trained Dish-YOLO model in the cloud to locate the new dishes in the plate images, extract the single dish images of the new dishes, regenerate labeled training samples and add them to the cloud training data set of the dish recognition model; after the cloud training data set of the dish recognition model is updated, the updated training data set is reused in the cloud to train the dish recognition model and update the network parameters, and after the training is completed, the restaurant local end is notified to update and synchronize.

7. The dish recognition system based on Dish-YOLO according to claim 6, characterized in that: Before the Dish-YOLO model is put into use at each restaurant's local end, it is pre-trained using an image dataset uploaded by the restaurant's local end, and the image dataset contains different plate images, each of which contains dishes served in bowls and placed on the plate. The dishes in all plate images should cover all dishes sold by the restaurant, and the image dataset needs to undergo data enhancement to expand the sample size.

8. The dish recognition system based on Dish-YOLO as claimed in claim 6, characterized in that: In the Dish-YOLO model, a total of 16 anchor boxes are set, namely (12,46), (16,111), (22,76), (39,72), (22,166), (30,113), (75,79), (33,198), (49,148), (40,282), (57,327), (291,53), (64,205), (94,294), (135,154) and (194,310). At each scale, 8 target bounding boxes are predicted with the help of the anchor boxes.

9. The dish recognition system based on Dish-YOLO according to claim 6, characterized in that: The training of the Dish-YOLO model and the dish recognition model are both performed in the cloud server, and the trained models are saved in the cloud server for downloading by the restaurant local end.

10. The dish recognition system based on Dish-YOLO according to claim 6, characterized in that: The dish recognition model is a VGG16 network model.

Images