A smart canteen food detection method based on deep model and quantification technology

Abstract

The present invention discloses a smart canteen food detection method based on deep model and quantization technology, which belongs to the field of computer vision technology. While improving the accuracy and speed of current food detection, it reduces the hardware requirements for model deployment. The present invention includes the following steps: first, a large number of canteen tray food images are collected, and the types and positions of the food in the images are marked; then the data is sent to a convolutional neural network designed for food detection for training until the network converges and a weight file is obtained; then the size of the model is reduced by quantization technology and deployed to an embedded motherboard; finally, the food in the food image can be quickly detected by the neural network and the weight file. The present invention solves the current problems of low food recognition accuracy and slow reasoning speed, and effectively solves the problems of difficulty in deploying deep models and high dependence on hardware computing power and video memory.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a smart canteen food detection method based on deep model and quantization technology. Background Art

[0002] The rapid development of computer vision in recent years has made intelligent transportation, unmanned supermarkets and other scenarios a reality. While people are enjoying convenience, the concept of smart cities is gradually gaining popularity. In some large-scale food trading places, such as canteens, many transactions are processed every day. According to surveys, many canteens currently use self-service meal selection and manual settlement methods. Since the settlement platform is currently manually operated, it is often inefficient and leads to long queues. This not only wastes customer time, but also precious human resources. Therefore, we proposed a practical solution and designed a canteen unmanned fee settlement system to make the settlement process intelligent. This not only improves efficiency and liberates manpower, but also promotes the construction of smart canteens in smart cities.

[0003] At present, the main methods for canteen food recognition are detection methods based on convolutional neural networks and recognition methods based on RFID. Among them, the detection algorithm based on convolutional neural networks is divided into one-stage detection algorithm and two-stage detection algorithm. The two-stage algorithm is a convolutional neural network based on region proposal. It first calculates the region where objects may exist (proposal region), and then detects the specific type and location of the objects in this region. Another one-stage algorithm only uses convolutional neural network CNN to directly predict the category and location of different targets. In comparison, the single-stage algorithm has obvious advantages in detection speed, but its accuracy is not as good as the two-stage algorithm in many tasks. The RFID-based food recognition method uses wireless radio frequency technology to identify food. This is a very cumbersome and traditional method. The smart canteen based on this technology needs to mark each food with RFID.

[0004] From the perspective of technological development, canteen food recognition based on convolutional neural networks is a better choice. In 2018, the paper "Grab, pay, and eat: Semantic food detection for smart restaurants" in IEEE Transactions on Multimedia studied food recognition in smart restaurants based on convolutional neural networks. In their method, a picture passes through a CNN food segmentation module and a food detection module, which is a parallel module. Then background elimination and non-maximum suppression are performed to achieve semantic food detection. Among them, it is mainly based on the improvement of the one-stage algorithm YOLOv2. In this process, multiple modules are required for processing, which obviously performs poorly in terms of reasoning speed. At the same time, it is actually used to help restaurants charge, and its accuracy is not convincing enough. In 2020, in the journal IEEE Transactions on Industrial Informatics, the paper "A Deep Transfer Learning Solution for Food Material Recognition Using Electronic Scales" proposed an automated procurement solution for food raw materials. The types of food raw materials are identified through a deep model, and the weighing function of the electronic scale is combined to achieve the purpose of automated procurement of food raw materials using electronic scales. In order to improve the accuracy, the author used a relatively deep neural network, which resulted in a large model weight, which greatly increased the difficulty of deployment.

[0005] Therefore, the main defects of the methods in the prior art are: the model accuracy and robustness of the method based on multi-module convolutional neural network are not high, and it is time-consuming. At the same time, the deep model has a large weight and is difficult to deploy. Summary of the invention

[0006] In order to solve the technical problems mentioned in the above background technology, the present invention proposes a smart canteen food detection method based on deep model and quantization technology, which solves the current problems of low food recognition accuracy and slow reasoning speed, and effectively solves the difficulties in deploying deep models and the high dependence on hardware computing power and video memory.

[0007] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0008] A smart canteen food detection method based on deep model and quantification technology, comprising the following steps:

[0009] (1) Image acquisition process: Collect a large number of images of food on canteen trays and mark the types and locations of the food in the images;

[0010] (2) Neural network training process: the image data annotated in step (1) is sent to the convolutional neural network designed for food detection for training until the network converges and a weight file is obtained;

[0011] (3) Deep model quantization and deployment process: Use quantization technology to appropriately reduce the size of the model and deploy it to the embedded motherboard;

[0012] (4) Test image detection process: Use the neural network and weight file to detect food targets in the test image and output the detection results.

[0013] In the above steps, in step (1), the collected images are cleaned and processed to filter out blurry, incomplete food and other photos that do not meet the requirements, and then the dish targets in the remaining images are marked to mark the type and location of the food in the image;

[0014] The neural network training method in step (2) is a one-stage target detection YOLO method. Compared with other target detection methods, the YOLO method has a relatively high accuracy and an obvious advantage in reasoning speed. Step (2) specifically includes the following steps:

[0015] (21) The residual module is used in the backbone network to effectively extract image information and significantly reduce the amount of calculation;

[0016] (22) Use the SPP module to perform multiple receptive field fusion to help the network perform multi-scale recognition;

[0017] (23) Change the neck part of the yolo network and replace PANet with BIFPN;

[0018] (24) Before training the network, recalculate the anchors of the data set to make the network easier to converge, converge faster, and improve the IoU of the model;

[0019] (25) When training the network, the food images input to the network are augmented. While changing the color gamut of the images, four images are spliced ​​together by random scaling, random cropping, and random arrangement and input into the network;

[0020] (26) Using CIoU loss as the network loss function can effectively improve the problem that the original IoU Loss calculation method converges slowly and cannot converge in some cases, making the network converge faster;

[0021] (27) setting the training hyperparameters of the network, performing training, and obtaining a network file and a weight file that can be used for food detection;

[0022] Furthermore, in step (21), a residual convolution module similar to CSPNet is used to output feature maps of different sizes through multiple dimensionality reduction.

[0023] Furthermore, in step (23), Neck uses a three-layer BIFPN structure to perform feature fusion, fusing the feature maps of four different levels output from the backbone network;

[0024] Furthermore, in step (27), the ImageNet pre-trained weights are used as the initial weights, the learning rate is set to 0.0001, the number of iterations is set to 200,000 times, and the batch size is set to 128. When the loss function converges or the maximum number of iterations is reached, the training is stopped to obtain the network file and weight file that can be used for cafeteria food detection;

[0025] Step (3) specifically includes the following steps:

[0026] (31) By quantizing the saved floating-point model, it is quantized into an integer model to reduce the size of the model;

[0027] (32) Through model conversion, the weight model is converted into an RKNN model and deployed on the embedded motherboard AIO-3399Pro;

[0028] Furthermore, the quantization method used in step (31) is Post training quantization, a method of training first and then quantizing, quantizing the float64 model to asymmetric_quantized-u8, and the calculation formula is as follows:

[0029]

[0030] quant=cast_to_bw

[0031] Among them, quant represents the quantized number, float_num represents the floating point value, scale is a float32 type, zero_points is an int32 type, representing the quantized value corresponding to the real number 0, and finally saturates quant to [range_min, range_max]

[0032] rang_max=255

[0033] range_min=0

[0034] The corresponding inverse quantization:

[0035] float_num=scale(quant-zero_point)

[0036] Step (4) specifically includes the following steps:

[0037] (41) Send the test image into the improved YOLO backbone network to obtain the convolution feature map;

[0038] (42) The convolution feature map is processed by the Yolo algorithm to output the predicted boundary value and classification value;

[0039] (43) Set the threshold and filter out the final detection results through non-maximum suppression.

[0040] Beneficial effects: The present invention proposes a smart canteen food detection method based on deep models and quantification technology. By improving the backbone network of the one-stage method yolo, the computational complexity of the network is reduced and the detection speed is improved. By improving the feature fusion module, more useful features can be extracted, thereby improving the accuracy and speed of food detection. Combined with quantification technology, the hardware requirements for model deployment are reduced, the difficulty of model deployment is greatly reduced, and a practical smart canteen charging plan is proposed, thereby providing an important technological advancement for the construction of smart canteens in smart cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0042] Figure 2 is a flow chart of step 2 of an embodiment of the present invention;

[0043] Figure 3 is a flowchart of step 3 of an embodiment of the present invention;

[0044] Figure 4 is a flowchart of step 4 of an embodiment of the present invention;

[0045] Figure 5 and Figure 6 is a test result diagram in an embodiment of the present invention;

[0046] Figure 7 It is an augmented detection result diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0048] like Figure 1 As shown in the figure, the smart canteen food detection method based on deep model and quantification technology includes the following steps:

[0049] Step 1: Collect a large number of images of food on canteen trays and mark the types and locations of the food in the images;

[0050] Step 2: Send the data to the convolutional neural network designed for food detection for training until the network converges and obtains the weight file;

[0051] Step 3: Use quantization technology to appropriately reduce the size of the model and deploy it to the embedded motherboard;

[0052] Step 4: Use the neural network and weight file to detect food objects in the test image and output the detection results.

[0053] In this embodiment, step 1 adopts the following scheme:

[0054] The collected images are cleaned and processed to filter out blurry photos, incomplete food photos and other photos that do not meet the requirements. Then, the food targets in the remaining images are marked, and the types and locations of the food in the images are marked.

[0055] In this embodiment, step 2 adopts the following scheme:

[0056] First of all, the neural network method is a one-stage target detection YOLO method. Compared with other target detection methods, the YOLO method has a very high accuracy and an obvious advantage in reasoning speed.

[0057] like Figure 2 As shown, the specific expansion steps of step 2 are as follows:

[0058] Step 201: Improve the network structure of yolo, mainly improving the structure of its backbone network and neck part;

[0059] Furthermore, the backbone network in step 201 mainly uses a residual convolution module similar to CSPNet to output feature maps of different sizes through multiple dimensionality reduction. The Neck part uses a three-layer BIFPN structure for feature fusion, fusing the feature maps of four different levels output from the backbone network;

[0060] Step 202: before training the network, recalculate the anchors of the data set and replace the original anchors to make the network converge more easily and faster, thereby improving the IoU of the model;

[0061] Step 203: Using CIoU loss as the loss function of the network can effectively improve the problem that the original IoU Loss calculation method converges slowly in calculation and cannot converge in some cases, so that the network converges faster;

[0062] Step 204: Use the ImageNet pre-trained weights as the initial weights, set the learning rate to 0.0001, set the number of iterations to 200,000, and set the batch size to 128;

[0063] Step 205: Mosaic enhancement is performed on the input image and training is performed. When the loss function converges or the maximum number of iterations is reached, the training is stopped to obtain a network file and a weight file that can be used for cafeteria food detection.

[0064] like Figure 3 As shown, the specific expansion steps of step 3 are as follows:

[0065] Step 301: Use the Post training quantization method and set the quantization type to asymmetric_quantized-u8 type;

[0066] Furthermore, the quantization calculation formula in step 301 is as follows:

[0067]

[0068] quant=cast_to_bw

[0069] Among them, quant represents the quantized number, float_num represents the floating point value, scale is a float32 type, zero_points is an int32 type, representing the quantized value corresponding to the real number 0, and finally saturates quant to [range_min, range_max]

[0070] rang_max=255

[0071] range_min=0

[0072] The corresponding inverse quantization:

[0073] float_num=scale(quant-zero_point)

[0074] Step 302: Import the original model into the RKNN Toolkit and convert it into an rknn model file that can be used by NPU;

[0075] Step 303: Deploy the converted RKNN model to the embedded motherboard AIO-3399PRO.

[0076] like Figure 4 As shown, the specific expansion steps of step 4 are as follows:

[0077] (401) Sending the test image into the network to obtain a convolutional feature map;

[0078] (402) Processing the convolution feature map through the Yolo algorithm, outputting the predicted boundary value and classification value;

[0079] (403) The threshold is set to 0.5, and the final detection result is filtered out by non-maximum suppression.

[0080] Figure 5 and Figure 6 They are canteen food images and detection results using the method of the present invention. After testing, the above method can achieve 99.6% food detection accuracy with a small number of samples and has a faster reasoning speed. Figure 7 This is the result of detecting the image after augmentation. The augmented image with significant changes in color, position and shape also has excellent detection effect, showing that the food detection model has extremely high robustness.

[0081] The above-described embodiments are only for illustrating the preferred implementation modes of the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solutions in accordance with the technical ideas proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A smart canteen food detection method based on deep model and quantification technology, characterized in that: The following steps are involved: (1) Image acquisition process: Collect a large number of images of food on canteen trays and mark the types and locations of the food in the images; (2) Neural network training process: The image data annotated in step (1) is sent to the convolutional neural network designed for food detection for training until the network converges and a weight file is obtained, which is specifically: (21) Using the residual convolution module in the backbone network, the feature maps of different sizes are output through multiple dimensionality reduction; (22) Use the SPP module to perform multiple receptive field fusion to help the network perform multi-scale recognition; (23) Change the neck part of the yolo network and replace PANet with BIFPN. Neck uses a three-layer BIFPN structure for feature fusion, fusing the feature maps of four different levels output from the backbone network. (24) Before training the network, recalculate the anchors of the data set to make the network easier to converge, converge faster, and improve the IoU of the model; (25) When training the network, the food images input to the network are augmented. While changing the color gamut of the images, four images are spliced ​​together by random scaling, random cropping, and random arrangement and input into the network; (26) Use CIoU loss as the network loss function, use ImageNet pre-trained weights as initial weights, set the network training hyperparameters, and perform training. When the loss function converges or reaches the maximum number of iterations, stop training to obtain network files and weight files that can be used for cafeteria food detection; (3) Deep model quantization and deployment process: The saved floating-point model is quantized into an integer model through quantization technology, and the size of the model is appropriately reduced. The weight model is converted into an RKNN model through model conversion and deployed to the embedded motherboard. The quantization method used is Post training quantization, and the float64 model is quantized into asymmetric_quantized-u8. The calculation formula is as follows: quant=cast_to_bw Among them, quant represents the quantized number, float_num represents the floating point value, scale is a float32 type, zero_points is an int32 type, representing the quantized value corresponding to the real number 0, and finally saturates quant to [range_min, range_max] rang_max=255 range_min=0 The corresponding dequantization: float_num=scale(quant-zero_point); (4) Test image detection process: Use the neural network and weight file to detect food targets in the test image and output the detection results.

2. The smart canteen food detection method based on deep model and quantification technology according to claim 1 is characterized in that: In step (1), the collected images are cleaned and processed to filter out photos that do not meet the requirements, and then the food targets in the remaining images are marked to indicate the type and location of the food in the image.

3. The smart canteen food detection method based on deep model and quantification technology according to claim 1 is characterized in that: In step (26), the learning rate is set to 0.0001, the number of iterations is set to 200,000, and the batch size is set to 128.

4. The smart canteen food detection method based on deep model and quantification technology according to claim 1 is characterized in that: Step (4) specifically includes the following steps: (41) Send the test image into the improved YOLO backbone network to obtain the convolution feature map; (42) The convolution feature map is processed by the Yolo algorithm to output the predicted boundary value and classification value; (43) Set the threshold and filter out the final detection results through non-maximum suppression.

Images