Multi-type dish identification method and system capable of being used for virtual equipment

Abstract

The invention discloses a multi-type dish identification method and system capable of being used for virtual equipment, and belongs to the field of target identification of deep learning. The method comprises the following steps: collecting dish images, and analyzing and processing; constructing a dish image data set; constructing a target recognition model based on a YOLOv8 algorithm; using the dish image data set to obtain a trained dish identification model, and combining any dish image to obtain a final prediction result; converting the improved dish identification model into an ONNX format; a Pico MR environment perception application is developed by Unity, and a real environment is obtained; obtaining the dish identification model output in the ONNX format by using a Unity Sensitive plug-in; and packaging the application into an apk format, and installing the apk format in Pico 4Pro to identify and display real dishes. According to the invention, the virtual reality device is used for dish identification, consumers can more intelligently and visually know dish information by means of the virtual-reality combination visualization effect, and the service quality of catering enterprises is also positively influenced.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and more specifically, relates to a method and system for recognizing multiple types of dishes that can be used in virtual devices. Background Art

[0002] Dishes are not only the presentation of food, but also the embodiment of culture, history, geography and people's lifestyle. Through dishes, people can understand and experience the unique culture of different regions and nationalities, and promote cultural exchanges and mutual understanding. With the increasingly close integration of virtual reality technology and the catering industry, the application of neural networks in the field of dish recognition is becoming more and more extensive. Using virtual reality equipment for dish recognition, with its virtual and real combination of visualization effects, consumers can understand dish information more intelligently and intuitively, which will also have a positive impact on the service quality of catering companies.

[0003] However, the accuracy of existing dish recognition models still needs to be improved, especially for Chinese food, where even the same dishes have large feature differences. In the actual dining process, factors such as occlusion between dishes and irregular small target dishes will also affect the recognition rate and detection speed of the model. In addition, the memory and computing power of VR devices are limited, and common algorithms are difficult to meet their deployment requirements and fast reasoning.

[0004] At present, it is difficult to achieve a win-win situation in terms of lightweight and accuracy in the research of dish recognition. Therefore, it is urgent to ensure the original recognition accuracy of the model while increasing the recognition speed of the model as much as possible and reducing the model size, so that it can be applied smoothly on hardware devices with limited computing power.

[0005] After searching, the Chinese patent application number is 2024107479523, the application date is June 11, 2024, and the name of the invention is: dish recognition method, device, terminal equipment and storage medium; after obtaining the target dish image, the application extracts the feature vector of the target dish image through the convolutional neural network model optimized by the cross entropy loss function and the triplet loss function, and the extraction efficiency is higher, and the dish feature vector that can be used for subsequent retrieval in the retrieval library is obtained, so as to determine the dish category corresponding to the target dish image, and the error between the dish category finally obtained and the actual dish category of the target dish image is smaller, and the accuracy is higher. However, the application device processor and memory of this application are not limited, so there is a lack of research on the lightweight convolutional neural network model. Summary of the invention

[0006] 1. Technical problem to be solved by the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a method and system for identifying multiple types of dishes that can be used in virtual devices. The present invention improves the YOLOv8 model to improve the accuracy of dish recognition, performs knowledge distillation on the improved model, reduces the number of model parameters, and improves the inference speed, so that the dish recognition function can be integrated into mobile devices with limited computing power such as VR.

[0008] 2. Technical solution

[0009] In order to achieve the above object, the technical solution provided by the present invention is:

[0010] A method for identifying multiple types of dishes that can be used in a virtual device of the present invention comprises the following steps:

[0011] Step 1: Data collection and processing;

[0012] The dishes are photographed by mobile phones and augmented with images from the Internet, and the collected images are analyzed and processed to match the resolution of the model input images.

[0013] Step 2: Build a food image dataset;

[0014] Use the processed images in step 1 to create a dataset, annotate the dish categories and bounding boxes in the images, and call the annotated dish categories and bounding boxes the true categories and true boxes. Then divide the annotated dataset into training set, validation set, and test set. In addition to the self-created dataset mentioned above, the public dish dataset VieroFood-172 is also used, and its dataset is divided in the same way.

[0015] Step 3: Build a dish recognition model;

[0016] Based on YOLOv8, a lightweight dish recognition algorithm SCR-YOLO is proposed, whose structure is as follows Figure 2 As shown. First, the backbone network is changed to the SG-Conv structure, which adopts the SPD layer enhanced downsampling technology of SPD-Conv and replaces the non-step convolution in SPD-Conv with GhostConv to reduce the computational complexity of the model. Then, the feature fusion network adds the attention mechanism CBAM. At the same time, the repulsive loss function is added to the original loss function to improve the detection accuracy under the occlusion of dishes. Finally, using the channel knowledge distillation (CWD) technology, the student model YOLOv8s learns the output and intermediate representation of the teacher model SCR-YOLO, so that the student model can keep the model size small while being as close to the performance of the teacher model as possible.

[0017] Step 4: Model migration;

[0018] Convert the model to ONNX format and package it into the Pico application created by Unity.

[0019] Step 5: Develop a dish recognition system;

[0020] Develop Pico MR environment perception function, obtain real food images, use Unity Sentis plug-in to obtain the food recognition model output in ONNX format, convert it into rendering texture and copy it to the screen.

[0021] Step 6: Application packaging;

[0022] Package the dish recognition system developed in step 5 into an apk file and install it on the Pico 4Pro device. Start the application, first see the real environment, take a screenshot of the dish screen, the system recognizes the image, and displays the result on the desktop.

[0023] 3. Beneficial effects

[0024] Compared with the existing known technologies, the technical solution provided by the present invention has the following significant effects:

[0025] (1) The present invention provides a method for identifying multiple types of dishes that can be used in virtual devices. The YOLOv8 model is improved to address the problems of occlusion between dishes and difficulty in identifying small target dishes in real dishes. Specifically, the backbone network is changed to the SG-Conv structure, which uses the SPD layer enhanced downsampling technology of SPD-Conv and replaces the non-stride convolution in SPD-Conv with GhostConv to reduce the computational complexity of the model; the feature fusion network adds the attention mechanism CBAM; and the repulsion loss function is added to the original loss function to improve the detection accuracy under dish occlusion. Experimental results show that the proposed algorithm improves mAP50 by 4.1% and 2.6% respectively compared with YOLOv8 on the self-created Chinese food dataset CF-8 and the public dataset VireoFood-172, providing a higher accuracy rate;

[0026] (2) A method for identifying multiple types of dishes that can be used in virtual devices of the present invention performs CWD knowledge distillation on a dish recognition model with high accuracy. As a result, while ensuring high accuracy, the model parameters are reduced by 72.8%, the computational complexity is reduced by 70.9%, and the model reasoning is accelerated by about 6 times on the mobile terminal;

[0027] (3) The present invention provides a multi-category dish recognition system that can be used for virtual devices, which can integrate the dish recognition model into mobile devices with limited computing power, so that the dish recognition function can be more quickly applied to the catering industry. The virtual and real combined visualization effect provided by the virtual reality device can enhance the user experience and have a positive impact on the service quality of the catering industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method for identifying multiple types of dishes that can be used in a virtual device according to the present invention;

[0029] Figure 2 This is the network structure diagram of the improved dish recognition model SCR-YOLO;

[0030] Figure 3 This is the structural diagram of the SPD layer in SG-Conv;

[0031] Figure 4 This is the structure diagram of GhostConv in SG-Conv;

[0032] Figure 5 It is the structure diagram of CWD knowledge distillation;

[0033] Figure 6 This is the effect diagram of occlusion recognition between dishes;

[0034] Figure 7 Identify the effect diagram of small target dishes;

[0035] Figure 8 This is the effect diagram of dish recognition in VR equipment. DETAILED DESCRIPTION

[0036] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments.

[0037] Example 1

[0038] Combination Figure 1 In this embodiment, a method for identifying multiple types of dishes that can be used in a virtual device comprises the following steps:

[0039] Step 1: Data collection and processing;

[0040] The dishes were photographed by mobile phones and expanded through the Internet. The collected images were analyzed and processed to match the resolution of the model input image. In this embodiment, 5,644 Chinese food images of 8 categories were obtained.

[0041] Step 2: Build a food image dataset;

[0042] Use the processed images in step 1 to create a YOLO format dataset, use LabelImg software to annotate the dish categories and bounding boxes in the image, and call the annotated dish categories and bounding boxes the true categories and true boxes. Then randomly divide the annotated dataset into training set, validation set, and test set in a ratio of 7:2:1. In addition to the above-mentioned self-created dataset, the public dish dataset VieroFood-172 is also used, which is annotated and divided into datasets in the same way.

[0043] Step 3: Build a dish recognition model;

[0044] Based on YOLOv8, a lightweight dish recognition algorithm SCR-YOLO is proposed, whose structure is as follows Figure 2 As shown. First, the backbone network is changed to the SG-Conv structure, which adopts the SPD layer enhanced downsampling technology of SPD-Conv and replaces the non-step convolution in SPD-Conv with GhostConv to reduce the computational complexity of the model. Then, the feature fusion network adds the attention mechanism CBAM. At the same time, the repulsion loss function Repulsion Loss is added to the original loss function to improve the detection accuracy under the occlusion of dishes. Finally, using the channel knowledge distillation (CWD) technology, the student model YOLOv8s learns the output and intermediate representation of the teacher model SCR-YOLO, so that the student model can keep the model size small while being as close to the performance of the teacher model as possible.

[0045] It is worth noting that the dish recognition model constructed based on the YOLOv8 algorithm in this embodiment adopts the SG-Conv structure in the backbone network. It uses the SPD layer of SPD-Conv for downsampling, and avoids information loss in traditional stride convolution and pooling operations by converting information in the spatial dimension into the depth dimension. This enhances the detection performance of small objects in complex scenes and improves the generalization and robustness of the model. GhostConv is used to replace the non-stride convolution in SPD-Conv, and linear transformation is used to generate redundant feature maps, which reduces the amount of network calculation and model parameters, and enhances the feasibility of actual deployment of the model.

[0046] Specifically, the dish recognition model first pre-processes the input dish image, adjusts the image to 640*640 pixels, and then normalizes the image and maps the pixel value to between 0-1, which speeds up the convergence of the model and improves the training and detection accuracy.

[0047] The processed dish images enter the backbone network, and SG-Conv is used to replace the last four ordinary convolution Convs of the backbone network in the YOLOv8 model. Its structure is as follows Figure 2As shown in the upper right corner. For the YOLOv8s model, w=0.5, so the number of convolution kernels after the first ordinary convolution operation of the image to be processed is 64*w, and the size becomes (320*320*64*w). After the SG-Conv operation, in the SPD layer, the feature map X is downsampled by 2 times, and the size is reduced to 160*160. Four sub-maps are obtained each time the sampling is downsampled. These sub-feature maps are connected along the channel dimension. The number of channels is four times that of the original feature map, that is, 256*w. ChostConv first uses 1*1 convolution to reduce the number of channels of the SPD layer output feature map, and the size becomes (160*160*128*w), and then uses 3*3 depth convolution to extract features from each channel feature map, and the size becomes (160*160*128*w). Finally, the output feature maps of the two convolutions are stacked on the channel dimension, and the size becomes (160*160*256*w).

[0048] Figure 3 The process of downsampling the SPD layer is shown. Figure 3 The input of the original feature map is (W, H, C). First, the input feature map is divided into four feature maps of size (W / 2, H / 2, C). Next, these feature maps are concatenated in the channel dimension, and the final feature map size is (W / 2, H / 2, 4C), where the width and height are reduced by half, and the number of channels is four times that of the original feature map.

[0049] Using GhostConv to replace the non-stride convolution in SPD-Conv solves the computational inefficiency of generating redundant feature maps while maintaining the same number of feature maps as conventional convolution. The structure of GhostConv is as follows Figure 4 As shown in Figure 2. Unlike regular convolution, GhostConv uses a two-step operation: first, a reduced portion of the feature map is generated, and then a linear transformation is used to generate the remaining feature map. Compared with regular convolution, the amount of computation required to generate feature maps through linear transformation is significantly reduced, thereby reducing model parameters, weights, and computational costs.

[0050] The C2f module mainly includes a 1*1 Conv layer, n Bottleneck layers, shortcut=True, the input is the output of SG-Conv (160*160*256*w), n=3*d=3*0.33=1, and its structure is as follows Figure 2As shown in the upper right corner. First, the first convolutional layer is used for transformation to generate an intermediate feature map; the generated intermediate feature map is split into two parts, one part is directly passed to the final Concat block, and the other part is passed to n Bottleneck blocks for further processing. Each Bottleneck block contains two convolutional layers, which transform the input feature map and extract higher-level feature representations; the feature map processed by the Bottleneck block and the part of the feature map directly passed are spliced ​​in the Concat block to form a fused feature map; the feature map is then processed by the second convolutional layer, and the final output feature map of the C2f module is (160*160*256*w).

[0051] After three more SG-Conv and C2f operations, the output feature map size becomes (80*80*1024*w), (40*40*1024*w), (20*20*1024*w), and the maximum number of channels of YOLOv8s is 1024. The SPPF structure is as follows Figure 2 As shown in the upper right corner, 1x1Conv is used to reduce the dimension to half the size; then nn.MaxPool2d is performed three times with a kernel of 5, and then connected with itself; the connected feature map is reduced to the number of channels in the output layer (20*20*1024*w).

[0052] In the neck network feature fusion, the feature maps of different scales output by the backbone network are fused to combine the deep semantic information with the shallow detail information, enhance the expressive power of the features, and enable the model to better combine semantic information at different levels, thereby more accurately detecting dishes of different sizes and positions.

[0053] For example, the 20×20 feature map output by the backbone network is upsampled to 40×40, and then concatenated with the 40×40 feature map in the channel dimension to obtain a fused 40×40 feature map; then, the fused 40×40 feature map is upsampled to 80×80, and then concatenated with the 80×80 feature map in the channel dimension to obtain a fused 80×80 feature map. The attention mechanism CBAM is introduced after the C2f module, which combines the channel attention and spatial attention mechanisms to enhance the performance of the convolutional neural network. The channel attention module distinguishes the feature representations between different channels by evaluating the importance of each channel. The spatial attention module calculates the spatial importance of each pixel to better capture the spatial structure in the image.

[0054] On the feature map after feature fusion, a series of convolution operations are performed to generate prediction results, which include information such as the category probability of the dish, the position and size of the bounding box, etc. For each grid unit, the model predicts multiple bounding boxes, each of which has corresponding category probability and position information. This embodiment adds a rejection loss function on the basis of the original loss function to improve the positioning accuracy of the target box.

[0055] By introducing the exclusion loss function to adjust the confidence of the prediction box, a large number of bounding boxes generated by the prediction are screened and optimized, and the bounding boxes with low confidence are removed; then, the non-maximum suppression algorithm is used to remove the bounding boxes with high overlap, and the bounding boxes that are most likely to be the dishes are retained to obtain the final detection results, including information such as the category, location and confidence of the dish.

[0056] It is worth noting that this embodiment studies Chinese dishes, and one dish involves multiple raw materials. In addition to the overlap of multiple raw materials, the mutual occlusion between dishes poses a major challenge to the accurate detection of dishes. The hybrid loss function of YOLOv8 has strong universality, but lacks specificity for small targets. The attraction term in the repulsion loss function is used to narrow the gap between the prediction box and its target box. The repulsion term keeps the predicted bounding box as far away as possible from adjacent objects of the non-real annotation box, thereby improving the algorithm's ability to detect occluded small targets. Therefore, this embodiment combines Repulsion Loss with the loss function of YOLOv8 to improve the positioning accuracy of the target box.

[0057] The improved model reduces model parameters and improves the model's reasoning speed through the CWD knowledge distillation algorithm. After ablation and comparative experiments, the excellent detection accuracy of the SCR-YOLO algorithm proposed in this embodiment has been verified, but its parameters are large and the reasoning speed is slow. For VR devices such as Pico, memory and chip computing power are limited, so it is still difficult to run the SCR-YOLO algorithm to process data on mobile devices. In order to solve this problem, this embodiment uses CWD technology to let the student model YOLOv8s learn the output and intermediate representation of the teacher model SCR-YOLO to reduce the complexity of the model, so that it can be deployed on mobile devices and achieve satisfactory food detection effects.

[0058] The CWD knowledge distillation structure is as follows: Figure 5As shown in the figure, the implementation of CWD knowledge distillation can be divided into two steps. First, the channel attention distribution corresponding to the teacher network and the student network is converted into a probability distribution, and the KL divergence is used to compare the difference in channel information between the two. Then the student network uses the information of the teacher network to try to minimize the difference, so as to retain high-level semantic information while paying attention to channel-level detail features. Finally, the student model maintains a small size and has the higher detection accuracy of the teacher model. By conducting a lightweight comparison experiment on the dish recognition model before and after distillation, the data shown in Table 1 are obtained. While ensuring high accuracy, the model parameters are reduced by 72.8%, the computational complexity is reduced by 70.9%, and the model inference is accelerated by about 6 times on the mobile terminal. The dish recognition results on the Pico device are as follows: Figure 8 In complex dining environments, the problems of occlusion between dishes and difficulty in identifying small target dishes have also been improved, such as Figure 6 and Figure 7 shown.

[0059] Table 1 Lightweight comparison test

[0060]

[0061] Step 4: Model migration;

[0062] Convert the model to ONNX format and package it into the Pico application created by Unity.

[0063] Step 5: Develop a dish recognition system;

[0064] Develop Pico MR environment perception function, obtain real food images, use Unity Sentis plug-in to obtain the food recognition model output in ONNX format, convert it into rendering texture and copy it to the screen.

[0065] Step 6: Application packaging;

[0066] Package the dish recognition system developed in step 5 into an apk file and install it on the Pico 4Pro device. Start the application, first see the real environment, take a screenshot of the dish screen, the system recognizes the image, and displays the result on the desktop.

[0067] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.

Claims

1. A method for identifying multiple types of dishes that can be used in a virtual device, characterized in that: The steps are: Step 1: Take pictures with your mobile phone and expand the images of dishes through the Internet, and analyze and process the collected images; Step 2: Use the images processed in step 1 to construct a dish image dataset; Step 3: Build a target recognition model SCR-YOLO based on the YOLOv8 algorithm. The target recognition model SCR-YOLO changes the backbone network of the YOLOv8 model to the SG-Conv structure, introduces the attention mechanism CBAM into the feature fusion network, and adds the repulsion loss function Repulsion Loss on the basis of the original loss function. Step 4: Use the dish image dataset to train a dish recognition model, perform CWD channel knowledge distillation on it, and then migrate it to the virtual reality device. The virtual reality device combines any dish image to obtain the final prediction result.

2. The method for identifying multiple types of dishes that can be used in a virtual device according to claim 1, characterized in that: The SG-Conv structure described in step 3 replaces the last four ordinary convolution Convs of the backbone network in the YOLOv8 model with SG-Conv, uses the SPD layer of SPD-Conv for downsampling, and replaces the non-step convolution in SPD-Conv with GhostConv.

3. The method for identifying multiple types of dishes that can be used in a virtual device according to claim 2, characterized in that: In the backbone network of the target recognition model SCR-YOLO, the image to be processed undergoes the first ordinary convolution operation and then the SG-Conv operation. In the SPD layer, the feature map is downsampled by 2 times, and four sub-maps are obtained each time the downsampling is performed. These sub-feature maps are connected along the channel dimension, and the number of channels is four times that of the original feature map; ChostConv first uses convolution to reduce the number of channels of the feature map output by the SPD layer, and then uses deep convolution to extract features from each channel feature map. Finally, the output feature maps of the two convolutions are stacked in the channel dimension.

4. The method for identifying multiple types of dishes that can be used in a virtual device according to claim 3, characterized in that: After the image to be processed is operated by SG-Conv, it is output to the C2f module; the C2f module first transforms it through the first convolutional layer to generate an intermediate feature map; the generated intermediate feature map is split into two parts, one part is directly passed to the final Concat block, and the other part is passed to n Bottleneck blocks for further processing. Each Bottleneck block contains two convolutional layers, which transform the input feature map and extract higher-level feature representations; the feature map processed by the Bottleneck block and the part of the feature map that is directly passed are spliced ​​in the Concat block to form a fused feature map; the fused feature map is then processed by the second convolutional layer and finally output.

5. The method for identifying multiple types of dishes that can be used in a virtual device according to claim 4, characterized in that: After 4 SG-Conv and C2f operations, the image to be processed is output to the SPPF module. The SPPF module first reduces the number of channels to half the size through a convolutional layer. Then it performs nn.MaxPool2d three times and connects itself. The connected feature map is reduced to the number of channels in the output layer.

6. The method for identifying multiple types of dishes that can be used in a virtual device according to claim 5, characterized in that: In the neck network feature fusion, the feature maps of different scales output by the backbone network are fused. Among them, the attention mechanism CBAM is introduced after the C2f module of the neck network. The attention mechanism CBAM combines the channel attention and spatial attention mechanisms to enhance the performance of the convolutional neural network.

7. The method for identifying multiple types of dishes that can be used in a virtual device according to claim 6, characterized in that: The target recognition model SCR-YOLO adds an exclusion loss function on the basis of the original loss function. By introducing the exclusion loss function, the confidence of the predicted box is adjusted, and a large number of bounding boxes generated by the prediction are screened and optimized to remove bounding boxes with low confidence. Then, the non-maximum suppression algorithm is used to remove bounding boxes with high overlap and retain the bounding boxes that are most likely to be dishes.

8. The method for identifying multiple types of dishes that can be used in a virtual device according to claim 7, characterized in that: The target recognition model SCR-YOLO reduces model parameters through the CWD knowledge distillation algorithm. Specifically, the student model YOLOv8s learns the output and intermediate representation of the teacher model SCR-YOLO to reduce the complexity of the model.

9. A method for identifying multiple types of dishes that can be used in a virtual device according to any one of claims 1 to 8, characterized in that: Step 4: Convert the target recognition model SCR-YOLO to ONNX format and package it into the Pico application created by Unity; Develop Pico MR environmental perception function, obtain real dish images, use Unity Sentis plug-in to obtain the dish recognition model output in ONNX format, convert it into rendering texture and copy it to the screen; package the developed dish recognition system into an apk file and install it on the Pico 4Pro device.

10. A multi-category dish recognition system that can be used in a virtual device, characterized in that: Execute the method for identifying multiple types of dishes as described in any one of claims 1-8.

Images