A multi-category dish recognition method and system applicable to virtual devices
By improving the YOLOv8 model, adopting the SG-Conv structure, replacing convolution with GhostConv, adding CBAM and repulsive loss function, and utilizing channel knowledge distillation technology, the problems of accuracy and lightweighting of food recognition on virtual reality devices were solved, achieving efficient food recognition.
Patent Information
- Application Number
- CN202411719819.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2044-11-28
AI Technical Summary
Existing food recognition models have low accuracy on virtual reality devices, especially for Chinese food, and are limited by memory and computing power, making it difficult to achieve a win-win situation between lightweight design and high accuracy.
The YOLOv8 model is improved by adopting the SG-Conv structure, replacing non-stride convolution with GhostConv, adding the CBAM attention mechanism and repulsive loss function, and combining channel knowledge distillation technology to reduce model parameters and improve detection accuracy.
While maintaining high accuracy, the model parameters were reduced by 72.8%, the computational complexity was reduced by 70.9%, and the inference speed was increased by 6 times, making it suitable for mobile devices with limited computing power.
Smart Images
Figure CN119672704B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image recognition technology, and more specifically, relates to a method and system for recognizing various types of dishes that can be used in virtual devices. Background Technology
[0002] Dishes are more than just food; they embody culture, history, geography, and people's lifestyles. Through dishes, people can learn about and experience the unique cultures of different regions and ethnic groups, promoting cultural exchange and mutual understanding. With the increasingly close integration of virtual reality technology and the catering industry, neural networks are being applied more widely in the field of dish recognition. Using virtual reality devices for dish recognition, with its combined virtual and real visualization effects, allows consumers to understand dish information more intelligently and intuitively, which also has a positive impact on the service quality of catering businesses.
[0003] However, the accuracy of existing food recognition models still needs improvement, especially for Chinese cuisine, where even the same dish can have significant feature differences. In actual dining, factors such as occlusion between dishes and irregularly shaped small dishes can also affect the model's recognition rate and detection speed. Furthermore, VR devices have limited storage and computing power, making it difficult for common algorithms to meet their deployment requirements and rapid inference.
[0004] Currently, research on food recognition struggles to achieve a balance between lightweight design and high accuracy. Therefore, there is an urgent need to maintain the model's original recognition accuracy while maximizing recognition speed and minimizing model size, enabling smooth application on hardware with limited computing power.
[0005] A search revealed Chinese patent application number 2024107479523, filed on June 11, 2024, entitled "Method, Apparatus, Terminal Device, and Storage Medium for Food Recognition." This application, after acquiring a target food image, extracts its feature vectors using a convolutional neural network model optimized with cross-entropy and triplet loss functions. This extraction is more efficient, yielding food feature vectors that can be used for subsequent retrieval in a database, thereby determining the food category corresponding to the target food image. The final determined food category has a smaller error compared to the actual food category in the target food image, resulting in higher accuracy. However, the application device in this application has no limitations in processor and memory, thus lacking research on lightweighting the convolutional neural network model. Summary of the Invention
[0006] 1. The technical problem that the invention aims to solve
[0007] In view of the shortcomings of the existing technology, the present invention provides a method and system for recognizing various types of dishes that can be used in virtual devices. The present invention improves the YOLOv8 model to enhance the accuracy of dish recognition, performs knowledge distillation on the improved model to reduce the number of model parameters and improve the inference speed, thereby enabling the dish recognition function to be integrated into mobile devices with limited computing power, such as VR.
[0008] 2. Technical Solution
[0009] To achieve the above objectives, the technical solution provided by the present invention is as follows:
[0010] The present invention provides a method for recognizing various types of dishes that can be used in virtual devices, comprising the following steps:
[0011] Step 1: Data acquisition and processing;
[0012] By taking photos of dishes with a mobile phone and expanding the images online, the collected images are analyzed and processed to match the resolution of the input images for the model.
[0013] Step 2: Construct a dataset of food images;
[0014] The dataset was created using the images processed in step 1. The food categories and bounding boxes in the images were labeled. The labeled food categories and bounding boxes were called the ground truth categories and ground truth boxes. The labeled dataset was then divided into training set, validation set and test set. In addition to the self-created dataset mentioned above, the publicly available food dataset VieroFood-172 was also used and its dataset was divided in the same way.
[0015] Step 3: Construct a dish recognition model;
[0016] Based on YOLOv8, a lightweight food identification algorithm, SCR-YOLO, is proposed, with the following structure: Figure 2 As shown, firstly, the backbone network is changed to an SG-Conv structure, which uses the SPD layer enhancement downsampling technique of SPD-Conv and replaces the non-strut convolutions in SPD-Conv with GhostConv to reduce the computational complexity of the model. Then, the feature fusion network incorporates the attention mechanism CBAM. Simultaneously, a repulsive loss function is added to the original loss function to improve detection accuracy under food occlusion. Finally, channel knowledge distillation (CWD) is used to allow the student model YOLOv8s to learn the output and intermediate representations of the teacher model SCR-YOLO, enabling the student model to approach the performance of the teacher model as closely as possible while maintaining a smaller model size.
[0017] Step 4, Model Transfer;
[0018] Convert the model to ONNX format and package it into a Pico application created with Unity.
[0019] Step 5: Develop a dish recognition system;
[0020] Develop Pico MR environment awareness functionality to obtain real food images, use the Unity Sentis plugin to obtain the ONNX format food recognition model output, convert it into a rendering texture and copy it to the screen.
[0021] Step 6: Package the application;
[0022] Package the food recognition system developed in step 5 into an APK file and install it on your Pico 4 Pro device. Launch the application; you'll first see a real-world environment. Take a screenshot of the food image, and the system will recognize the image and display the result on your desktop.
[0023] 3. Beneficial effects
[0024] Compared with existing known technologies, the technical solution provided by this invention has the following significant advantages:
[0025] (1) This invention provides a method for recognizing various types of dishes in virtual devices. Addressing the challenges of occlusion between dishes and difficulty in recognizing small target dishes in real-world settings, the YOLOv8 model is improved. Specifically, the backbone network is changed to an SG-Conv structure, which employs the SPD layer enhancement downsampling technique of SPD-Conv. GhostConv replaces the non-stride convolutions in SPD-Conv, reducing the computational complexity of the model. The feature fusion network incorporates the attention mechanism CBAM. A repulsive loss function is added to the original loss function to improve detection accuracy under dish occlusion. Experimental results show that the proposed algorithm improves mAP50 by 4.1% and 2.6% compared to YOLOv8 on the self-created Chinese food dataset CF-8 and the public dataset VireoFood-172, respectively, providing higher accuracy.
[0026] (2) The present invention provides a method for identifying various types of dishes that can be used in virtual devices. The method performs CWD knowledge distillation on a high-accuracy dish identification model. As a result, while ensuring high accuracy, the model parameters are reduced by 72.8%, the computational complexity is reduced by 70.9%, and the model inference is accelerated by about 6 times on mobile devices.
[0027] (3) The present invention provides a multi-type dish recognition system that can be used in virtual devices. It can integrate the dish recognition model into mobile devices with limited computing power, so that the dish recognition function can be applied to the catering industry more quickly. Furthermore, through the virtual reality device's virtual reality combined visualization effect, it can improve the user experience and have a positive impact on the service quality of the catering industry. Attached Figure Description
[0028] Figure 1 This is a flowchart of a method for recognizing various types of dishes that can be used in virtual devices according to the present invention;
[0029] Figure 2 The network structure diagram of the improved dish recognition model SCR-YOLO;
[0030] Figure 3 This is a structural diagram of the SPD layer in SG-Conv;
[0031] Figure 4 Here is a structural diagram of GhostConv in SG-Conv;
[0032] Figure 5 A structural diagram of CWD knowledge distillation;
[0033] Figure 6 Image showing the effect of identifying obstructions between food items;
[0034] Figure 7 Image showing the effect of identifying small target dishes;
[0035] Figure 8 This is an image showing the effect of food recognition within a VR device. Detailed Implementation
[0036] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments.
[0037] Example 1
[0038] Combination Figure 1 The method for recognizing various types of dishes that can be used in virtual devices according to this embodiment includes the following steps:
[0039] Step 1: Data acquisition and processing;
[0040] By taking pictures of dishes with a mobile phone and expanding the images online, the collected images are analyzed and processed to match the resolution of the input images of the model. In this embodiment, a total of 5,644 Chinese food images in 8 categories are obtained.
[0041] Step 2: Construct a dataset of food images;
[0042] Using the images processed in step 1, a YOLO format dataset was created. LabelImg software was used to label the food categories and bounding boxes in the images. The labeled food categories and bounding boxes were called the ground truth categories and ground truth boxes. The labeled dataset was then randomly divided into training, validation, and test sets in a ratio of 7:2:1. In addition to the self-created dataset, the publicly available food dataset VieroFood-172 was also used, and it was labeled and divided in the same way.
[0043] Step 3: Construct a dish recognition model;
[0044] Based on YOLOv8, a lightweight food identification algorithm, SCR-YOLO, is proposed, with the following structure: Figure 2 As shown, firstly, the backbone network is changed to an SG-Conv structure, which uses the SPD layer enhancement downsampling technique of SPD-Conv and replaces the non-strut convolutions in SPD-Conv with GhostConv to reduce the computational complexity of the model. Then, the feature fusion network incorporates the attention mechanism CBAM. Simultaneously, a repulsion loss function is added to the original loss function to improve detection accuracy under food occlusion. Finally, channel knowledge distillation (CWD) is used to allow the student model YOLOv8s to learn the output and intermediate representations of the teacher model SCR-YOLO, enabling the student model to approach the performance of the teacher model as closely as possible while maintaining a smaller model size.
[0045] It is worth noting that the food recognition model built based on the YOLOv8 algorithm in this embodiment adopts the SG-Conv structure in the backbone network. It uses the SPD layer of SPD-Conv for downsampling, converting spatial dimension information into depth dimension, avoiding information loss in traditional stride convolution and pooling operations. This enhances the detection performance of small objects in complex scenes and improves the model's generalization and robustness. GhostConv is used instead of the non-stride convolution in SPD-Conv, using linear transformation to generate redundant feature maps, reducing network computation and model parameter count, and enhancing the feasibility of practical model deployment.
[0046] Specifically, the food recognition model first preprocesses the input food image by adjusting it to 640*640 pixels, and then normalizes the image to map the pixel values between 0 and 1, thereby accelerating the convergence speed of the model and improving training and detection accuracy.
[0047] The processed food image is fed into the backbone network, where the last four ordinary convolutional Convs in the YOLOv8 model backbone are replaced with SG-Convs, as shown in the structure below. Figure 2As shown in the upper right corner. For the YOLOv8s model, w = 0.5, so the number of convolution kernels after the first ordinary convolution operation is 64*w, and the size becomes (320*320*64*w). Then, after the SG-Conv operation, in the SPD layer, the feature map X is downsampled by a factor of 2, reducing its size to 160*160. Each downsampling yields four sub-images, which are then connected along the channel dimension. The number of channels is four times that of the original feature map, i.e., 256*w. ChostConv first uses a 1*1 convolution to reduce the number of channels in the SPD layer output feature map, changing its size to (160*160*128*w). Then, it uses a 3*3 depthwise convolution to extract features from each channel feature map, changing its size to (160*160*128*w). Finally, the output feature maps from the two convolutions are stacked along the channel dimension, changing their size to (160*160*256*w).
[0048] Figure 3 The process of SPD layer downsampling is demonstrated. Figure 3 The input to the original feature map is (W, H, C). First, the input feature map is divided into four feature maps of size (W / 2, H / 2, C). Next, these feature maps are concatenated along the channel dimension, resulting in a final feature map of size (W / 2, H / 2, 4C), where the width and height are reduced by half, and the number of channels is four times that of the original feature map.
[0049] Replacing the non-strut convolution in SPD-Conv with GhostConv solves the problem of low computational efficiency in generating redundant feature maps, while maintaining the same number of feature maps as regular convolution. The structure of GhostConv is as follows: Figure 4 As shown, unlike regular convolution, GhostConv employs a two-step operation: first, it generates a reduced portion of the feature map, and then it uses a linear transformation to generate the remaining feature map. Compared to regular convolution, the computational cost required to generate the feature map through linear transformation is significantly reduced, thereby lowering the model parameters, weights, and computational cost.
[0050] The C2f module mainly consists of one 1*1 Conv layer, n Bottleneck layers, shortcut=True, and the input is the output of SG-Conv (160*160*256*w), where n=3*d=3*0.33=1. Its structure is as follows: Figure 2As shown in the upper right corner. First, the input feature map is transformed by the first convolutional layer to generate an intermediate feature map. The generated intermediate feature map is split into two parts. One part is directly passed to the final Concat block, and the other part is passed to n Bottleneck blocks for further processing. Each Bottleneck block contains two convolutional layers. These convolutional layers transform the input feature map and extract higher-level feature representations. The feature map processed by the Bottleneck block is concatenated with the directly passed feature map in the Concat block to form a fused feature map. The feature map is then processed by the second convolutional layer. Finally, the output feature map of the C2f module is (160*160*256*w).
[0051] After three more SG-Conv and C2f operations, the output feature map size becomes (80*80*1024*w), (40*40*1024*w), and (20*20*1024*w), with a maximum of 1024 channels in YOLOv8s. The SPPF structure is as follows... Figure 2 As shown in the upper right corner, the dimensionality is reduced to half the size using 1x1Conv; then, nn.MaxPool2d is performed three times with a kernel of 5, and then the kernel is added and connected; the dimensionality of the connected feature map is reduced to the number of output layer channels (20*20*1024*w).
[0052] In the neck network feature fusion, feature maps of different scales output by the backbone network are fused together, combining deep semantic information with shallow detail information to enhance the expressive power of features. This enables the model to better combine semantic information at different levels, thereby more accurately detecting dishes of different sizes and positions.
[0053] For example, the 20×20 feature map output from the backbone network is upsampled to 40×40, and then concatenated with the 40×40 feature map along the channel dimension to obtain a fused 40×40 feature map. Next, the fused 40×40 feature map is upsampled to 80×80, and then concatenated with the 80×80 feature map along the channel dimension to obtain a fused 80×80 feature map. An attention mechanism, CBAM, is introduced after the C2f module. It combines channel attention and spatial attention mechanisms to enhance the performance of the convolutional neural network. The channel attention module distinguishes feature representations between different channels by evaluating the importance of each channel. The spatial attention module calculates the spatial importance of each pixel to better capture the spatial structure in the image.
[0054] On the feature map after feature fusion, a series of convolutional operations are used to generate prediction results, including information such as the category probability of the dish, the position and size of the bounding box, etc. For each grid cell, the model predicts multiple bounding boxes, each with corresponding category probability and position information. This embodiment adds a repulsion loss function to the original loss function to improve the localization accuracy of the target box.
[0055] By introducing a rejection loss function to adjust the confidence of the predicted bounding boxes, a large number of predicted bounding boxes are filtered and optimized to remove bounding boxes with low confidence. Then, a non-maximum suppression algorithm is used to remove bounding boxes with high overlap and retain the bounding boxes that are most likely to be dishes, so as to obtain the final detection results, including information such as the category, location and confidence of the dishes.
[0056] It is worth noting that this embodiment studies Chinese dishes, where a single dish involves multiple ingredients. Besides the overlap of these ingredients, the mutual occlusion between the ingredients poses a significant challenge to accurate detection. While YOLOv8's hybrid loss function is highly general, it lacks specificity for small targets. The attraction term in the repulsion loss function is used to reduce the gap between the predicted bounding box and its target bounding box, while the repulsion term keeps the predicted bounding box as far away as possible from adjacent objects that are not in the actual labeled bounding box, thus improving the algorithm's ability to detect occluded small targets. Therefore, this embodiment combines Repulsion Loss with the YOLOv8 loss function to improve the accuracy of target box localization.
[0057] The improved model reduces model parameters and improves inference speed through the CWD (Knowledge Distillation) algorithm. The SCR-YOLO algorithm proposed in this embodiment has demonstrated excellent detection accuracy after ablation and comparison experiments; however, it has a large number of parameters and a slow inference speed. For VR devices such as Pico, memory and chip computing power are limited, making it difficult to run the SCR-YOLO algorithm on mobile devices to process data. To address this issue, this embodiment utilizes CWD technology, allowing the student model YOLOv8s to learn the output and intermediate representations of the teacher model SCR-YOLO to reduce model complexity, enabling it to be deployed on mobile devices and achieve satisfactory food detection results.
[0058] CWD knowledge distillation structure such as Figure 5As shown, the implementation of CWD knowledge distillation can be divided into two steps. First, the channel attention distributions corresponding to the teacher network and student network are transformed into probability distributions, and the differences in channel information between the two are compared using KL divergence. Then, the student network uses information from the teacher network to try to minimize the differences, achieving the goal of retaining high-level semantic information while focusing on channel-level detailed features. Ultimately, the student model maintains a small size and has the high detection accuracy of the teacher model. Through lightweight comparison experiments on the dish recognition models before and after distillation, the data shown in Table 1 are obtained. While maintaining high accuracy, the model parameters are reduced by 72.8%, the computational complexity is reduced by 70.9%, and the model inference is accelerated by about 6 times on mobile devices. The dish recognition results on the Pico device are as follows. Figure 8 As shown. In complex dining environments, the difficulties in identifying dishes that are obscured or small are also improved, such as... Figure 6 and Figure 7 As shown.
[0059] Table 1 Lightweight Comparison Test
[0060]
[0061] Step 4, Model Transfer;
[0062] Convert the model to ONNX format and package it into a Pico application created with Unity.
[0063] Step 5: Develop a dish recognition system;
[0064] Develop Pico MR environment awareness functionality to obtain real food images, use the Unity Sentis plugin to obtain the ONNX format food recognition model output, convert it into a rendering texture and copy it to the screen.
[0065] Step 6: Package the application;
[0066] Package the food recognition system developed in step 5 into an APK file and install it on your Pico 4 Pro device. Launch the application; you'll first see a real-world environment. Take a screenshot of the food image, and the system will recognize the image and display the result on your desktop.
[0067] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.
Claims
1. A multi-class dish recognition method for a virtual device, the method comprising: The steps are: Step 1, taking pictures of dishes by mobile phone and expanding the image through network, and analyzing and processing the collected images; Step 2, using the processed images in step 1 to construct a dish image dataset; Step 3, constructing a target recognition model SCR-YOLO based on the YOLOv8 algorithm, changing the backbone network of the YOLOv8 model to an SG-Conv structure, introducing an attention mechanism CBAM in the feature fusion network, and adding a repulsion loss function Repulsion Loss based on the original loss function; Step 4, training a dish recognition model using the dish image dataset, migrating it to a virtual reality device after CWD channel knowledge distillation, and obtaining the final prediction result by combining the virtual reality device with any dish image; specifically, converting the target recognition model SCR-YOLO to ONNX format and packaging it into the Pico application created by Unity; Develop Pico MR environment perception function, get real dish image, use Unity Sentis plug-in to get ONNX format dish recognition model output, convert it to rendering texture and copy it to the screen; package the developed dish recognition system into an apk file and install it into a Pico4 Pro device.
2. The multi-category dish recognition method for virtual devices of claim 1, wherein: The SG-Conv structure described in step 3 replaces the last four normal convolutions Conv in the backbone network of the YOLOv8 model with SG-Conv, adopts SPD The SPD layer of Conv is down-sampled, and the non-step convolution in SPD-Conv is replaced with GhostConv. 3.The multi-category dish recognition method for virtual devices of claim 2, wherein: In the backbone network of the target recognition model SCR-YOLO, the picture to be processed is first subjected to a normal convolution operation and then subjected to an SG-Conv operation. In the SPD layer, the feature map is down-sampled by 2, and four sub-maps are obtained each time. The sub-feature maps are connected along the channel dimension, and the number of channels is four times that of the original feature map. ChostConv first uses convolution to reduce the number of channels of the SPD layer output feature map, then uses deep convolution to extract features from each channel feature map, and finally stacks the output feature maps of the two convolutions in the channel dimension.
4. The multi-category dish recognition method for virtual devices of claim 3, wherein: After the SG-Conv operation of the picture to be processed, the output is sent to the C2f module. The C2f module first transforms the input through a convolution layer to generate an intermediate feature map. The generated intermediate feature map is split into two parts, one of which is directly passed to the final Concat block, and the other of which is passed to n Bottleneck blocks for further processing. Each Bottleneck block contains two convolution layers that transform the input feature map to extract higher-level feature representations. The feature map processed by the Bottleneck block is concatenated with the directly transmitted feature map in the Concat block to form a fused feature map. The fused feature map is then processed by a second convolution layer and finally output.
5. The method of claim 4, wherein: After the SG-Conv operation and C2f operation of the picture to be processed four times, the output is sent to the SPPF module. The SPPF module first reduces the number of channels to half the size through a convolution layer. Then, three nn.MaxPool2d operations are performed, and the original feature map is connected. The connected feature map is reduced to the number of output layer channels.
6. The multi-category dish recognition method for virtual devices of claim 5, wherein: In the neck network feature fusion, the feature maps of different scales output by the backbone network are fused, wherein the attention mechanism CBAM is introduced after the C2f module of the neck network, and the attention mechanism CBAM fuses the channel attention and spatial attention mechanisms to enhance the performance of the convolutional neural network.
7. The method of claim 6, wherein the method is used for virtual kitchen. The target recognition model SCR-YOLO increases the repulsion loss function on the basis of the original loss function, adjusts the confidence of the prediction box by introducing the repulsion loss function, screens and optimizes a large number of boundary boxes generated by prediction, and removes the boundary boxes with low confidence; then through the non-maximum suppression algorithm, the boundary boxes with high overlap are removed, and the boundary boxes most likely to be dishes are reserved. 8.The multi-category dish recognition method for virtual devices of claim 7, wherein: The target recognition model SCR-YOLO reduces the model parameters through the CWD knowledge distillation algorithm, specifically, the student model YOLOv8s learns the output and intermediate representation of the teacher model SCR-YOLO to reduce the complexity of the model.
9. A multi-class menu recognition system for a virtual device, the system comprising: Performing the multi-category dish recognition method according to any one of claims 1-8.