Tableware contour learning and identification method and system based on deep neural network

Abstract

The invention relates to the technical field of tableware recognition, and discloses a tableware contour learning and recognition method and system based on a deep neural network, and the method comprises the steps: 1, carrying out the tableware contour learning, recognizing the target coordinate position information of warehousing tableware in an image based on a target detection algorithm, carrying out the external expansion of a target frame, and segmenting the subgraph of the warehousing tableware; using a contour segmentation algorithm to carry out contour segmentation on the warehousing tableware sub-graph to obtain a contour area actual image of the warehousing tableware; using the tableware measurement learning model to extract the contour feature information of the put-in-storage tableware, and storing the contour feature information into a tableware feature general library; and 2, tableware contour identification: processing the image of the to-be-identified tableware to obtain contour feature information of the to-be-identified tableware, and comparing and matching the contour feature information with feature information in the tableware feature general library to obtain an identification result. According to the invention, features of various tableware can be put in storage based on the deep neural network, so that subsequent tableware identification is facilitated, and self-learning and rapid identification of various tableware contours are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of tableware recognition, and in particular to a tableware contour learning and recognition method and system based on a deep neural network. Background Art

[0002] In the catering industry, tableware identification can help monitor the cleanliness and usage of tableware to ensure food safety and hygiene standards. Tableware comes in many types, shapes, and materials (such as ceramic, plastic, metal, etc.), which increases the difficulty of identification.

[0003] There are two existing methods for tableware recognition. One is to use traditional image processing methods to extract template features of the tableware to be used and store them in the database. During recognition, the feature points of the entered plate template features are used to match the tableware that needs to be recognized for template matching. Common methods for extracting feature points include traditional SIFT, FAST and other algorithms. The other is based on deep neural networks. First, pictures of various types of tableware to be used are collected. After these pictures are classified and trained, a recognition model for the currently specified type of dishes can be obtained, and this model will be used for subsequent detection and recognition.

[0004] Among them, the traditional image processing method is to extract template features first, and then perform feature matching for identification. Due to the large amount of extracted features, the amount of calculation during matching also increases accordingly. It is less adaptable to scenes with high real-time requirements. In addition, since it uses traditional image algorithms to extract features, it is less adaptable to some scenes and special tableware. The deep neural network method requires first collecting a large number of pictures of various tableware under different lighting and positions, and then classifying and labeling these pictures for training, and using the trained model for identification. Since the model only detects and identifies specific dishes, if different types of dishes are added, it is still necessary to collect pictures and retrain the new model. The process is relatively cumbersome and requires a large amount of manpower to collect pictures and train the model.

[0005] Therefore, how to improve the tableware recognition technology to achieve self-learning and rapid recognition of tableware contours and be compatible with various types of tableware has become an important research topic for those skilled in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a tableware contour learning and recognition method and system based on a deep neural network, which takes into account both real-time and robustness. Based on the deep neural network, the features of various types of tableware can be stored in the database to facilitate subsequent tableware recognition, thereby realizing self-learning and rapid recognition of tableware contours, and is compatible with multiple types of tableware, and can be applied to a wider range of application scenarios.

[0007] To achieve the above object, the present invention provides the following solutions: A tableware contour learning and recognition method based on a deep neural network, the method comprising the following steps: S1, tableware outline learning, including: S101, acquiring an image of tableware entering the warehouse, and identifying target coordinate position information of the tableware entering the warehouse in the image based on a target detection algorithm; S102, expanding a target frame based on target coordinate position information of the stored tableware, and segmenting the image of the stored tableware within the expanded target frame to obtain a sub-image of the stored tableware; S103, using a contour segmentation algorithm to perform contour segmentation on the incoming tableware sub-image, and obtaining a mask image of the contour area of ​​the incoming tableware, and obtaining an actual image of the contour area of ​​the incoming tableware based on the mask image; S104, using MobileNet as the backbone network model, distillation training is performed to obtain the tableware metric learning model; S105, using the tableware metric learning model to extract feature information from the actual image of the contour area of ​​the tableware entering the warehouse, and store it in the tableware feature library, wherein the tableware entering the warehouse serves as a tableware template of the tableware feature library; S2, tableware contour recognition, including: S201, acquiring an image of tableware to be identified, and identifying target coordinate position information of the tableware to be identified in the image based on a target detection algorithm; S202, expanding a target frame based on target coordinate position information of the tableware to be identified, and segmenting the image of the tableware to be identified within the expanded target frame to obtain a sub-image of the tableware to be identified; S203, using a contour segmentation algorithm to perform contour segmentation on the tableware sub-image to be identified, and obtaining a mask image of the contour area of ​​the tableware to be identified, and obtaining an actual image of the contour area of ​​the tableware to be identified based on the mask image; S204, using the tableware metric learning model to extract feature information in the actual image of the contour area of ​​the tableware to be identified, and compare and match it with the feature information in the tableware feature library, and select the tableware template with the highest matching score as the identification result.

[0008] Furthermore, the S101, acquiring an image of the tableware entering the warehouse, and identifying the target coordinate position information of the tableware entering the warehouse in the image based on a target detection algorithm, specifically includes: (1) Collect multiple tableware images with different lighting conditions, backgrounds, and types, build a target detection dataset, annotate the target boxes, and divide the dataset into training set, validation set, and test set; (2) Use the training set to train the YoloV8 network model to obtain a tableware detection model; (3) Use the validation set to verify the tableware detection model, and select the tableware detection model with the best positioning detection effect as the target detection positioning model; (4) Use the target detection and positioning model to identify the image of the tableware entering the warehouse and obtain the target coordinate position information of the tableware entering the warehouse.

[0009] Further, in S102, a target frame is expanded based on the target coordinate position information of the stored tableware, and the image of the stored tableware within the expanded target frame is segmented to obtain a sub-image of the stored tableware; (1) Based on the target coordinate location information of the tableware entering the warehouse , expand the target frame rLen Length, get : ; (2) Based on The target box is used to segment the image of the incoming tableware and obtain the incoming tableware sub-image.

[0010] Furthermore, the S103, using a contour segmentation algorithm to perform contour segmentation on the stored tableware sub-image, and obtaining a mask image of the contour area of ​​the stored tableware, and obtaining an actual image of the contour area of ​​the stored tableware based on the mask image, specifically includes: (1) Use the trained cutlery contour segmentation model to perform contour segmentation on the incoming cutlery sub-image and obtain the mask image corresponding to the incoming cutlery sub-image; (2) Perform dilation and erosion operations on the mask images corresponding to each sub-image of the incoming tableware, and extract the mask image (counterMask) of the contour area of ​​the incoming tableware through the difference operation after dilation and erosion; (3) Extract the actual image (counterImg) of the contour area in the image of the stored tableware corresponding to the mask image.

[0011] Furthermore, the training process of the tableware contour segmentation model is as follows: (1) Collect multiple tableware images with different lighting conditions, different backgrounds, and different types, construct a dataset for contour segmentation, annotate the dataset with contours, and divide the dataset into training set, validation set, and test set; (2) Use the training set to train the FastSCNN model to obtain the tableware contour segmentation model; (3) Use the validation set to verify the cutlery contour segmentation model, and select the cutlery contour segmentation model with the best segmentation effect as the trained cutlery contour segmentation model.

[0012] Furthermore, the step S104 uses MobileNet as the backbone network model and performs distillation training to obtain a tableware metric learning model, specifically including: (1) Collect multiple tableware images with different lighting conditions, different backgrounds, and different types, and process them using the target detection and positioning model and the tableware contour segmentation model in turn to obtain the actual images of the contour areas of different types of tableware, annotate them by category, and construct a feature extraction data set, which is divided into a training set, a validation set, and a test set; (2) Select the Swin Transformer model as the teacher model, use the training set in (1) for training, and select the well-trained model with the best performance as the teacher model for distillation; (3) Select the MobileNet backbone network as the student model and use pre-trained weights for initialization to speed up the training process; (4) Obtain the output features of the teacher model and the student model respectively, and optimize the student model parameters based on the calculated classification loss and distillation loss; (5) Evaluate the performance of the student model on the validation set and fine-tune it to ensure that its ability to extract tableware features is optimized; (6) Select the best student model on the test set as the tableware metric learning model DML.

[0013] Furthermore, the feature dimension of the tableware metric learning model is 1×1280, that is, pFeatures (1×1280) In S105, the calculation method of the information features extracted by the tableware metric learning model is as follows: ; in, N is the number of the same type of tableware in storage, and each feature information is obtained by taking the average value.

[0014] Furthermore, in S204, the matching score Score The calculation method is as follows: ; in, pFeatures refers to the feature vector of the tableware to be identified. pFeatures i It refers to the first feature vector of the tableware to be identified. i eigenvalues; pFeaturesAvg j Refers to the stored j The feature vector of the cutlery template, Refers to the stored j The feature vector of the cutlery template i feature values.

[0015] The present invention also provides a tableware contour learning and recognition system based on a deep neural network, which is used to execute the tableware contour learning and recognition method based on a deep neural network, comprising: Cutlery outline learning unit, including: The target detection module is used to obtain an image of the tableware entering the warehouse and identify the target coordinate position information of the tableware entering the warehouse in the image based on the target detection algorithm; A target frame expansion module is used to expand the target frame based on the target coordinate position information of the incoming tableware, and to segment the image of the incoming tableware within the expanded target frame to obtain a sub-image of the incoming tableware; A contour segmentation module is used to perform contour segmentation on the incoming tableware sub-image using a contour segmentation algorithm, and obtain a mask image of the contour area of ​​the incoming tableware, and obtain an actual image of the contour area of ​​the incoming tableware based on the mask image; The tableware metric learning model construction module is used to use MobileNet as the backbone network model and perform distillation training to obtain the tableware metric learning model; A feature extraction module is used to extract feature information in the actual image of the contour area of ​​the tableware entering the warehouse by using the tableware metric learning model; A feature storage module is used to store the extracted feature information of the tableware entering the storage into the tableware feature library, wherein the tableware entering the storage serves as a tableware template of the tableware feature library; Tableware contour recognition unit, specifically comprising: A target detection module, used to obtain an image of tableware to be identified, and identify target coordinate position information of the tableware to be identified in the image based on a target detection algorithm; A target frame expansion module is used to expand the target frame based on the target coordinate position information of the tableware to be identified, and to segment the image of the tableware to be identified within the expanded target frame to obtain a sub-image of the tableware to be identified; A contour segmentation module, used to perform contour segmentation on the tableware sub-image to be identified using a contour segmentation algorithm, and obtain a mask image of the contour area of ​​the tableware to be identified, and obtain an actual image of the contour area of ​​the tableware to be identified based on the mask image; A feature extraction module, used to extract feature information in the actual image of the contour area of ​​the tableware to be identified using the tableware metric learning model; The comparison and analysis module is used to compare and match the extracted feature information of the tableware to be identified with the feature information in the tableware feature library, and select the tableware template with the highest matching score as the identification result.

[0016] According to the specific embodiments provided by the present invention, the tableware contour learning and recognition method and system based on deep neural network provided by the present invention disclose the following technical effects: the tableware contour learning can be performed on multiple stored tableware of different lighting conditions, different backgrounds and different types, so as to continuously enrich and improve the total tableware feature library. Specifically, the present invention respectively adopts the methods of detecting and locating the plate based on deep neural network, segmenting and extracting the plate contour area, and extracting the plate contour features and storing them for recognition. This method first uses the target detection algorithm to locate the plate area, and then uses the deep neural network to extract the plate contour area after cutting the sub-graph, and then uses the feature extraction model to extract features from the plate contour area. Subsequently, the features are used for storing, matching and identifying the plate. This method can ensure real-time performance while being compatible with various plate types, and can meet the actual online use in multiple scenarios.

[0017] The present invention uses different deep neural networks to locate the plate, extract the plate contour area, and extract the plate contour feature information. This method can ensure the robustness of the algorithm in different scenarios while also being compatible with the flexibility of tableware selection. In order to be compatible with the real-time performance of the algorithm on different devices, the models selected by the present invention are all relatively lightweight algorithm models, which can ensure its recognition accuracy while also being capable of actual use in terms of real-time performance, ensuring the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 4 is a flow chart of a tableware contour learning and recognition method based on a deep neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The purpose of the present invention is to provide a tableware contour learning and recognition method and system based on deep neural networks, which respectively use different deep neural networks to locate the plate, extract the plate contour area and extract the plate contour feature information, while ensuring the robustness of the algorithm in different scenarios and being compatible with the flexibility of tableware selection.

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] like Figure 1 As shown, the tableware contour learning and recognition method based on deep neural network provided by the present invention comprises the following steps: S1, tableware outline learning, including: S101, acquiring an image of tableware entering the warehouse, and identifying target coordinate position information of the tableware entering the warehouse in the image based on a target detection algorithm; S102, expanding a target frame based on target coordinate position information of the stored tableware, and segmenting the image of the stored tableware within the expanded target frame to obtain a sub-image of the stored tableware; S103, using a contour segmentation algorithm to perform contour segmentation on the incoming tableware sub-image, and obtaining a mask image of the contour area of ​​the incoming tableware, and obtaining an actual image of the contour area of ​​the incoming tableware based on the mask image; S104, using MobileNet as the backbone network model, distillation training is performed to obtain the tableware metric learning model; S105, using the tableware metric learning model to extract feature information from the actual image of the contour area of ​​the tableware entering the warehouse, and store it in the tableware feature library, wherein the tableware entering the warehouse serves as a tableware template of the tableware feature library; S2, tableware contour recognition, including: S201, acquiring an image of tableware to be identified, and identifying target coordinate position information of the tableware to be identified in the image based on a target detection algorithm; S202, expanding a target frame based on target coordinate position information of the tableware to be identified, and segmenting the image of the tableware to be identified within the expanded target frame to obtain a sub-image of the tableware to be identified; S203, using a contour segmentation algorithm to perform contour segmentation on the tableware sub-image to be identified, and obtaining a mask image of the contour area of ​​the tableware to be identified, and obtaining an actual image of the contour area of ​​the tableware to be identified based on the mask image; S204, using the tableware metric learning model to extract feature information in the actual image of the contour area of ​​the tableware to be identified, and compare and match it with the feature information in the tableware feature library, and select the tableware template with the highest matching score as the identification result.

[0024] Embodiment 1: The tableware contour learning and recognition method based on a deep neural network provided in an embodiment of the present invention is performed by a tableware contour learning and recognition system based on a deep neural network. The method specifically includes a tableware contour learning method based on a deep neural network and a tableware contour learning method based on a deep neural network, which are specifically as follows: First, based on the tableware contour learning method of deep neural network, the tableware to be used is entered into the tableware feature library, which specifically includes the following steps: (1) Call the camera to capture an image of the tableware that needs to be stored, use the YoloV8 detection model to detect the tableware target, and obtain the target coordinate position of the tableware that needs to be stored in the current image information; (2) Expand the above tableware ROI outside the target frame rLen Length (default rLen =50), and get the new ; ; (3) Cutting the above The corresponding tableware sub-image in the target frame; (4) Use the trained FastSCNN model to perform contour segmentation on the tableware sub-image in (3) to obtain the mask image of the corresponding tableware in storage; (5) The mask image of the incoming tableware is expanded and eroded respectively. The mask image (counterMask) corresponding to the contour area of ​​the tableware can be extracted through the difference operation after expansion and erosion, and the actual image (counterImg) of the contour area of ​​the incoming tableware in the original image corresponding to the mask image area of ​​the contour is extracted; (6) Using the tableware metric learning model (DML) obtained by distillation training with MobileNet as the backbone network model, extract the feature information (pFeatures) of the actual image of the contour area of ​​the tableware stored in step (5); (7) The feature information of the tableware (e.g., bowls and plates) extracted above is stored in the tableware feature database. The same tableware can be recorded in N (default N=5) pieces. The average value of their feature values ​​(pFeaturesAvg) is taken as the tableware entry feature: .

[0025] Second, a tableware contour learning method based on a deep neural network is used to identify the category information of the tableware to be identified in the order graph, including the following steps: (1) Call the camera to capture the order image of the current settlement area, including the image of the tableware to be identified. Use the YoloV8 detection model to perform target detection on the tableware to be identified in the current order image and obtain the coordinate position of the tableware to be identified. information; (2) Expand the target frame of the above tableware template ROI rLen Length (default rLen =50), and get the new ; ; (3) cutting the tableware sub-graphs corresponding to the tableware to be identified in all target frames in step (2); (4) Use the trained FastSCNN model to perform contour segmentation on all tableware sub-images in (3) and obtain the mask images of the tableware to be identified in the corresponding order image; (5) The order image tableware mask image in step (4) is respectively subjected to dilation and erosion operations. The mask image (counterMask) corresponding to the contour area of ​​the tableware can be extracted through the difference operation after dilation and erosion, and the actual tableware contour area image (counterImg) corresponding to the contour mask image area in the original image is extracted; (6) Using the tableware metric learning model (DML) obtained by distillation training with MobileNet as the backbone network model, the feature information of the actual tableware contour area image in step (5) is extracted respectively; (7) Calculate the matching score (score) between each tableware feature to be identified in the order graph and the tableware features already in stock, and select the tableware template with the highest matching score (maxScore) as the identification result.

[0026] ; in, score is the matching score value with the tableware template in the warehouse; pFeatures refers to the feature vector of the tableware to be identified. pFeatures i It refers to the first feature vector of the tableware to be identified. i eigenvalues; pFeaturesAvg j Refers to the stored j The feature vector of the tableware, It means that you have entered j The tableware feature vector i feature values.

[0027] In the tableware contour learning and recognition method based on a deep neural network in an embodiment of the present invention, the tableware contour learning and recognition steps use the same target detection algorithm, contour segmentation algorithm, and contour feature extraction algorithm, and collect multiple tableware images with different lighting conditions, different backgrounds, and different types as data sets used in building models in the target detection algorithm, contour segmentation algorithm, and contour feature extraction algorithm. Each algorithm is specifically described as follows: 1. Object Detection Algorithm (1) Collect a large number of datasets of tableware images with different lighting conditions, backgrounds, and types, and annotate the target boxes for the datasets; (2) Use the YoloV8 network architecture to train the tableware detection model, split the data set in (1) into training set, validation set and test set, and perform data augmentation (such as rotation, scaling, color jitter, etc.) when loading the data to improve the robustness of the model. At the same time, load the pre-trained model as the basic model for training to speed up the convergence speed; the loss is optimized using the SGD optimization algorithm to optimize the comprehensive loss function including classification and bounding box regression loss; (3) Verify the trained model through the validation set and select the detection model with the best positioning detection effect as the target detection and positioning model for tableware.

[0028] In actual applications, the model can be converted to the corresponding platform according to different usage scenarios and deployed to the application. In actual use, the tableware target detection algorithm detects and locates the tableware target in the incoming image. The target detection result contains the location information of the tableware, that is, the target , the image corresponding to the target area position is cut into sub-images and provided to the subsequent algorithm for contour segmentation.

[0029] 2. Contour Segmentation Algorithm (1) Using the tableware sub-image data processed by the target detection algorithm and image segmentation as the data set of the contour segmentation algorithm, and dividing it into a training set, a validation set and a test set, wherein the data annotation method adopted is to perform contour annotation on the image of each tableware target area; (2) Use the training set to train the FastSCNN (Fast Semantic Segmentation Framework) model to obtain a cutlery contour segmentation model; use the data set in (1) to train and verify and select the optimal model as the cutlery contour segmentation model; (3) The above-mentioned tableware contour segmentation model is used to extract the contour of the tableware sub-image to obtain the corresponding mask image. The mask image can be used to extract the mask image (counterMask) corresponding to the contour area of ​​the tableware through the difference operation after expansion and corrosion. By extracting the actual tableware contour area image (counterImg) corresponding to the mask image area of ​​the contour in the original image, the actual tableware contour image (counterImg) can be used by the feature extraction model for feature extraction.

[0030] 3. Contour feature extraction algorithm (1) First, the tableware detected by the target detection algorithm is cut into sub-images, and then the sub-images are segmented using the contour segmentation algorithm. The actual tableware contour image is labeled with the tableware classification and the dataset is divided into training set, validation set, and test set; (2) Select the Swin Transformer model as the teacher model, use the data set in (1) for training, and select the well-trained model with the best performance as the teacher model for distillation; (3) Select the MobileNet backbone network as the student model and use pre-trained weights for initialization to speed up the training process; (4) Obtain the output features of the teacher model and the student model respectively, and optimize the student model parameters based on the calculated classification loss and distillation loss; (5) Evaluate the performance of the student model on the validation set and fine-tune it to ensure that its ability to extract tableware features is optimized; (6) The best training feature extraction model on the test set is selected as the tableware metric learning model DML. The feature dimension of tableware extracted by this model is 1×1280, that is, pFeatures (1×1280) ; The metric learning model DML based on the MobileNet model extracted through the above distillation training extracts the tableware feature vector. The tableware feature vector can be used as the feature of the tableware entering the warehouse. The similarity score between the feature and the feature of the tableware model that has been stored can also be calculated during tableware recognition, and the tableware template corresponding to the highest score is selected as the recognition result.

[0031] In the method of the present invention, the deep neural network models in different steps can be replaced with models with similar functions. In addition, the method can also use other segmentation models for positioning and segmentation to replace the detection model and segmentation model therein, such as the YOLACT segmentation model.

[0032] Embodiment 2: Another aspect of the present invention provides a tableware contour learning and recognition system based on a deep neural network, which is used to execute the tableware contour learning and recognition method based on a deep neural network, comprising: Cutlery outline learning unit, including: The target detection module is used to obtain an image of the tableware entering the warehouse and identify the target coordinate position information of the tableware entering the warehouse in the image based on the target detection algorithm; A target frame expansion module is used to expand the target frame based on the target coordinate position information of the incoming tableware, and to segment the image of the incoming tableware within the expanded target frame to obtain a sub-image of the incoming tableware; A contour segmentation module is used to perform contour segmentation on the incoming tableware sub-image using a contour segmentation algorithm, and obtain a mask image of the contour area of ​​the incoming tableware, and obtain an actual image of the contour area of ​​the incoming tableware based on the mask image; The tableware metric learning model construction module is used to use MobileNet as the backbone network model and perform distillation training to obtain the tableware metric learning model; A feature extraction module is used to extract feature information in the actual image of the contour area of ​​the tableware entering the warehouse by using the tableware metric learning model; A feature storage module is used to store the extracted feature information of the tableware entering the storage into the tableware feature library, wherein the tableware entering the storage serves as a tableware template of the tableware feature library; Tableware contour recognition unit, specifically comprising: A target detection module, used to obtain an image of tableware to be identified, and identify target coordinate position information of the tableware to be identified in the image based on a target detection algorithm; A target frame expansion module is used to expand the target frame based on the target coordinate position information of the tableware to be identified, and to segment the image of the tableware to be identified within the expanded target frame to obtain a sub-image of the tableware to be identified; A contour segmentation module, used to perform contour segmentation on the tableware sub-image to be identified using a contour segmentation algorithm, and obtain a mask image of the contour area of ​​the tableware to be identified, and obtain an actual image of the contour area of ​​the tableware to be identified based on the mask image; A feature extraction module, used to extract feature information in the actual image of the contour area of ​​the tableware to be identified using the tableware metric learning model; The comparison and analysis module is used to compare and match the extracted feature information of the tableware to be identified with the feature information in the tableware feature library, and select the tableware template with the highest matching score as the identification result.

[0033] In the present invention, the tableware contour learning unit and the tableware contour recognition unit share the same set of target detection module, target frame expansion module, contour segmentation module, and feature extraction module.

[0034] In summary, the tableware contour learning and recognition method and system based on deep neural network provided by the present invention have the following characteristics: (1) Based on the optimization and improvement of the model-based contour extraction, a deep neural network model is used to extract the cutlery contour. Combined with the dilation, erosion and difference operations in the traditional image processing method, a cutlery contour edge area mask can be obtained. Through the cutlery edge contour mask, the actual image of the cutlery edge contour area can be accurately extracted, which is compatible with a variety of cutlery of different specifications.

[0035] (2) Based on the optimization and improvement of feature extraction, a deep neural network model is used to extract features from the actual image of the edge contour area of ​​the tableware. The extracted feature dimension is 1×1280. Matching can be achieved quickly through matrix operation, and the recognition speed can meet the needs of actual use.

[0036] The present invention also provides an electronic device, comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the tableware contour learning and recognition method based on a deep neural network as described above.

[0037] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A tableware contour learning and recognition method based on deep neural network, characterized in that: The following steps are involved: S1, tableware outline learning, including: S101, acquiring an image of tableware entering the warehouse, and identifying target coordinate position information of the tableware entering the warehouse in the image based on a target detection algorithm; S102, expanding a target frame based on target coordinate position information of the stored tableware, and segmenting the image of the stored tableware within the expanded target frame to obtain a sub-image of the stored tableware; S103, using a contour segmentation algorithm to perform contour segmentation on the incoming tableware sub-image, and obtaining a mask image of the contour area of ​​the incoming tableware, and obtaining an actual image of the contour area of ​​the incoming tableware based on the mask image; S104, using MobileNet as the backbone network model, distillation training is performed to obtain the tableware metric learning model; S105, extracting feature information from the actual image of the contour area of ​​the incoming tableware using the tableware metric learning model, and storing the feature information in the tableware feature library, wherein the incoming tableware serves as a tableware template of the tableware feature library; S2, tableware contour recognition, specifically including: S201, acquiring an image of tableware to be identified, and identifying target coordinate position information of the tableware to be identified in the image based on a target detection algorithm; S202, expanding a target frame based on target coordinate position information of the tableware to be identified, and segmenting the image of the tableware to be identified within the expanded target frame to obtain a sub-image of the tableware to be identified; S203, using a contour segmentation algorithm to perform contour segmentation on the tableware sub-image to be identified, and obtaining a mask image of the contour area of ​​the tableware to be identified, and obtaining an actual image of the contour area of ​​the tableware to be identified based on the mask image; S204, using the tableware metric learning model to extract feature information in the actual image of the contour area of ​​the tableware to be identified, and compare and match it with the feature information in the tableware feature library, and select the tableware template with the highest matching score as the identification result.

2. The tableware contour learning and recognition method based on deep neural network according to claim 1 is characterized in that: The step S101, obtaining an image of the tableware entering the warehouse and identifying the target coordinate position information of the tableware entering the warehouse in the image based on a target detection algorithm, specifically includes: (1) Collect multiple tableware images with different lighting conditions, different backgrounds, and different types, build a dataset for target detection, annotate the dataset with target boxes, and divide the dataset into training set, validation set, and test set; (2) Use the training set to train the YoloV8 network model to obtain a tableware detection model; (3) Use the validation set to verify the tableware detection model, and select the tableware detection model with the best positioning detection effect as the target detection positioning model; (4) Use the target detection and positioning model to identify the image of the tableware entering the warehouse and obtain the target coordinate position information of the tableware entering the warehouse.

3. The tableware contour learning and recognition method based on deep neural network according to claim 1, characterized in that: S102, expanding the target frame based on the target coordinate position information of the stored tableware, and segmenting the image of the stored tableware within the expanded target frame to obtain a sub-image of the stored tableware; (1) Based on the target coordinate location information of the tableware entering the warehouse , expand the target frame rLen Length, get : ； (2) Based on The target box is used to segment the image of the incoming tableware and obtain the incoming tableware sub-image.

4. The tableware contour learning and recognition method based on deep neural network according to claim 1, characterized in that: The step S103, using a contour segmentation algorithm to perform contour segmentation on the stored tableware sub-image, and obtaining a mask image of the contour area of ​​the stored tableware, and obtaining an actual image of the contour area of ​​the stored tableware based on the mask image, specifically includes: (1) Use the trained cutlery contour segmentation model to perform contour segmentation on the incoming cutlery sub-image and obtain the mask image corresponding to the incoming cutlery sub-image; (2) Perform dilation and erosion operations on the mask images corresponding to each sub-image of the tableware in storage, and extract the mask image of the contour area of ​​the tableware in storage through the difference operation after dilation and erosion; (3) Extract the actual image of the contour area in the image of the stored tableware corresponding to the mask image.

5. The tableware contour learning and recognition method based on deep neural network according to claim 4 is characterized in that: The training process of the tableware contour segmentation model is as follows: (1) Collect multiple tableware images with different lighting conditions, different backgrounds, and different types, construct a dataset for contour segmentation, annotate the dataset with contours, and divide the dataset into training set, validation set, and test set; (2) Use the training set to train the FastSCNN model to obtain the tableware contour segmentation model; (3) Use the validation set to verify the cutlery contour segmentation model, and select the cutlery contour segmentation model with the best segmentation effect as the trained cutlery contour segmentation model.

6. The tableware contour learning and recognition method based on deep neural network according to claim 1, characterized in that: S104, using MobileNet as the backbone network model, and performing distillation training to obtain a tableware metric learning model, specifically includes: (1) Collect multiple tableware images with different lighting conditions, different backgrounds, and different types, and process them using the target detection and positioning model and the tableware contour segmentation model in turn to obtain the actual images of the contour areas of different types of tableware, annotate them by category, and construct a feature extraction data set, which is divided into a training set, a validation set, and a test set; (2) Select the Swin Transformer model as the teacher model, use the training set in (1) for training, and select the well-trained model with the best performance as the teacher model for distillation; (3) Select the MobileNet backbone network as the student model and use pre-trained weights for initialization to speed up the training process; (4) Obtain the output features of the teacher model and the student model respectively, and optimize the student model parameters based on the calculated classification loss and distillation loss; (5) Evaluate the performance of the student model on the validation set and fine-tune it to ensure that its ability to extract tableware features is optimized; (6) Select the best student model on the test set as the tableware metric learning model DML.

7. The tableware contour learning and recognition method based on deep neural network according to claim 1, characterized in that: The feature dimension of the tableware metric learning model is 1×1280, that is, Features (1×1280) In S105, the calculation method of the information features extracted by the tableware metric learning model is as follows: ； in, N is the number of the same type of tableware in storage, and each feature information is obtained by taking the average value.

8. The tableware contour learning and recognition method based on deep neural network according to claim 2, characterized in that: In S204, the matching score Score The calculation method is as follows: ； in, Features refers to the feature vector of the tableware to be identified. Features i It refers to the first feature vector of the tableware to be identified. i eigenvalues; pFeaturesAvg j Refers to the stored j The feature vector of the cutlery template, Refers to the stored j The feature vector of the cutlery template i feature values.

9. A tableware contour learning and recognition system based on a deep neural network, used to execute the tableware contour learning and recognition method based on a deep neural network according to any one of claims 1 to 8, characterized in that: include: Cutlery outline learning unit, including: The target detection module is used to obtain an image of the tableware entering the warehouse and identify the target coordinate position information of the tableware entering the warehouse in the image based on the target detection algorithm; A target frame expansion module is used to expand the target frame based on the target coordinate position information of the incoming tableware, and to segment the image of the incoming tableware within the expanded target frame to obtain a sub-image of the incoming tableware; A contour segmentation module is used to perform contour segmentation on the incoming tableware sub-image using a contour segmentation algorithm, and obtain a mask image of the contour area of ​​the incoming tableware, and obtain an actual image of the contour area of ​​the incoming tableware based on the mask image; The tableware metric learning model construction module is used to use MobileNet as the backbone network model and perform distillation training to obtain the tableware metric learning model; A feature extraction module is used to extract feature information in the actual image of the contour area of ​​the tableware entering the warehouse by using the tableware metric learning model; A feature storage module is used to store the extracted feature information of the tableware entering the storage into the tableware feature library, wherein the tableware entering the storage serves as a tableware template of the tableware feature library; Tableware contour recognition unit, specifically comprising: A target detection module, used to obtain an image of tableware to be identified, and identify target coordinate position information of the tableware to be identified in the image based on a target detection algorithm; A target frame expansion module is used to expand the target frame based on the target coordinate position information of the tableware to be identified, and to segment the image of the tableware to be identified within the expanded target frame to obtain a sub-image of the tableware to be identified; A contour segmentation module, used to perform contour segmentation on the tableware sub-image to be identified using a contour segmentation algorithm, and obtain a mask image of the contour area of ​​the tableware to be identified, and obtain an actual image of the contour area of ​​the tableware to be identified based on the mask image; A feature extraction module, used to extract feature information in the actual image of the contour area of ​​the tableware to be identified using the tableware metric learning model; The comparison and analysis module is used to compare and match the extracted feature information of the tableware to be identified with the feature information in the tableware feature library, and select the tableware template with the highest matching score as the identification result.

Images