Processing method, device and equipment for intelligently identifying categories of dishes in dinner plate

Abstract

The invention relates to the technical field of dish identification, in particular to a processing method, device and equipment for intelligently identifying the category of dishes in a dinner plate, and the method comprises the steps: collecting dish image data of a target dish placed on a target dinner plate through an image collector arranged on the dish identification equipment; based on a pre-trained dish identification model, performing identification operation on the acquired dish image data to obtain a dish identification result output by the dish identification model; and when the dish identification result is used for indicating that the corresponding dish name is identified, determining the identified dish name as the dish name of the target dish, so that the category of the dish can be accurately and rapidly identified, the efficiency of accurately calculating the dish price is improved, the identification accuracy of the dish weight is improved, and the user experience is improved. Therefore, the analysis accuracy of the nutritional value of the dishes ingested by the user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of dish recognition, and in particular to a processing method, device and equipment for intelligently identifying the types of dishes on a plate. Background Art

[0002] With the acceleration of urbanization, a faster pace of life, and an aging population (e.g., increased dining needs among dual-income families and the elderly), the government is promoting the construction of community canteens through policy support to address residents' food shortages. These canteens, with their proximity to communities and affordable prices, have quickly become a vital supplement to people's livelihood needs.

[0003] At present, there are usually two ways to identify dishes in large cafeterias: when the serving staff places the dishes selected by the user on the user's plate, the user's selected dishes are entered into the serving equipment at that time; or, after the user has ordered all the dishes, he or she goes to the cashier counter, where the cashier counter staff identifies the dishes on the plate and then enters the information into the price device for billing. However, practice has found that due to human subjectivity and fatigue, it is impossible to achieve efficient and accurate identification of dishes through manual dish identification, which is especially unsuitable for large cafeteria scenarios. Therefore, it is particularly important to improve the efficiency of accurately identifying dishes, improve the accuracy of identifying the weight of dishes, and thus improve the accuracy of analyzing the nutritional value of the dishes consumed by users. Summary of the Invention

[0004] The present invention provides a processing method, device and equipment for intelligently identifying the category of dishes on a plate. The method can analyze the image data of dishes based on a model obtained from a universal basic dish recognition model, and accurately and quickly identify the category of dishes.

[0005] In order to solve the above technical problems, a first aspect of an embodiment of the present invention discloses a method for intelligently identifying the category of dishes on a plate, the method comprising:

[0006] The image collector provided on the dish recognition device collects dish image data of the target dish placed on the target plate;

[0007] Performing a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model;

[0008] When the dish recognition result is used to indicate that the corresponding dish name is recognized, the recognized dish name is determined to be the dish name of the target dish.

[0009] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0010] When the dish recognition result indicates that the corresponding dish name is not recognized, the dish name of the target dish is set to another dish name, or the dish name of the dish with the highest similarity to the target dish is determined to be the dish name of the target dish;

[0011] The method further comprises:

[0012] Obtaining a dish sample set, the dish sample set consisting of dish data of a plurality of sample dishes, wherein the dish data of each sample dish includes the dish name of the sample dish and dish image data obtained by photographing the sample dish from different angles when the sample dish is placed on a sample plate;

[0013] Training a predetermined basic dish recognition model based on the dish data of each of the sample dishes to obtain a similarity between each of the sample dishes and a plurality of basic dishes in the basic dish recognition model;

[0014] According to the similarity of all the basic dishes corresponding to each sample dish, the basic dish recognition model is trained to obtain the dish recognition model.

[0015] As an optional embodiment, in the first aspect of the present invention, the training of the basic dish recognition model based on the similarity of all the basic dishes corresponding to each sample dish to obtain the dish recognition model includes:

[0016] For any of the sample dishes, selecting a target basic dish having the greatest similarity to the sample dish from the similarities of all the basic dishes;

[0017] Determining whether the similarity corresponding to the target basic dish is greater than or equal to a preset dish similarity; if so, determining the dish name of the target basic dish as the dish name of the sample dish, and updating the dish image data corresponding to the sample dish to the dish image data of the target basic dish;

[0018] When the result of the judgment is no, the dish name of the sample dish and the dish image data of the sample dish are associated to obtain the associated information of the sample dish;

[0019] The dish name of the sample dish, the dish image data of the sample dish, and the associated information of the sample dish are updated into the basic dish recognition model to obtain the dish recognition model.

[0020] As an optional embodiment, in the first aspect of the present invention, before performing a recognition operation on the collected dish image data based on a pre-trained dish recognition model and obtaining a dish recognition result output by the dish recognition model, the method further includes:

[0021] Analyzing the number of dish categories of dishes in the target plate according to the dish image data;

[0022] Determine whether the number of dish categories in the target plate is greater than or equal to the preset number of dish categories. When the judgment result is no, execute the recognition operation on the collected dish image data based on the pre-trained dish recognition model to obtain the dish recognition result output by the dish recognition model.

[0023] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0024] When it is determined that the number of dish categories in the target plate is greater than or equal to the preset number of dish categories, analyzing RGB information of the dish image data according to the dish image data to obtain an RGB analysis result of the dish image data;

[0025] Divide the area on the target plate where the target dish is placed according to the RGB analysis result of the dish image data and the number of dish categories to obtain a dish placement area for each sub-target dish, wherein the target dish includes all the sub-target dishes;

[0026] Dividing the dish image data according to the dish placement area of ​​each sub-target dish to obtain dish image data of each sub-target dish;

[0027] The step of performing a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model includes:

[0028] Based on a pre-trained dish recognition model, a recognition operation is independently performed on the dish image data of each sub-target dish to obtain a dish recognition result output by the dish recognition model.

[0029] As an optional embodiment, in the first aspect of the present invention, analyzing the number of dish categories of dishes in the target plate based on the dish image data includes:

[0030] determining, based on the dish image data, whether the dish on the target plate is stored in a separate dish container;

[0031] When it is determined that the dishes on the target plate are stored in separate dish containers, identifying the number of dish containers containing the dishes on the target plate based on the dish image data;

[0032] The number of containers corresponding to the target plate is determined as the number of dish categories of the dishes in the target plate.

[0033] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0034] When it is determined that the dish on the target plate is not stored in a separate dish container, analyzing the dish attribute data on the target plate based on the dish image data, the dish attribute data on the target plate including the dish texture data, dish shape data, and dish color data on the target plate;

[0035] Classifying dishes with the same attributes according to the attribute data of the dishes on the target plate to obtain a dish classification result;

[0036] Based on the dish classification results, determine whether dishes with the same attributes are distributed together. When it is determined that dishes with the same attributes are distributed together, determine the number of dish classifications on the target plate based on the dish classification results, as the number of dish categories of the dishes in the target plate.

[0037] A second aspect of an embodiment of the present invention discloses a processing device for intelligently identifying the category of dishes on a plate, the device comprising:

[0038] An acquisition module is used to acquire image data of a target dish placed on a target plate through an image acquisition device provided on the dining device;

[0039] A recognition module, configured to perform a recognition operation on the collected dish image data based on a pre-trained dish recognition model, and obtain a dish recognition result output by the dish recognition model;

[0040] a determination module, configured to, when the dish recognition result indicates that a corresponding dish name has been recognized, determine that the recognized dish name is the dish name of the target dish;

[0041] The determination module is also used to set the dish name of the target dish to another dish name when the dish recognition result is used to indicate that the corresponding dish name is not recognized, or to determine the dish name of the dish with the highest similarity to the target dish as the dish name of the target dish.

[0042] As an optional embodiment, in the second aspect of the present invention, the determination module is further configured to, when the dish recognition result indicates that the corresponding dish name is not recognized, set the dish name of the target dish to another dish name, or determine the dish name of the dish with the highest similarity to the target dish as the dish name of the target dish;

[0043] The determining module is further configured to obtain a dish sample set, the dish sample set comprising dish data of a plurality of sample dishes, wherein the dish data of each sample dish comprises the dish name of the sample dish and dish image data obtained by photographing the sample dish from different angles when the sample dish is placed on a sample plate;

[0044] The device further comprises:

[0045] A training module is used to train a predetermined basic dish recognition model based on the dish data of each sample dish to obtain the similarity between each sample dish and multiple basic dishes in the basic dish recognition model; according to the similarity of all the basic dishes corresponding to each sample dish, the basic dish recognition model is trained to obtain the dish recognition model.

[0046] As an optional embodiment, in the second aspect of the present invention, the training module trains the basic dish recognition model according to the similarity of all the basic dishes corresponding to each sample dish, and a specific method of obtaining the dish recognition model includes:

[0047] For any of the sample dishes, selecting a target basic dish having the greatest similarity to the sample dish from the similarities of all the basic dishes;

[0048] Determining whether the similarity corresponding to the target basic dish is greater than or equal to a preset dish similarity; if so, determining the dish name of the target basic dish as the dish name of the sample dish, and updating the dish image data corresponding to the sample dish to the dish image data of the target basic dish;

[0049] When the result of the judgment is no, the dish name of the sample dish and the dish image data of the sample dish are associated to obtain the associated information of the sample dish;

[0050] The dish name of the sample dish, the dish image data of the sample dish, and the associated information of the sample dish are updated into the basic dish recognition model to obtain the dish recognition model.

[0051] As an optional embodiment, in the second aspect of the present invention, the device further includes:

[0052] an analysis module configured to analyze the number of dish categories of dishes on the target plate based on the dish image data before the recognition module performs a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model;

[0053] The judgment module is used to judge whether the number of dish categories in the target plate is greater than or equal to the preset number of dish categories. When the judgment result is no, the recognition module is triggered to perform the recognition operation on the collected dish image data based on the pre-trained dish recognition model to obtain the dish recognition result output by the dish recognition model.

[0054] As an optional embodiment, in the second aspect of the present invention, the analysis module is further configured to, when it is determined that the number of dish categories in the target plate is greater than or equal to the preset number of dish categories, analyze RGB information of the dish image data based on the dish image data to obtain an RGB analysis result of the dish image data;

[0055] The device further comprises:

[0056] a division module, configured to divide the area on the target plate where the target dish is placed according to the RGB analysis result of the dish image data and the number of dish categories, to obtain a dish placement area for each sub-target dish, wherein the target dish includes all the sub-target dishes;

[0057] The division module is further configured to divide the dish image data according to the dish placement area of ​​each sub-target dish to obtain dish image data of each sub-target dish;

[0058] The recognition module performs a recognition operation on the collected dish image data based on a pre-trained dish recognition model, and obtains the dish recognition result output by the dish recognition model in a specific manner including:

[0059] Based on a pre-trained dish recognition model, a recognition operation is independently performed on the dish image data of each sub-target dish to obtain a dish recognition result output by the dish recognition model.

[0060] As an optional embodiment, in the second aspect of the present invention, the specific manner in which the analysis module analyzes the number of dish categories of the dishes on the target plate based on the dish image data includes:

[0061] determining, based on the dish image data, whether the dish on the target plate is stored in a separate dish container;

[0062] When it is determined that the dishes on the target plate are stored in separate dish containers, identifying the number of dish containers containing the dishes on the target plate based on the dish image data;

[0063] The number of containers corresponding to the target plate is determined as the number of dish categories of the dishes in the target plate.

[0064] As an optional embodiment, in the second aspect of the present invention, the specific manner in which the analysis module analyzes the number of dish categories of the dishes on the target plate based on the dish image data further includes:

[0065] When it is determined that the dish on the target plate is not stored in a separate dish container, analyzing the dish attribute data on the target plate based on the dish image data, the dish attribute data on the target plate including the dish texture data, dish shape data, and dish color data on the target plate;

[0066] Classifying dishes with the same attributes according to the attribute data of the dishes on the target plate to obtain a dish classification result;

[0067] Based on the dish classification results, determine whether dishes with the same attributes are distributed together. When it is determined that dishes with the same attributes are distributed together, determine the number of dish classifications on the target plate based on the dish classification results, as the number of dish categories of the dishes in the target plate.

[0068] A third aspect of the present invention discloses another processing device for intelligently identifying the category of dishes on a plate, the device comprising:

[0069] a memory storing executable program code;

[0070] a processor coupled to a memory;

[0071] The processor calls the executable program code stored in the memory to execute part or all of the steps in any one of the processing methods for intelligently identifying the category of dishes on a plate disclosed in the first aspect of the present invention.

[0072] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in any one of the processing methods for intelligently identifying the category of dishes on a plate disclosed in the first aspect of the present invention.

[0073] The fifth aspect of the present invention discloses a dish identification device, which includes a food retrieval device or a plate return device, wherein the dish identification device is used to execute part or all of the steps of any one of the processing methods for intelligently identifying the category of dishes in a plate disclosed in the first aspect of the present invention.

[0074] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0075] In an embodiment of the present invention, an image collector provided on a dish recognition device is used to collect dish image data of a target dish placed on a target plate; a recognition operation is performed on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model; when the dish recognition result is used to indicate that the corresponding dish name has been recognized, the recognized dish name is determined to be the dish name of the target dish. It can be seen that the embodiment of the present invention can accurately and quickly identify the category of the dish by analyzing the image data of the dish based on a model obtained based on a general basic dish recognition model, so as to improve the efficiency of accurately calculating the price of the dish and improve the recognition accuracy of the dish weight, thereby improving the accuracy of the analysis of the nutritional value of the dish consumed by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0077] Figure 1 This is a flow chart of a method for intelligently identifying the types of dishes on a plate disclosed in an embodiment of the present invention;

[0078] Figure 2 This is a schematic structural diagram of a processing device for intelligently identifying the category of dishes on a plate disclosed in an embodiment of the present invention;

[0079] Figure 3 This is a schematic structural diagram of another processing device for intelligently identifying the category of dishes on a plate disclosed in an embodiment of the present invention;

[0080] Figure 4 This is a structural diagram of another processing device for intelligently identifying the category of dishes on a plate disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0082] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or device.

[0083] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0084] The present invention discloses a processing method, device, and apparatus for intelligently identifying the category of dishes on a plate. The method includes collecting dish image data of a target dish placed on a target plate through an image collector provided on the dish recognition device. A recognition operation is performed on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model. When the dish recognition result indicates that a corresponding dish name has been identified, the identified dish name is determined to be the target dish name. This method enables accurate and rapid identification of the dish category, thereby improving the efficiency of accurately calculating dish prices and the accuracy of identifying dish weights, thereby improving the accuracy of analyzing the nutritional value of the dishes consumed by the user. These are described in detail below.

[0085] Example 1

[0086] See also Figure 1 , Figure 1 It is a flow chart of a processing method for intelligently identifying the category of dishes on a plate disclosed in an embodiment of the present invention. The method can be applied to any canteen scene where dish identification is required, wherein the canteen scene is provided with a corresponding processing device, wherein the processing device includes one of a dish identification device, a dish identification system (local system or cloud system) and a dish identification server (local server or cloud server). The dish identification device includes a meal pickup device or a return device. The dish identification device can be connected to the dish identification system or the dish identification server in communication to obtain corresponding data for dish category analysis. Figure 1 As shown, the method may include the following operations:

[0087] 101. Collect image data of a target dish placed on a target plate through an image collector provided on a dish recognition device.

[0088] In an embodiment of the present invention, the dish recognition device may optionally be provided with multiple image collectors, wherein all image collectors may be perpendicular to the plate placement position. The plate placement position is used for a user to place a plate with dishes on it. When the user places a target plate on the plate placement position, the image collectors capture images of the dishes on the target plate and obtain corresponding dish image data.

[0089] 102. Perform recognition operations on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model.

[0090] In an embodiment of the present invention, optionally, the dish recognition model is obtained by training a basic dish recognition model based on dish data of sample dishes.

[0091] 103. When the dish recognition result is used to indicate that the corresponding dish name is recognized, the recognized dish name is determined to be the dish name of the target dish.

[0092] In an embodiment of the present invention, optionally, when the dish recognition model identifies a dish that matches the dish image data, a unique corresponding dish name will be obtained.

[0093] It can be seen that implementation Figure 1 The described method collects dish image data of a target dish placed on a target plate through an image collector provided on a dish recognition device; performs a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model; when the dish recognition result is used to indicate that the corresponding dish name is recognized, the recognized dish name is determined to be the dish name of the target dish, and the category of the dish can be accurately and quickly identified to improve the efficiency of accurately calculating the dish price, and improve the recognition accuracy of the dish weight, thereby improving the analysis accuracy of the nutritional value of the dish consumed by the user.

[0094] In an optional embodiment, the method may further include the following steps:

[0095] When the dish recognition result indicates that the corresponding dish name is not recognized, the dish name of the target dish is set to another dish name, or the dish name with the highest similarity to the target dish is determined to be the dish name of the target dish.

[0096] In this optional embodiment, the other dish names can be one category, i.e., only one dish category, or multiple dish categories. Furthermore, when the target dish name is identified as another dish name, its accurate dish name can be manually confirmed, and dish image data of the target dish from multiple angles can be collected. Finally, the dish recognition model is further optimized based on the dish name of the target dish and the dish image data from multiple different angles to obtain an optimized dish recognition model.

[0097] In this optional embodiment, the dish recognition model may optionally match the identified dish features with those of dishes in the database during the recognition process of the dish image data to determine the similarity between the target dish and the dishes in the database. Furthermore, only when the maximum dish similarity is determined to be greater than or equal to a preset similarity, such as 98%, is the dish name determined to be the target dish.

[0098] It can be seen that when this optional embodiment fails to recognize the name of a dish, it will also summarize it with other dish names or directly use the name corresponding to the most similar dish as the recognized dish name, which can ensure that even if it is a new dish, the corresponding dish name can be recognized so as to perform subsequent operations, such as billing and naming the dish name.

[0099] In another optional embodiment, the method may further include the following steps:

[0100] Obtaining a dish sample set, where the dish sample set consists of dish data of a plurality of sample dishes, where the dish data of each sample dish includes the dish name of the sample dish and dish image data obtained by photographing the sample dish from different angles when the sample dish is placed on a sample plate;

[0101] Training a predetermined basic dish recognition model based on the dish data of each sample dish to obtain a similarity between each sample dish and multiple basic dishes in the basic dish recognition model;

[0102] According to the similarity of all basic dishes corresponding to each sample dish, the basic dish recognition model is trained to obtain the dish recognition model.

[0103] In an optional embodiment, optionally, a basic dish recognition model is trained based on the similarity of all basic dishes corresponding to each sample dish to obtain a dish recognition model, including:

[0104] For any sample dish, select the target basic dish with the greatest similarity to the sample dish from the similarities of all basic dishes;

[0105] Determine whether the similarity corresponding to the target basic dish is greater than or equal to a preset dish similarity, such as 97%. If the result is yes, determine the dish name of the target basic dish as the dish name of the sample dish, and update the dish image data corresponding to the sample dish to the dish image data of the target basic dish;

[0106] When the result of the judgment is no, the dish name of the sample dish and the dish image data of the sample dish are associated to obtain the associated information of the sample dish;

[0107] The dish name of the sample dish, the dish image data of the sample dish, and the associated information of the sample dish are updated into the basic dish recognition model to obtain a dish recognition model.

[0108] In this optional embodiment, optionally, all angles include XYZ angles and multiple angles in directions formed by the three.

[0109] In this optional embodiment, the similarity between each sample dish and the base dish optionally includes multiple similarities among texture similarity, color similarity, shape similarity, and gloss similarity. Optionally, the base dish recognition model can be trained based on a deep learning model such as a CNN. Furthermore, the base dish recognition model is provided with corresponding spatial attention modules and channel attention modules to emphasize the features involved in the similarity.

[0110] In this optional embodiment, optionally, the association information of the sample dish is used to represent the association relationship between the dish name and the dish image data.

[0111] It can be seen that this optional embodiment trains the basic dish recognition model with the dish data of the sample dish, and updates the dish image data of the sample dish to the corresponding dish name in the basic dish recognition model based on the similarity between the trained sample dish and the basic dish, or updates its dish name, dish image data and the association between the two to the basic dish recognition model to obtain the required dish recognition model, thereby improving the accuracy of the dish recognition model training and the training efficiency, thereby quickly obtaining an accurate dish recognition model.

[0112] In another optional embodiment, before performing a recognition operation on the collected dish image data based on a pre-trained dish recognition model and obtaining a dish recognition result output by the dish recognition model, the method may further include the following steps:

[0113] Analyze the number of dish categories in the target plate based on the dish image data;

[0114] Determine whether the number of dish categories in the target plate is greater than or equal to the preset number of dish categories, such as 3 categories. When the judgment result is no, perform a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain the dish recognition result output by the dish recognition model.

[0115] In this optional embodiment, optionally, analyzing the number of dish categories of dishes in the target plate based on the dish image data includes:

[0116] Based on the dish image data, determine whether the dishes on the target plate are stored in separate dish containers;

[0117] When it is determined that the dishes on the target plate are stored in separate dish containers, the number of dish containers containing the dishes on the target plate is identified based on the dish image data;

[0118] The number of containers corresponding to the target plate is determined as the number of dish categories of the dishes in the target plate.

[0119] In this optional embodiment, when the dishes on the target plate are stored in separate dish containers, the obtained dish image data is used to indicate that the boundaries between each dish are particularly clear. The separate dish container can be provided on the target plate itself or a bowl separate from the target plate. Furthermore, based on the color change before and after the dish is placed in the dish container, the collected dish image data is analyzed with the image data of the separate dish container when no dish is placed in it to obtain a container analysis result. Based on the container analysis result, the number of dish containers with dishes placed on the target plate is determined.

[0120] As can be seen, this optional embodiment first analyzes the number of dish categories on the plate before analyzing the dish image data for dish names. If the number is small, the dish image data is further analyzed based on the dish recognition model. This reduces the possibility of unreliable dish recognition due to directly using the entire dish image data when there are many dish categories. Furthermore, when dishes on the plate are stored in individual containers, the number of containers identified is directly used as the number of dish categories, improving the efficiency of accurately identifying dish categories.

[0121] In another optional embodiment, the method may further include the following steps:

[0122] When it is determined that the dish on the target plate is not stored in a separate dish container, the dish attribute data on the target plate is analyzed based on the dish image data, where the dish attribute data on the target plate includes the dish texture data, dish shape data, and dish color data on the target plate;

[0123] According to the attribute data of dishes on the target plate, dishes with the same attributes are classified to obtain the dish classification results;

[0124] Based on the dish classification results, determine whether dishes with the same attributes are distributed together. When it is determined that dishes with the same attributes are distributed together, determine the number of dish classifications on the target plate based on the dish classification results, as the number of dish categories of the dishes in the target plate.

[0125] In this optional embodiment, optionally, when it is determined that there are dishes with the same attributes that are not distributed together, a corresponding dish classification prompt is generated based on the sub-dishes that are not distributed together to prompt the user to classify the sub-dishes that are not distributed together.

[0126] It can be seen that this optional embodiment classifies and analyzes dishes according to their multiple attributes when dishes are not placed in separate dish containers, and when dishes with the same attributes are classified together, the number of dish classifications is used as the number of dish categories. This is particularly suitable for scenarios where dishes are not placed in containers individually, enriches the method of determining the number of dish categories, and improves the accuracy and reliability of determining the number of dish categories.

[0127] In another optional embodiment, the method may further include the following steps:

[0128] When it is determined that the number of dish categories in the target plate is greater than or equal to the preset number of dish categories, such as 3, the RGB information of the dish image data is analyzed based on the dish image data to obtain an RGB analysis result of the dish image data;

[0129] Based on the RGB analysis results of the dish image data and the number of dish categories, the area on the target plate where the target dish is placed is divided into regions to obtain the dish placement area of ​​each sub-target dish, where the target dish includes all sub-target dishes;

[0130] Divide the dish image data according to the dish placement area of ​​each sub-target dish to obtain the dish image data of each sub-target dish;

[0131] Among them, based on the pre-trained dish recognition model, the recognition operation is performed on the collected dish image data to obtain the dish recognition result output by the dish recognition model, including:

[0132] Based on the pre-trained dish recognition model, a recognition operation is independently performed on the dish image data of each sub-target dish to obtain the dish recognition result output by the dish recognition model.

[0133] In this optional embodiment, optionally, the RGB analysis result includes analysis results of the connectivity of the three colors R, G, and B, and the color connectivity areas are divided into dish placement areas according to the number of dish categories.

[0134] In this optional embodiment, optionally, the RGB information of the dish image data is normalized to obtain the normalized RGB information of the dish image data; and the color mean of the three RGB channels of the dish image data is calculated based on the normalized RGB information of the dish image data;

[0135] The color mean of the three RGB channels is determined as the reference center of the dish division area. Based on the reference center of the dish division area and the number of dish categories, the dish image data is color clustered to obtain a color clustering result. The color clustering result is used to represent the clustering of dishes of different colors. The more colors a dish has, the more complex the color clustering result.

[0136] According to the color clustering results, the different color areas of the dish image data are segmented to obtain multiple sub-segmented areas, and the pixel mean of each sub-segmented area is analyzed separately;

[0137] Based on the pixel mean of each sub-segmented area, the pixel difference of adjacent sub-segmented areas is calculated. Based on the pixel difference of adjacent sub-segmented areas and the number of dish categories, a region generation operation is performed on the sub-segmented areas to obtain the dish placement area of ​​each sub-target dish;

[0138] For the dish placement area of ​​each sub-target dish, the coordinates of its minimum circumscribed rectangle and / or polygon vertex coordinates are determined in the dish image data, the area coordinates of the dish placement area in the dish image data are obtained, and based on the corresponding area coordinates, the corresponding dish image data is cropped out from the dish image data.

[0139] It can be seen that when this optional embodiment determines that there are dishes of many categories on the plate, the area where each dish is placed is divided based on the RGB analysis results of the dish image data and the number of dish categories, and then the dish image data is divided based on the divided area division results. Finally, the divided dish image data is analyzed based on the dish recognition model, thereby further improving the recognition efficiency and accuracy of the dish; and the dish image data is color clustered by the color mean of the three RGB channels of the dish image data and the number of dish categories, and then the dish image data is regionally segmented based on the color clustering results. Finally, the dish image data is divided based on the pixel situation of each area after segmentation and the number of dish categories, thereby improving the division accuracy of the image data corresponding to each sub-dish, thereby further improving the dish recognition accuracy and reliability of each sub-dish.

[0140] Example 2

[0141] See also Figure 2 , Figure 2 This is a structural diagram of a processing device for intelligently identifying the category of dishes on a plate disclosed in an embodiment of the present invention. The device can be applied to any canteen scene where dish identification is required, wherein the processing device includes one of a dish identification device, a dish identification system (local system or cloud system) and a dish identification server (local server or cloud server). The dish identification device includes a meal pickup device or a return device. The dish identification device can be connected to the dish identification system or the dish identification server in communication to obtain corresponding data for dish category analysis. Figure 2 As shown, the device includes:

[0142] The acquisition module 201 is used to acquire image data of a target dish placed on a target plate through an image acquisition device provided on the dining device;

[0143] The recognition module 202 is used to perform a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model;

[0144] The determination module 203 is used to determine that the identified dish name is the dish name of the target dish when the dish recognition result is used to indicate that the corresponding dish name is recognized.

[0145] In an embodiment of the present invention, the dish recognition device may optionally be equipped with multiple image collectors, all of which may be positioned perpendicular to the tray placement position. The tray placement position is used for a user to place a tray containing dishes on it. When the user places a target tray on the tray position, the image collectors capture images of the dishes on the target tray and obtain corresponding dish image data.

[0146] In an embodiment of the present invention, optionally, the dish recognition model is obtained by training a basic dish recognition model based on dish data of sample dishes.

[0147] In an embodiment of the present invention, optionally, when the dish recognition model identifies a dish that matches the dish image data, a unique corresponding dish name will be obtained.

[0148] It can be seen that implementation Figure 2The described device collects dish image data of a target dish placed on a target plate through an image collector provided on a dish recognition device; performs a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model; when the dish recognition result is used to indicate that the corresponding dish name is recognized, the recognized dish name is determined to be the dish name of the target dish, and the category of the dish can be accurately and quickly identified to improve the efficiency of accurately calculating the dish price, as well as to improve the recognition accuracy of the dish weight, thereby improving the accuracy of the analysis of the nutritional value of the dish consumed by the user.

[0149] In an optional embodiment, if Figure 2 As shown, the determination module 203 is also used to set the dish name of the target dish to another dish name when the dish recognition result is used to indicate that the corresponding dish name is not recognized, or to determine the dish name of the dish with the highest similarity to the target dish as the dish name of the target dish.

[0150] In this optional embodiment, optionally, the other dish names can be one category, that is, there is only one dish category, or can be multiple dish categories.

[0151] In this optional embodiment, when the dish recognition model recognizes the dish image data, it may optionally match the recognized dish features with the dish features of dishes in the database to determine the similarity between the target dish and the dishes in the database. Furthermore, only when it is determined that the maximum dish similarity is greater than or equal to a preset similarity is the dish name determined to be the target dish.

[0152] It can be seen that when this optional embodiment fails to recognize the name of a dish, it will also summarize it with other dish names or directly use the name corresponding to the most similar dish as the recognized dish name, which can ensure that even if it is a new dish, the corresponding dish name can be recognized so as to perform subsequent operations, such as billing and naming the dish name.

[0153] In another optional embodiment, Figure 2 As shown, the determination module 203 is further used to obtain a dish sample set, which is composed of dish data of multiple sample dishes. The dish data of each sample dish includes the dish name of the sample dish and dish image data taken from different angles when the sample dish is placed on a sample plate;

[0154] Figure 3 This is a schematic diagram of the structure of another processing device for intelligently identifying the category of dishes on a plate disclosed in an embodiment of the present invention. Figure 3 As shown, the device may also include:

[0155] The training module 204 is used to train a predetermined basic dish recognition model based on the dish data of each sample dish to obtain the similarity between each sample dish and multiple basic dishes in the basic dish recognition model; according to the similarity of all basic dishes corresponding to each sample dish, the basic dish recognition model is trained to obtain a dish recognition model.

[0156] In this optional embodiment, the training module 204 optionally trains a basic dish recognition model based on the similarities of all basic dishes corresponding to each sample dish. The specific method of obtaining the dish recognition model includes:

[0157] For any sample dish, select the target basic dish with the greatest similarity to the sample dish from the similarities of all basic dishes;

[0158] Determine whether the similarity corresponding to the target basic dish is greater than or equal to the preset dish similarity. If the result is yes, determine the dish name of the target basic dish as the dish name of the sample dish, and update the dish image data corresponding to the sample dish to the dish image data of the target basic dish;

[0159] When the result of the judgment is no, the dish name of the sample dish and the dish image data of the sample dish are associated to obtain the associated information of the sample dish;

[0160] The dish name of the sample dish, the dish image data of the sample dish, and the associated information of the sample dish are updated into the basic dish recognition model to obtain a dish recognition model.

[0161] In an optional embodiment, optionally, all angles include multiple angles of the direction formed by XYZ.

[0162] In this optional embodiment, the similarity between each sample dish and the base dish optionally includes multiple of texture similarity, color similarity, shape similarity, and gloss similarity. Optionally, the base dish recognition model may be trained based on a deep learning model such as a CNN. Furthermore, the base dish recognition model is provided with corresponding spatial attention modules and channel attention modules to emphasize the features involved in the similarity.

[0163] In this optional embodiment, optionally, the association information of the sample dish is used to represent the association relationship between the dish name and the dish image data.

[0164] It can be seen that this optional embodiment trains the basic dish recognition model with the dish data of the sample dish, and updates the dish image data of the sample dish to the corresponding dish name in the basic dish recognition model based on the similarity between the trained sample dish and the basic dish, or updates its dish name, dish image data and the association between the two to the basic dish recognition model to obtain the required dish recognition model, thereby improving the accuracy of the dish recognition model training and the training efficiency, thereby quickly obtaining an accurate dish recognition model.

[0165] In another optional embodiment, Figure 3 As shown, the device may also include:

[0166] The analysis module 205 is configured to analyze the number of dish categories of dishes on the target plate based on the dish image data before the recognition module 202 performs a recognition operation on the collected dish image data based on the pre-trained dish recognition model to obtain the dish recognition result output by the dish recognition model;

[0167] The judgment module 206 is used to judge whether the number of dish categories in the target plate is greater than or equal to the preset number of dish categories. When the judgment result is no, the recognition module 202 is triggered to perform a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain the dish recognition result output by the dish recognition model.

[0168] In this optional embodiment, the specific method in which the analysis module 205 analyzes the number of dish categories of dishes in the target plate based on the dish image data includes:

[0169] Based on the dish image data, determine whether the dishes on the target plate are stored in separate dish containers;

[0170] When it is determined that the dishes on the target plate are stored in separate dish containers, the number of dish containers containing the dishes on the target plate is identified based on the dish image data;

[0171] The number of containers corresponding to the target plate is determined as the number of dish categories of the dishes in the target plate.

[0172] In this optional embodiment, when the dishes on the target plate are stored in separate dish containers, the obtained dish image data is used to indicate that the boundaries between each dish are particularly clear. The separate dish container can be provided on the target plate itself or a bowl separate from the target plate. Furthermore, based on the color change before and after the dish is placed in the dish container, the collected dish image data is analyzed with the image data of the separate dish container when no dish is placed in it to obtain a container analysis result. Based on the container analysis result, the number of dish containers with dishes placed on the target plate is determined.

[0173] As can be seen, this optional embodiment first analyzes the number of dish categories on the plate before analyzing the dish image data for dish names. If the number is small, the dish image data is further analyzed based on the dish recognition model. This reduces the possibility of unreliable dish recognition due to directly using the entire dish image data when there are many dish categories. Furthermore, when dishes on the plate are stored in individual containers, the number of containers identified is directly used as the number of dish categories, improving the efficiency of accurately identifying dish categories.

[0174] In another optional embodiment, Figure 3 As shown, the analysis module 205 is further configured to, when it is determined that the number of dish categories in the target plate is greater than or equal to the preset number of dish categories, analyze the RGB information of the dish image data according to the dish image data to obtain an RGB analysis result of the dish image data;

[0175] like Figure 3 As shown, the device may also include:

[0176] A division module 207 is configured to divide the area on the target plate where the target dish is placed according to the RGB analysis results of the dish image data and the number of dish categories, to obtain a dish placement area for each sub-target dish, wherein the target dish includes all sub-target dishes;

[0177] The division module 207 is further configured to divide the dish image data according to the dish placement area of ​​each sub-target dish to obtain dish image data of each sub-target dish;

[0178] The recognition module 202 performs a recognition operation on the collected dish image data based on a pre-trained dish recognition model, and obtains a dish recognition result output by the dish recognition model in the following specific ways:

[0179] Based on the pre-trained dish recognition model, a recognition operation is independently performed on the dish image data of each sub-target dish to obtain the dish recognition result output by the dish recognition model.

[0180] In this optional embodiment, optionally, the RGB analysis result includes analysis results of the connectivity of the three colors R, G, and B, and the color connectivity areas are divided into dish placement areas according to the number of dish categories.

[0181] It can be seen that when this optional embodiment determines that there are many types of dishes placed on the plate, the area where each dish is placed is divided based on the RGB analysis results of the dish image data and the number of dish categories, and then the dish image data is divided based on the divided area division results. Finally, the divided dish image data is analyzed based on the dish recognition model, which further improves the efficiency and accuracy of dish recognition.

[0182] In another optional embodiment, the specific method of analyzing the number of dish categories of dishes in the target plate according to the dish image data by the analysis module 205 further includes:

[0183] When it is determined that the dish on the target plate is not stored in a separate dish container, the dish attribute data on the target plate is analyzed based on the dish image data, where the dish attribute data on the target plate includes the dish texture data, dish shape data, and dish color data on the target plate;

[0184] According to the attribute data of dishes on the target plate, dishes with the same attributes are classified to obtain the dish classification results;

[0185] Based on the dish classification results, determine whether dishes with the same attributes are distributed together. When it is determined that dishes with the same attributes are distributed together, determine the number of dish classifications on the target plate based on the dish classification results, as the number of dish categories of the dishes in the target plate.

[0186] In this optional embodiment, optionally, when it is determined that there are dishes with the same attributes that are not distributed together, a corresponding dish classification prompt is generated based on the sub-dishes that are not distributed together to prompt the user to classify the sub-dishes that are not distributed together.

[0187] It can be seen that this optional embodiment classifies and analyzes dishes according to their multiple attributes when dishes are not placed in separate dish containers, and when dishes with the same attributes are classified together, the number of dish classifications is used as the number of dish categories. This is particularly suitable for scenarios where dishes are not placed in containers individually, enriches the method of determining the number of dish categories, and improves the accuracy and reliability of determining the number of dish categories.

[0188] Example 3

[0189] See also Figure 4 , Figure 4This is a structural diagram of another processing device for intelligently identifying the category of dishes on a plate disclosed in an embodiment of the present invention. The device can be used in any canteen scene where dish identification is required, wherein the processing device includes one of a dish identification device, a dish identification system (local system or cloud system) and a dish identification server (local server or cloud server). The dish identification device includes a meal pickup device or a return device. The dish identification device can be connected to the dish identification system or the dish identification server in communication to obtain corresponding data for dish category analysis. Figure 4 As shown, the device may include:

[0190] A memory 301 storing executable program code;

[0191] a processor 302 coupled to the memory 301;

[0192] Furthermore, it may also include an input interface 303 and an output interface 304 coupled to the processor 302;

[0193] The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the method for intelligently identifying the category of dishes on a plate described in the first embodiment.

[0194] Example 4

[0195] An embodiment of the present invention discloses a computer storage medium storing computer instructions. When the computer instructions are called, they are used to execute the steps of the processing method for intelligently identifying the category of dishes on a plate described in the first embodiment.

[0196] Example 5

[0197] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the processing method for intelligently identifying the category of dishes on a plate described in Example 1.

[0198] Example 6

[0199] The embodiment of the present invention discloses a dish identification device, which includes a meal picking device or a plate returning device. Regardless of the device, it can be as follows: Figure 2-4 Any one of the processing devices for intelligently identifying the categories of dishes on a plate is used to execute the steps of the processing method for intelligently identifying the categories of dishes on a plate described in the first embodiment.

[0200] The device embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0201] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, in essence, or the portion that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0202] Finally, it should be noted that the method, device and equipment for intelligently identifying the category of dishes on a plate disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for intelligently identifying the types of dishes on a plate, characterized in that: The method comprises: The image collector provided on the dish recognition device collects dish image data of the target dish placed on the target plate; Performing a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model; When the dish recognition result is used to indicate that the corresponding dish name is recognized, the recognized dish name is determined to be the dish name of the target dish.

2. The method for intelligently identifying the category of dishes on a plate according to claim 1, further comprising: When the dish recognition result indicates that the corresponding dish name is not recognized, the dish name of the target dish is set to another dish name, or the dish name of the dish with the highest similarity to the target dish is determined to be the dish name of the target dish; The method further comprises: Obtaining a dish sample set, the dish sample set consisting of dish data of a plurality of sample dishes, wherein the dish data of each sample dish includes the dish name of the sample dish and dish image data obtained by photographing the sample dish from different angles when the sample dish is placed on a sample plate; Training a predetermined basic dish recognition model based on the dish data of each of the sample dishes to obtain a similarity between each of the sample dishes and a plurality of basic dishes in the basic dish recognition model; According to the similarity of all the basic dishes corresponding to each sample dish, the basic dish recognition model is trained to obtain the dish recognition model.

3. The method for intelligently identifying the category of dishes on a plate according to claim 2, characterized in that: The step of training the basic dish recognition model based on the similarity of all the basic dishes corresponding to each sample dish to obtain the dish recognition model includes: For any of the sample dishes, selecting a target basic dish having the greatest similarity to the sample dish from the similarities of all the basic dishes; Determining whether the similarity corresponding to the target basic dish is greater than or equal to a preset dish similarity; if so, determining the dish name of the target basic dish as the dish name of the sample dish, and updating the dish image data corresponding to the sample dish to the dish image data of the target basic dish; When the result of the judgment is no, the dish name of the sample dish and the dish image data of the sample dish are associated to obtain the associated information of the sample dish; The dish name of the sample dish, the dish image data of the sample dish, and the associated information of the sample dish are updated into the basic dish recognition model to obtain the dish recognition model.

4. The method for intelligently identifying the category of dishes on a plate according to any one of claims 1 to 3, characterized in that: Before performing a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model, the method further includes: Analyzing the number of dish categories of dishes in the target plate according to the dish image data; Determine whether the number of dish categories in the target plate is greater than or equal to the preset number of dish categories. When the judgment result is no, execute the recognition operation on the collected dish image data based on the pre-trained dish recognition model to obtain the dish recognition result output by the dish recognition model.

5. The method for intelligently identifying the category of dishes on a plate according to claim 4, characterized in that: The method further comprises: When it is determined that the number of dish categories in the target plate is greater than or equal to the preset number of dish categories, analyzing RGB information of the dish image data according to the dish image data to obtain an RGB analysis result of the dish image data; Divide the area on the target plate where the target dish is placed according to the RGB analysis result of the dish image data and the number of dish categories to obtain a dish placement area for each sub-target dish, wherein the target dish includes all the sub-target dishes; Dividing the dish image data according to the dish placement area of ​​each sub-target dish to obtain dish image data of each sub-target dish; The step of performing a recognition operation on the collected dish image data based on a pre-trained dish recognition model to obtain a dish recognition result output by the dish recognition model includes: Based on a pre-trained dish recognition model, a recognition operation is independently performed on the dish image data of each sub-target dish to obtain a dish recognition result output by the dish recognition model.

6. The method for intelligently identifying the category of dishes on a plate according to claim 4, characterized in that: Analyzing the number of dish categories of dishes in the target plate based on the dish image data includes: determining, based on the dish image data, whether the dish on the target plate is stored in a separate dish container; When it is determined that the dishes on the target plate are stored in separate dish containers, identifying the number of dish containers containing the dishes on the target plate based on the dish image data; The number of containers corresponding to the target plate is determined as the number of dish categories of the dishes in the target plate.

7. The method for intelligently identifying the category of dishes on a plate according to claim 6, characterized in that: The method further comprises: When it is determined that the dish on the target plate is not stored in a separate dish container, analyzing the dish attribute data on the target plate based on the dish image data, the dish attribute data on the target plate including the dish texture data, dish shape data, and dish color data on the target plate; Classifying dishes with the same attributes according to the attribute data of the dishes on the target plate to obtain a dish classification result; Based on the dish classification results, determine whether dishes with the same attributes are distributed together. When it is determined that dishes with the same attributes are distributed together, determine the number of dish classifications on the target plate based on the dish classification results, as the number of dish categories of the dishes in the target plate.

8. A processing device for intelligently identifying the type of dishes on a plate, characterized in that: The device comprises: An acquisition module is used to acquire image data of a target dish placed on a target plate through an image acquisition device provided on the dining device; A recognition module, configured to perform a recognition operation on the collected dish image data based on a pre-trained dish recognition model, and obtain a dish recognition result output by the dish recognition model; a determination module, configured to, when the dish recognition result indicates that a corresponding dish name has been recognized, determine that the recognized dish name is the dish name of the target dish; The determination module is also used to set the dish name of the target dish to another dish name when the dish recognition result is used to indicate that the corresponding dish name is not recognized, or to determine the dish name of the dish with the highest similarity to the target dish as the dish name of the target dish.

9. A processing device for intelligently identifying the type of dishes on a plate, characterized in that: The processing device comprises: a memory storing executable program code; a processor coupled to a memory; The processor calls the executable program code stored in the memory to execute the processing method for intelligently identifying the category of dishes on a plate as claimed in any one of claims 1 to 7.

10. A dish identification device, characterized in that: The dish identification device includes a meal retrieval device or a plate return device, wherein the dish identification device is used to execute the processing method for intelligently identifying the category of dishes in a plate as claimed in any one of claims 1 to 7.

Images