A dish recognition method based on cascade feature fusion network

Through the dish recognition method based on the cascading feature fusion network, the problem of errors in judgment when obtaining dish information in the fast food industry is solved, and more accurate and efficient dish recognition is achieved to adapt to the complex characteristics of diverse dishes.

CN118038444BActive Publication Date: 2024-11-15杭州正跃信息技术有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311863278.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-29
Publication Date
2024-11-15
Estimated Expiration
2043-12-29

AI Technical Summary

Technical Problem

In hotel operations, especially in the fast food industry, when obtaining and checking out dishes, it is easy to lead to dish judgment and price errors, and it is difficult to improve identification efficiency.

Method used

The dish recognition method based on the cascading feature fusion network is adopted. By obtaining the dish images, labeling the dish image information, training the cascading convolutional neural network model, and using two-dimensional camera calibration technology to obtain the dish image information to be identified, and the target cascading volume is used. The neural network model outputs dish identification information.

Benefits of technology

It improves the quality of dish image acquisition and recognition accuracy, can obtain dish information more accurately, adapt to the complex characteristics of diverse dishes, and improves the efficiency and accuracy of dish recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118038444B_ABST
    Figure CN118038444B_ABST
Patent Text Reader

Abstract

The present invention discloses a dish recognition method based on a cascade feature fusion network, which relates to the technical field of dish recognition, and includes obtaining a dish image; labeling the dish image to obtain dish image information; training a cascade convolutional neural network model based on the dish image information to obtain a target cascade convolutional neural network model; obtaining a dish image to be recognized and dish image information to be recognized based on a two-dimensional camera calibration technology; outputting the dish image to be recognized and the dish image information to be recognized through a target cascade convolutional neural network model to obtain dish recognition information. The present invention can more accurately obtain information of a dish image to be recognized, improves the acquisition quality of a dish image, avoids affecting dish recognition due to the acquisition of dish image quality, and can improve the accuracy of dish recognition by a subsequent cascade convolutional neural network model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of dish recognition, and in particular to a dish recognition method based on a cascade feature fusion network. Background Art

[0002] In today's digital age, image recognition technology plays an important role in various fields. Among them, food image recognition, as an important application field, has received widespread attention. As people pay more and more attention to dietary health and food culture, it has become more urgent to develop an efficient and accurate dish recognition method;

[0003] In the process of restaurant operation, it is necessary to obtain dish information such as dishes and prices. Especially in the fast food industry, dish information acquisition and checkout are mostly done manually, which can easily lead to incorrect dish and price judgments. The process of judging dish prices and categories also makes it difficult to improve the efficiency of identifying dishes. Summary of the invention

[0004] The purpose of the present invention is to provide a dish recognition method based on a cascade feature fusion network to solve the shortcomings of the background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a dish recognition method based on a cascade feature fusion network, comprising the following steps:

[0006] Get the dish image;

[0007] Annotating the dish image to obtain dish image information, wherein the dish image information includes basic dish information and dish image feature information;

[0008] The cascade convolutional neural network model is trained based on the dish image information to obtain the target cascade convolutional neural network model;

[0009] Acquire the dish image to be identified and the dish image information to be identified based on the two-dimensional camera calibration technology;

[0010] The dish image to be identified and the dish image information to be identified are output through the target cascade convolutional neural network model to obtain the dish identification information.

[0011] In a preferred embodiment, the step of labeling the dish image to obtain the dish image information includes:

[0012] Based on the dish image, the dish and ingredient name, cooking method, cuisine, and price are marked as the basic information of the dish;

[0013] Acquire tableware features in the dish image, wherein the tableware features include tableware shape and tableware color;

[0014] Cutting out the image of tableware features in the dish image to obtain a partial image of the dish;

[0015] Obtain the center point of the dish portion of the screen, determine the measurement direction of the dish portion of the screen, and establish a two-dimensional coordinate system of the dish portion of the screen based on the center point;

[0016] Based on the two-dimensional coordinate system, the coordinates of the bounding box of the dish image and the coordinates of the bounding box of each ingredient are obtained as the feature information of the dish image;

[0017] The dish image feature information and the dish basic information are used as the dish image information.

[0018] In a preferred embodiment, the step of training a cascaded convolutional neural network model based on dish image information to obtain a target cascaded convolutional neural network model comprises:

[0019] Divide the dish image information into training set and validation set;

[0020] The training set is input into the cascade convolutional neural network model to extract features of different scales of the dish images through different convolution kernel sizes and strides;

[0021] Based on the feature fusion basis, the features of different scales of the dish image are deeply fused to obtain multi-scale features;

[0022] Based on the convolution layer, the multi-scale features are mapped to the target space, and the final output result is generated to obtain the cascade convolutional neural network model to be verified;

[0023] The cascade convolutional neural network model to be verified is verified based on the verification set to obtain the target cascade convolutional neural network model.

[0024] In a preferred embodiment, the step of acquiring the image of the dish to be identified and the image information of the dish to be identified based on the two-dimensional camera calibration technology includes:

[0025] Acquire picture information of the image to be captured, wherein the picture information includes color information and shape information;

[0026] determining the tableware in the image to be captured based on the picture information of the image to be captured;

[0027] Based on the tableware in the image to be captured, an image of the dishes in the tableware is obtained;

[0028] Obtain the center point of the image where the dish is located, and based on the two-dimensional camera calibration technology, correspond the two-dimensional coordinate center point to the center point of the image where the dish is located to obtain the dish image to be identified;

[0029] Obtaining the bounding box coordinates of the dish image to be identified based on two-dimensional camera calibration technology;

[0030] Extracting color information and shape information of multiple ingredients in the dish image to be identified as ingredient feature information;

[0031] The food feature information is measured based on the two-dimensional camera calibration technology to obtain the bounding box coordinates of each food;

[0032] The coordinates of the bounding box of the dish image and the coordinates of the bounding box of each ingredient are used as the dish image information to be identified.

[0033] In a preferred embodiment, the step of determining the tableware in the image to be captured based on the picture information of the image to be captured includes:

[0034] Grid-dividing the image to be captured into a plurality of unit grids, and extracting a plurality of unit grids in the image that have the same color as the tableware;

[0035] The frame where the tableware is located in the image to be captured is determined in a plurality of unit grids according to the shape of the tableware.

[0036] In a preferred embodiment, the step of outputting the dish image to be identified and the dish image information to be identified through a target cascade convolutional neural network model to obtain dish identification information comprises:

[0037] Inputting the dish image to be identified and the dish image information to be identified into the target cascade convolutional neural network model to obtain the output result;

[0038] Dish identification information is obtained based on the output result, wherein the dish identification information is basic information of the dish.

[0039] A dish recognition system based on a cascade feature fusion network, comprising:

[0040] A first acquisition module, used to acquire a dish image;

[0041] a labeling module, connected to the acquisition module, for labeling the dish image to obtain dish image information, wherein the dish image information includes basic dish information and dish image feature information;

[0042] A construction module, connected to the annotation module, is used to train a cascade convolutional neural network model based on the dish image information to obtain a target cascade convolutional neural network model;

[0043] A second acquisition module acquires the image of the dish to be identified and the image information of the dish to be identified based on the two-dimensional camera calibration technology;

[0044] The recognition module is connected to the construction module and the second acquisition module, and is used to obtain dish recognition information by outputting the dish image to be recognized and the dish image information to be recognized through the target cascade convolutional neural network model.

[0045] In a preferred embodiment, the second acquisition module includes:

[0046] An acquisition unit, used to acquire picture information of the image to be captured, wherein the picture information includes color information and shape information;

[0047] A shooting unit, connected to the acquisition unit, for determining the tableware in the image to be shot based on the picture information of the image to be shot;

[0048] A collection unit connected to the shooting unit, used to obtain an image of the dishes in the tableware based on the tableware in the image to be shot;

[0049] A corresponding unit, connected to the acquisition unit, is used to obtain the center point of the image where the dish is located, and based on the two-dimensional camera calibration technology, the center point of the two-dimensional coordinate corresponds to the center point of the image where the dish is located to obtain the dish image to be identified;

[0050] The recognition unit is connected to the corresponding unit and is used to obtain the bounding box coordinates of the dish image to be recognized based on the two-dimensional camera calibration technology;

[0051] An extraction unit, connected to the recognition unit, for extracting color information and shape information of a plurality of ingredients in the dish image to be recognized as ingredient feature information;

[0052] The calibration unit is connected to the extraction unit and is used to measure the food feature information based on the two-dimensional camera calibration technology to obtain the bounding box coordinates of each food.

[0053] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the above method.

[0054] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0055] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0056] 1. The present invention uses the bounding box coordinates of the dish image and the bounding box coordinates of each ingredient as the dish image information to be identified, which can more accurately obtain the information of the dish image to be identified, improve the acquisition quality of the dish image, avoid the acquisition of the dish image quality affecting the dish recognition, and improve the accuracy of subsequent recognition. The above method can be used to more accurately obtain the information of the dish image to be identified, improve the acquisition quality of the dish image, avoid the acquisition of the dish image quality affecting the dish recognition, and improve the accuracy of the subsequent cascade convolutional neural network model in recognizing the dish;

[0057] 2. The present invention can improve the accuracy of feature acquisition through cascade fusion between sub-convolutional layers and convolutional layers, thereby improving the final output result of the model and having a good recognition effect. Multi-scale features are generated by introducing cascade stages, and a lightweight transition network module is used in each stage for feature integration. This design can not only more effectively fuse multi-scale features, but also adapt to the complex features of diverse dishes. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0059] Figure 1 A schematic diagram of a method flow chart proposed in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of a dish recognition system based on a cascade feature fusion network proposed in an embodiment of the present application;

[0061] Figure 3 This is a schematic diagram of a second acquisition module in a dish recognition system based on a cascade feature fusion network proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Example 1, please refer to Figure 1As shown, the dish recognition method based on the cascade feature fusion network described in this embodiment includes the following steps:

[0064] S1, obtain dish images;

[0065] S2. Annotating the dish image to obtain dish image information, wherein the dish image information includes basic dish information and dish image feature information;

[0066] S3, training the cascade convolutional neural network model based on the dish image information to obtain a target cascade convolutional neural network model;

[0067] S4, obtaining the dish image to be identified and the dish image information to be identified based on the two-dimensional camera calibration technology;

[0068] S5. Output the dish image to be identified and the dish image information to be identified through the target cascade convolutional neural network model to obtain dish identification information.

[0069] As described in the above steps S1-S6, the information of the dish image to be identified can be obtained more accurately, the acquisition quality of the dish image is improved, and the recognition of the dish is prevented from being affected by the acquisition quality of the dish image, which can improve the accuracy of the subsequent cascade convolutional neural network model in identifying the dish;

[0070] Multi-scale features are generated by introducing a cascade stage, and a lightweight transition network module is used in each stage for feature integration; this design can not only more effectively fuse multi-scale features, but also adapt to the complex features of diverse dishes;

[0071] In one embodiment, the step S2 of labeling the dish image to obtain dish image information includes:

[0072] S21, labeling the dish image with the name of the dish and the ingredients, the cooking method, the cuisine, and the price of the dish as basic dish information;

[0073] S22, obtaining tableware features in the dish image, wherein the tableware features include tableware shape and tableware color;

[0074] S23, cutting out the image of tableware features in the dish image to obtain a partial dish image;

[0075] S24, obtaining the center point of the dish portion picture, determining the measurement direction of the dish portion picture, and establishing a two-dimensional coordinate system of the dish portion picture based on the center point;

[0076] S25, obtaining the coordinates of the bounding box of the dish image and the coordinates of the bounding box of each ingredient as the dish image feature information based on the two-dimensional coordinate system;

[0077] S26, using the dish image feature information and the dish basic information as dish image information;

[0078] As described in the above steps S21-S26, for the finished dishes, the identified dish information includes the ingredient information and the cuisine information; the ingredient information not only covers the necessary ingredient names to meet the basic ingredient identification requirements, but also adds the cooking method information. By adding the cooking method to the marked ingredient information, it not only helps to improve the accuracy of ingredient identification, but also can distinguish the ingredients processed by different cooking methods, so as to more accurately analyze the nutritional components of the dishes. These cooking methods can specifically include at least one of stir-frying, deep-frying, pan-frying, baking, roasting, steaming, boiling, stewing, etc., and the more cooking methods covered, the more beneficial it is to improve the accuracy of identification. It should be understood that the purpose of introducing cooking methods in the ingredient information is that, on the one hand, for the same ingredient, it can be distinguished according to different shapes, so as to improve the accuracy of identification; on the other hand, it can more accurately analyze the nutritional components of the ingredients, and can also improve the accuracy of subsequent identification. Here, by collecting the dish information, the shape of the dish can be accurately obtained, which is also conducive to the subsequent identification of the dish;

[0079] In one embodiment, the step S3 of training the cascade convolutional neural network model based on the dish image information to obtain the target cascade convolutional neural network model includes:

[0080] S31, dividing the dish image information into a training set and a validation set;

[0081] S32, inputting the training set into the cascade convolutional neural network model to extract features of different scales of the dish image through different convolution kernel sizes and strides;

[0082] S33, based on the feature fusion basis, deeply fuse the features of different scales of the dish image to obtain multi-scale features;

[0083] S34, mapping the multi-scale features to the target space based on the convolution layer, generating the final output result to obtain the cascade convolutional neural network model to be verified;

[0084] S35, verifying the cascade convolutional neural network model to be verified based on the verification set to obtain a target cascade convolutional neural network model;

[0085] As described in the above steps S31-S35, the input dish image is feature extracted through the main backbone network to generate high-level features. The backbone can adopt various existing classification backbones, such as ResNet50. The output feature map of the main backbone has high resolution and semantic information, but lacks multi-scale information; feature expansion. In this step, multiple network modules are included, each of which has the same network architecture. For each sub-network module, a sub-feature extraction module and a feature fusion module are included. The sub-feature extraction module is used to extract multi-scale features, and the feature fusion module is used to aggregate these features together. In each sub-network module, the feature extraction module can use different scales and resolutions to generate multi-scale features. These sub-network modules constitute a cascaded feature fusion network; the sub-feature extraction module can use different convolution kernel sizes and strides to extract features of different scales. For example, a smaller convolution kernel and a larger stride can be used to extract low-resolution features, and a larger convolution kernel and a smaller stride can be used to extract high-resolution features. The feature fusion module can adopt different fusion strategies, such as weighted average, cascade fusion, etc., to integrate multi-scale features together; the feature integration operation is integrated into the main backbone network to fuse multi-scale features at a deeper level; specifically, the feature integration operation deeply fuses the multi-scale features obtained in the cascade fusion network with the features in the main backbone network to generate more representative dish ingredient features. The feature integration operation can adopt different fusion strategies, such as weighted average, cascade fusion, etc. Different from the existing multi-scale feature fusion methods; predict the output, map the features to the target space through a fully connected layer or a convolutional layer to generate the final predicted output; different output layer structures, such as a fully connected layer, a convolutional layer, etc., can be used to generate different types of predicted outputs. For example, in the target detection task, a fully connected layer and a softmax layer can be used to generate category probabilities and bounding box coordinates. Through the cascade fusion between the sub-convolutional layer and the convolutional layer, the accuracy of feature acquisition can be improved, thereby improving the final output results of the model, and having a better recognition effect;

[0086] In one embodiment, the step S4 of acquiring the image of the dish to be identified and the image information of the dish to be identified based on the two-dimensional camera calibration technology includes:

[0087] S41, obtaining picture information of the image to be captured, wherein the picture information includes color information and shape information;

[0088] S42, determining the tableware in the image to be captured based on the picture information of the image to be captured;

[0089] S43, obtaining an image of the dishes in the tableware based on the tableware in the image to be captured;

[0090] S44, obtaining the center point of the image where the dish is located, and using a two-dimensional camera calibration technology to correspond the two-dimensional coordinate center point to the center point of the image where the dish is located to obtain an image of the dish to be identified;

[0091] S45, obtaining the coordinates of the bounding box of the dish image to be identified based on the two-dimensional camera calibration technology;

[0092] S46, extracting color information and shape information of multiple ingredients in the dish image to be identified as ingredient feature information;

[0093] S47, measuring the food feature information based on the two-dimensional camera calibration technology to obtain the bounding box coordinates of each food;

[0094] S48, using the coordinates of the bounding box of the dish image and the coordinates of the bounding box of each ingredient as the dish image information to be identified;

[0095] As described in the above steps S41-S48, for example, during use, it is necessary to temporarily identify the dishes through the image acquisition device. Therefore, during the acquisition process, the image of the dish needs to be processed. First, the image of the location of the dish is dynamically acquired, and the picture information of the image to be captured is correspondingly acquired, wherein the picture information includes color information and shape information. At this time, it is not determined that the picture is the final captured image. Then, it is necessary to distinguish between tableware and dishes. After the image of the dish is obtained in the image to be captured, it is immediately captured and photographed. Then, the center point of the image of the dish is obtained, and the center point of the two-dimensional coordinate is corresponded to the center point of the image of the dish based on the two-dimensional camera calibration technology to obtain the image to be identified. The qualified dish image has completed the collection and calibration according to the two-dimensional camera calibration technology. At this time, it is only necessary to collect the bounding box coordinates of the dish image, and use the color information and shape information of multiple ingredients in the dish image as the ingredient feature information. Based on the two-dimensional camera calibration technology, the ingredient feature information is measured to obtain the bounding box coordinates of each ingredient, and the bounding box coordinates of each ingredient are obtained. The bounding box coordinates of the dish image and the bounding box coordinates of each ingredient are used as the dish image information to be identified, which can more accurately obtain the information of the dish image to be identified, improve the collection quality of the dish image, avoid the influence of the dish recognition due to the collection of the dish image quality, and improve the accuracy of subsequent recognition;

[0096] In one embodiment, the step S42 of determining the tableware in the image to be captured based on the picture information of the image to be captured includes:

[0097] S421, grid-dividing the image to be captured into a plurality of unit grids, and extracting a plurality of unit grids in the image that have the same color as the tableware;

[0098] S422, determining the frame where the tableware is located in the image to be captured in a plurality of unit grids according to the shape of the tableware;

[0099] As described in the above steps S51 and S52, the shape of the tableware can be accurately determined, the outer frame of the entire dish in the picture can be obtained, and it can be divided into a number of unit grids to synchronously identify the color and shape to extract the tableware. Dividing into a plurality of unit grids can improve the efficiency of analysis, and at the same time can also improve the accuracy of color analysis, thereby improving the accuracy of dish feature extraction, making good preparations for recognition through the cascade convolutional neural network model, and improving the efficiency of dish recognition;

[0100] In one embodiment, the step S5 of outputting the dish image to be identified and the dish image information to be identified through the target cascade convolutional neural network model to obtain the dish identification information includes:

[0101] S51, inputting the dish image to be identified and the dish image information to be identified into the target cascade convolutional neural network model to obtain an output result;

[0102] S52, obtaining dish identification information based on the output result, wherein the dish identification information is basic dish information;

[0103] As described in the above steps S51 and S52, the identified dish image and the dish image information to be identified are input into the target cascade convolutional neural network model to obtain an output result, which corresponds to the name of the dish and the ingredients. The basic information of the dish is obtained by involving the name of the dish and the ingredients. The basic information of the dish includes the name of the dish and the ingredients, the cooking method of the dish, the cuisine, the price of the dish and other information. The accuracy of the dish image information recognition directly affects the recognition accuracy of the subsequent cascade convolutional neural network model for the dish. Therefore, the use of accurate dish image information acquisition means also greatly improves the subsequent recognition accuracy of the dish, can more effectively integrate multi-scale features, and can also adapt to the complex features of diverse dishes. It is worth noting that we use residual connections and GELU activation functions in the network. These elements help to speed up the convergence speed of the network and improve training efficiency. Finally, the Block4 network architecture outputs a feature tensor that has undergone multi-level feature extraction. The introduction of the Block4 network architecture enables our model to extract features at a deeper level, thereby capturing more abstract and advanced features and improving the recognition ability of the model. Secondly, the use of residual connections and GELU activation functions not only improves the convergence speed of the model, but also enhances the stability and generalization ability of the model. Finally, the design of the fusion network enables the features of different modules to be fully integrated, thereby improving the model's perception of global and local features. Overall, these advantages help improve the performance of our method in the dish recognition task.

[0104] The function of the fusion network is to integrate the features of Block 1-4 to better capture global and local feature information. Through convolution and upsampling operations, we unify the dimensions of the feature tensor to prepare for subsequent feature fusion. Finally, through the concatenate operation, we fuse the features from 4 different modules together and use the fused features as the output of the network;

[0105] The above method can more accurately obtain the information of the dish image to be identified, improve the acquisition quality of the dish image, avoid the influence of the dish recognition on the acquisition of the dish image quality, and improve the accuracy of the subsequent cascade convolutional neural network model in identifying the dish;

[0106] Multi-scale features are generated by introducing a cascade stage, and a lightweight transition network module is used in each stage for feature integration; this design can not only more effectively fuse multi-scale features, but also adapt to the complex features of diverse dishes;

[0107] See also Figure 2 As shown, the present application also provides a dish recognition system based on a cascade feature fusion network, which can be run on electronic products such as mobile phones and pads, including:

[0108] The first acquisition module 101 is used to acquire a dish image;

[0109] The labeling module 102 is connected to the acquisition module and is used to label the dish image to obtain dish image information, wherein the dish image information includes basic dish information and dish image feature information;

[0110] A construction module 103, connected to the labeling module, is used to train a cascade convolutional neural network model based on the dish image information to obtain a target cascade convolutional neural network model;

[0111] The second acquisition module 104 acquires the dish image to be identified and the dish image information to be identified based on the two-dimensional camera calibration technology;

[0112] The recognition module 105 is connected to the construction module and the second acquisition module, and is used to obtain dish recognition information by outputting the dish image to be recognized and the dish image information to be recognized through the target cascade convolutional neural network model;

[0113] See also Figure 3 As shown, in one embodiment, the second acquisition module includes:

[0114] The acquisition unit 1041 is used to acquire the picture information of the image to be captured, wherein the picture information includes color information and shape information;

[0115] A shooting unit 1042, connected to the acquisition unit, for determining the tableware in the image to be shot based on the picture information of the image to be shot;

[0116] The acquisition unit 1043 is connected to the shooting unit and is used to obtain an image of the dishes in the tableware based on the tableware in the image to be shot;

[0117] The corresponding unit 1044 is connected to the acquisition unit and is used to obtain the center point of the image where the dish is located, and to correspond the center point of the two-dimensional coordinate to the center point of the image where the dish is located based on the two-dimensional camera calibration technology to obtain the dish image to be identified;

[0118] The recognition unit 1045 is connected to the corresponding unit and is used to obtain the bounding box coordinates of the dish image to be recognized based on the two-dimensional camera calibration technology;

[0119] An extraction unit 1046, connected to the recognition unit, for extracting color information and shape information of multiple ingredients in the dish image to be recognized as ingredient feature information;

[0120] The calibration unit 1047 is connected to the extraction unit and is used to measure the food feature information based on the two-dimensional camera calibration technology to obtain the bounding box coordinates of each food;

[0121] Further explanation, continuous dish image data is collected through a camera installed on the device. After obtaining a large amount of different dish image data, these images are annotated, and the annotation content includes the name of the ingredients contained in the dish (for example, green vegetables and shredded pork contain two ingredients, green vegetables and shredded pork), and the cooking method. The second acquisition module is set on the device for collecting dishes, and the channel is located at the location of the dish at the checkout.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A dish recognition method based on a cascade feature fusion network, characterized in that: The following steps are involved: Get the dish image; Annotating the dish image to obtain dish image information, wherein the dish image information includes basic dish information and dish image feature information; The cascade convolutional neural network model is trained based on the dish image information to obtain the target cascade convolutional neural network model; Acquire the dish image to be identified and the dish image information to be identified based on the two-dimensional camera calibration technology; Outputting the dish image to be identified and the dish image information to be identified through the target cascade convolutional neural network model to obtain dish identification information; The step of acquiring the dish image to be identified and the dish image information to be identified based on the two-dimensional camera calibration technology includes: Acquire picture information of the image to be captured, wherein the picture information includes color information and shape information; determining the tableware in the image to be captured based on the picture information of the image to be captured; Based on the tableware in the image to be captured, an image of the dishes in the tableware is obtained; Obtain the center point of the image where the dish is located, and based on the two-dimensional camera calibration technology, correspond the two-dimensional coordinate center point to the center point of the image where the dish is located to obtain the dish image to be identified; Obtaining the bounding box coordinates of the dish image to be identified based on two-dimensional camera calibration technology; Extracting color information and shape information of multiple ingredients in the dish image to be identified as ingredient feature information; The food feature information is measured based on the two-dimensional camera calibration technology to obtain the bounding box coordinates of each food; The coordinates of the bounding box of the dish image and the coordinates of the bounding box of each ingredient are used as the dish image information to be identified.

2. The dish recognition method based on cascade feature fusion network according to claim 1 is characterized by: The step of labeling the dish image to obtain dish image information includes: Based on the dish image, the dish and ingredient name, cooking method, cuisine, and price are marked as the basic information of the dish; Acquire tableware features in the dish image, wherein the tableware features include tableware shape and tableware color; Cutting out the image of tableware features in the dish image to obtain a partial image of the dish; Obtain the center point of the dish portion of the screen, determine the measurement direction of the dish portion of the screen, and establish a two-dimensional coordinate system of the dish portion of the screen based on the center point; Based on the two-dimensional coordinate system, the coordinates of the bounding box of the dish image and the coordinates of the bounding box of each ingredient are obtained as the feature information of the dish image; The dish image feature information and the dish basic information are used as the dish image information.

3. The dish recognition method based on cascade feature fusion network according to claim 1 is characterized by: The step of training the cascade convolutional neural network model based on the dish image information to obtain a target cascade convolutional neural network model includes: Divide the dish image information into training set and validation set; The training set is input into the cascade convolutional neural network model to extract features of different scales of the dish images through different convolution kernel sizes and strides; Based on the feature fusion basis, the features of different scales of the dish image are deeply fused to obtain multi-scale features; Based on the convolution layer, the multi-scale features are mapped to the target space, and the final output result is generated to obtain the cascade convolutional neural network model to be verified; The cascade convolutional neural network model to be verified is verified based on the verification set to obtain the target cascade convolutional neural network model.

4. The dish recognition method based on cascade feature fusion network according to claim 1 is characterized by: The step of determining the tableware in the image to be captured based on the picture information of the image to be captured comprises: Grid-dividing the image to be captured into a plurality of unit grids, and extracting a plurality of unit grids in the image that have the same color as the tableware; The frame where the tableware is located in the image to be captured is determined in a plurality of unit grids according to the shape of the tableware.

5. The dish recognition method based on cascade feature fusion network according to claim 1 is characterized by: The step of outputting the dish image to be identified and the dish image information to be identified through the target cascade convolutional neural network model to obtain the dish identification information comprises: Inputting the dish image to be identified and the dish image information to be identified into the target cascade convolutional neural network model to obtain the output result; Dish identification information is obtained based on the output result, wherein the dish identification information is basic information of the dish.

6. A dish recognition system based on cascade feature fusion network, characterized in that: include: A first acquisition module, used to acquire a dish image; a labeling module, connected to the acquisition module, for labeling the dish image to obtain dish image information, wherein the dish image information includes basic dish information and dish image feature information; A construction module, connected to the annotation module, is used to train a cascade convolutional neural network model based on the dish image information to obtain a target cascade convolutional neural network model; The second acquisition module acquires the image of the dish to be identified and the image information of the dish to be identified based on the two-dimensional camera calibration technology; A recognition module, connected to the construction module and the second acquisition module, for obtaining dish recognition information by outputting the dish image to be recognized and the dish image information to be recognized through the target cascade convolutional neural network model; The second acquisition module includes: An acquisition unit, used to acquire picture information of the image to be captured, wherein the picture information includes color information and shape information; A shooting unit, connected to the acquisition unit, for determining the tableware in the image to be shot based on the picture information of the image to be shot; A collection unit connected to the shooting unit, used to obtain an image of the dishes in the tableware based on the tableware in the image to be shot; A corresponding unit, connected to the acquisition unit, is used to obtain the center point of the image where the dish is located, and based on the two-dimensional camera calibration technology, the center point of the two-dimensional coordinate corresponds to the center point of the image where the dish is located to obtain the dish image to be identified; The recognition unit is connected to the corresponding unit and is used to obtain the bounding box coordinates of the dish image to be recognized based on the two-dimensional camera calibration technology; An extraction unit, connected to the recognition unit, for extracting color information and shape information of a plurality of ingredients in the dish image to be recognized as ingredient feature information; The calibration unit is connected to the extraction unit and is used to measure the characteristic information of the ingredients based on the two-dimensional camera calibration technology to obtain the bounding box coordinates of each ingredient, and use the bounding box coordinates of the dish image and the bounding box coordinates of each ingredient as the dish image information to be identified.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of a dish recognition method based on a cascade feature fusion network as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a dish recognition method based on a cascade feature fusion network as described in any one of claims 1 to 5 are implemented.