Food Recognition Method, Device, Storage Medium and Electronic Device

Abstract

An embodiment of the present invention provides a food recognition method, device, storage medium, and electronic device. Among them, the method includes: obtaining a first picture containing a first food; using a first model to analyze the first picture to determine first region information of the first food contained in the first picture; cropping an image of the first food in the first picture based on the first region information, and using the cropped image as a second picture; using a second model to analyze the second picture to determine a first food name of the first food contained in the second picture; and outputting the first region information and the first food name. Through the present invention, the problem of low food recognition accuracy in the related art is solved, the effect of accurately recognizing food is achieved, and the accuracy of food recognition is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and more particularly, to a food recognition method, apparatus, storage medium, and electronic device. Background Art

[0002] Food recognition, as a new application field of computer vision, has received attention from both the academic and industrial communities. In related technologies, there are already a large number of technical methods for realizing food recognition applications. However, for food weight estimation, relevant research is almost non-existent. For food recognition methods, some are based on related hardware devices for food recognition, but their scenarios are fixed and the robustness is poor; the mainstream methods are based on deep learning methods for food recognition, but the training of deep models depends on a large amount of food data, and the data sources are very scarce. At the same time, there are a variety of deep models, and it is also challenging to select a suitable model for food recognition. Especially in the case of Chinese cuisine, since Chinese cuisine involves a wide variety of ingredients and cooking methods, there is a strong long-tail effect. The so-called long-tail effect means that a large number of food categories appear with a small frequency, but due to the huge number of categories, they still account for a non-negligible proportion of food appearances. Therefore, simply applying general food recognition methods to Chinese cuisine pictures is difficult to achieve satisfactory results.

[0003] It can be seen that there is a problem of low food recognition accuracy in related technologies.

[0004] For the above problems existing in related technologies, no effective solution has been proposed yet. Summary of the Invention

[0005] The embodiments of the present invention provide a food recognition method, apparatus, storage medium, and electronic device to at least solve the problem of low food recognition accuracy existing in related technologies.

[0006] According to an embodiment of the present invention, a food recognition method is provided, including: obtaining a first picture containing a first food; using a first model to analyze the first picture to determine first region information of the first food contained in the first picture, where the first model is obtained by training a first initial model through machine learning using multiple groups of first data, and each group of the multiple groups of first data includes: a picture and region information of the food contained in the picture; cropping an image of the first food in the first picture based on the first region information, and using the cropped image as a second picture; using a second model to analyze the second picture to determine a first food name of the first food contained in the second picture, where the second model is obtained by training a second initial model through machine learning using multiple groups of second data, and each group of the multiple groups of second data includes: a picture and the food name of the food included in the picture, and the food name includes ingredient information and cooking information for indicating the way of cooking the ingredients; outputting the first region information and the first food name.

[0007] According to another embodiment of the present invention, a food recognition device is provided, including: a first acquisition module for obtaining a first picture containing a first food; a first analysis module for using a first model to analyze the first picture to determine first region information of the first food contained in the first picture, where the first model is obtained by training a first initial model through machine learning using multiple groups of first data, and each group of the multiple groups of first data includes: a picture and region information of the food contained in the picture; a cropping module for cropping an image of the first food in the first picture based on the first region information, and using the cropped image as a second picture; a second analysis module for using a second model to analyze the second picture to determine a first food name of the first food contained in the second picture, where the second model is obtained by training a second initial model through machine learning using multiple groups of second data, and each group of the multiple groups of second data includes: a picture and the food name of the food included in the picture, and the food name includes ingredient information and cooking information for indicating the way of cooking the ingredients; a first output module for outputting the first region information and the first food name.

[0008] According to still another embodiment of the present invention, a computer-readable storage medium is further provided, where a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0009] According to another embodiment of the present invention, an electronic device is further provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0010] Through the present invention, after obtaining a first picture including a first food, the first picture is analyzed by using a first model to determine first region information of the first food included in the first picture. An image of the first food is cropped according to the first region information and determined as a second picture. The second picture is analyzed by using a second model to determine a first food name of the first food included in the second picture, and the first region information and the first food name are output. The first picture is analyzed by using a model obtained through machine learning to identify the food name. In addition, in the training dataset of the model for analyzing the food name, each set of training data includes: a picture and the food name of the food included in the picture. The food name includes ingredient information and cooking information for indicating a way of cooking the ingredients, and a hierarchical structure composed of the ingredient information and the cooking information. The application of this hierarchical structure enables more accurate and reliable food recognition during the food recognition process. Therefore, the problem of low food recognition accuracy in the related art can be solved, the effect of accurately recognizing food can be achieved, and the food recognition accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a hardware structure block diagram of a mobile terminal for a food recognition method according to an embodiment of the present invention;

[0012] Figure 2 is a flowchart of a food recognition method according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of a multi-food picture with border annotation information according to an embodiment of the present invention;

[0014] Figure 4 is a model training flowchart according to an exemplary embodiment of the present invention;

[0015] Figure 5 is a schematic diagram of Xception according to an exemplary embodiment of the present invention;

[0016] Figure 6 is a schematic diagram of convolution according to an exemplary embodiment of the present invention;

[0017] Figure 7 is a schematic diagram of Depthwise Separable Convolution according to an exemplary embodiment of the present invention;

[0018] Figure 8It is a training flow chart of the second model according to an exemplary embodiment of the present invention;

[0019] Figure 9 It is a schematic diagram of the YOLOV3 network structure according to an exemplary embodiment of the present invention;

[0020] Figure 10 It is a schematic diagram of the Darknet-53 network structure according to an exemplary embodiment of the present invention;

[0021] Figure 11 It is a training flow chart of the first model according to an exemplary embodiment of the present invention;

[0022] Figure 12 It is a training flow chart of the third model according to an exemplary embodiment of the present invention;

[0023] Figure 13 It is an effect diagram of the third model recognition according to an exemplary embodiment of the present invention;

[0024] Figure 14 It is a flow chart of food recognition and food weight detection according to an exemplary embodiment of the present invention;

[0025] Figure 15 It is a structural block diagram of a food recognition device according to an embodiment of the present invention. Detailed implementation manners

[0026] In the following, embodiments of the present invention will be described in detail with reference to the drawings and in conjunction with the embodiments.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence.

[0028] First, the terms that may be used in the present invention will be described:

[0029] Single food recognition: The process of recognizing the name of the food in the image by relevant technologies when the image to be recognized contains only one dish.

[0030] Multi-food recognition: The process of recognizing the names of all foods in the image by object detection technology when the image to be recognized contains at least one dish.

[0031] Food weight estimation: The process of estimating the weight of the food detected and recognized in the image to obtain the weight of the food in the image.

[0032] Food nutrition estimation: The process of estimating the nutrients based on the foods recognized in the image and their weights, and according to the information in the food nutrition database, to obtain the categories and contents of the nutrients included in the foods in the image.

[0033] Food hierarchy: A tree-like hierarchical structure formed by merging the ingredients and cooking methods of foods. The food types of the child nodes of the tree are specific sub-categories of the food types of their parent nodes. Specific food instances are at the leaf nodes of the tree. Among them, in the tree-like hierarchical structure, the top ones are the parent nodes, the end ones are the leaf nodes, and the middle ones are the child nodes.

[0034] In the method embodiments provided in the embodiments of the present application, they can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 It is a hardware structure block diagram of a mobile terminal for a food recognition method according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processor 102 may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above-mentioned mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0035] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the food recognition method in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor 102, and these remote memories can be connected to the mobile terminal through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of a mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] A food recognition method is provided in this embodiment. Figure 2 It is a flowchart of the food recognition method according to an embodiment of the present invention. As Figure 2 shown, the process includes the following steps:

[0038] Step S202, obtain a first picture containing a first food;

[0039] Step S204, use a first model to analyze the first picture to determine first region information of the first food contained in the first picture, wherein the first model is obtained by training a first initial model through machine learning using multiple groups of first data, and each group of the multiple groups of first data includes: a picture and region information of the food contained in the picture;

[0040] Step S206, crop an image of the first food in the first picture based on the first region information, and use the cropped image as a second picture;

[0041] Step S208, use a second model to analyze the second picture to determine a first food name of the first food contained in the second picture, wherein the second model is obtained by training a second initial model through machine learning using multiple groups of second data, and each group of the multiple groups of second data includes: a picture and the food name of the food included in the picture, and the food name includes ingredient information and cooking information for indicating the way of cooking the ingredients;

[0042] Step S210, output the first region information and the first food name. In the above embodiments, the first model may be a food localization model, and the second model may be a food classification model. Among them, both the first model and the second model are obtained through machine learning training. The pictures included in multiple groups of first data and multiple groups of second data may be web picture crawls. For example, pictures searched through a browser, pictures loaded in an App for providing food-making methods, pictures loaded in a meal delivery platform, etc.; they may also be food pictures taken manually; they may also be pictures in some publicly available datasets, such as food-101, UEC FOOD-256, etc. The region information included in the pictures in multiple groups of first data may be manually marked. The training data of the first model may include multiple (for example, 7936, this value is only an illustrative example, the quantity of the first data in the present invention is not limited, and it may also be 7500, 10000, etc.) multi-food pictures with border annotation information (center point coordinates, border length and width). Among them, a schematic diagram of the multi-food pictures with border annotation information can be seen in Appendix Figure 3 . The training data of the second model may include 408383 (this value is only an illustrative example, the present invention does not limit the quantity of the training data of the second model) single-food (only including one kind of food) pictures with category (for example, 600 categories, this value is only an illustrative example, the present invention does not limit the quantity of this category, and it may also be 580 categories, 650 categories, etc.) annotation. Among them, in a more preferred embodiment, the annotation information may be annotated by more than two annotators to further ensure accuracy. Exemplarily, the execution subject of the above steps may be a background processor, or other devices with similar processing capabilities, and may also be a machine at least integrated with an image acquisition device and a data processing device. Among them, the image acquisition device may include an image acquisition module such as a camera, and the data processing device may include terminals such as a computer and a mobile phone, but is not limited thereto.

[0043] Through the present invention, after obtaining the first picture including the first food, the first model is used to analyze the first picture to determine the first region information of the first food included in the first picture. According to the first region information, the image of the first food is cropped and determined as the second picture. The second model is used to analyze the second picture to determine the first food name of the first food included in the second picture, and the first region information and the first food name are output. By using the model obtained through machine learning to analyze the first picture and identify the food name, therefore, the problem of low food recognition accuracy existing in the related art can be solved, the effect of accurately recognizing food can be achieved, and the accuracy of food recognition is improved.

[0044] It should be noted that in single food recognition, there is only one type of food in the first picture, and the first food corresponds to one first region information and one food name; in multi-food recognition, the first food in the first picture is multiple types of food. Therefore, the first food corresponds to multiple first region information (one type of food corresponds to one type of region information) and multiple food names (one type of food corresponds to one food name).

[0045] In an exemplary embodiment, after cropping the image of the first food in the first picture based on the first region information and using the cropped image as the second picture, the method further includes: analyzing the second picture using a third model to determine the first weight value of the first food included in the second picture; wherein, the third model is obtained by training a third initial model through machine learning using multiple groups of third data, and each group of the multiple groups of third data includes: a picture and the weight value of the food included in the picture; outputting the first weight value. In this embodiment, the third model can be a food weight estimation model. The training data of the third model can include more than 100,000 single-food pictures with weight and category label information. Different from the food classification data set (corresponding to the training data of the second model above), this part of the data can all be generated by manual shooting. According to the first level of the food hierarchy (for example, 9 major categories, this value is an exemplary illustration, and the present invention does not limit the number of categories at the first level, and it can also be divided into 8 major categories, 12 major categories, etc.), according to the category number ratio of different branches, 100 (this value is only an exemplary illustration, and the present invention does not limit the number of typical foods, for example, it can also be 80, 120, etc.) different typical foods are selected, and each type of food can be photographed in different environments respectively. That is, the pictures in the third data can be taken in a variety of different shooting environments, and more than 100 pictures can be generated for one category of food. It should be noted that through the above data collection method, except for the need to reweigh when changing the quality, other transformation methods (such as taking pictures of the same food with the same quality at different angles, different backgrounds, etc.) no longer require weighing, greatly reducing the data collection difficulty.

[0046] In the above embodiment, the entire data set (that is, multiple groups of first data, multiple groups of second data, multiple groups of third data) can be divided into three parts by using a data partitioning module: a training set, a validation set, and a test set. The number ratio of the pictures in the three parts can be 3:1:1. The training set data is used to train the model, the validation set is used to screen the model, and the test set is used to evaluate the results of the model. The model training flowchart can be seen in the appendix Figure 4 。

[0047] In an exemplary embodiment, before analyzing the second picture using the second model to determine the first food name of the first food included in the second picture, the method further includes: obtaining the second initial model constructed in the following manner: adding a structural branch to the output of the GlobalAveragePooling of the Xception model to construct the second initial model, where the structural branch is used to identify the hierarchical categories of the food included in the input picture, the hierarchical categories include food types hierarchically divided in descending order of size, the hierarchical categories at least include a first hierarchical category and a second hierarchical category, the first hierarchical category at least includes a hierarchical category for indicating ingredient information, the second hierarchical category includes a hierarchical category for indicating cooking information, the level of the first hierarchical category is higher than that of the second hierarchical category, and the food types in the hierarchical category with a higher level include the food types in the hierarchical category with a lower level; using multiple sets of the second data to train the second initial model through machine learning to obtain the second model. In this embodiment, the first model can adopt Xcepiton (Deep Learning with Depthwise Separable Convolutions), and a schematic diagram of Xception can be seen in the appendix Figure 5 , and this network model mainly uses Depthwise Separable Convolution to replace ordinary convolution for feature extraction of pictures. Among them, a schematic diagram of convolution can be seen in the appendix Figure 6 , and a schematic diagram of Depthwise Separable Convolution can be seen in the appendix Figure 7. Specifically, Depthwise Separable Convolution has two steps. The first step: Depthwise convolution. For each channel of the input, perform K×K (where K represents the size of the convolution kernel, and K can take values such as 3, 5, 7, 9, 11, etc.) convolution operations respectively, and input the results into concat (a function for concatenating multiple arrays). The second step: Pointwise convolution. Perform 1×1 convolution operations on the concat results in the Depthwise convolution. This structure can separately process the correlation between channels and the spatial correlation, and can greatly reduce the number of model parameters and the amount of computation without affecting the effect. On this basis, a hierarchical structure is proposed and applied to the network, enabling the network to automatically learn this part of information. For the dietary characteristics of Chinese people, each kind of food can be hierarchically structured according to its ingredients. Each hierarchical category includes at least two levels. The first hierarchical category includes at least the first hierarchical category for indicating ingredient information, and the second hierarchical category includes the hierarchical category for indicating cooking freshness. In the above embodiment, the training flowchart of the second model can be seen in the appendix Figure 8 , the input image can be scaled to a square of 299px (this value is only an exemplary illustration, and the present invention does not limit the pixel size of the input image. For example, it can also take values such as 260px, 350px, etc.). Apply a random rotation within 15 degrees to the image, and perform data augmentation processing on the brightness and color saturation with + / -5%. Fine-tune based on the imagenet pre-trained weights. The training process is divided into two stages. In the first stage, set batch_size = 128 and the initial learning rate LR = 1e -3 , freeze all the model parameters before the GlobalAveragePooling layer, and only train the model parameters after the GlobalAveragePooling layer for 1 epoch. In the second stage, set batch_size = 64 and the initial learning rate LR = 1e -4 , the learning rate decays to 0.5LR every 15 epochs. The Adabound (Adaptive Gradient Methodswith Dynamic Bound of Learning Rate) can be used for both processes as the optimizer.

[0048] In an exemplary embodiment, the structural branch is further configured to, when the first-level category includes at least two level categories, and the level categories included in the first-level category are sequentially distributed in descending order of food ingredient types, and the level of the identified food is the third-level category included in the first-level category and the confidence of the third-level category is lower than the first threshold, determine the previous-level category corresponding to the ingredient information included in the third-level category and the cooking information included in the second-level category as the level category of the identified food, where the fourth-level category is further included in the first-level category and the level of the fourth-level category is higher than the level of the third-level category. In this embodiment, according to the dietary characteristics of Chinese people, a hierarchical structure can be divided for each food based on its ingredients. Each food can generate 4 levels of information (this level is only an exemplary illustration, and the number of multi-levels in the present invention is not limited, for example, it can also be 2 levels, 3 levels, 5 levels, etc.), including 3 levels of ingredient information (level 1-3) and one level of cooking information (level 4). The higher the level category, the larger the granularity, and the lower the level category, the smaller the granularity. The specific level categories can be seen in Table 1. Through this hierarchical information, the prior information of the food structure (ingredient and cooking information) can be combined, and the prediction of the model can be made more accurate through supervised training, and different results can be returned according to the actual situation. If the confidence of a certain category is too low (for example, less than 0.3), the previous-level category can be returned to make the granularity coarser and the classification more accurate. For example, when the category with the highest prediction confidence is braised pork, but its confidence is 0.2, we can combine the output results of the ingredients (pork) and cooking (braised) and return the result as braised pork category. A structural branch is added to the output of GlobalAveragePooling in the original network structure. Together with the picture category branch, end-to-end supervised training is performed on the network. For the hierarchical branch, by fusing all hierarchical information into a long vector, the sigmoid activation function can be used to output the result, and the loss function can adopt binary_crossentropy (binary cross-entropy loss function). The category branch can adopt a general method and use the softmax activation function to output the result, and the loss function adopts categorical_crossentropy (multi-class cross-entropy loss function). Finally, the total loss function is L_ total = 25 * L_ binary_crossentropy + L_ categorical_crossentropy .

[0049] Table 1

[0050]

[0051] In an exemplary embodiment, before using the first model to analyze the first picture and determine the first region information of the first food included in the first picture, the method further includes: configuring the YOLOV3 model as follows to obtain the first initial model: configuring the YOLOV3 model to include a first number of Res_blocks arranged in sequence from the back to the front, each of which outputs a second number of feature layers, where the second number of feature layers includes a feature layer for determining the bounding box coordinates of the food, a feature layer for determining the bounding box confidence, and a feature layer for determining the probability that the input picture belongs to the first food category; configuring the threshold of the bounding box confidence as a second threshold, and configuring the threshold of the IOU as a third threshold, where after filtering out the bounding boxes with a confidence less than the second threshold, the bounding box with the highest confidence among the bounding boxes with an IOU greater than the third threshold is determined and retained through the non-maximum suppression (NMS) method; using multiple sets of the first data to train the first initial model through machine learning to obtain the first model. In this embodiment, the food localization model, that is, the first model, can adopt an object detection model, for example, YOLOV3 (You only look once version 3), CenterNet (Objects as Points), etc. Taking the YOLOV3 model as an example, this model uses Darknet-53 as the basic network to extract features, and the schematic diagram of its network structure can be seen in Appendix Figure 9 , and the schematic diagram of the Darknet-53 network structure can be seen in Appendix Figure 10 . This network mainly consists of 5 Res_blocks (i.e., res1, res2, res3, res4, res5 in Appendix Figure 9 , or Appendix Figure 10It is stacked by rectangular frames (in the figure), and every time a Res_block is passed through, a convolutional layer with a stride of 2 is used for downsampling once, making the length and width of the feature map become half of the original, and the number of channels become twice the original. Each Res_block is stacked by different numbers of DBL (Darknetconv2d_BN_Leaky) and Res_unit, forming an efficient and stable feature extraction structure. Among them, DBL can include a conv convolutional structure, batch_normalization (batch normalization), and leaky_relu, and Res_unit can include two DBL and a residual structure. In order to increase the perception ability of foods of different sizes, finally, three branches can be formed by 3 Res_blocks (the 3 Blocks are only an exemplary description, and the present invention does not limit the number of Blocks, and the number of Blocks can be set according to actual applications. For example, it can also be 1, 4, 6, etc.). Nine prior boxes: (10x13), (16x30), (33x23), (30x61), (62x45), (59x119), (116x90), (156x198), (373x326) are used to predict the food position. Among them, the nine prior boxes can be respectively applied to the feature maps of the last 3 Blocks. Each feature point corresponding to each scale will generate 3 prior boxes. Different from the coco dataset, using the first model can only distinguish foods and the background. Therefore, for each branch, the number of feature layers finally output can be 4 + 1 + 1 = 6, that is, each branch can obtain six values: 4 coordinate data, 1 bounding box confidence, and the probability of the object category. Among them, the 4 coordinate data can be the bounding box coordinates (4 values). The bounding box confidence (1 value), the probability of the object category (food OR non-food, 0.5 - 1). The bounding box coordinates can be determined by the feature layer, that is, a feature layer can determine a vertex coordinate. The bounding box can be a conventional rectangular box, or a circular box, an oval box, or other shaped boxes and irregular graphic boxes. For the Res_block4 branch, it is necessary to upsample and fuse the intermediate result of the Res_block5 branch. Similarly, the Res_block3 branch needs to be fused with the feature map after the upsampling of the fused Res_block4 and Res_block5. For details, see the two concat structures in the appendix Figure 9 in the appendix.

[0052] In the above embodiments, the threshold for predicting the confidence of the bounding box, i.e., the second threshold, can be set to 0.3, and the IOU (Intersection over Union) threshold, i.e., the third threshold, can be set to 0.45. After filtering all the prediction boxes with a confidence less than 0.3, for the prediction boxes with an IOU greater than 0.45 by using the NMS (non maximum suppression) method, the one with the highest confidence is retained.

[0053] In the above embodiments, the training flowchart of the first model can be found in the appendix Figure 11 , when generating data with a batch_size (e.g., 16, 32, 64, 128) each time, that is, when generating the number of samples selected for each training (the number of samples can be obtained by performing subsequent processing on an image), after scaling the image to a square size of 416px, a rotation within a random range of 5 degrees can be applied to the image, and data augmentation processing with a +5% to -5% change in brightness and color saturation can be performed. Among them, rotation and brightness / color enhancement are two data augmentation methods, and generally both can be used simultaneously to increase the robustness of the model. The processed image is input into the convolutional layer. During the training process, fine-tuning can be performed based on the pre-trained weights of coco, with batch_size = 64 and the initial learning rate LR = 1e -4 , the learning rate decays to 0.5LR every 8 epochs (a complete dataset passing through the neural network once and returning once is called an epoch), and the optimizer can use Adam (adaptive moment estimation) for 30 epochs of optimization iteration. The final predicted output includes the rectangular box coordinates of the food and the food position information of the length and width. It should be noted that 5 degrees and + / - 5% are continuous variable values, indicating a change within this range, not fixed values.

[0054] It should be noted that the above feature layer for determining the probability that the input image is of the first food category can identify the food to be recognized as a certain type of food and determine the confidence (probability) of determining the food to be recognized as this type of food. In a more preferred embodiment, this feature layer can identify the food to be recognized as multiple different types of food, respectively determine the confidence of determining it as different types of food, and determine the food category with the highest confidence (maximum probability) and its confidence (probability) as the finally recognized food category and its probability.

[0055] In an exemplary embodiment, before analyzing the second picture using the third model to determine the first weight value of the first food contained in the second picture, the method further includes: obtaining the third initial model constructed in the following manner: configuring a regression branch and a classification branch on the output side of the GlobalAveragePooling of the Xception model to construct the third initial model; training the third initial model using multiple sets of the third data through machine learning to obtain the third model. In this embodiment, the third model can be based on the original Xception model. Similar to the second model, after the GlobalAveragePooling layer, there are two branches, a regression branch and a classification branch. For the regression branch, the number of neurons in the last fully connected layer is 1, and the relu activation function is used to ensure that the output is positive, and the loss function uses MSE (mean square error); for the classification branch, the number of neurons in the last fully connected layer is 100, and the softmax activation function is used, and the loss function uses categorical_crossentropy. Finally, the total loss function is L_ total = 2 * L_ MSE + L_ categorical_crossentropy .

[0056] In the above embodiment, the training flowchart of the third model can be seen in Appendix Figure 12 , scale the input picture to a square with a size of 299px, and apply a random rotation within 15 degrees to the image. We fine-tune based on the training weights of the food classification model. The training process is divided into two stages. In the first stage, set batch_size = 128 and the initial learning rate LR = 1e -3 , freeze all model parameters in front of the GlobalAveragePooling layer, and only train the model parameters behind the GlobalAveragePooling layer for 1 epoch; in the second stage, set batch_size = 64 and the initial learning rate LR = 1e -4 , the learning rate decays to 0.5LR every 10 epochs. The optimizer used in both processes is Adabound (Adaptive Gradient Methods with Dynamic Bound of Learning Rate). The recognition effect diagram of the third model can be seen in Appendix Figure 13 , as Figure 13 shown, for each food picture, locate the food position, and give the food weight and the corresponding food category name.

[0057] In the above embodiment, the flowchart of food recognition and food weight detection can be seen in Appendix Figure 14, the food image to be evaluated is input into the model, and the following can be output: a. the position of the food; b. the category of the food; c. the weight of the food. Through the above information, by looking up the food nutrition database, the nutritional elements contained in the food can be further calculated, thereby realizing diet entry and nutrition calculation to guide users to control their nutritional intake. Among them, the input food picture can be a picture containing one food or a picture containing multiple foods. After the food positioning module (corresponding to the above first model) determines the position of the food, if the food is detected, the food is cropped according to the obtained food frame, and the cropped pictures are respectively input into the food classification model (corresponding to the above second model) and the food weight estimation model (corresponding to the above third model) for prediction results. Finally, the food position box, food name, and weight value in the picture are obtained.

[0058] In an exemplary embodiment, before training the third initial model with multiple sets of the third data through machine learning to obtain the third model, the method further includes: obtaining pictures of the same food taken under different shooting conditions and the weight value of the food to obtain the third data; where the different shooting conditions include at least one of the following: different table backgrounds for placing the food, lighting conditions with different brightness, different shooting angles, different tableware for holding the food, and different weights of the same food. In this embodiment, the shooting conditions may include table backgrounds of different colors (such as black, white, brown, etc.), lighting conditions with different brightness, different angles but the whole food needs to be captured (for example, the shooting angle is between 30 degrees and 90 degrees relative to the tabletop), different tableware (such as plates, bowls, glass bowls, trays, etc.), and different masses (such as between 50g and 1100g).

[0059] In an exemplary embodiment, the method further includes: determining the nutritional value of the first food from the food nutrition database based on the first food name and the first weight value; feeding back the nutritional value of the first food to the target terminal. In this embodiment, according to the first food name and the first weight value, the nutritional value of the first food can be determined from the food nutrition database, and the nutritional value of the first food is fed back to the target terminal. Among them, the target terminal can be a smart terminal with a camera function such as a mobile phone, a tablet computer, a smart bracelet, etc. The user can know the nutritional value of the food by taking a picture of the food through the target terminal. Among them, the nutritional value may include the content of trace elements, calories, etc.

[0060] In an exemplary embodiment, after determining the nutritional value of the first food from the food nutrition database based on the first food name and the first weight value, the method further includes: when it is determined that the deviation between the nutritional value of the first food and the expected nutritional value of a pre-configured target object is greater than a fourth threshold, formulating a diet plan, where the diet plan includes adjusting the weight value of the first food and / or replacing the first food. In this embodiment, when it is determined that the deviation between the nutritional value of the first food and the expected nutritional value of a pre-configured target object is greater than the fourth threshold, a diet plan can be specified for the user. For example, the weight value of the first food can be adjusted and / or the first food can be replaced. Among them, the target object can be a person, an animal, etc. The fourth threshold can be set to different values according to the user, and the present invention does not limit this threshold, and this threshold can be customized through an Application (APP).

[0061] In the foregoing embodiment, a food localization model based on three scenarios: plates and bowls, single plate with multiple dishes (such as Chinese fast food), and mixed food (without clear boundaries) is proposed, which can determine the position frames of different foods in a picture for further recognition, achieving the purpose of recognizing multiple foods; based on a large-scale and multi-category food dataset mainly consisting of Chinese cuisine, through the analysis of existing user records, on the premise of 600 kinds of foods, it can try its best to meet the needs of more users. Therefore, the finally selected food types can cover Chinese foods, fruits, and Western foods. Among them, Chinese foods cover common foods in various regions of China, and an innovative food hierarchy structure is designed to improve the recognition accuracy. In addition, a unique method for estimating the weight based on 2D pictures is adopted. In the absence of a reference object, a single food picture content is used to implement an end-to-end weight prediction model. By combining three models: a food localization model, a food classification model, and a food weight estimation model, food nutrition calculation is combined to form a more practical and complete function. At the same time, the food recognition accuracy and the food nutrition prediction accuracy are optimized, so as to better serve the actual nutrition management scenario. The problem that in the food recognition scenario, only a single food can be recognized, the category recognition effect is poor, and it is difficult to estimate the amount of food is solved.

[0062] In the foregoing embodiment, relying on a Chinese food hierarchy structure that combines ingredients and cooking methods, based on this structure, data collection, cleaning, and model training are optimized, so as to obtain a better food database and model recognition results. Through artificial intelligence methods, based on the food pictures taken by the mobile phone, in the absence of a reference object, the weight of the food can be estimated through a single picture; multiple foods in a picture can be detected at the same time, and with the assistance of the food hierarchy structure, the food type can be recognized more accurately. And by comparing the food nutrition database, simple and objective food and nutrition records are made.

[0063] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0064] In this embodiment, a food recognition device is further provided. This device is used to implement the above embodiments and preferred embodiments, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0065] Figure 15 is a structural block diagram of a food recognition device according to an embodiment of the present invention. As Figure 15 shown, the device includes:

[0066] A first acquisition module 1502, configured to acquire a first picture including a first food;

[0067] A first analysis module 1504, configured to analyze the first picture using a first model to determine first region information of the first food included in the first picture. Among them, the first model is obtained by training a first initial model through machine learning using multiple groups of first data. Each group of first data in the multiple groups of first data includes: a picture and region information of the food included in the picture;

[0068] A cropping module 1506, configured to crop an image of the first food in the first picture based on the first region information, and use the cropped image as a second picture;

[0069] A second analysis module 1508, configured to analyze the second picture using a second model to determine a first food name of the first food included in the second picture. Among them, the second model is obtained by training a second initial model through machine learning using multiple groups of second data. Each group of second data in the multiple groups of second data includes: a picture and the food name of the food included in the picture. The food name includes ingredient information and cooking information for indicating the way of cooking the ingredients;

[0070] The first output module 1510 is configured to output the first region information and the first food name.

[0071] In the above embodiment, the first model may be a food localization model, and the second model may be a food classification model. Among them, both the first model and the second model are obtained through machine learning training. The pictures included in multiple groups of first data and multiple groups of second data can be web picture crawls. For example, pictures searched through browsers, pictures loaded in apps for providing food-making methods, pictures loaded in meal delivery platforms, etc.; they can also be food pictures taken manually; they can also be pictures in some publicly available datasets, such as food-101, UEC FOOD-256, etc. The region information included in the pictures in multiple groups of first data can be manually labeled. The training data of the first model can include multiple (for example, 7936, this value is only an exemplary illustration, the quantity of the first data in the present invention is not limited, and it can also take values such as 7500, 10000, etc.) multi-food pictures with border annotation information (center point coordinates, border length and width). Among them, a schematic diagram of multi-food pictures with border annotation information can be seen in the appendix Figure 3 . The training data of the second model can include 408383 (this value is only an exemplary illustration, the present invention does not limit the quantity of the training data of the second model) single-food (only including one kind of food) pictures with category (for example, 600 categories, this value is only an exemplary illustration, the present invention does not limit the quantity of this category, and it can also be 580 categories, 650 categories, etc.) annotation. Among them, in a more preferred embodiment, the annotation information can be annotated by more than two annotators to further ensure accuracy.

[0072] Through the present invention, after obtaining the first picture including the first food, the first model is used to analyze the first picture to determine the first region information of the first food included in the first picture. According to the first region information, the image of the first food is cropped and determined as the second picture. The second model is used to analyze the second picture to determine the first food name of the first food included in the second picture, and the first region information and the first food name are output. By using the model obtained through machine learning to analyze the first picture and identify the food name, therefore, the problem of low food recognition accuracy existing in the related art can be solved, the effect of accurately recognizing food can be achieved, and the accuracy of food recognition is improved.

[0073] It should be noted that in single food recognition, the first food included in the first picture is only one kind of food, and the first food corresponds to a first region information and a food name; in multi-food recognition, the first food included in the first picture is multiple kinds of food. Therefore, the first food corresponds to multiple first region information (one kind of food corresponds to one kind of region information) and multiple food names (one kind of food corresponds to one food name).

[0074] In an exemplary embodiment, the apparatus further includes: a third analysis module configured to, after cropping the image of the first food in the first picture based on the first region information and using the cropped image as the second picture, analyze the second picture using a third model to determine a first weight value of the first food included in the second picture; wherein, the third model is obtained by training a third initial model through machine learning using multiple groups of third data, and each group of the multiple groups of third data includes: a picture and a weight value of the food included in the picture; a second output module, configured to output the first weight value. In this embodiment, the third model may be a food weight estimation model, and the training data of the third model may include more than 100,000 single-food pictures with weight and category label information. Different from the food classification data set (corresponding to the training data of the above-mentioned second model), this part of the data may all be generated by manual shooting. According to the first level of the food hierarchy (for example, 9 major categories, this value is an exemplary illustration, and the present invention does not limit the number of categories in the first level, and it may also be divided into 8 major categories, 12 major categories, etc.), according to the category number ratio of different branches, 100 different typical foods are selected (this value is only an exemplary illustration, and the present invention does not limit the number of typical foods, for example, 80 or 120, etc.). Each kind of food can be photographed in different environments respectively. That is, the pictures in the third data may be photographed in a variety of different shooting environments, and more than 100 pictures can be generated for one category of food. It should be noted that through the above-mentioned data collection method, except for the need to reweigh when changing the quality, other transformation methods (such as taking pictures of the same food with the same quality at different angles, different backgrounds, etc.) no longer require weighing, greatly reducing the data collection difficulty. In the above embodiment, the data division module can be used to divide all the data sets (that is, multiple groups of first data, multiple groups of second data, multiple groups of third data) into three parts: a training set, a validation set, and a test set. The quantity ratio of the pictures in the three parts can be 3:1:1. The training set data is used to train the model, the validation set is used to screen the model, and the test set is used to evaluate the result of the model. The model training flow chart can be seen in the appendix Figure 4 。

[0075] In an exemplary embodiment, the device further includes: a second acquisition module, configured to, before determining a first food name of the first food included in the second picture by analyzing the second picture using a second model, acquire the second initial model constructed by the following units: a first construction unit, configured to add a structural branch to an output of a GlobalAveragePooling of an Xception model to construct the second initial model, where the structural branch is configured to identify hierarchical categories of foods included in an input picture, the hierarchical categories include food types hierarchically divided in a descending order, the hierarchical categories at least include a first hierarchical category and a second hierarchical category, the first hierarchical category at least includes a hierarchical category for indicating ingredient information, the second hierarchical category includes a hierarchical category for indicating cooking information, a rank of the first hierarchical category is higher than a rank of the second hierarchical category, and food types in a hierarchical category with a higher rank include food types in a hierarchical category with a lower rank; a first training module, configured to train the second initial model using multiple groups of the second data through machine learning to obtain the second model. In this embodiment, the first model may adopt Xcepiton (Deep Learning with DepthwiseSeparable Convolutions). A schematic diagram of Xception can be seen in Appendix Figure 5 , and this network model mainly uses DepthwiseSeparable Convolution to replace ordinary convolution for feature extraction of pictures. Among them, a schematic diagram of convolution can be seen in Appendix Figure 6 , and a schematic diagram of Depthwise Separable Convolution can be seen in Appendix Figure 7Specifically, Depthwise Separable Convolution has two steps. The first step is Depthwise convolution, where for each channel of the input, a K×K convolution operation is performed separately (K represents the size of the convolution kernel, and K can take values such as 3, 5, 7, 9, 11, etc.), and the result is input into concat (a function for concatenating multiple arrays). The second step is Pointwise convolution, where a 1×1 convolution operation is performed on the concat result in the Depthwise convolution. This structure can separately process the correlation between channels and the spatial correlation, and can greatly reduce the number of model parameters and the amount of computation without affecting the effect. On this basis, a hierarchical structure is proposed and applied to the network to enable the network to automatically learn this part of the information. For the dietary characteristics of Chinese people, each food can be hierarchically structured according to its ingredients. Each hierarchical category includes at least two levels. The first hierarchical category includes at least the first hierarchical category for indicating ingredient information, and the second hierarchical category includes the hierarchical category for indicating cooking freshness.

[0076] In the above embodiment, the training flowchart of the second model can be seen in the appendix Figure 8 , the input image can be scaled to a square of 299px (this value is only an exemplary illustration, and the present invention does not limit the pixel size of the input image. For example, 260px, 350px, etc. can also be taken). The image is rotated randomly within 15 degrees, and data augmentation processing of + / -5% is performed on the brightness and color saturation. Fine-tuning is performed based on the imagenet pre-trained weights. The training process is divided into two stages. In the first stage, batch_size = 128 and the initial learning rate LR = 1e -3 , all model parameters before the GlobalAveragePooling layer are frozen, and only the model parameters after the GlobalAveragePooling layer are trained for 1 epoch. In the second stage, batch_size = 64 and the initial learning rate LR = 1e -4 , the learning rate decays to 0.5LR every 15 epochs. In both processes, the optimizer can use Adabound (Adaptive Gradient Methods with Dynamic Bound of Learning Rate).

[0077] In an exemplary embodiment, the structural branch is further configured to, when the first-level category includes at least two level categories, and the level categories included in the first-level category are sequentially distributed in descending order of food ingredient types, and the identified food is at the third-level category included in the first-level category, and the confidence of the third-level category is lower than the first threshold, determine the previous-level category corresponding to the ingredient information included in the third-level category and the cooking information included in the second-level category as the level category of the identified food, where the fourth-level category is further included in the first-level category, and the rank of the fourth-level category is higher than the rank of the third-level category. In this embodiment, according to the dietary characteristics of Chinese people, a hierarchical structure can be divided for each kind of food from the food ingredients. Each kind of food can generate at most 4 (this level is only an exemplary illustration, and the present invention does not limit the number of levels. For example, it can also be 2 levels, 3 levels, 5 levels, etc.) levels of information, including 3 levels of ingredient information (level 1 to 3) and one level of cooking information (level 4). The higher the level category, the larger the granularity, and the lower the level category, the smaller the granularity. For the specific level categories, see Table 1. Through this hierarchical information, the prior information (ingredient and cooking information) of the food structure can be combined, and the prediction of the model can be made more accurate through supervised training, and different results can be returned according to the actual situation. If the confidence of a certain category is too low (for example, less than 0.3), the previous-level category can be returned to make the granularity coarser and the classification more accurate. For example, when the category with the highest prediction result confidence is braised pork, but its confidence is 0.2, we can combine the output results of the ingredients (pork) and cooking (braised) and return the result as the braised pork category. A structural branch is added to the output of GlobalAveragePooling in the original network structure. Together with the picture category branch, end-to-end supervised training of the network is performed. For the hierarchical branch, by fusing all hierarchical information into a long vector, the sigmoid activation function can be used to output the result, and the binary_crossentropy (binary cross-entropy loss function) can be used as the loss function. The category branch can adopt a general method, use the softmax activation function to output the result, and the categorical_crossentropy (multi-class cross-entropy loss function) is used as the loss function. Finally, the total loss function is L_ total = 25 * L_ binary_crossentropy + L_ categorical_crossentropy .

[0078] In an exemplary embodiment, the apparatus further includes: a configuration module, configured to, before using a first model to analyze the first picture and determine first region information of the first food included in the first picture, configure the YOLOV3 model by means of the following units to obtain the first initial model: a configuration unit, configured to configure the YOLOV3 model to include a first number of Res_blocks arranged in sequence from back to front, each outputting a second number of feature layers, where the second number of feature layers includes a feature layer for determining the bounding box coordinates of food, a feature layer for determining the bounding box confidence, and a feature layer for determining the probability that the input picture is of the first food category; configure the threshold of the bounding box confidence to be a second threshold, and configure the threshold of the IOU to be a third threshold, where, after filtering out the bounding boxes with a confidence less than the second threshold, the bounding box with the highest confidence among the bounding boxes with an IOU greater than the third threshold is determined and retained by means of non-maximum suppression (NMS); a second training unit, configured to train the first initial model by means of machine learning using multiple sets of the first data to obtain the first model. In this embodiment, the food localization model, i.e., the first model, may adopt an object detection model, for example, YOLOV3 (You only look once version 3), CenterNet (Objects as Points), etc. Taking the YOLOV3 model as an example, this model uses Darknet-53 as the basic network to extract features, and the schematic diagram of its network structure can be seen in Appendix Figure 9 , and the schematic diagram of the Darknet-53 network structure can be seen in Appendix Figure 10 . This network mainly consists of 5 Res_blocks (i.e., res1, res2, res3, res4, res5 in Appendix Figure 9 , or Appendix Figure 10It is stacked by rectangular frames (in the figure), and every time a Res_block is passed through, a convolutional layer with a stride of 2 is used for downsampling once, making the length and width of the feature map become half of the original, and the number of channels become twice the original. Each Res_block is stacked by different numbers of DBL (Darknetconv2d_BN_Leaky) and Res_unit to form an efficient and stable feature extraction structure. Among them, DBL can include a conv convolutional structure, batch_normalization (batch normalization), and leaky_relu, and Res_unit can include two DBLs and a residual structure. In order to increase the perception ability for foods of different sizes, finally, three branches can be formed by 3 Res_blocks (the 3 Blocks are only an exemplary description, and the present invention does not limit the number of Blocks, and the number of Blocks can be set according to actual applications. For example, it can also be 1, 4, 6, etc.). Nine prior boxes: (10x13), (16x30), (33x23), (30x61), (62x45), (59x119), (116x90), (156x198), (373x326) are used to predict the food position. Among them, the nine prior boxes can be respectively applied to the feature maps of the last 3 Blocks. Each feature point corresponding to each scale will generate 3 prior boxes. Different from the coco dataset, using the first model can only distinguish foods and the background. Therefore, the number of feature layers finally output for each branch can be 4 + 1 + 1 = 6, that is, each branch can obtain six values: 4 coordinate data, 1 bounding box confidence, and the probability of the object category. Among them, the 4 coordinate data can be the bounding box coordinates (4 values). The bounding box confidence (1 value), the probability of the object category (food OR non-food, 0.5 - 1). The bounding box coordinates can be determined by the feature layer, that is, a feature layer can determine a vertex coordinate. The bounding box can be a conventional rectangular box, or a circular box, an oval box, or other shaped boxes and irregular graphic boxes. For the Res_block4 branch, it is necessary to upsample and fuse the intermediate result of the Res_block5 branch. Similarly, the Res_block3 branch needs to be fused with the feature map after the upsampling of the fused Res_block4 and Res_block5. For details, please refer to the two concat structures in the appendix Figure 9 in the appendix.

[0079] In the above embodiments, the threshold for predicting the confidence of the bounding box, i.e., the second threshold, can be set to 0.3, and the IOU (Intersection over Union) threshold, i.e., the third threshold, can be set to 0.45. After filtering all the prediction boxes with a confidence less than 0.3, for the prediction boxes with an IOU greater than 0.45 through NMS (non maximum suppression), the one with the highest confidence is retained.

[0080] In the above embodiments, the training flowchart of the first model can be seen in the appendix Figure 11 , when generating data with a batch_size (e.g., 16, 32, 64, 128) each time, that is, when generating the number of samples selected for each training (the number of samples can be obtained by performing subsequent processing on an image), after scaling the image to a square size of 416px, a random rotation within 5 degrees can be applied to the image, and data augmentation processing with a +5% to -5% change in brightness and color saturation can be performed. Among them, rotation and brightness and color enhancement processing are two data augmentation methods, and generally both can be used simultaneously to increase the robustness of the model. The processed image is input into the convolutional layer. During the training process, fine-tuning can be performed based on the pre-trained weights of coco, setting batch_size = 64, and the initial learning rate LR = 1e -4 , the learning rate decays to 0.5LR every 8 epochs (a complete dataset passing through a neural network once and returning once is called an epoch), and the optimizer can use Adam (adaptive moment estimation) to optimize and iterate for 30 epochs. The final predicted output includes the rectangular box coordinates of the food and the food position information of the length and width. It should be noted that 5 degrees and + / - 5% are continuous variable values, which means changing within this range, not fixed values.

[0081] It should be noted that the above feature layer for determining the probability that the input image is of the first food category can identify the food to be recognized as a certain type of food and determine the confidence (probability) of determining the food to be recognized as this type of food. In a more preferred embodiment, this feature layer can identify the food to be recognized as multiple different types of food, respectively determine the confidence of determining it as different types of food, and determine the food category with the highest confidence (maximum probability) and its confidence (probability) as the final recognized food category and its probability.

[0082] In an exemplary embodiment, the device further includes a third acquisition module, configured to acquire the third initial model constructed by the following units before using a third model to analyze the second picture to determine a first weight value of the first food included in the second picture: a second replacement unit, configured to configure a regression branch and a classification branch on an output side of GlobalAveragePooling of an Xception model to construct the third initial model; a third training unit, configured to train the third initial model through machine learning using multiple groups of the third data to obtain the third model. In this embodiment, the third model may be based on the original Xception model. Similar to the second model, after the GlobalAveragePooling layer, there are two branches, a regression branch and a classification branch. For the regression branch, the number of neurons in the last fully connected layer is 1, and the relu activation function is used to ensure that the output is positive. The loss function uses MSE (mean squared error); for the classification branch, the number of neurons in the last fully connected layer is 100, and the softmax activation function is used. The loss function uses categorical_crossentropy. Finally, the total loss function is L_ total = 2 * L_ MSE + L_ categorical_crossentropy .

[0083] In the above embodiment, for the training flowchart of the third model, please refer to Appendix Figure 12 , scale the input picture to a square with a size of 299px, and apply a random rotation within 15 degrees to the image. We fine-tune based on the training weights of the food classification model. The training process is divided into two stages. In the first stage, set batch_size = 128, initial learning rate LR = 1e -3 , freeze all model parameters in front of the GlobalAveragePooling layer, and only train the model parameters behind the GlobalAveragePooling layer for 1 epoch; in the second stage, set batch_size = 64, initial learning rate LR = 1e -4 , the learning rate decays to 0.5LR every 10 epochs. The optimizer used in both processes is Adabound (Adaptive Gradient Methods with Dynamic Bound of Learning Rate). For the recognition effect diagram of the third model, please refer to Appendix Figure 13 , as Figure 13 shown, for each food picture, locate the food position, and give the food weight and the corresponding food category name.

[0084] In the above embodiments, the flowchart of food recognition and food weight detection can be found in the appendix Figure 14 When the food image to be evaluated is input into the model, the following can be output: a. the position of the food; b. the category of the food; c. the weight of the food. Based on the above information, by searching the food nutrition database, the nutritional elements contained in the food can be further calculated, thereby realizing diet entry and nutrition calculation to guide users to control their nutritional intake. Among them, the input food picture can be a picture containing one food or a picture containing multiple foods. After the position of the food is determined by the food positioning module (corresponding to the above first model), if the food is detected, the food is cropped according to the obtained food frame, and the cropped pictures are respectively input into the food classification model (corresponding to the above second model) and the food weight estimation model (corresponding to the above third model) for prediction results. Finally, the food position box, food name, and weight value in the picture are obtained.

[0085] In an exemplary embodiment, the device further includes: a fourth acquisition module, configured to obtain pictures of the same food taken under different shooting conditions and the weight value of the food before training the third initial model with multiple groups of the third data through machine learning to obtain the third model, so as to obtain the third data; wherein, the different shooting conditions include at least one of the following: different table backgrounds for placing the food, illumination conditions with different brightness, different shooting angles, different tableware for holding the food, and different weights of the same food. In this embodiment, the shooting conditions may include table backgrounds of different colors (such as black, white, brown, etc.), illumination conditions with different brightness, different angles but the whole food needs to be captured (for example, the shooting angle is between 30 degrees and 90 degrees relative to the tabletop), different tableware (such as plates, bowls, glass bowls, trays, etc.), and different masses (such as between 50g and 1100g).

[0086] In an exemplary embodiment, the device further includes: a determination module, configured to determine the nutritional value of the first food from the food nutrition database based on the first food name and the first weight value; a feedback module, configured to feedback the nutritional value of the first food to the target terminal. In this embodiment, the nutritional value of the first food can be determined from the food nutrition database according to the first food name and the first weight value, and the nutritional value of the first food is feedback to the target terminal. Among them, the target terminal can be a smart terminal with a camera function such as a mobile phone, a tablet computer, a smart bracelet, etc. The user can know the nutritional value of the food by taking a picture of the food with the target terminal. Among them, the nutritional value may include the content of trace elements, calories, etc.

[0087] In an exemplary embodiment, the apparatus further includes: a formulation module configured to formulate a diet plan when it is determined that the deviation between the nutritional value of the first food determined from the food nutrition database based on the first food name and the first weight value and the expected nutritional value of a pre-configured target object is greater than a fourth threshold. The diet plan includes adjusting the weight value of the first food and / or replacing the first food. In this embodiment, when it is determined that the deviation between the nutritional value of the first food and the expected nutritional value of the pre-configured target object is greater than the fourth threshold, a diet plan can be specified for the user. For example, the weight value of the first food can be adjusted and / or the first food can be replaced. The target object can be a person, an animal, etc. The fourth threshold can be set to different values according to the user, and the present invention does not limit this threshold. Moreover, this threshold can be customized through an Application (APP).

[0088] It should be noted that the above-mentioned respective modules can be implemented by software or hardware. For the latter, it can be achieved in the following ways, but not limited thereto: all the above modules are located in the same processor; or, the above-mentioned respective modules are separately located in different processors in any combination form.

[0089] An embodiment of the present invention further provides a computer-readable storage medium in which a computer program is stored. The computer program is configured to execute the steps in any one of the above method embodiments when running.

[0090] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disc that can store a computer program.

[0091] An embodiment of the present invention further provides an electronic device including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0092] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device. The transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0093] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be elaborated herein.

[0094] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a sequence different from that here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0095] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A food recognition method, characterized in that, Including: Obtaining a first picture including a first food; Analyzing the first picture using a first model to determine first region information of the first food included in the first picture, wherein the first model is obtained by training a first initial model through machine learning using multiple groups of first data, and each group of the multiple groups of first data includes: a picture and region information of the food included in the picture; Cropping an image of the first food in the first picture based on the first region information, and using the cropped image as a second picture; analyzing the second picture using a second model to determine a first food name of the first food included in the second picture, wherein the second model is obtained by training a second initial model through machine learning using multiple groups of second data, and each group of the multiple groups of second data includes: a picture and a food name of the food included in the picture, and the food name includes ingredient information and cooking information for indicating a way of cooking the ingredient; Outputting the first region information and the first food name; The method further includes: adding a structural branch to the output of the GlobalAveragePooling of the Xception model to construct the second initial model, the structural branch being used to identify a hierarchical category of the food included in the input picture, the hierarchical category including food types hierarchically divided in descending order, the hierarchical category at least including a first hierarchical category and a second hierarchical category, the first hierarchical category at least including a hierarchical category for indicating ingredient information, the second hierarchical category including a hierarchical category for indicating cooking information, the rank of the first hierarchical category being higher than that of the second hierarchical category, and the food types in the hierarchical category with a higher rank including the food types in the hierarchical category with a lower rank; training the second initial model through machine learning using multiple groups of the second data to obtain the second model; the structural branch is further used to, when the first hierarchical category includes at least two hierarchical categories, the hierarchical categories included in the first hierarchical category are sequentially distributed in descending order of ingredient types, the identified food is in a third hierarchical category included in the first hierarchical category, and the confidence of the third hierarchical category is lower than a first threshold, determine the hierarchical category of the ingredient information corresponding to the previous hierarchical category included in the third hierarchical category and the cooking information included in the second hierarchical category as the hierarchical category of the identified food, wherein the first hierarchical category further includes a fourth hierarchical category, and the rank of the fourth hierarchical category is higher than that of the third hierarchical category.

2. The method according to claim 1, wherein After cropping an image of the first food in the first picture based on the first region information and using the cropped image as a second picture, the method further includes: Analyze the second picture using a third model to determine the first weight value of the first food contained in the second picture; wherein, the third model is obtained by training a third initial model through machine learning using multiple groups of third data, and each group of third data in the multiple groups of third data includes: a picture and the weight value of the food included in the picture; Output the first weight value.

3. The method according to claim 1, wherein Before using the first model to analyze the first picture to determine the first region information of the first food contained in the first picture, the method further includes: Configure the YOLOV3 model as follows to obtain the first initial model: Configure the YOLOV3 model to include a first number of Res_block arranged in sequence from back to front, each outputting a second number of feature layers, where the second number of feature layers includes a feature layer for determining the bounding box coordinates of the food, a feature layer for determining the bounding box confidence, and a feature layer for determining the probability that the input picture is of the first food category; configure the threshold of the bounding box confidence as a second threshold, and configure the threshold of the IOU as a third threshold, where after filtering out the bounding boxes with a confidence less than the second threshold, the bounding box with the highest confidence among the bounding boxes with an IOU greater than the third threshold is determined and retained through the non-maximum suppression (NMS) method; Train the first initial model through machine learning using multiple groups of the first data to obtain the first model.

4. The method according to claim 2, characterized in that, Before using the third model to analyze the second picture to determine the first weight value of the first food contained in the second picture, the method further includes: Obtain the third initial model constructed in the following manner: configure a regression branch and a classification branch on the output side of the GlobalAveragePooling of the Xception model to construct the third initial model; Train the third initial model through machine learning using multiple groups of the third data to obtain the third model.

5. The method according to claim 4, characterized in that, Before training the third initial model through machine learning using multiple groups of the third data to obtain the third model, the method further includes: Obtain pictures taken of the same food under different shooting conditions and the weight values of the food to obtain the third data; Wherein, the different shooting conditions include at least one of the following: different table backgrounds for placing the food, different lighting conditions of different brightness, different shooting angles, different tableware for serving the food, and different weights of the same food.

6. The method according to claim 2, wherein The method further includes: Determine the nutritional value of the first food from the food nutrition database based on the first food name and the first weight value; Feed back the nutritional value of the first food to the target terminal.

7. The method according to claim 6, wherein After determining the nutritional value of the first food from the food nutrition database based on the first food name and the first weight value, the method further includes: When it is determined that the deviation between the nutritional value of the first food and the expected nutritional value of the pre-configured target object is greater than the fourth threshold, a diet plan is formulated, where the diet plan includes adjusting the weight value of the first food and / or replacing the first food.

8. A food recognition device, characterized in that, Including: A first acquisition module, configured to acquire a first picture including a first food; A first analysis module, configured to analyze the first picture using a first model to determine first region information of the first food included in the first picture, where the first model is obtained by training a first initial model using machine learning with multiple groups of first data, and each group of the multiple groups of first data includes: a picture and region information of the food included in the picture; A cropping module, configured to crop an image of the first food in the first picture based on the first region information and use the cropped image as a second picture; A second analysis module, configured to analyze the second picture using a second model to determine a first food name of the first food included in the second picture, where the second model is obtained by training a second initial model using machine learning with multiple groups of second data, and each group of the multiple groups of second data includes: a picture and a food name of the food included in the picture, and the food name includes ingredient information and cooking information for indicating the way of cooking the ingredient; A first output module, configured to output the first region information and the first food name; The device is further configured to: add a structural branch to the output of the GlobalAveragePooling of the Xception model to construct the second initial model, where the structural branch is used to identify the hierarchical categories of the food included in the input picture, and the hierarchical categories include food types hierarchically divided in descending order, and the hierarchical categories at least include a first hierarchical category and a second hierarchical category. The first hierarchical category at least includes a hierarchical category for indicating ingredient information, and the second hierarchical category includes a hierarchical category for indicating cooking information. The level of the first hierarchical category is higher than that of the second hierarchical category, and the food types in the hierarchical category with a higher level include the food types in the hierarchical category with a lower level; use multiple sets of the second data to train the second initial model through machine learning to obtain the second model; when the structural branch includes at least two hierarchical categories in the first hierarchical category, and the hierarchical categories included in the first hierarchical category are distributed in descending order of ingredient types, and the identified food is in the third hierarchical category included in the first hierarchical category, and the confidence level of the third hierarchical category is lower than the first threshold, determine the hierarchical category of the previous level corresponding to the ingredient information included in the third hierarchical category and the cooking information included in the second hierarchical category as the hierarchical category of the identified food, where the first hierarchical category further includes a fourth hierarchical category, and the level of the fourth hierarchical category is higher than that of the third hierarchical category.

9. The device according to claim 8, wherein The device further includes: A third analysis module, configured to, after cropping the image of the first food in the first picture based on the first region information and using the cropped image as the second picture, analyze the second picture using a third model to determine a first weight value of the first food included in the second picture; where the third model is obtained by training a third initial model through machine learning using multiple sets of third data, and each set of the multiple sets of third data includes: a picture and the weight value of the food included in the picture; A second output module, configured to output the first weight value.

10. The device according to claim 8, characterized in that, The device further includes: Configuration module, before using the first model to analyze the first picture and determine the first region information of the first food contained in the first picture, facilitating the following unit to configure the YOLOV3 model to obtain the first initial model: Configuration unit, used to configure the YOLOV3 model to include a first number of Res_blocks arranged in sequence from back to front, each outputting a second number of feature layers, where the second number of feature layers includes a feature layer for determining the bounding box coordinates of the food, a feature layer for determining the bounding box confidence, and a feature layer for determining the probability that the input picture belongs to the first food category; configuring the threshold of the bounding box confidence as the second threshold, and configuring the threshold of the IOU as the third threshold, where after filtering out the bounding boxes with a confidence less than the second threshold, the bounding box with the highest confidence among the bounding boxes with an IOU greater than the third threshold is determined and retained through the non-maximum suppression (NMS) method; Second training module, used to train the first initial model through machine learning using multiple groups of the first data to obtain the first model.

11. The device according to claim 9, characterized in that, The device further includes: Third acquisition module, before using the third model to analyze the second picture and determine the first weight value of the first food contained in the second picture, acquiring the third initial model constructed by the following unit: Second construction unit, configuring a regression branch and a classification branch on the output side of the GlobalAveragePooling of the Xception model to construct the third initial model; Third training module, used to train the third initial model through machine learning using multiple groups of the third data to obtain the third model.

12. The device according to claim 11, wherein The device further includes: Fourth acquisition module, used to acquire the pictures obtained by photographing the same food under different shooting conditions and the weight values of the food to obtain the third data before training the third initial model through machine learning using multiple groups of the third data to obtain the third model; wherein, the different shooting conditions include at least one of the following: different table backgrounds for placing the food, different lighting conditions with different brightness, different shooting angles, different tableware for serving the food, and different weights of the same food.

13. The device according to claim 9, characterized in that, The device further includes: Determination module, used to determine the nutritional value of the first food from the food nutrition database based on the first food name and the first weight value; Feedback module, used to feedback the nutritional value of the first food to the target terminal.

14. The device according to claim 13, characterized in that, The device further includes: Formulation module, used to formulate a diet plan after determining the nutritional value of the first food from the food nutrition database based on the first food name and the first weight value, when the deviation between the nutritional value of the first food and the expected nutritional value of the pre-configured target object is greater than the fourth threshold, where the diet plan includes adjusting the weight value of the first food and / or replacing the first food.

15. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the method described in any one of claims 1 to 7 when running.

16. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method described in any one of claims 1 to 7.

Images