Food material recognition method and device, storage medium and electronic device

Abstract

This disclosure provides a method and apparatus for food ingredient identification, an electronic device, and a computer-readable storage medium, relating to the fields of image recognition and health technology. The method includes: receiving an image to be identified; identifying the food ingredients in the image using a pre-trained food recognition model to obtain a preliminary identification result; obtaining the feature vector of the image to be identified; determining the feature similarity between the feature vector of the image to be identified and the feature vectors of historical images, where historical images are images where food ingredients have been identified and the identification results have been corrected; obtaining similar images from the historical images based on the feature similarity; and correcting the preliminary identification result based on the correction data of the similar images to obtain the final identification result of the food ingredients in the image to be identified. This disclosure improves the accuracy of food ingredient identification by correcting the preliminary identification result of the food ingredients in the image using similar images.

Description

Technical Field

[0001] This disclosure relates to the fields of image recognition and health technology, and more specifically, to a food ingredient recognition method, a food ingredient recognition device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] The development of social production levels and the improvement of living standards have led people to pay increasing attention to dietary health. Rationally planning the energy and nutrient structure of dietary intake is an important way to ensure good health. Especially for people who want to control their weight, accurately measuring the types and weights of food consumed, combined with molecular analysis of nutrients, can prevent excessive energy intake while ensuring a balanced nutritional intake, thereby achieving the goal of maintaining or reducing weight while ensuring good health. For non-professionals, who lack the relevant knowledge, professional nutritional analysis tools are usually needed to perform the above-mentioned nutritional analysis of food.

[0003] Among related technologies, one approach involves users manually entering food information into nutritional analysis software, which then performs the nutritional analysis. This method is rather cumbersome. Another approach uses image recognition to identify ingredients in images. While this avoids the tedious manual entry process, existing methods for identifying ingredients in images suffer from low accuracy when dealing with complex food images, and require model retraining to improve accuracy.

[0004] Therefore, in order to solve the above problems, this disclosure provides a food ingredient identification method, a food ingredient identification device, an electronic device, and a computer-readable storage medium. Summary of the Invention

[0005] The purpose of this disclosure is to provide a food ingredient identification method, a food ingredient identification device, an electronic device, and a computer-readable storage medium, thereby solving to some extent the problems of cumbersome operation, low identification accuracy, and the need to retrain the model to improve accuracy in related technologies.

[0006] According to a first aspect of this disclosure, a method for identifying food ingredients is provided, comprising: receiving an image to be identified; identifying the food ingredients in the image to be identified using a pre-trained food recognition model to obtain a preliminary identification result; obtaining a feature vector of the image to be identified; determining the feature similarity between the feature vector of the image to be identified and the feature vector of a historical image, wherein the historical image is an image in which the food ingredients have been identified and the identification result has been corrected; obtaining a similar image to the image to be identified from the historical image based on the feature similarity; and, when a similar image is obtained, correcting the preliminary identification result based on the correction data of the similar image to obtain a final identification result of the food ingredients in the image to be identified.

[0007] In one exemplary embodiment of this disclosure, after obtaining the final recognition result of the food in the image to be recognized, the method further includes: displaying the final recognition result to the user; modifying the final recognition result in response to the user's modification operation; and archiving the image to be recognized as a historical image in response to the user's confirmation operation of uploading the image to be recognized.

[0008] In one exemplary embodiment of this disclosure, when no similar image is available, the method further includes: displaying the preliminary recognition result to the user; modifying the preliminary recognition result in response to the user's modification operation on the preliminary recognition result, and using the modified result as the final recognition result of the food in the image to be recognized; and archiving the image to be recognized as a historical image in response to the user's confirmation operation of uploading the image to be recognized.

[0009] In one exemplary embodiment of this disclosure, the above-mentioned archiving of the image to be identified as a historical image includes: associating the feature vector of the image to be identified with the user's modification of the identification result, wherein the identification result is the preliminary identification result or the final identification result; and archiving the image to be identified associated with the modification as a historical image.

[0010] In one exemplary embodiment of this disclosure, the modification of the preliminary identification result in response to a user's modification operation includes: adding the ingredient to the preliminary identification result in response to a user's addition operation to the preliminary identification result; deleting the ingredient from the preliminary identification result in response to a user's deletion operation to the preliminary identification result; and

[0011] The above-mentioned response to the user's modification operation on the final recognition result includes: in response to the user's addition operation on the final recognition result, adding the ingredient to the final recognition result; and in response to the user's deletion operation on the final recognition result, deleting the ingredient from the final recognition result.

[0012] In one exemplary embodiment of this disclosure, the above-mentioned correction of the preliminary recognition result based on the correction data of similar images includes: using feature similarity as correction data for adding a food ingredient in the corresponding similar images, and using the negative of feature similarity as correction data for deleting the food ingredient in the corresponding similar images; adding the correction data for adding and deleting the food ingredient in each similar image; if the sum is greater than a first preset threshold, adding the food ingredient in the preliminary recognition result of the image to be recognized; if the sum is less than a second preset threshold, deleting the food ingredient in the preliminary recognition result of the image to be recognized.

[0013] In one exemplary embodiment of this disclosure, the above-mentioned method of obtaining similar images of the image to be identified from historical images based on feature similarity includes: sorting the historical images in descending order of feature similarity; and obtaining the first preset number of historical images whose feature similarity satisfies a preset threshold condition as similar images.

[0014] According to a second aspect of this disclosure, a food ingredient recognition device is provided, comprising: a preliminary recognition module for receiving an image to be recognized, recognizing the food ingredients in the image to be recognized using a pre-trained food recognition model, and obtaining a preliminary recognition result; a feature similarity determination module for acquiring a feature vector of the image to be recognized, determining the feature similarity between the feature vector of the image to be recognized and the feature vector of a historical image, wherein the historical image is an image in which the food ingredients have been recognized and the recognition result has been corrected; a similar image acquisition module for acquiring similar images of the image to be recognized from the historical images based on the feature similarity; and a food ingredient recognition module for correcting the preliminary recognition result based on the correction data of the similar image when a similar image is acquired, thereby obtaining a final recognition result of the food ingredients in the image to be recognized.

[0015] In one exemplary embodiment of this disclosure, the food ingredient recognition device further includes a correction and archiving module, which is used to: display the final recognition result to the user; modify the final recognition result in response to the user's modification operation; and archive the image to be recognized as a historical image in response to the user's confirmation operation of uploading the image to be recognized; and

[0016] When no similar image is available, the correction and archiving module is also used to: display the preliminary recognition results to the user; modify the preliminary recognition results in response to the user's modification operation on the preliminary recognition results, and use the modified results as the final recognition results of the ingredients in the image to be recognized; and archive the image to be recognized as a historical image in response to the user's confirmation operation of uploading the image to be recognized.

[0017] In an exemplary embodiment of this disclosure, the food ingredient recognition module specifically includes a correction data determination unit and a final recognition unit, wherein: the correction data determination unit is used to use feature similarity as correction data for adding a food ingredient in corresponding similar images, and to use the negative of feature similarity as correction data for deleting the food ingredient in corresponding similar images; the final recognition unit is used to add the correction data for adding and deleting the food ingredient in each similar image; if the sum is greater than a first preset threshold, the food ingredient is added to the preliminary recognition result of the image to be recognized; if the sum is less than a second preset threshold, the food ingredient is deleted from the preliminary recognition result of the image to be recognized.

[0018] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any of the preceding methods by executing the executable instructions.

[0019] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0020] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0021] In the food identification method provided in this exemplary embodiment, an image to be identified is received, and the food in the image is identified using a pre-trained food identification model to obtain a preliminary identification result. The feature vector of the image to be identified is obtained, and the feature similarity between the feature vector of the image to be identified and the feature vectors of historical images is determined. Based on the feature similarity, similar images to the image to be identified are obtained from the historical images, where the food has already been identified and the identification result has been corrected. When a similar image is obtained, the preliminary identification result is corrected based on the correction data of the similar image to obtain the final identification result of the food in the image to be identified. On the one hand, the food identification method provided in this exemplary embodiment improves the accuracy of food identification by correcting the preliminary identification result of the food in the image to be identified using the correction data of similar images. On the other hand, it eliminates the need to retrain the food identification model, thus avoiding manual data processing, model training, deployment iterations, etc., and solving the problem of not being able to complete the model retraining process fully automatically in a short period. Furthermore, the system obtains preliminary recognition results of ingredients in the image to be recognized through a pre-trained food recognition model, and corrects the recognition results through correction data from similar images. This eliminates the need for manual input of ingredients and can be applied to scenarios that require nutritional analysis based on food images, simplifying user operations and optimizing the user experience.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1A schematic diagram of an exemplary system architecture for a food ingredient identification method and apparatus to which embodiments of the present disclosure can be applied is shown;

[0025] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure is shown;

[0026] Figure 3 The illustration shows a schematic flowchart of a food ingredient identification method according to an embodiment of the present disclosure;

[0027] Figure 4 The illustration shows a schematic diagram of an image to be identified using a food identification method according to an embodiment of the present disclosure;

[0028] Figure 5 A block diagram of a food ingredient recognition device according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0030] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] Figure 1 A schematic diagram of a system architecture for an exemplary application environment in which an ingredient identification method and apparatus according to embodiments of the present disclosure can be applied is shown.

[0032] like Figure 1As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Terminal devices 101, 102, and 103 include, but are not limited to, desktop computers, laptops, smartphones, and tablets. It should be understood that... Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0033] The food identification method provided in this embodiment can be executed by terminal devices 101, 102, and 103, and correspondingly, the food identification device can be installed in terminal devices 101, 102, and 103. Alternatively, the food identification method can be jointly executed by terminal devices 101, 102, and 103 and server 105, and correspondingly, the food identification device can be installed in terminal devices 101, 102, and 103 and server 105. Furthermore, the food identification method provided in this embodiment can also be executed by server 105, and correspondingly, the food identification device can be installed in server 105. This exemplary embodiment does not impose any special limitations on this approach.

[0034] For example, in this exemplary embodiment, the above-described food ingredient recognition method can be jointly executed by terminal devices 101, 102, and 103 and server 105. Users can take pictures of food using terminal devices 101, 102, and 103 and upload these pictures as images to be recognized to server 105. Upon receiving the image to be recognized, server 105 uses a pre-trained food recognition model to identify the food ingredients in the image, obtaining a preliminary recognition result; it acquires the feature vector of the image to be recognized, determines the feature similarity between the feature vector of the image to be recognized and the feature vector of historical images (images where food ingredients have been identified and the recognition results have been corrected); it obtains similar images from the historical images based on the feature similarity; and when similar images are obtained, it corrects the preliminary recognition result based on the correction data of the similar images, obtaining the final recognition result of the food ingredients in the image to be recognized and transmitting it to terminal devices 101, 102, and 103.

[0035] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0036] It should be noted that, Figure 2The computer system 200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0037] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 202 or programs loaded from storage section 208 into random access memory (RAM) 203. The RAM 203 also stores various programs and data required for system operation. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0038] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0039] The development of social production levels and the improvement of living standards have led people to pay increasing attention to dietary health. Rationally planning the energy and nutrient structure of dietary intake is an important way to ensure good health. By accurately measuring the types and weights of food consumed, and combining this with molecular analysis of nutrients, it is possible to prevent excessive energy intake while ensuring a balanced nutritional intake.

[0040] Among related technologies, one approach involves users manually entering food information into nutritional analysis software, which then performs the nutritional analysis. This method is rather cumbersome. Another approach uses image recognition to identify ingredients in images. While this avoids the tedious manual entry process, existing methods for identifying ingredients in images suffer from low accuracy when dealing with complex food images, and require model retraining to improve accuracy.

[0041] To address the problems existing in the above methods, this exemplary embodiment proposes a food ingredient identification method, a food ingredient identification device, an electronic device, and a computer-readable storage medium. The technical solutions of this disclosure embodiment are described in detail below:

[0042] This example implementation first provides a method for identifying food ingredients. (See reference...) Figure 3 As shown, this food ingredient identification method specifically includes the following steps:

[0043] Step S310: Receive the image to be identified, and use the pre-trained food recognition model to identify the ingredients in the image to obtain preliminary recognition results;

[0044] Step S320: Obtain the feature vector of the image to be identified, and determine the feature similarity between the feature vector of the image to be identified and the feature vector of the historical images. The historical images are those images in which the food ingredients have been identified and the identification results have been corrected.

[0045] Step S330: Obtain similar images to the image to be identified from historical images based on feature similarity;

[0046] Step S340: When a similar image is obtained, the preliminary recognition result is corrected based on the correction data of the similar image to obtain the final recognition result of the food in the image to be recognized.

[0047] In the food identification method provided by the exemplary embodiments of this disclosure, on the one hand, the food identification method provided by this exemplary embodiment corrects the preliminary identification results of the food in the image to be identified by using correction data from similar images, which can improve the accuracy of food identification. On the other hand, it eliminates the need to retrain the food identification model, thus avoiding manual intervention in data processing, model training, deployment iteration, and other steps, solving the problem of not being able to complete the model retraining process fully automatically in a short period of time. In addition, by obtaining the preliminary identification results of the food in the image to be identified through the already trained food identification model, and correcting the identification results by using correction data from similar images, there is no need to manually enter the food information. This simplifies user operation and optimizes the user experience when applied to scenarios that require nutritional analysis based on food images.

[0048] The above steps will now be described in more detail in another embodiment.

[0049] In step S310, the image to be identified is received, and the food ingredients in the image are identified using a pre-trained food recognition model to obtain preliminary recognition results.

[0050] The images to be identified are those that require food ingredient recognition. For example, the images to be identified could be photos of meals taken and uploaded by the user, such as... Figure 4 The image shown is an image to be identified, containing ingredients such as tomatoes, green vegetables, eggs, and noodles. It should be noted that the above scenario is merely an example; any image that conforms to the above definition of an image to be identified falls within the protection scope of this example implementation.

[0051] The aforementioned food recognition model is trained on a large number of sample images and is used to identify ingredients in the images. For example, this food recognition model can be a convolutional neural network model, trained on tens of thousands of labeled food images. The ingredient recognition method provided in this example embodiment inputs the image to be recognized into the food recognition model to identify the ingredients in the image. The recognition result of this food recognition model is the aforementioned preliminary recognition result.

[0052] It should be noted that a nutritionally balanced meal typically consists of a combination of various carbohydrates, meats, vegetables, and soy products. Therefore, the variety of ingredients in the image to be identified is quite large, and the placement of the ingredients is not fixed. This has a significant negative impact on the accuracy of the food recognition model, resulting in inaccurate preliminary recognition results and the possibility of missed, misidentified, or incorrectly identified ingredients.

[0053] To address the issue of inaccurate food recognition by the aforementioned food recognition model, the food recognition method provided in this example embodiment further requires correction of the preliminary recognition results through the following steps S320 to S340:

[0054] In step S320, the feature vector of the image to be identified is obtained, and the feature similarity between the feature vector of the image to be identified and the feature vector of the historical image is determined. The historical image is an image in which the food has been identified and the identification result has been corrected.

[0055] The aforementioned feature vector is used to characterize the features of the image to be identified. For example, the feature vector may contain information such as the color, texture, and shape of the image to be identified. The aforementioned historical images are images for which food identification results have been obtained using the food identification method provided in this example embodiment. That is, historical images are images for which preliminary food identification results have been obtained using the aforementioned food identification model, and which have been corrected according to the method of this example embodiment to obtain the final identification result. The aforementioned feature similarity refers to the degree of similarity between the feature vector of the image to be identified and the feature vector of the historical images.

[0056] For example, the acquisition of the feature vector of the image to be identified can be implemented based on another convolutional neural network. This convolutional neural network is a feature extraction model trained on multiple sets of similar food images. The model output is a 128-dimensional real vector, which contains image information such as color, texture, and shape of the image input to the feature extraction model. It can be understood that this feature extraction model can transform the feature vectors of similar food images to the same dimension for subsequent feature similarity calculation. It should be noted that the above scenario is only an illustrative example, and this example implementation does not impose any special limitations on it. Other techniques that can achieve feature vector extraction also fall within the protection scope of this example implementation.

[0057] For example, the above method for determining the feature similarity between the feature vector of the image to be identified and the feature vectors of historical images can be implemented as follows: The feature vectors of the image to be identified and the historical images are obtained through the feature extraction model described above; the cosine similarity between the feature vector of the image to be identified and the feature vectors of each historical image is calculated. Assume the feature vector of the image to be identified is v, and the feature vector of the historical image i stored in the database is V. i The cosine similarity, or feature similarity s, between the image to be identified and the historical image can be calculated using the following formula. i :

[0058]

[0059] It should be noted that the above scenario is only an example, and the scope of protection of this example implementation is not limited thereto. For example, feature similarity can also be a numerical value that can represent the degree of feature similarity other than cosine similarity.

[0060] In step S330, similar images to the image to be identified are obtained from historical images based on feature similarity.

[0061] The aforementioned similar images refer to historical images whose feature similarity to the feature vector of the image to be identified meets a preset condition. For example, this preset condition could be that the feature similarity is greater than a certain preset threshold. The process of obtaining similar images to the image to be identified from historical images based on feature similarity can be implemented as follows: sort the historical images in descending order of feature similarity; and obtain the first preset number of historical images whose feature similarity meets the preset threshold condition as similar images.

[0062] Specifically, assuming there are n historical images in the database, where n is a positive integer, the feature similarity between each historical image and the image to be identified is calculated through step S320 above as (s1, s2, ..., s...). n If ), then the above process can be described as follows: for (s1, S2, ..., s nSort in descending order and take the similarity score s. i The top 20 historical images with a value ≥0.6 are considered as similar images. It should be noted that the above scenario is merely illustrative, and the scope of protection of this example implementation is not limited thereto.

[0063] In step S340, when a similar image is obtained, the preliminary recognition result is corrected based on the correction data of the similar image to obtain the final recognition result of the food in the image to be recognized.

[0064] To improve the accuracy of food identification, this example implementation requires correcting the initial identification results of the food obtained through the above food identification model. Depending on whether a similar image to the image to be identified is obtained in step S330, the correction method is divided into the following two cases:

[0065] In the first scenario: if a similar image is obtained through step S330, the preliminary identification result is corrected through step S340. Specifically, the preliminary identification result is corrected based on the correction data of the similar image to obtain the final identification result of the food in the image to be identified.

[0066] The aforementioned correction data refers to the recorded data used to correct the initial recognition results of similar images. For example, this correction data may include records of adding missed-identification ingredients and deleting misidentified ingredients. The final recognition result is the recognition result of the ingredients in the image to be identified after correcting the initial recognition result.

[0067] For example, the above-mentioned correction of the preliminary recognition result based on the correction data of similar images can be achieved as follows: the feature similarity is used as the correction data for adding a food ingredient in the corresponding similar images, and the negative number of the feature similarity is used as the correction data for deleting the food ingredient in the corresponding similar images; the correction data for adding and deleting the food ingredient in each similar image are added together; if the sum is greater than a first preset threshold, the food ingredient is added to the preliminary recognition result of the image to be recognized; if the sum is less than a second preset threshold, the food ingredient is deleted from the preliminary recognition result of the image to be recognized.

[0068] The above correction process will be explained in detail below with reference to Table 1:

[0069] As shown in Table 1, assuming that four similar images (images 1 to 4) are identified from the historical images through steps S320 and S330, and the feature similarities between images 1 to 4 and the image to be identified are s1, s2, s3, and s4 respectively. These four similar images contain correction data for four ingredients: A, B, C, and D. When an ingredient is added to a similar image's modification record, the feature similarity of that similar image is used as the correction data for that addition operation. For example, if ingredient A is missed in the initial identification result of image 1, and ingredient A needs to be added during correction, then the feature similarity s1 corresponding to image 1 is used as the correction data for adding ingredient A when correcting the image to be identified. If ingredient D is misidentified in image 4, and ingredient D needs to be deleted during correction, then the negative of the feature similarity of image 4 (-s4) is used as the correction data for deleting ingredient D when correcting the image to be identified. The same correction data can be obtained for other deletion and addition operations of similar images.

[0070] After obtaining the above correction data, the correction data for each ingredient in the four similar images (Images 1 to 4) are added together to obtain the sum of the correction data. For example, if Image 1 misidentifies ingredient C, a deletion operation is required, and the corresponding correction data is -s1. If Image 3 misses identifying ingredient C, an addition operation is required, and the corresponding correction data is s3. If Images 2 and 4 correctly identify ingredient C, then the sum of the correction data for ingredient C in the image to be identified is -s1 + s3.

[0071] After obtaining the sum of the corrected data, the sum of the corrected data is compared with a preset threshold. If the sum is greater than the first preset threshold, the ingredient is added to the preliminary recognition result of the image to be recognized. For example, assuming the first preset threshold is 2, if the result of -s1+s3 is greater than 2, the ingredient C is added to the preliminary recognition result of the image to be recognized. If the sum is less than the second preset threshold, the ingredient is deleted from the preliminary recognition result of the image to be recognized. For example, assuming the second preset threshold is -2, if the result of -s1+s3 is less than -2, the ingredient C is deleted from the preliminary recognition result of the image to be recognized.

[0072] Table 1

[0073]

[0074] It should be noted that the above scenarios are all illustrative examples, and this example implementation is not intended to be particularly limited. For example, the number of types of ingredients and the modification operations on the ingredients are not limited to the above scenarios.

[0075] After correcting the preliminary recognition results as described above, the food identification method provided in this example embodiment outputs the final recognition result of the food in the image to be identified.

[0076] Optionally, to further improve recognition accuracy, and considering the image to be recognized as a historical image after correction, the example implementation may further correct the result manually after obtaining the final recognition result. Specifically, the final recognition result may be displayed to the user; the final recognition result may be modified in response to the user's modification operation; and the image to be recognized may be archived as a historical image in response to the user's confirmation operation of uploading the image to be recognized.

[0077] The above-mentioned response to the user's modification operation on the final recognition result may include: in response to the user's addition operation on the final recognition result, adding the ingredient to the final recognition result; in response to the user's deletion operation on the final recognition result, deleting the ingredient from the final recognition result.

[0078] For example, after obtaining the final recognition result of the ingredients in the image to be recognized through steps S310 to S340, the final recognition result can be displayed to the user, and a modification interface can be provided to the user so that the user can modify the incorrect recognition result. For ingredients that were missed, the user can add them through a search operation. For example, if the final recognition result of the image to be recognized missed ingredient E, then in response to the user's operation of searching for ingredient E in the ingredient database and adding it, ingredient E is added to the final recognition result. For misidentified food, the user can remove it through a deletion operation. For example, if the final recognition result of the image to be recognized missed ingredient F, then in response to the user's operation of deleting ingredient E, ingredient F is removed from the final recognition result of the image to be recognized. In addition, in response to the user's confirmation operation to confirm the completion of the modification, the modifications made by the user to the final recognition result can also be summarized and recorded.

[0079] In response to the user's confirmation of uploading an image to be recognized, the image to be recognized is archived as a historical image. Specifically, this can be achieved by associating the feature vector of the image to be recognized with the user's modifications to the final recognition result. For example, the feature vector of the image to be recognized can be associated with the summary of user modifications by using the image number. The image to be recognized associated with the modifications is then archived as a historical image.

[0080] The second scenario: No similar image was obtained in step S330. For example, the image to be identified is the first uploaded image, and there are no historical images in the database; or, there are no similar images that meet the preset conditions among the historical images. In the above cases, the preliminary identification result is corrected using the following methods: the preliminary identification result obtained in step S310 is displayed to the user; in response to the user's modification operation on the preliminary identification result, the preliminary identification result is modified, and the modified result is used as the final identification result of the food in the image to be identified; in response to the user's confirmation operation of uploading the image to be identified, the image to be identified is archived as a historical image.

[0081] For example, in response to a user's modification operation on the preliminary identification result, modifying the preliminary identification result may include: adding the ingredient to the preliminary identification result in response to a user's addition operation on the preliminary identification result; or deleting the ingredient from the preliminary identification result in response to a user's deletion operation on the preliminary identification result.

[0082] For example, assuming the above Figure 4 The image shown is identified by the food recognition model as leafy greens, cilantro, eggs, and noodles. It can be seen that the initial recognition missed identifying tomatoes and incorrectly identified cilantro. This recognition result is displayed to the user, providing an interface to modify the result. In response to the user's action of searching for and adding tomatoes in the food database, tomatoes are added to the initial recognition result. In response to the user's action of deleting cilantro, cilantro is removed from the initial recognition result.

[0083] After correcting the initial identification, the process of archiving the image to be identified as a historical image in response to the user's confirmation of the uploaded image can be implemented as follows: associating the feature vector of the image to be identified with the user's modifications to the initial identification result; and archiving the image to be identified with the associated modifications as a historical image. Since this process has already been described in detail in the corresponding section of the first scenario, it will not be repeated here.

[0084] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0085] Furthermore, in this exemplary embodiment, a food ingredient recognition device is also provided, with reference to... Figure 5As shown, the food ingredient recognition device 500 may include a preliminary recognition module 510, a feature similarity determination module 520, a similar image acquisition module 530, and an Op food ingredient recognition module 540. Wherein:

[0086] The preliminary recognition module 510 can be used to receive the image to be recognized, identify the ingredients in the image using a pre-trained food recognition model, and obtain preliminary recognition results.

[0087] The feature similarity determination module 520 can be used to obtain the feature vector of the image to be identified and determine the feature similarity between the feature vector of the image to be identified and the feature vector of historical images. The historical images are images in which the food ingredients have been identified and the identification results have been corrected.

[0088] The similar image acquisition module 530 can be used to obtain similar images to the image to be identified from historical images based on feature similarity.

[0089] The food ingredient recognition module 540 can be used to correct the preliminary recognition results based on the correction data of similar images when similar images are obtained, so as to obtain the final recognition result of the food ingredients in the image to be recognized.

[0090] In one exemplary embodiment of this disclosure, the food ingredient recognition device further includes a correction and archiving module, which is used to: display the final recognition result to the user; modify the final recognition result in response to the user's modification operation on the final recognition result; and archive the image to be recognized as a historical image in response to the user's confirmation operation of uploading the image to be recognized.

[0091] In addition, when no similar image is available, the correction and archiving module is also used to: display the preliminary recognition results to the user; modify the preliminary recognition results in response to the user's modification operation on the preliminary recognition results, and use the modified results as the final recognition results of the ingredients in the image to be recognized; and archive the image to be recognized as a historical image in response to the user's confirmation operation of uploading the image to be recognized.

[0092] In an exemplary embodiment of this disclosure, the food ingredient recognition module specifically includes a correction data determination unit and a final recognition unit, wherein: the correction data determination unit is used to use feature similarity as correction data for adding a food ingredient in corresponding similar images, and to use the negative of feature similarity as correction data for deleting the food ingredient in corresponding similar images; the final recognition unit is used to add the correction data for adding and deleting the food ingredient in each similar image; if the sum is greater than a first preset threshold, the food ingredient is added to the preliminary recognition result of the image to be recognized; if the sum is less than a second preset threshold, the food ingredient is deleted from the preliminary recognition result of the image to be recognized.

[0093] The specific details of each module or unit in the above-mentioned food identification device have been described in detail in the corresponding food identification method, so they will not be repeated here.

[0094] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0095] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0096] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0097] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A food material recognition method characterized by comprising: The method comprises: receiving a picture to be identified, identifying food materials in the picture to be identified by a trained food recognition model to obtain a preliminary identification result; obtaining a feature vector of the picture to be identified, determining a feature similarity between the feature vector of the picture to be identified and a feature vector of a historical picture, the historical picture being a picture on which identification of food materials has been performed and on which an identification result has been corrected; obtaining similar pictures of the picture to be identified from the historical pictures according to the feature similarity; when a plurality of similar pictures are obtained, the similar pictures containing correction records of corresponding food materials, wherein, when the correction records contain an addition operation on the food materials, the feature similarity is taken as correction data of an addition operation on the food materials in the corresponding similar pictures, and when the correction records contain a deletion operation on the food materials, an inverse of the feature similarity is taken as correction data of a deletion operation on the food materials in the corresponding similar pictures; adding the correction data of the addition operation and the correction data of the deletion operation of the similar pictures on the food materials; if the addition result is greater than a first preset threshold, adding the food materials in the preliminary identification result of the picture to be identified; if the addition result is less than a second preset threshold, deleting the food materials in the preliminary identification result of the picture to be identified to obtain a final identification result of food materials in the picture to be identified.

2. The food material recognition method according to claim 1, characterized by, After the final identification result of the food materials in the picture to be identified is obtained, the method further comprises: displaying the final identification result to a user; in response to a modification operation of the user on the final identification result, modifying the final identification result; in response to a confirmation operation of the user uploading the picture to be identified, archiving the picture to be identified as the historical picture. 3.The food material recognition method according to claim 2, characterized by, When the similar pictures cannot be obtained, the method further comprises: displaying the preliminary identification result to a user; in response to a modification operation of the user on the preliminary identification result, modifying the preliminary identification result and taking the modified result as the final identification result of the food materials in the picture to be identified; in response to a confirmation operation of the user uploading the picture to be identified, archiving the picture to be identified as the historical picture.

4. The food material recognition method according to claim 2 or 3, characterized by, The archiving of the picture to be identified as the historical picture comprises: associating the feature vector of the picture to be identified with modification content of an identification result by the user, the identification result being the preliminary identification result or the final identification result; archiving the picture to be identified associated with the modification content as the historical picture.

5. The food material recognition method according to claim 3, wherein The modifying of the preliminary identification result in response to the modification operation of the user on the preliminary identification result comprises: in response to an addition operation of the user on a food material in the preliminary identification result, adding the food material in the preliminary identification result; in response to a deletion operation of the user on a food material in the preliminary identification result, deleting the food material in the preliminary identification result; and The modifying of the final identification result in response to the modification operation of the user on the final identification result comprises: In response to a user adding operation on a food material in the final recognition result, the food material is added in the final recognition result; In response to a user deleting operation on a food material in the final recognition result, the food material is deleted in the final recognition result. 6.The food material recognition method according to claim 1, characterized by, The obtaining of the similar pictures of the to-be-recognized picture from the historical pictures according to the feature similarity degree comprises: The historical pictures are sorted in descending order of the feature similarity degree; The historical pictures with a feature similarity degree satisfying a preset threshold condition are obtained as the similar pictures.

7. A food material recognition device characterized by comprising: Comprise: The preliminary recognition module is used for receiving a to-be-recognized picture, recognizing a food material in the to-be-recognized picture through a food recognition model that has been trained, and obtaining a preliminary recognition result; The feature similarity degree determination module is used for obtaining a feature vector of the to-be-recognized picture, and determining a feature similarity degree between the feature vector of the to-be-recognized picture and a feature vector of a historical picture, the historical picture being a picture on which a food material has been recognized and on which a recognition result has been corrected; The similar picture obtaining module is used for obtaining, from the historical pictures, similar pictures of the to-be-recognized picture according to the feature similarity degree; The food material recognition module is used for, when a plurality of similar pictures are obtained, the similar pictures containing a correction record of a corresponding food material, correcting the preliminary recognition result according to correction data of the similar pictures when the correction record contains an adding operation on the food material, and obtaining a final recognition result of a food material in the to-be-recognized picture when the correction record contains a deleting operation on the food material; The food material recognition module comprises a correction data determination unit and a final recognition unit, wherein: The correction data determination unit is used for taking the feature similarity degree as correction data of an adding operation on a food material in a corresponding similar picture, and taking an inverse number of the feature similarity degree as correction data of a deleting operation on the food material in the corresponding similar picture; The final recognition unit is used for adding the correction data of the adding operation and the correction data of the deleting operation of each similar picture on the food material, adding the food material in the preliminary recognition result of the to-be-recognized picture when the addition result is greater than a first preset threshold, and deleting the food material in the preliminary recognition result of the to-be-recognized picture when the addition result is less than a second preset threshold. 8.The food recognition device according to claim 7, characterized in that, The food material recognition device further comprises a correction and archiving module; The correction and archiving module is used for displaying the final recognition result to a user, and modifying the final recognition result in response to a modification operation of the user on the final recognition result; In response to a user uploading the to-be-recognized picture, the to-be-recognized picture is archived as the historical picture; And When the similar pictures cannot be obtained, the correction and archiving module is further used for displaying the preliminary recognition result to a user, modifying the preliminary recognition result in response to a modification operation of the user on the preliminary recognition result, and taking the modified result as the final recognition result of a food material in the to-be-recognized picture; In response to a confirmation operation of the user uploading the picture to be recognized, the picture to be recognized is archived as the historical picture.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-6.

10. An electronic device, comprising: Comprise: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to implement the method of any one of claims 1-6 via execution of the executable instructions.

Images