Food image retrieval method, device, electronic device and storage medium

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for retrieving food images. The retrieval method includes: acquiring a food image; determining an initial retrieval result based on a matching result between the food image and a preset image in an image library; determining, based on the food image, at least one second food category corresponding to the food image and the confidence level of the food image for each second food category; and adjusting the initial retrieval result based on the confidence level, the correspondence between the first food category and the second food category to generate a final retrieval result. The embodiment of the present disclosure can reorder the initial retrieval results based on the confidence level of the second food category corresponding to each food image, so that the retrieved first food category is more consistent with the actual food category of the food image, thereby improving the classification accuracy of the food image and facilitating improving the final retrieval accuracy of the food image.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to a food image retrieval method, device, electronic device, and storage medium. Background Art

[0002] Food image retrieval has a wide range of applications in production and daily life, such as dietary recommendations and querying food nutritional composition. However, due to the low retrieval accuracy of food images, food image retrieval has been difficult to popularize and hindered its practical application and deployment. Summary of the Invention

[0003] This disclosure proposes a technical solution for retrieving food images.

[0004] According to one aspect of the present disclosure, a method for retrieving food images is provided, the retrieval comprising: acquiring a food image; determining an initial retrieval result based on a matching result between the food image and a preset image in an image library; wherein the initial retrieval result comprises at least one first food category corresponding to the food image; determining, based on the food image, at least one second food category corresponding to the food image and a confidence level of the food image for each second food category; adjusting the initial retrieval result based on the confidence level and the correspondence between the first food category and the second food category to generate a final retrieval result.

[0005] In a possible implementation manner, the second food category corresponding to the first food category includes at least one higher-level category to which the first food category belongs.

[0006] In a possible embodiment, each first food category in the initial retrieval result is sorted according to the size of the score corresponding to the first food category, and the score is positively correlated with the similarity between the food image and the preset image corresponding to the first food category. The initial retrieval result is adjusted according to the confidence level and the correspondence between the first food category and the second food category to generate a final retrieval result, including: for any first food category, according to the confidence level of the second food category corresponding to the any first food category, updating the score corresponding to the any first food category; and re-sorting the first food categories according to the size of the updated score to generate a final retrieval result.

[0007] In one possible embodiment, determining at least one second food category corresponding to the food image based on the food image includes: inputting the food image into a target model, determining the confidence that the food image corresponds to each output category of the target model; and determining at least one second food category corresponding to the food image in the output category based on the size of the confidence.

[0008] In a possible embodiment, determining the initial retrieval result based on the matching result between the food image and the preset images in the image library includes: determining the similarity between each preset image in the image library and the food image; screening out at least one first image from the preset images based on the similarity; and determining at least one first food category corresponding to the food image based on the food category corresponding to the first image.

[0009] In one possible embodiment, determining the similarity between each preset image in the image library and the food image includes: obtaining each preset image in the image library; for any preset image, determining the manifold space similarity and Euclidean space similarity between the preset image and the food image, and determining the similarity between the any preset image and the food image based on the manifold space similarity and Euclidean space similarity corresponding to the any preset image.

[0010] In a possible embodiment, the retrieval method further includes: obtaining the image to be stored and the first food category corresponding to the image to be stored; inputting the image to be stored into the target model to obtain the second food category corresponding to the image to be stored; using the image to be stored as the preset image, the first food category corresponding to the image to be stored as the first food category corresponding to the preset image, and the second food category corresponding to the image to be stored as the second food category corresponding to the preset image, and saving them into the image library.

[0011] In a possible embodiment, the inputting the image to be stored into the target model to obtain the second food category corresponding to the image to be stored includes: inputting the image to be stored into the target model to obtain a first confidence level for each output category corresponding to the image to be stored; and taking the output category whose first confidence level is greater than a preset confidence level as the second food category corresponding to the image to be stored.

[0012] In a possible implementation, acquiring the food image includes: acquiring an image to be processed; and determining at least one food region image including a food object in the image to be processed, and using the image as the food image.

[0013] In a possible implementation, after generating the final retrieval result, the retrieval method further includes: displaying the final retrieval result according to the position of the food image in the image to be processed.

[0014] According to one aspect of the present disclosure, a food image retrieval device is provided, the retrieval device comprising: a food image acquisition module for acquiring food images; an initial retrieval result determination module for determining an initial retrieval result based on a matching result between the food image and a preset image in an image library, the initial retrieval result including at least one first food category corresponding to the food image; a confidence determination module for determining, based on the food image, at least one second food category corresponding to the food image and a confidence of the food image for each second food category; and a final retrieval result generation module for adjusting the initial retrieval result based on the confidence and the correspondence between the first food category and the second food category to generate a final retrieval result.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned retrieval method.

[0016] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned retrieval method is implemented.

[0017] In an embodiment of the present disclosure, a food image can be obtained, and then an initial search result can be determined based on the matching result between the food image and a preset image in an image library. Then, based on the food image, at least one second food category corresponding to the food image and the confidence of the food image for each second food category can be determined. Finally, based on the confidence, the correspondence between the first food category and the second food category, the initial search result can be adjusted to generate a final search result. The embodiment of the present disclosure can reorder the initial search results based on the confidence of the second food category corresponding to each food image. Compared with the initial search result, the final search result obtained thereby has the first food category retrieved that is more consistent with the actual food category of the food image, thereby improving the classification accuracy of the food image, which is beneficial to improving the final retrieval accuracy of the food image.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0020] Figure 1 A flowchart of a food image retrieval method provided according to an embodiment of the present disclosure is shown.

[0021] Figure 2 A flowchart of a food image retrieval method provided according to an embodiment of the present disclosure is shown.

[0022] Figure 3 The embodiment provided by the present disclosure shows Figure 1 Reference schematic diagram of step S200 in FIG.

[0023] Figure 4 The embodiment provided by the present disclosure Figure 1 Reference schematic diagram of step S300 in FIG.

[0024] Figure 5 A reference schematic diagram of a food image retrieval method provided according to an embodiment of the present disclosure is shown.

[0025] Figure 6 A reference schematic diagram of a food image retrieval method provided according to an embodiment of the present disclosure is shown.

[0026] Figure 7 A block diagram of a food image retrieval device provided according to an embodiment of the present disclosure is shown.

[0027] Figure 8 A block diagram of an electronic device provided according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0029] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0030] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0031] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0032] In related technologies, an image retrieval model is usually used to directly search for several images with the highest similarity in the image library as the retrieval results. However, this approach is prone to the following problems: the similarity between images is obtained by comparing the image features of the query image with the image features of the preset images in the image library. Although the image features can be used to determine whether the image contents of the two are similar, the similarity obtained in this way cannot guarantee whether the food types in the retrieval results are similar to the food types in the query image, that is, the retrieval accuracy is low.

[0033] In view of this, the embodiment of the present disclosure provides a method for retrieving food images, which can obtain food images, and then determine the initial retrieval results based on the matching results of the food images with preset images in the image library, and then determine at least one second food category corresponding to the food image and the confidence of the food image for each second food category based on the food image, and finally adjust the initial retrieval results based on the confidence, the correspondence between the first food category and the second food category to generate the final retrieval results. The embodiment of the present disclosure can re-sort the initial retrieval results based on the confidence of the second food category corresponding to each food image. Compared with the initial retrieval results, the retrieved first food category of the final retrieval results is more consistent with the actual food category of the food image, thereby improving the classification accuracy of the food image, which is conducive to improving the final retrieval accuracy of the food image.

[0034] In one possible implementation, the retrieval method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method can be executed by a server. In combination with actual application scenarios, the user can capture food images through the camera on the terminal device, and then directly search and match them by the terminal device or upload them to the server. After the search and matching is completed, the terminal device displays the search results, for example: displaying the food category, nutritional value, recommended recipes, etc. of the food image.

[0035] See Figure 1 As shown, Figure 1 A flowchart of a food image retrieval method provided according to an embodiment of the present disclosure is shown.

[0036] like Figure 1 As shown, the above retrieval method may include:

[0037] Step S100: Acquire a food image. Exemplarily, the food image includes at least one food object, which can be any kind of food.

[0038] In a possible embodiment, step S100 may include: acquiring an image to be processed, and then determining at least one food area image including a food object in the image to be processed, and using it as the food image. Exemplarily, the above-mentioned image to be processed may be collected by a camera of a terminal device, or directly obtained from a database storing captured images, and the embodiments of the present disclosure are not limited thereto. In one example, the image to be processed may be input into a food object detection model in the related art, and then the model may determine the area where the food object is located in the image to be processed, and use it as the above-mentioned food image. The above-mentioned image to be processed may also undergo other image processing, such as image clarity processing, image size processing, etc., to facilitate processing by the food object detection model, and the embodiments of the present disclosure are not limited thereto.

[0039] Step S200, determining the initial retrieval result based on the matching result between the food image and the preset image in the image library. The initial retrieval result includes at least one first food category corresponding to the food image. Exemplarily, the image library stores a plurality of preset images, wherein each preset image corresponds to at least one first food category. For example, the first food category may include miso soup, sushi, rice, eel rice, udon noodles and other food categories with more detailed classifications. In one example, each first food category in the initial retrieval result is sorted according to its corresponding score size (from large to small, or from small to large), and the score is positively correlated with the similarity between the food image and the preset image corresponding to the first food category. Combined Figure 3 As shown, Figure 3 The embodiment provided by the present disclosure Figure 1 Reference diagram of step S200 in FIG. Figure 3 The small box area in the food picture in the image is retrieved by the retrieval model of the relevant technology in the image library (i.e. Figure 3 The food database in Figure 3 The search results and scores are collected for subsequent processing.

[0040] See Figure 2 As shown, Figure 2 A flowchart of a food image retrieval method provided according to an embodiment of the present disclosure is shown.

[0041] like Figure 2 As shown, in a possible implementation, step S200 may include:

[0042] Step S210 determines the similarity between each preset image in the image library and the food image. For example, the similarity can be expressed as cosine similarity, manifold similarity, or other similarity metrics used in related technologies to indicate whether the image content of the two images is similar. In one example, the food image can be input into a feature extraction model used in related technologies to extract image features of the food image. These features are then compared with the image features of the preset image using a retrieval model used in related technologies to obtain the similarity.

[0043] In a possible implementation, step S210 may include: obtaining each preset image in the image library. Then, for any preset image, determine the manifold space similarity and Euclidean space similarity between the preset image and the food image, and determine the similarity between the preset image and the food image based on the manifold space similarity and Euclidean space similarity corresponding to the preset image. Exemplarily, the calculation process of the above-mentioned manifold space similarity and Euclidean space similarity can refer to the relevant technology, and the embodiments of the present disclosure will not be described here. For example, the Euclidean space similarity can be expressed as cosine similarity, and the manifold space similarity can be obtained by the similarity matrix between different image features. In one example, the above-mentioned manifold space similarity and Euclidean space similarity can be averaged or weighted averaged, and the average value can be used as the above-mentioned similarity. The embodiments of the present disclosure can make the generated similarity more representative by fusing the similarities of different feature spaces, which is beneficial to improving the generation accuracy of the first food category.

[0044] Step S220 selects at least one first image from the preset images based on the similarity. For example, the preset images may be sorted based on their similarity to the food image, and all preset images with a similarity greater than a predetermined similarity and a predetermined number of similarities are selected as the first images.

[0045] Step S230: Determine at least one first food category corresponding to the food image based on the food category corresponding to the first image. For example, the food category corresponding to the first image may be used as the first food category corresponding to the food image. Developers may also define specific rules to filter the first food category within the food category, but this is not a limitation in the present embodiment.

[0046] Continue reading Figure 1 , step S300, based on the food image, determining at least one second food category corresponding to the food image and the confidence of the food image for each second food category. The second food category corresponding to the first food category includes at least one higher-level category to which the first food category belongs. Exemplarily, each first food category may belong to at least one higher-level category. For example, if the first food category is eel rice, its corresponding higher-level category may include rice, fried food, and meat. If the first food category is udon noodles, its corresponding higher-level category may include noodles. The above-mentioned second food category is a subset of the above-mentioned higher-level category.

[0047] In one possible implementation, step S300 may include inputting the food image into a target model and determining the confidence level of each output category of the target model. Exemplarily, the output categories are also the higher-level categories. In one example, the target model may be a multi-classification model known in the related art. Based on the input image, the multi-classification model may determine the confidence level (or confidence score) of each output category. Then, based on the confidence level, at least one second food category corresponding to the food image is determined within the output categories. Exemplarily, a developer may set a preset confidence level. If the confidence level of an output category corresponding to a food image exceeds the preset confidence level, the output category may be designated as the second food category to improve the accuracy of subsequent score generation, thereby improving the accuracy of food image retrieval. Developers may also set a preset number to limit the number of second food categories, which is not a limitation in the present embodiment. In the related art, if only the correlation model is used to obtain the final search results, the correlation model may output more categories (because users typically use detailed categories as search results, rather than basic food categories), which increases processing time and training costs. The disclosed embodiment combines confidence with scores to generate final search results that are not only more detailed and accurate, but also helps reduce model processing time and training costs because the target model has fewer categories than related models.

[0048] Combine Figure 4 , Figure 4 The embodiment provided by the present disclosure shows Figure 1Reference diagram of step S300 in step S300. The above output categories may include: soup (i.e., soup in the figure), noodles (i.e., noodles in the figure), fried food (i.e., fried food in the figure), meat (i.e., meat in the figure), rice (i.e., rice in the figure), fruits (i.e., fruits in the figure), and vegetables (i.e., vegetables in the figure). If the actual first food category of the food image is eel rice, then after the output of the multi-classification model, the confidence of the food image corresponding to the seven output categories is obtained, and the fried food, meat, and rice categories with confidence higher than the preset confidence (e.g., 0.6) are used as the second food category corresponding to the food image. In the above case, the classification task of the target model can be divided into seven binary classification tasks, that is, the annotation labels (which can be manually labeled) corresponding to the training samples used in each binary classification task of the target model can be: [soup, non-soup], [noodles, non-noodles], [fried food, non-fried food], [meat, non-meat], [vegetables, non-vegetables], [fruits, non-fruits], [rice, non-rice], and then after the target model training is completed, the confidence level of the food image corresponding to each output category can be obtained in step S300.

[0049] In one possible embodiment, if the output category of the target model is determined, the above-mentioned image library can be established in the following manner: obtain the image to be stored and the first food category corresponding to the image to be stored. Exemplarily, the first food category corresponding to the image to be stored can be obtained by manual annotation. Then, the image to be stored is input into the target model to obtain the second food category corresponding to the image to be stored. In one example, the image to be stored can be input into the target model to obtain the first confidence level of each output category corresponding to the image to be stored, and then the output category with the first confidence level greater than the preset confidence level is used as the second food category corresponding to the image to be stored. Exemplarily, the developer can set other preset conditions to filter the second food category of the image to be stored, such as limiting the number of second food categories corresponding to the image to be stored by a preset number, so as to improve the representativeness of the second food category corresponding to the image to be stored. In one example, if labor costs are not considered, the above-mentioned second food category can also be manually labeled by the developer. Finally, the image to be stored is taken as the preset image, the first food category corresponding to the image to be stored is taken as the first food category corresponding to the preset image, and the second food category corresponding to the image to be stored is taken as the second food category corresponding to the preset image, and saved in the image library. The disclosed embodiment can automatically establish an image library through the target model, which can save labor costs. In addition, the above-mentioned second food category corresponds to the output category of the target model, which reduces the probability of incorrect classification of the second food category (such as: if the second food category is manually classified, the image to be stored may be assigned a non-output category of the target model or an incorrect output category due to an operational error, thereby making it impossible to correctly calculate the score of the subsequent image to be stored).

[0050] Step S400 : adjusting the initial search result according to the confidence level and the corresponding relationship between the first food category and the second food category to generate a final search result.

[0051] See Figure 5 As shown, Figure 5 A reference schematic diagram of a food image retrieval method according to an embodiment of the present disclosure is shown. Figure 5 As shown, the electronic device can fuse the retrieval model with the preliminary retrieval results of the image library and the classification results output by the multi-classification model to obtain the re-sorted preliminary retrieval results (i.e., the re-sorted results in the figure), thereby improving the retrieval accuracy of food images.

[0052] In one possible implementation, if the first food categories in the initial search results are sorted by their corresponding scores, step S400 may include: updating the score corresponding to each first food category based on the confidence level of the second food category corresponding to the first food category. The first food categories are then re-sorted based on the updated scores to generate the final search results. For example, the confidence levels and scores may be weighted and summed, and the scores updated to the summed value.

[0053] See Figure 6 As shown, Figure 6 A reference diagram of a method for retrieving food images according to an embodiment of the present disclosure is shown. Figure 6 As shown, if the initial search results of the food image (i.e., the small box in the food image in the figure) include: pork cutlet rice (i.e., katsudon in the figure) with a corresponding score of 0.85, rice (i.e., rice in the figure) with a corresponding score of 0.84, sushi (i.e., sushi in the figure) with a corresponding score of 0.81, pilaf (i.e., pilaf in the figure) with a corresponding score of 0.80, eel rice (i.e., eels on rice in the figure) with a corresponding score of 0.79, tempura (i.e., tempura bowl in the figure) with a corresponding score of 0.76, and udon noodles (i.e., udon noodles in the figure) with a score of 0.72. Among them, the output categories corresponding to pork cutlet rice include meat and rice, the output categories corresponding to rice include rice, the output categories corresponding to sushi include meat and rice, the output categories corresponding to pilaf include rice and vegetables, the output categories corresponding to eel rice include meat, rice, and fried, the output categories corresponding to tempura include meat and fried, and the output categories corresponding to udon noodles include noodles. If the second food category corresponding to the food image includes: fried food (confidence is 0.67), meat (confidence is 0.90), and rice (confidence is 0.86), the weight of the confidence added to the score is 0.1 (i.e., the updated score = original score + 0.1 times the confidence). The electronic terminal then updates the corresponding score for each first food category that contains the second food category. For example: the score of pork chop rice = 0.85 (original score) + 0.1*0.90 (confidence of meat) + 0.1*0.86 (confidence of rice) = 1.026 (updated score). The electronic device then rearranges the first food category in the initial search results according to the updated score and uses it as the final search category, thereby improving the retrieval accuracy of the first food category of the food object in the food image. In addition, the retrieval method provided by the embodiment of the present disclosure is more optimizable, that is, if the developer wants to optimize the final retrieval results, the above-mentioned weights can be adjusted, and the optimization workload is smaller.

[0054] In one possible embodiment, after generating the final retrieval result, the retrieval method further includes: displaying the final retrieval result based on the position of the food image in the image to be processed. For example, the final retrieval result may be represented by the first food category with the highest score. In this case, the electronic device may mark a food frame for each food image in the image to be processed and display the first food category in the food frame. For another example, the electronic device may determine the nutritional content of the food object in the food image based on the final retrieval result and display the first food category and the nutritional content of the food object in the food frame.

[0055] The present disclosure provides several application scenarios for reference:

[0056] The above-mentioned food image retrieval method can be applied to the food preparation method recommendation function. The user uses a smartphone to take a picture of the food object to be detected, and then transmits it to the server. The server determines the final retrieval result of the food object through the retrieval method provided by the embodiment of the present disclosure, and automatically queries the corresponding food preparation method based on the first food category with the highest score in the final retrieval result, and places a web page link control with the food preparation method in the area where the food object is located in the above-mentioned image to be detected, and sends it to the smartphone to display the image, so that the user can know the preparation method of the food object by clicking the control on the smartphone.

[0057] The above-mentioned food image retrieval method can also be applied to the diet management function. The user uses a smartphone to take a photo of an image to be detected with a food object, and then transmits it to the server. The server determines the final retrieval result of the food object through the retrieval method provided by the embodiment of the present disclosure, and automatically queries the corresponding calorie content based on the first food category with the highest score in the final retrieval result, and displays the calorie content of the food object in the area where the food object is located in the above-mentioned image to be detected, and sends it to the smartphone to display the image, so that the user can know the calorie content of the food object in the smartphone, which is convenient for the user to formulate a diet management plan.

[0058] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0059] In addition, the present disclosure also provides a food image retrieval device, electronic device, computer-readable storage medium, and program, all of which can be used to implement any retrieval method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0060] Figure 7 A block diagram of a food image retrieval device according to an embodiment of the present disclosure is shown. Figure 7 As shown, the retrieval device 100 includes: a food image acquisition module 110, which is used to acquire food images. An initial retrieval result determination module 120, which is used to determine the initial retrieval result based on the matching result between the food image and the preset image in the image library, and the initial retrieval result includes at least one first food category corresponding to the food image. A confidence determination module 130, which is used to determine at least one second food category corresponding to the food image and the confidence of the food image for each second food category based on the food image. A final retrieval result generation module 140, which is used to adjust the initial retrieval result based on the confidence, the correspondence between the first food category and the second food category, and generate a final retrieval result.

[0061] In a possible implementation manner, the second food category corresponding to the first food category includes at least one higher-level category to which the first food category belongs.

[0062] In a possible embodiment, each first food category in the initial retrieval result is sorted according to the size of the score corresponding to the first food category, and the score is positively correlated with the similarity between the food image and the preset image corresponding to the first food category. The initial retrieval result is adjusted according to the confidence level and the correspondence between the first food category and the second food category to generate a final retrieval result, including: for any first food category, according to the confidence level of the second food category corresponding to the any first food category, updating the score corresponding to the any first food category; and re-sorting the first food categories according to the size of the updated score to generate a final retrieval result.

[0063] In one possible embodiment, determining at least one second food category corresponding to the food image based on the food image includes: inputting the food image into a target model, determining the confidence that the food image corresponds to each output category of the target model; and determining at least one second food category corresponding to the food image in the output category based on the size of the confidence.

[0064] In a possible embodiment, determining the initial retrieval result based on the matching result between the food image and the preset images in the image library includes: determining the similarity between each preset image in the image library and the food image; screening out at least one first image from the preset images based on the similarity; and determining at least one first food category corresponding to the food image based on the food category corresponding to the first image.

[0065] In one possible embodiment, determining the similarity between each preset image in the image library and the food image includes: obtaining each preset image in the image library; for any preset image, determining the manifold space similarity and Euclidean space similarity between the preset image and the food image, and determining the similarity between the any preset image and the food image based on the manifold space similarity and Euclidean space similarity corresponding to the any preset image.

[0066] In a possible embodiment, the retrieval device also includes: a to-be-stored image processing module, used to perform any one of the following steps: obtaining the to-be-stored image and the first food category corresponding to the to-be-stored image; inputting the to-be-stored image into the target model to obtain the second food category corresponding to the to-be-stored image; using the to-be-stored image as the preset image, the first food category corresponding to the to-be-stored image as the first food category corresponding to the preset image, and the second food category corresponding to the to-be-stored image as the second food category corresponding to the preset image, and saving them in the image library.

[0067] In a possible embodiment, the inputting the image to be stored into the target model to obtain the second food category corresponding to the image to be stored includes: inputting the image to be stored into the target model to obtain a first confidence level for each output category corresponding to the image to be stored; and taking the output category whose first confidence level is greater than a preset confidence level as the second food category corresponding to the image to be stored.

[0068] In a possible implementation, acquiring the food image includes: acquiring an image to be processed; and determining at least one food region image including a food object in the image to be processed, and using the image as the food image.

[0069] In a possible implementation, after generating the final search result, the search device further includes: a final search result display module, configured to display the final search result according to the position of the food image in the image to be processed.

[0070] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0071] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0072] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0073] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0074] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0075] The electronic device may be provided as a server or other form of device.

[0076] Figure 8 FIG1 shows a block diagram of an electronic device 1900 provided according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0077] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0078] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0079] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0080] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0081] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0082] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0083] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0084] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0085] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0086] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0087] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0088] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0089] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0090] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0091] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A food image retrieval method, characterized in that: The retrieval method comprises: Get food images; Determining an initial search result based on a matching result between the food image and a preset image in an image library; wherein the initial search result includes at least one first food category corresponding to the food image; determining, based on the food image, at least one second food category corresponding to the food image and a confidence score of the food image for each second food category, wherein the second food category corresponding to the first food category includes at least one higher-level category to which the first food category belongs; The initial search result is adjusted according to the confidence level and the corresponding relationship between the first food category and the second food category to generate a final search result.

2. The search method according to claim 1, wherein: The first food categories in the initial search results are sorted according to the scores corresponding to the first food categories, and the scores are positively correlated with the similarity between the food images and the preset images corresponding to the first food categories. The adjusting the initial search result according to the confidence level and the corresponding relationship between the first food category and the second food category to generate a final search result includes: For any first food category, updating the score corresponding to the first food category according to the confidence level of the second food category corresponding to the first food category; According to the updated scores, the first food categories are re-sorted to generate the final search results.

3. The search method according to claim 1 or 2, wherein: The determining, based on the food image, at least one second food category corresponding to the food image includes: Inputting the food image into a target model, and determining a confidence score that the food image corresponds to each output category of the target model; At least one second food category corresponding to the food image is determined in the output category according to the confidence level.

4. The search method according to claim 1 or 2, wherein: The determining of the initial search result according to the matching result between the food image and the preset images in the image library includes: Determining the similarity between each preset image in the image library and the food image; Selecting at least one first image from the preset images according to the similarity; At least one first food category corresponding to the food image is determined according to the food category corresponding to the first image.

5. The search method according to claim 4, wherein: Determining the similarity between each preset image in the image library and the food image includes: Get each preset image in the image library; For any preset image, determine the manifold space similarity and Euclidean space similarity between the preset image and the food image, and determine the similarity between the preset image and the food image based on the manifold space similarity and Euclidean space similarity corresponding to the preset image.

6. The search method according to claim 1 or 2, wherein: The retrieval method further comprises: Acquire an image to be stored and a first food category corresponding to the image to be stored; Inputting the image to be stored into the target model to obtain a second food category corresponding to the image to be stored; The image to be stored is used as the preset image, the first food category corresponding to the image to be stored is used as the first food category corresponding to the preset image, and the second food category corresponding to the image to be stored is used as the second food category corresponding to the preset image, and are saved in the image library.

7. The search method according to claim 6, wherein: Inputting the image to be stored into the target model to obtain the second food category corresponding to the image to be stored includes: Inputting the image to be stored into the target model to obtain a first confidence score for each output category corresponding to the image to be stored; The output category whose first confidence level is greater than a preset confidence level is used as the second food category corresponding to the image to be stored.

8. The search method according to claim 1 or 2, wherein: The obtaining of food images comprises: Get the image to be processed; At least one food region image including a food object in the image to be processed is determined as the food image.

9. The search method according to claim 8, wherein: After generating the final search result, the search method further includes: The final retrieval result is displayed according to the position of the food image in the image to be processed.

10. A food image retrieval device, characterized in that: The retrieval device comprises: A food image acquisition module, used to acquire food images; an initial search result determination module, configured to determine an initial search result based on a matching result between the food image and a preset image in an image library, wherein the initial search result includes at least one first food category corresponding to the food image; a confidence determination module, configured to determine, based on the food image, at least one second food category corresponding to the food image and a confidence level of the food image for each second food category, wherein the second food category corresponding to the first food category includes at least one higher-level category to which the first food category belongs; The final search result generating module is used to adjust the initial search result according to the confidence level and the corresponding relationship between the first food category and the second food category to generate a final search result.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the retrieval method according to any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the search method according to any one of claims 1 to 9 is implemented.

Images