Image recognition methods and related devices

Abstract

This application provides an image recognition method and related apparatus. The method includes: acquiring a first image; the first image including a first object; inputting the first image into a first model to obtain a first category of the first object and a confidence level for the first category; if the confidence level of the first category does not meet a confidence level requirement, obtaining a second category of the first object based on the first image and a second model; the training samples of the first model are less than the training samples of the second model; and outputting the second category of the first object. In this way, electronic devices can expand the range of food categories they can recognize based on a large language model, thereby improving the richness and accuracy of food category recognition, reducing scenarios where food categories cannot be recognized or are misidentified, and ultimately improving the accuracy of food information.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image recognition method and related apparatus. Background Technology

[0002] Some electronic devices can support dietary health management functions. For example, users can use electronic devices to obtain food information, such as the name of the food, its calories, and its nutritional components, in order to appropriately control dietary calories and balance the intake of nutrients.

[0003] In some implementations, electronic devices can identify food based on image recognition capabilities. For example, a user can use an electronic device to take a photo or video of food, and the device can then identify the food based on the image.

[0004] However, in some implementations, when using electronic devices to recognize food images, the devices may fail to identify the food, resulting in users being unable to obtain relevant food information and affecting the user experience. Summary of the Invention

[0005] This application provides an image recognition method and related apparatus, applied in the field of terminal technology. By accurately identifying food categories, it improves the accuracy of food information (e.g., food calories and / or food nutritional components), thereby providing users with dietary advice and health management solutions, and enhancing the user experience.

[0006] In a first aspect, embodiments of this application propose an image recognition method. The method includes: acquiring a first image (e.g., ... Figure 1A The food image 101 shown in Figure a) includes a first object (which may correspond to a first food, for example, a carrot); the first image is input into a first model to obtain a first category of the first object (for example, a carrot is the first food) and a confidence score for the first category (for example, the confidence score for a carrot is 0.35, see [reference]). Figure 5 If the confidence level of the first category does not meet the confidence requirement (the confidence requirement can be greater than the first threshold of 0.95, or other confidence requirements), then based on the first image and the second model, the second category of the first object is obtained (the second category can be the same as or different from the first category; for example, if both the first and second categories are carrots, see [reference]). Figure 5 After the output of the second model, the electronic device can determine the category of the first food as carrot; the training samples of the first model are less than the training samples of the second model (the second model can be a large language model trained with massive training data); output the second category of the first object.

[0007] It should be noted that when the first and second categories are the same, the confidence level of the first category output by the first model may not meet the confidence requirements due to the limited training samples. The electronic device may then output "cannot identify food" instead of outputting the first category. Therefore, the second model is needed to re-detect the first object to improve the accuracy of the first category. Conversely, when the first and second categories are different, the first model may misidentify the first object. The electronic device can use the second model to verify the first object, thus outputting the accurate second category.

[0008] In this way, electronic devices can expand the range of food categories they can recognize based on large language models, thereby improving the richness and accuracy of food categories, reducing scenarios where food categories cannot be recognized or are misidentified, and thus improving the accuracy of food information.

[0009] Optionally, based on the first image and the second model, the second category of the first object is obtained, including: obtaining the image of the first object (e.g., an image of the first food), the mask of the first object (e.g., the mask of the first food), and the bounding box of the first object (e.g., the border of the first food) in the first image; wherein, the image of the first object is an image cropped from the first image based on the bounding box of the first object; the mask and the bounding box of the first object are both obtained based on the first model; the image and the mask of the first object are input into a third model (e.g., mask-Q-former) to obtain a fusion vector of the first object (e.g., a fused vector); the fusion vector of the first object is input into the second model to obtain the second category of the first object and the conditional probability of the second category; the second category of the first object is output, including: outputting the second category of the first object when the conditional probability of the second category meets the conditional probability requirements.

[0010] It should be noted that, in one possible implementation, the conditional probability of the second category satisfies the conditional probability requirement by having the conditional probability of the second category greater than a certain threshold. In another possible implementation, the conditional probability of the second category satisfies the conditional probability requirement by having the target probability, after correcting the conditional probability of the second category using the confidence level of the second category, satisfy the target probability requirement.

[0011] In this way, the electronic device can use the third model as a bridge between the first and second models to fuse the data output by the first model, so that the output of the first model is adapted to the input of the second model, so that the second model outputs the second category of the first object.

[0012] Optionally, the image and mask of the first object are input into the third model to obtain the fusion vector of the first object, including: converting the image of the first object into a first image vector (e.g., vector 3) based on the first image encoder, converting the mask of the first object into a first mask vector (e.g., vector 4) based on the mask encoder, and converting a preset prompt word into a first prompt word vector (e.g., vector 5) based on the first prompt word encoder; wherein, the preset prompt word is used to guide the third model to fuse the first image vector and the first mask vector; the third model includes a first image encoder, a mask encoder, and a first prompt word encoder; based on mask query vector, context query vector, and mode query vector... The query vector and the first prompt word vector are fused to the first image vector and the first mask vector to obtain the fused vector of the first object; the mask query vector, the content query vector and the pattern query vector are all preset vectors. The mask query vector is used to set the weight of the mask area of ​​the first object, the content query vector is used to set the weight of the context area of ​​the first object, and the pattern query vector is used to set the weight of the mask query vector and the weight of the content query vector; the context area of ​​the first object corresponds to the remaining area in the border of the first object after removing the mask area of ​​the first object.

[0013] This process can correspond to Figure 7 And step S404. In this way, the electronic device can fuse the multimodal information related to the first object through the third model to obtain the input data of the second model.

[0014] Optionally, the first model includes a first sub-model (e.g., a self-attention mechanism (SAM model) and a second sub-model (e.g., a groundingDINO object detection framework model); the first sub-model is used to output the category of the object based on the image; the second sub-model is used to perform mask segmentation on the image; the first image is input into the first model to obtain the first category of the first object and the confidence score of the first category, including: obtaining the candidate bounding box information (proposal) of the first object in the first image; the candidate bounding box information of the first object is used to define the position of the first object in the first image; the first image and the candidate bounding box information of the first object are input into the first sub-model to obtain the bounding box of the first object, the first category of the first object, and the confidence score of the first category; the matching degree between the bounding box of the first object and the first object is higher than the matching degree between the candidate bounding box of the first object and the first object (which can be understood as the candidate bounding box being a coarse selection of the first object, and the bounding box being a precise selection of the first object); after inputting the first image into the first model, the model further includes: inputting the bounding box of the first object and the first image into the second sub-model to obtain the mask of the first object.

[0015] This process can correspond to Figure 6 In step S402, the GroundingDINO model can obtain the bounding box, food category, and confidence level of the first object; the SAM model can obtain the mask of the first object. In this way, the electronic device can perform fine-grained segmentation of the first image based on the first model and obtain the food category and confidence level corresponding to the mask for subsequent judgment and calculation.

[0016] Optionally, the bounding box and the first image of the first object are input into the second sub-model to obtain the mask of the first object, including: converting the first image into a second image vector (vector 1) based on the second image encoder, and converting the bounding box of the first object into a second cue word vector (vector 2) based on the second cue word encoder; the second sub-model includes a second image encoder, a second cue word encoder and a mask decoder; the second image vector and the second cue word vector are input into the mask decoder to obtain the mask of the first object.

[0017] This process can correspond to Figure 6 And step S402. In this way, the electronic device can learn the mask of the first object by combining more accurate border information, thereby obtaining a fine-grained division of the first image.

[0018] Optionally, before outputting the second category of the first object, the method further includes: fusing the confidence level of the second category in the first model with the conditional probability of the second category in the second model to obtain the target probability of the second category; the weights of the confidence level of the second category in the first model and the weights of the conditional probability of the second category in the second model are different; and outputting the second category of the first object when the conditional probability of the second category meets the conditional probability requirement, including: outputting the second category of the first object when the target probability of the second category meets the target probability condition.

[0019] Understandably, after obtaining the second category of the first object through the large language model, the prediction result of the large language model can be corrected through, for example, step S406, to improve the accuracy of the target probability.

[0020] Optional, target probability Satisfy the following formula:

[0021]

[0022] Where p is the image of the first object; c is the category of the first object; c ′ To determine which of the following categories of the first object is obtained when the first model identifies the first object; P c (p,c) represents the conditional probability that the first object in the second model is of category c. The confidence score of class c for the first object output by the first model; P c (p,c ′ The category of the first object output by the second model is c. ′ The probability value; λ is the preset exponent; C B Here is the set of categories corresponding to the training samples of the first model; bg is the background category; C N Not belonging to C B A collection of other categories, including the background category.

[0023] Understandably, in the second category, which belongs to C B When the electronic device uses formula (1) to obtain the target probability; in the second category belonging to C N When the target probability is obtained by formula (2), the electronic device uses formula (3) to obtain the target probability when the second category belongs to bg. In this way, after obtaining the second category of the first object through the large language model, the accuracy of the target probability can be improved by correcting the prediction results of the large language model.

[0024] Optionally, the first image may also include a second object (which may correspond to a second food item, for example, Figure 1A The food image 101 (a boiled egg) is used as an example. After inputting the first image into the first model, the method further includes: obtaining the third category of the second object (e.g., boiled egg) and the confidence level of the third category (e.g., 0.96); outputting the second category of the first object, including: if the confidence level of the third category meets the confidence level requirement (e.g., the confidence level is greater than the first threshold 0.95), outputting the second category of the first object and the third category of the second object.

[0025] It is understood that in this embodiment of the application, for foods that the first model can identify, the electronic device can directly determine the food category; for foods that the first model cannot accurately identify, the electronic device can identify them through the second model. Thus, after obtaining the first image, the electronic device can obtain the categories of one or more foods through the first model, and / or through the second model; thereby, it can identify all food categories in the first image, and thus improve the accuracy of food information by improving the accuracy of food category identification.

[0026] Optionally, after outputting the second category of the first object and the third category of the second object, the following may also be included:

[0027] Display the first interface ( Figure 1AThe interface shown in b) includes a first image (food image 101) and first information (e.g., relevant information in pop-up window 102; the first information may also be displayed in other forms); wherein the first image displays a first object (e.g., a carrot), an identifier of the first object (e.g., ②), a second object (e.g., a boiled egg), and an identifier of the second object (e.g., ①); wherein the identifier of the first object partially or completely overlaps with the first object; the identifier of the second object partially or completely overlaps with the second object; the first information includes at least one of the following: the total calories of the first object and the second object, and / or the total nutritional composition of the first object and the second object; upon receiving a trigger operation on the identifier of the first object (e.g., a click operation on food identifier ②, or a sliding operation to switch food identifiers to food identifier ②), the second interface is displayed. Figure 1A The interface shown in d); the second interface includes second information (e.g., the relevant information in pop-up 104, the second information can also be displayed in other forms), the second information includes at least one of the following: the second category of the first object (e.g., carrot), the calories of the first object, and / or the nutritional components of the first object; wherein, the calories of the first object and / or the nutritional components of the first object are calculated based on the second category of the first object and the volume of the first object.

[0028] In this way, electronic devices can display total food information and individual food information in food images through the interface, so that users can understand the calories and nutritional components of the food they consume and adjust their diet accordingly.

[0029] Optionally, the first model includes a Ground-SAM model, and the second model includes a large language model. This allows for more accurate prediction of open-category food types based on the massive data and generalization capabilities of the large language model, reducing scenarios where food categories cannot be detected.

[0030] Optionally, before acquiring the first image, the method further includes: displaying a camera application interface (e.g., in response to an operation used to launch the camera application) Figure 1B The camera application interface (as shown in b) includes a first control (e.g., image recognition control 106); in response to a trigger operation on the first control, a third interface (e.g., ...) is displayed. Figure 1B The interface shown in c); the third interface includes multiple controls for identifying items (such as object recognition, text recognition, translation, barcode scanning, ID card, calorie control, etc.); in response to a selection operation on the control for identifying food (calorie control 107), the fourth interface is displayed (e.g., Figure 1BThe interface shown in d); the fourth interface includes a second control (a control for taking pictures, such as the shutter button 108 in the interface; it can also be a trigger operation for the volume button on the side of the device, which is not limited in this embodiment); obtaining the first image (food image 101) includes: obtaining the first image in response to the trigger operation for the second control.

[0031] In this way, users can take pictures of food and learn about the corresponding food information through the camera app, thus enriching the usage scenarios of the camera app.

[0032] Optionally, before acquiring the first image, the method further includes: in response to an operation for launching a health management application (e.g., a sports and health application), displaying the interface of the health management application (e.g., a sports and health application). Figure 1C The interface shown in c) of the health management application includes a third control (photo recognition control 112); the third control is used to access the camera; in response to a trigger operation on the third control, a fifth interface is displayed (e.g., Figure 1C The interface shown in d); the fifth interface includes a fourth control (a control for taking pictures, such as the shutter button 108 in the interface; it can also be a trigger operation for the volume button on the side of the device, which is not limited in this embodiment); obtaining the first image includes: obtaining the first image in response to the trigger operation for the fourth control.

[0033] In this way, users can take pictures of food and learn about the corresponding food information through health management applications, enriching the usage scenarios for obtaining food information.

[0034] Secondly, embodiments of this application provide an electronic device, which may also be referred to as a terminal device, terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. Terminal devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on.

[0035] The electronic device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform the method as described in the first aspect.

[0036] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method as described in the first aspect.

[0037] Fourthly, embodiments of this application provide a computer program product, which includes a computer program that, when run, causes a computer to perform the method as described in the first aspect.

[0038] Fifthly, embodiments of this application provide a chip system including at least one processor and a communication interface, the communication interface and at least one processor being interconnected via a line, the at least one processor being used to run computer programs or instructions to perform the method as described in the first aspect.

[0039] It should be understood that the second to fifth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0040] Figure 1A A schematic diagram of the interface of the image recognition method provided in the embodiments of this application;

[0041] Figure 1B A schematic diagram of an interface for an image recognition method provided in an embodiment of this application;

[0042] Figure 1C A schematic diagram of an interface for another image recognition method provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0044] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0045] Figure 4 A schematic flowchart illustrating the image recognition method provided in this application embodiment;

[0046] Figure 5 A schematic diagram illustrating the process of identifying food categories provided in an embodiment of this application;

[0047] Figure 6 A schematic diagram illustrating the process of obtaining the mask, food category, and confidence level of food provided in an embodiment of this application;

[0048] Figure 7 A schematic diagram illustrating the multimodal information fusion process of food provided in an embodiment of this application;

[0049] Figure 8 A schematic diagram illustrating the training process of the SAM model provided in the embodiments of this application;

[0050] Figure 9 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application. Detailed Implementation

[0051] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0052] 1. Electronic equipment

[0053] The electronic devices in this application embodiment may include handheld devices with image processing functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices (such as vehicle infotainment systems), wearable devices, electronic devices in 5G, or future evolutions of public land mobile communications. The embodiments of this application do not limit the scope of electronic devices in a network (PLMN).

[0054] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as hearing aids, glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0055] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects through communication technology, thereby realizing the intelligent interconnection of humans and machines and the interconnection of things.

[0056] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0057] 2. Other terms

[0058] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0059] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0060] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the display interface provided in the embodiments of this application is merely an example, and the display interface may include more or less content.

[0061] In some implementations, electronic devices can identify food based on image recognition capabilities. For example, a user can use an electronic device to take a photo or video of food, obtaining an image of the food, and the electronic device can then identify the food based on the image.

[0062] However, in a possible implementation, when using electronic devices to recognize food images, the electronic devices may fail to recognize the food, thus failing to provide users with relevant food information, such as the nutritional components of the food; thus affecting the user experience.

[0063] This is because, in a possible implementation, the electronic device is equipped with a pre-trained model; this model can be used to identify food categories in images; this model can be, for example, a classifier model or a deep learning model. The foods that this model can identify are related to the sample data during the model's pre-training process; that is, the types of food the model can identify are preset. For example, if the sample data includes multiple sample images of eggs, then the trained model can accurately identify the food "eggs".

[0064] However, due to the diversity of food, it is difficult for the training sample set of a model to cover all foods. This results in the model being unable to identify foods that are not included in the training dataset, or misidentifying untrained foods as foods that the model has been trained on. Consequently, electronic devices may fail to display or display incorrect food categories, and food information may be inaccurate due to inaccurate food category settings.

[0065] In view of this, embodiments of this application provide an image recognition method. After obtaining a first image, the first image is input into a first model, wherein the first image includes one or more food items. The first model performs fine-grained segmentation of the first image according to different food types and outputs one or more food categories and confidence scores. The training set of the first model may not include the food categories in the first image; for example, the first model may not be able to recognize the first food item in the first image. For the first food item, a second model capable of recognizing open-category items can be used for recognition. For example, the second model can be a large language model (LLM). The training data of the second model is much larger than that of the first model, and the generalization ability of the second model is greater than that of the first model. Here, the second model can be used to infer the first food item that has not been recognized, thereby outputting the food category of the first food item. In this way, a more accurate trained food category can be obtained through the first model; at the same time, the untrained food category can be predicted through the second model; thereby reducing scenarios where electronic devices cannot recognize food categories.

[0066] Let's combine the following... Figure 1AThe applicable scenarios of the image recognition method in the embodiments of this application are described, such as... Figure 1A As shown:

[0067] In this embodiment, the electronic device supports image recognition. The target application of the electronic device can recognize objects in preview images, objects in captured images, and / or objects in stored images. The target application can be a camera application, a health management application that supports photography, or an image recognition application, etc. Taking a camera application as an example, after the camera application is opened, the electronic device can display images captured by the camera. For example, the electronic device displays a food photography interface; the food photography interface can be, for example... Figure 1A The interface shown in Figure a includes a food image 101, which may include, for example, boiled eggs, carrots, sausages, and nuts.

[0068] It is understandable that users may want to know the total calories of their food intake or the calories of a single food item while dining; therefore, users may use electronic devices to photograph one or more food items. For ease of explanation of the technical solutions in this application's embodiments, this application's embodiments use a single video image including multiple food items as an example. The food image 101 in this application's embodiments may also include a single food item; this application's embodiments do not impose any limitations on this.

[0069] After the electronic device completes the food image recognition, it can display the food information. In one possible implementation, the electronic device can display the total food information for multiple foods; in another possible implementation, the electronic device can also display the food information for any one of the multiple foods sequentially.

[0070] Specifically, after the electronic device completes the recognition of the food image, it can display the total food information for multiple foods, such as... Figure 1A As shown in interface b. Figure 1A In the interface shown in b, the electronic device can display food image 101 and pop-up window 102; each food in food image 101 corresponds to a food label, which can be used to mark and distinguish food. The food label can be a circular label with numbers, for example, food labels are ①, ②, ③, ④, etc.; the correspondence between food and food label can be: boiled egg is labeled as ①, carrot is labeled as ②, nuts are labeled as ③, sausage is labeled as ④, etc.

[0071] Pop-up window 102 can be used to explain food information for one or more foods in food image 101. Pop-up window 102 may include food name, calorie information, nutritional value information, and nutritional recommendations. The nutritional value information in pop-up window 102 can be the combined nutritional value of multiple foods in the current food image; for example, total energy, total protein content, total fat content, and total carbohydrate content. The nutritional recommendations in pop-up window 102 can be comprehensive nutritional recommendations derived from the combination of the nutritional components of one or more foods.

[0072] In this way, users can use electronic devices to recognize food images and thus understand information such as the total calories and total nutrients of the food they consume, so as to control and adjust their diet.

[0073] Optionally, if the text of the food nutrition recommendations is too long and the pop-up window 102 cannot display all the food nutrition recommendations, the electronic device can display more information based on the swiping operation in the pop-up window 102; for example, in response to the up and down swiping operation, the electronic device can scroll to display more details of the food; or, for example, in response to the left and right swiping operation, the electronic device can turn pages to display more details of the food.

[0074] Optionally, the embodiments of this application do not limit the specific implementation of food labeling. For example, food labels can be circular labels, rectangular labels, labels with letters, labels with text, labels of different colors, and border labels, etc.

[0075] Optionally, in this embodiment of the application, the electronic device can also display the food information of any one of the multiple foods in the food image 101 and the total food information more intuitively in the form of a list. For example, the row header of the list can be multiple foods and the total, and the column header of the list can be multiple nutrients; or, the column header of the list can be multiple foods and the total, and the row header of the list can be multiple nutrients.

[0076] exist Figure 1A Based on the interface shown in b, the electronic device can further display food information for any one of multiple food items. For example, when a trigger operation is received for a food label, the electronic device can display the food information corresponding to that food label. For instance, in... Figure 1A In the interface shown in b, the electronic device receives a trigger operation for the food identifier ① for boiled eggs, and the electronic device enters the following state: Figure 1A The interface shown in c is shown in the image. Figure 1AIn the interface shown in c, the electronic device displays a pop-up window 103 and highlights the selected food label (e.g., food label ①). The pop-up window 103 may include the food information corresponding to the selected food label; for example, if food label ① is selected, the food corresponding to food label ① is a boiled egg, and the pop-up window 103 displays the food information of the boiled egg, including the food name, calorie information, nutritional value information, and nutritional advice information.

[0077] Highlighting can be, for example, when the color of a food label changes, such as... Figure 1A In the interface shown in Figure c, food label ① is a different color from other food labels to indicate that food label ① is selected, while other food labels are optional; alternatively, the size of the food labels may differ, for example, food label ① is larger than the size of other food labels, to indicate that food label ① is selected, while other food labels are optional. Alternatively, the electronic device may use other methods to highlight the selected food label.

[0078] In this way, users can identify food images through electronic devices to understand the calories, nutritional components, and other information of any food they consume, so as to control and adjust their diet.

[0079] Optionally, in some embodiments, the electronic device may display different food information based on the selection of different food tags. For example, in Figure 1A In the interface shown in c, when a selection operation is received for food label ②, the electronic device can display as follows: Figure 1A The interface shown in d is shown in the image. Figure 1A In the interface shown as d, the electronic device displays a pop-up window 104 and highlights the selected food label (e.g., food label ②). The pop-up window 104 may include the food information corresponding to the selected food label; for example, if food label ② is selected, and the food corresponding to food label ② is carrot, the pop-up window 104 displays food information about carrots, including food name, calorie information, nutritional value information, and nutritional advice information.

[0080] Optionally, in other embodiments, the electronic device can switch between displaying different food information based on left and right swiping. For example, in Figure 1A In the interface shown in c, when a right swipe operation is received, the electronic device can highlight the selected food icon ④; for example, the electronic device can display a pop-up window to show food information about sausages. Figure 1A (Not shown in the image). Or, in Figure 1A In the interface shown in Figure c, when a left swipe operation is received, the electronic device can display as follows: Figure 1A The interface shown in d is shown in the image. Figure 1AIn the interface shown in d, the electronic device displays pop-up window 104, which highlights food label ② and displays the food information corresponding to food label ②.

[0081] Optionally, in this embodiment, the electronic device can display the food information of any one of the multiple food items individually, without displaying the combined food information of the multiple food items. For example, after the electronic device completes the recognition of the food image, it can display food image 101, which displays multiple food icons, all of which are unselected; furthermore, the electronic device may not display pop-up window 102. Upon receiving a selection operation for any food icon, the electronic device can display as follows: Figure 1A The interface shown in c or as shown in the image Figure 1A The interface shown in d is not limited in this embodiment.

[0082] The following is combined Figure 1B and Figure 1C This section introduces two ways to access the food category recognition function. Figure 1B This diagram illustrates the interface of an electronic device that enables food category recognition based on a camera application. Figure 1C The diagram shows the interface of an electronic device that enables the food category recognition function based on a health management application.

[0083] In the first possible implementation, see Figure 1B Electronic devices can display a desktop, such as Figure 1B The interface shown in Figure 'a' may include a camera application 105 on the desktop. When a trigger operation is received for the camera application 105, the electronic device can enter... Figure 1B The interface shown in b is a standard shooting interface. For example, this interface includes multiple controls that support different shooting modes, such as aperture, night scene, portrait, photo, video, movie, and professional. Figure 1B The interface shown in b may also include an image recognition control 106. When a trigger operation is received for the image recognition control 106, the electronic device can display the image recognition interface, which may include multiple image recognition modes, such as... Figure 1B The interface shown in 'c' displays an object recognition control, a text recognition control, a translation control, a barcode scanning control, an ID card control, and a calorie control, among others.

[0084] In some embodiments, when entering the initial interface of the image recognition interface, the default mode control of the electronic device is a barcode scanning control. At this time, the user can trigger the calorie control, and in response to the user's triggering operation, the electronic device can display as follows: Figure 1B The interface shown in d. In Figure 1BIn the interface shown as d, the calorie control 107 is selected. Upon receiving a trigger operation on the shutter button 108, the electronic device can capture an image and identify the food in the image. The identification process and results can be referenced as follows: Figure 1A .

[0085] In the second possible implementation, see Figure 1C Electronic devices can display a desktop, such as Figure 1C The interface shown in Figure 'a' may include a fitness and health application 109 on the desktop. When a trigger operation is received for the fitness and health application 109, the electronic device can enter... Figure 1C The interface shown in b includes multiple controls for exercise and / or diet; for example, a diet diary control 111. When a trigger operation is received for the diet diary control 111, the electronic device can display a calorie recording interface, as shown in Figure 111. Figure 1C The interface shown in 'c' can include dietary records for three meals a day, dietary analysis, and nutritionist advice.

[0086] In some embodiments, the electronic device can obtain the food category and estimated weight input by the user, and calculate the food's calories and nutritional value; alternatively, the electronic device can obtain the food's calories and nutritional value based on food image recognition analysis. For the second embodiment, the electronic device can display a photo recognition control 112, such as... Figure 1C The interface shown in c is as follows. When a trigger operation is received for the identification control 112, the electronic device can display the following: Figure 1C The interface shown as d in the figure includes a shutter button 113, which can be used to capture images of food. Upon receiving a trigger operation on the shutter button 113, the electronic device can capture an image and identify the food in the image. The identification process and results can be referred to as follows: Figure 1A .

[0087] Alternatively, electronic devices can also access the system through other means. Figure 1A The scenario shown is not limited to this example. This application's embodiments do not impose any restrictions on this.

[0088] To better understand the embodiments of this application, the structure of the electronic device of this application is described below:

[0089] Figure 2A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and an embedded secure element (eSE) chip 196, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0090] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0091] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. In this embodiment, processor 110 is used to execute information processing-related procedures during image recognition.

[0092] Electronic device 100 implements display functions through GPU, display screen 194, and application processor.

[0093] The display screen 194 is used to display images, videos, etc. In this embodiment, the electronic device can display food images and food information through the display screen 194. For example, the display screen 194 can display... Figure 1A , Figure 1B and / or Figure 1C The interface shown.

[0094] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0095] Camera 193 is used to capture still images or videos. In this embodiment, camera 193 can be used to acquire images of food.

[0096] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0097] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.

[0098] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0099] Figure 3 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0100] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system may include: an application layer, an application framework layer, a hardware abstraction layer (HAL), and a kernel layer, where the kernel layer may also be called the driver layer.

[0101] The application layer can include a series of application packages.

[0102] like Figure 3 As shown, the application package may include a camera app and a fitness and health app. Both apps can identify food items.

[0103] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0104] like Figure 3 As shown, the application framework layer may include a window manager, content provider, resource manager, view system, notification manager, camera access interface, etc.

[0105] The camera access interface allows applications to manage the camera and access the camera device, such as managing the camera to capture images.

[0106] The hardware abstraction layer can contain multiple library modules, such as camera library modules and algorithm library modules. The Android system can load the corresponding library modules for the device hardware, thereby enabling the application framework layer to access the device hardware. In this embodiment, the algorithm library may include relevant models for identifying food categories and relevant algorithms for calculating food information.

[0107] The kernel layer is the layer between hardware and software. It drives the hardware, enabling it to function. The kernel layer may contain camera device drivers, display drivers, audio drivers, etc., but this embodiment does not limit this. The hardware layer may include various types of sensors, such as time-of-flight (TOF) cameras for image capture.

[0108] For example, a camera device driver can drive camera-type sensors in the hardware layer to perform tasks such as image capture.

[0109] The following is combined Figure 3This paper describes possible implementations of the image recognition method in the embodiments of this application.

[0110] Taking food recognition via a camera application as an example, when performing food image recognition, the camera application can call the camera access interface. The camera access interface manages the camera hardware abstraction layer to acquire food images through the camera driver. The acquired food image is then processed by the relevant model for food category identification in the algorithm library of the hardware abstraction layer, as described in this application embodiment, to obtain the food category. After further processing by the relevant algorithm for calculating food information, as described in this application embodiment, the food information is obtained. Subsequently, the food category and food information are reported to the camera application through the camera access interface. The electronic device can then display the food image and its corresponding food category and information in the camera application interface. Figure 1A As shown.

[0111] The image recognition method of this application will be described in detail below through specific embodiments. The following embodiments can be implemented in combination with each other or independently, and the same or similar concepts or processes may not be described again in some embodiments.

[0112] It should be noted that in real-world scenarios, there are many types of food, and the overall and detailed differences between them vary greatly. Furthermore, the proportion of different foods in a food product varies, and all of these factors can affect the accuracy of food information.

[0113] For example, there are significant overall differences between porridge and stir-fried dishes. Electronic devices can train models based on the characteristics of the food itself to enable the models to distinguish between different types of food. The training samples can be coarse-grained sample images. Furthermore, for similar types of food, electronic devices need to mine local features between different foods to differentiate them. For example, similar foods could be preserved egg and lean pork porridge and rice and lotus seed porridge. In this case, the electronic device needs to perform fine-grained food recognition to more precisely distinguish between preserved egg and lean pork porridge and rice and lotus seed porridge based on ingredients such as preserved egg, meat, and lotus seeds. Therefore, to improve the accuracy of food recognition, this application embodiment can employ fine-grained image recognition technology. The process of fine-grained food image segmentation will be explained later in this application embodiment, and will not be elaborated here.

[0114] Furthermore, even for the same two food items, differences in the specific gravity or volume of their constituent ingredients can lead to variations in their nutritional composition and calorie content. In this embodiment, to improve the accuracy of food information, the electronic device can calculate the nutritional composition and calorie content of the food based on its volume and the food information per unit volume. However, due to the wide variety of food types, the electronic device may be unable to predict untrained foods, resulting in deviations in the food information per unit volume during calculation, ultimately affecting the accuracy of the food information output by the electronic device.

[0115] Therefore, based on the above analysis, this application provides an image recognition method that can classify food into fine-grained categories. At the same time, it uses a model with the ability to identify open-category items to identify food, expands the types of food that electronic devices can identify, improves the accuracy of food category identification, and thus improves the accuracy of food information.

[0116] The following will combine Figure 4 The usage process of the image recognition method provided in the embodiments of this application will be described. The specific implementation of the image recognition method may include:

[0117] S401, The electronic device obtains a first image; the first image may include one or more food items.

[0118] The first image can be Figure 1A The food image mentioned above. The first image can be the food image recognized by the electronic device based on the preview interface after the camera is activated (the preview interface can display the camera's captured image in real time; in the preview interface, the electronic device has not received any photo-taking or video-recording operations via the shutter button, for example...). Figure 1A (See interface shown in 'a'). The first image can also be a food image captured by the electronic device in response to a shutter button press after the camera is activated. Alternatively, the first image can be a stored food image exported from a gallery application.

[0119] The first image can be an RGB format image.

[0120] S402, The electronic device inputs the first image into the first model to obtain information about the first image; wherein, the information about the first image may include: multiple masks, multiple food bounding boxes, multiple food categories, and multiple food category confidence levels.

[0121] The first model can be used to classify the positions of each food item in the first image and to predict the food category of each food item. The position of each food item in the first image can be represented by a mask and a border.

[0122] A mask can be used to block or select images of different food items in a first image. For example, if the first image includes a first food item and a second food item, the electronic device can mask the position of the first food item in the first image to obtain a mask for the first food item; and mask the position of the second food item in the first image to obtain a mask for the second food item. For details, please refer to... Figure 5 The mask in the first image can include four types of food: boiled egg, carrot, nuts, and sausage. For example, the mask for a boiled egg can mark the position of the boiled egg in the first image.

[0123] Understandably, for a non-whole food like carrot chunks, the first model can identify one or more masks for that food category. Correspondingly, the first food can correspond to the mask of the first carrot chunk, the second food can correspond to the mask of the second carrot chunk, and so on. Subsequent electronic devices can then label multiple carrot chunks with different food tags. Figure 1A (Not shown in the image), block-shaped areas of the same food category can also be combined and labeled with the same food label, for example, Figure 1A In the interface shown in b, the area containing the carrot is marked with food label ②. This application's embodiments do not impose limitations on this.

[0124] Borders can be used to define the position of different food items within the first image or mask. For example, the border can be a rectangular border; see reference. Figure 5 In the mask, the top, bottom, left, and rightmost points of any food item are all located on the food item's propsal. For example, taking a carrot as the first food item, refer to... Figure 5 The border of the first food item can correspond to the area where the carrot is located in the first image.

[0125] The food category can be the name of the food item, such as boiled eggs, carrots, nuts, sausages, etc. See the food category list for reference. Figure 5 The text portion of the Chinese mask.

[0126] Confidence score measures the accuracy of the first model's predictions of food categories. The confidence score is positively correlated with the accuracy of the predicted food categories; that is, the higher the confidence score, the more reliable the predictions of the food categories, and vice versa. The confidence score for food categories can be found by referring to... Figure 5 The numerical part corresponding to the mask.

[0127] It is understandable that the first model can achieve the above functions based on three sub-models. The first model may include a first sub-model, a second sub-model, and a third sub-model. The usage of the first model can be as follows: Figure 6 As shown:

[0128] The first sub-model can be used to identify food categories. For example, the first sub-model can be GroundingDINO. The GroundingDINO model can detect the category and confidence level of target objects in a specific region. Specifically, the first image can be input into the GroundingDINO model, which performs image recognition on the candidate bounding box (proposal) regions of each food item in the first image to obtain the food category and confidence level corresponding to each bounding box.

[0129] The second sub-model can be used to convert the first image into a mask for segmenting food regions at different granularities. For example, the second sub-model can be a SAM model. The SAM model can capture the relationships between various pixel regions (pixels or combinations of pixels) in the first image using deep learning. It generates attention weights by calculating the correlation between each pixel region and other pixel regions, thereby distinguishing food and obtaining masks of different granularities.

[0130] It is understandable that before inputting the first image into the first sub-model, it is necessary to mark the candidate boxes of each food item in the first image. The process of generating candidate boxes can rely on the third sub-model. Here, the bounding box can be understood as the more accurate bounding area of ​​the image obtained by the electronic device based on the proposal.

[0131] The third sub-model can be used to add candidate bounding boxes for each food item onto the first image. The third sub-model can select an algorithm model for the candidate regions, such as a selective search model and / or an edge box algorithm model. After the first image is input into the third sub-model, it can output candidate region information.

[0132] Specifically, see below. Figure 6 The usage process of the first model will be explained in detail. For example... Figure 6 As shown:

[0133] After obtaining the first image, the electronic device can input the first image into the candidate region selection algorithm model. The candidate region selection algorithm can use selective search and / or edge box algorithms to calculate the first image and obtain multiple candidate boxes corresponding to multiple food items in the first image.

[0134] On one hand, after obtaining candidate bounding boxes for each food item, the first image and candidate bounding box information are input into the GroundingDINO model to obtain each bounding box in the first image, as well as the food category and confidence score corresponding to the bounding box. The GroundingDINO model can obtain an image feature of the candidate bounding box region by averaging the attention weights of all candidate bounding box regions. The image feature can be any one of the following: color feature, texture feature, or edge feature. Based on the image feature, the coordinates of the bounding box containing that image feature are determined.

[0135] On the other hand, the electronic device inputs the first image and bounding box coordinates into the SAM model, respectively. Specifically, the image encoder (e.g., a Vision Transformer (VIT)) obtains the first image and converts it into an image vector (e.g., vector 1); the cue word encoder obtains the bounding box coordinates and converts them into cue word vectors (e.g., vector 2). After inputting both vector 1 and vector 2 into the mask decoder, the mask decoder outputs a mask of the first image.

[0136] Understandably, the SAM model can output multiple mask segmentation results; multiple mask segmentation results can correspond to multiple mask segmentation methods with different granularities, and each mask segmentation result corresponds to a mask segmentation confidence level; the SAM model can output the mask segmentation result corresponding to the highest mask segmentation confidence level to improve the accuracy of mask segmentation results.

[0137] Optionally, the first model may also include a first sub-model and a second sub-model, but the first model does not include a third sub-model. For example, the third sub-model may be, for instance, a proposed region model, and the first model is a model obtained by jointly or not jointly training the first and second sub-models. The first model may be, for example, Ground-SAM. This application does not impose any limitations on this.

[0138] At this point, the electronic device obtains the mask of the first image, the food category of each food item in the first image, and the confidence level based on the first model.

[0139] S403. The electronic device determines whether the confidence level of the food category is less than the first threshold.

[0140] The first threshold could be, for example, 0.95.

[0141] Confidence level can be used to measure the accuracy of the prediction results. When the confidence level of a food category is less than the first threshold, it indicates that the food category has a low match with the food, and the accuracy of using this food category to describe the food is low; it can be considered that the food category output by the first model is incorrect or has low accuracy. In this case, the food category needs to be re-predicted through subsequent steps S404-S406. Conversely, when the confidence level of a food category is greater than or equal to the first threshold, it indicates that the food category has a high match with the food, and the accuracy of using this food category to describe the food is high; it can be considered that the food category output by the first model is correct or has high accuracy. In this case, the food category needs to be output through subsequent step S406.

[0142] Reference Figure 5 The food categories and confidence levels for each food item were as follows: boiled eggs 0.96; carrots 0.35; nuts 0.95; and sausages 0.97. Among these, the confidence levels for boiled eggs, nuts, and sausages were all greater than or equal to the first threshold; the confidence level for carrots was less than the first threshold.

[0143] S404. If the confidence level of the food category of the first food is less than the first threshold, the electronic device inputs the image of the first food, the mask of the first food, and the first preset prompt into the third model to obtain the fused vector (token).

[0144] The third model can be used to integrate information between multimodal information; multimodal information may include, for example, an image of the first food (RGB image), a mask of the first food (the position and boundaries of the first food), and a first preset prompt word; the image of the first food may be, for example, an image of a region defined in a first image based on the border of the first food. The third model may be, for example, a mask fusion model (mask Q-former).

[0145] Understandably, the reason for not directly using a large language model to identify food in the first image after obtaining it is that the visual features of the first image cannot directly interact with the large language model. These visual features include, for example, color, texture, shape, brightness, and darkness. The Mask Q-former model, however, can extract visual features from a frozen visual model using a learnable set of query vectors and establish a connection between the visual model and the large language model. This structure aims to learn the most relevant visual representations to the text and make these representations interpretable by the large language model. Therefore, the Mask Q-former model is introduced here as a bridge between visual features and the large language model.

[0146] The first preset prompt can be text used to describe the question and answer. For example, the first preset prompt can be "What is a segmentation mask?" or "What is a segmentation mask?" The first preset prompt can be used to guide the third model to learn the visual features related to the first preset prompt in the mask region and / or bounding box region.

[0147] Specifically, see below. Figure 7 Let me introduce the process of the third model fusing the multimodal information of the first food, such as... Figure 7 As shown:

[0148] The third model includes mask query vectors, context query vectors, mode query vectors, self-attention layers, cross-attention layers, and feedforward layers. The cross-attention layers can include mask cross-attention layers and context cross-attention layers.

[0149] The mask query vector, content query vector, and pattern query vector can all be pre-trained vectors. The mask query vector is used to calculate the attention weights of the mask for the first food item. The content query vector represents the contextual information of the mask for the first food item, which can be, for example, the area remaining after removing the mask area from the candidate box area (or border area) of the first food item. The content query vector is used to calculate the contextual weights of the mask for the first food item to improve the accuracy of the mask text label. The pattern query vector is used to learn prior information, enabling the model to maintain generality while exhibiting better recognition capabilities for specific food categories.

[0150] Reference Figure 7 The input vectors of the third model may include vectors 3, 4, and 5. Vector 3 may include the image vector of the candidate box region of the first food and the position vector of the candidate box; vector 4 may include the mask vector of the first food; and vector 5 may include the prompt word vector of the first preset prompt word.

[0151] Vector 3 can be obtained by: obtaining the candidate bounding box information and the image of the first food; inputting the image of the first food and the candidate bounding box information of the first food into the visual encoder (contrastive language-image pre-training, CLIP) to obtain vector 3, which may include the image vector and the position vector of the first food.

[0152] If the confidence level of the food category of the first food is less than a first threshold, the electronic device can fuse multimodal information related to the first food. (See [link to relevant documentation]). Figure 7 :

[0153] The mask query vector, content query vector, pattern query vector, and vector 5 are input into the self-attention layer to obtain a fusion vector 1 containing a weighted mask query vector, a weighted content query vector, and vector 5. Fusion vector 1 integrates information between different modalities. It is understandable that, in representing the characteristics of the first food item, the importance of the mask region features is greater than the importance of the mask region's context; therefore, the weight of the mask query vector can be greater than the weight of the content query vector.

[0154] The fused vectors 1, 3, and 4 are input into the cross-attention mechanism layer to obtain weighted vectors 6 and 7.

[0155] For example, vector 3 may include the image vector of the masked region and the image vector of the context region, and vector 4 may include the mask vector of the masked region and the mask vector of the context region. In the cross-attention layer, the cross-attention layer extracts vectors 6 and 7 from vectors 3 and 4. Vector 6 may be a fusion vector of the masked region in vector 3 and the masked region in vector 4; vector 7 may be a fusion vector of the context region in vector 3 and the context region in vector 4.

[0156] Fusion vector 1 can be used to weight vectors 6 and 7. For example, using the mask query vector in fusion vector 1 to weight vector 6 results in a weighted vector 6. Using the content query vector in fusion vector 1 to weight vector 7 results in a weighted vector 7. Fusion vector 2 includes weighted vector 6, weighted vector 7, and vector 5.

[0157] The third model uses a feedforward network layer to reduce the dimensionality of the fused vector 2, resulting in a fused vector. The dimension of the fused vector matches the dimension of the input vector of the large language model.

[0158] At this point, the electronic device fuses the multimodal information of the first food based on the third model and obtains a feature vector suitable for the second model.

[0159] S405 The electronic device inputs the fused vector and the second preset prompt word into the second model to obtain the food category and conditional probability of the first food.

[0160] The second model can be a large language model, which can transform the fused input vector into a corresponding food category using conditional probabilities. For example, the second model can output the food category c of the first food and the conditional probability P(c) of the food category c. This conditional probability can be understood as the probability of predicting the food category as the first food. If the conditional probability is greater than the first and second thresholds, it can be determined that the food category of the first food is accurate.

[0161] The second preset prompt is used to describe the text of the question and answer. For example, the second preset prompt could be "What is a segmentation mask?" or "What is a segmentation mask?" The second preset prompt can be used to guide the second model to learn the features of the mask region related to the second preset prompt. The second preset prompt can be the same as or different from the first preset prompt.

[0162] P(c) can be calculated using the beam search algorithm of the second model, or it can be calculated using a greedy algorithm of the second model.

[0163] The conditional probability that the first food item is food category c can be satisfied by the following formula:

[0164]

[0165] Here, 'c' can be understood as the food category corresponding to the mask of the first food item; the fused vector may include multiple tokens, c j-1 Let N be the j-th token of food category c, and N be the number of tokens. This formula can be understood as, given the previous conditions c0,…,c… j-1 Under the condition that c occurs, the probability of c occurring.

[0166] Understandably, large language models learn rich linguistic knowledge through massive amounts of sample data and can capture the deep structure and semantic relationships of language through complex neural network architectures. Based on this, large language models can predict items in open categories. For example, large language models can predict the category of a first food item based on its multimodal information.

[0167] It should be noted that after the fused vectors are input into the large language model, the large language model converts multiple tokens into corresponding food categories in the form of conditional probabilities. The large language model is pre-trained on massive amounts of data, and its parameters can be frozen when training the model for food category recognition. There is a certain degree of data category distribution deviation between the training data of the large language model and the training set for food categories. The new category probabilities output by the large language model may not perfectly match the test set for food category recognition. Here, the conditional probabilities output by the large language model can be corrected to eliminate this deviation. In one possible implementation, the electronic device can use step S406 to correct the conditional probabilities.

[0168] S406. Use the confidence level of the first food output by the first model to correct the conditional probability output by the second model, and obtain the food category and target probability of the first food.

[0169] In this embodiment of the application, the matching degree between the output of the second model and the food category identification can be enhanced by fusing the confidence level of the first model output with the conditional probability of the second model output.

[0170] Specifically, target probability It can satisfy the following formula:

[0171]

[0172] Where p is the image of the first food item, c is the food category corresponding to the mask of the first food item, and c ′ P represents any one of the multiple food categories identified by the first model. c (p,c) represents the conditional probability that the food category of the first food item output by the second model is c. C represents the confidence level of the food category c for the first food output by the first model; λ is a preset index used to characterize the relative importance of confidence level and conditional probability in the target probability, λ can be, for example, 2 / 3; C B This represents the set of food categories corresponding to the training set of the first model; bg represents the background category, and C... N Not belonging to C B And a collection of other food categories in bg.

[0173] Food category c belongs to set C. B In this case, the electronic device calculates the target probability using formula (1). This applies when food category c belongs to set C. N In the case where food category c belongs to set bg, the electronic device uses formula (2) to calculate the target probability.

[0174] Specifically, the confidence level obtained by the first model It can satisfy the following formula:

[0175]

[0176] Among them, l O (p,c) is used to measure the similarity between the image of the first food and food category c. O (p,c′) is used to measure the similarity between the image of the first food and any one of the multiple food categories c′.

[0177] The two similarity scores can be represented by the following formulas:

[0178]

[0179] in, Let t be the image vector of the first food item. c Let c be the text vector for food category c.

[0180] Understandably, after correcting the two probability values ​​of the two models to obtain the target probability, the food category of the first food can be output based on the relationship between the target probability and the third threshold. For example, if the target probability of food category c is greater than the third threshold, the food category of the first food is output as c; where the third threshold can be greater than the first threshold.

[0181] In this way, electronic devices can expand the range of food categories they can recognize based on large language models, thereby improving the richness and accuracy of food categories and reducing scenarios where food categories cannot be recognized.

[0182] And / or, after step S403, the method further includes:

[0183] S407. If the confidence level of the food category of the second food is greater than or equal to the first threshold, the electronic device obtains the food category of the second food.

[0184] It is understandable that if the confidence level of the food category of the second food is greater than or equal to the first threshold, the output of the first model can be considered accurate, and the output of the first model can be used directly here.

[0185] For example, refer to Figure 5 In the output of the first model, the confidence scores for boiled eggs are 0.96, nuts are 0.95, and sausages are 0.97. Since the confidence scores are all greater than the first threshold, the electronic device can identify the food category of the second food relatively accurately.

[0186] S408. The electronic device displays the food category and food information of one or more food items in the first image.

[0187] For example, after the electronic device identifies food in the first image, it can display the food categories and information of multiple food items, such as... Figure 1A The interface shown in b is an example. Electronic devices can also display a food category and food information, for example... Figure 1A The interface shown in c is or Figure 1A The interface shown in d is an example. Food information may be displayed in pop-ups 102, 103, or 104.

[0188] It is understood that the embodiments of this application illustrate the food category recognition process using an example where the first image includes multiple food items. In some embodiments, the food image captured by the electronic device may include a single food item. In this case, the electronic device may execute steps S401-S406 and S408, for example, to identify the food category in the food image. Alternatively, the electronic device may execute steps S401-S403, S407, and S408, for example, to identify the food category in the food image. The embodiments of this application do not impose limitations on this.

[0189] It should be noted that while the electronic device uses a camera to capture the first image, it can also use, for example, a time-of-flight (TOF) camera to capture the depth information of the first image, thus obtaining a depth map of the first image. In step S408, the electronic device can obtain the food categories of all the food items in the first image. The electronic device can calculate the volume of any food item in the first image based on the depth map of the first image. The electronic device can query the food information of a food category per unit volume from a preset list for any food category; and calculate the corresponding volume of food information based on the volume of that food; the electronic device can sum the food information of all the food items in the first image to obtain the food information corresponding to the first image. The preset list may include information such as calories and nutritional components corresponding to the food categories.

[0190] In this way, the electronic device can accurately calculate information such as calories and nutritional components of the food in the first image based on the accurate identification results of the food category, so as to facilitate the user's health management.

[0191] The above embodiments illustrate the process of electronic devices identifying food categories. The following, in conjunction with... Figure 8 The training process of the SAM model in the first model to achieve fine-grained partitioning is briefly explained.

[0192] It should be noted that in the embodiments of this application, the device used to train the model is not the same device as the device used to identify food categories; for example, the device that performs steps S401-S408 can be a portable electronic device with low computing power, such as a mobile phone or tablet; the device that performs the model training process can be an electronic device with high computing power, large memory, and the ability to process massive amounts of data, such as an edge server.

[0193] S801. Obtain the training sample set, which includes multiple sample images and corresponding prompts for the sample images.

[0194] The training sample set includes multiple training samples. Each training sample may include a sample image and a corresponding prompt word. The sample images can be images of various food categories. The prompt word corresponding to the sample image can be obtained using any of the following methods: mask, point, scratch, box, and / or text, or a combination of multiple types. The sample images can be multiple food images; the prompt word can be location information obtained by clicking, swiping, or selecting on the food image, or text information after inputting text.

[0195] S802. Use the training sample set to train the first model to be trained, and obtain the first model.

[0196] Reference Figure 8 The sample image is input into the image encoder to obtain the image vector (vector 8); the corresponding prompt word of the sample image is input into the prompt word encoder to obtain the prompt word vector (vector 9); vector 8 and vector 9 are input into the mask decoder so that the mask decoder learns the mapping relationship between the prompt word and the image element; after training, the mask of the sample image at different granularities and the confidence of the mask at that granularity can be obtained, for example, the mask and the corresponding confidence of the first granularity; the mask and the corresponding confidence of the second granularity; the mask and the corresponding confidence of the third granularity, and so on, and the mask corresponding to the highest confidence is output.

[0197] Specifically, the prompt word encoder performs different transformations on the input prompt words according to their type, resulting in different transformation results. For example:

[0198] If the input cue words are of a sparse type, such as point or box, then sincos positional encoding and learned category encoding are used as encoders to encode point and / or box into positional vectors.

[0199] If the input prompt is a text-type prompt such as "text", then the CLIP pre-trained text encoder is used to encode the text into a text vector.

[0200] If the input prompt word is a dense type such as "mask", then convolutional embedding (conv) is used to encode the mask into a mask vector (also known as dense embeddings). The mask vector is then summed pixel-by-pixel with the sample image vector, and the mask vector and the sample image vector are concatenated. The mask prompt word can be used to roughly indicate the target region to be segmented.

[0201] After training, the first model to be trained can gradually converge or reach the required number of iterations. At this point, the error between the output of the first model and the correct value is relatively small, and relatively accurate mask segmentation can be performed, thus obtaining the first model.

[0202] It should be noted that the embodiments of this application are applicable to the identification process of the aforementioned food categories, and can also be applied to other scenarios. For example, the identification of various items and the detection of new targets. Another example is object recognition in open scenes. Yet another example is text classification and entity recognition. Yet another example is anomaly detection and behavior recognition, etc. The embodiments of this application do not impose any limitations on these.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0204] The image recognition method according to the embodiments of this application has been described above. The apparatus for performing the above-described image recognition method, provided in the embodiments of this application, will now be described. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced in each other, and the related apparatus provided in the embodiments of this application can perform the steps in the above-described image recognition method.

[0205] like Figure 9 As shown, the image recognition device 900 can be used in communication equipment, circuits, hardware components, or chips. The image recognition device includes a display unit 901 and a processing unit 902. The display unit 901 supports the display steps performed by the image recognition device 900; the processing unit 902 supports the information processing steps performed by the image recognition device 900.

[0206] In a possible implementation, the image recognition device 900 may also include a communication unit 903. Specifically, the communication unit supports the image recognition device 900 in performing data transmission and data reception steps. The communication unit 903 may be an input or output interface, pins, or circuitry, etc.

[0207] In a possible embodiment, the image recognition device may further include a storage unit 904. The processing unit 902 and the storage unit 904 are connected via a line. The storage unit 904 may include one or more memories, which may be devices in one or more devices or circuits used for storing programs or data. The storage unit 904 may exist independently and be connected to the processing unit 902 of the image recognition device via a communication line. Alternatively, the storage unit 904 may be integrated with the processing unit 902.

[0208] Storage unit 904 can store computer-executable instructions for the methods in the terminal device, so that processing unit 902 can execute the methods in the above embodiments. Storage unit 904 can be a register, cache, or RAM, etc., and storage unit 904 can be integrated with processing unit 902. Storage unit 904 can be read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and storage unit 904 can be independent of processing unit 902.

[0209] The image recognition method provided in this application can be applied to electronic devices with communication functions. Electronic devices include electronic devices in general; the specific device form of an electronic device can be referred to the above-described related information, and will not be repeated here.

[0210] This application provides an electronic device, which includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform the above-described method.

[0211] This application provides a chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to execute the above-described method. Its implementation principle and technical effects are similar to the related embodiments described above, and will not be repeated here.

[0212] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0213] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0214] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0215] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0216] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image recognition method, characterized in that, Applied to electronic devices, including: Obtain a first image; the first image includes a first object; The first image is input into the first model to obtain the first category of the first object, the confidence level of the first category, and the mask of the first object; If the confidence level of the first category does not meet the confidence level requirement, a fusion vector of the first object is obtained based on the mask of the first object and the first image. The fusion vector of the first object is input into the second model to obtain the second category of the first object; wherein, the second model is a large language model with open category recognition capability, and the training samples of the first model are less than the training samples of the second model; Output the second category of the first object.

2. The method according to claim 1, characterized in that, The process of obtaining a fusion vector of the first object based on the mask of the first object and the first image, and inputting the fusion vector of the first object into a second model to obtain a second category of the first object includes: Obtain the image of the first object in the first image and the bounding box of the first object; wherein, the image of the first object is an image cropped from the first image based on the bounding box of the first object; the bounding box of the first object is obtained based on the first model; The image and mask of the first object are input into the third model to obtain the fusion vector of the first object; The fusion vector of the first object is input into the second model to obtain the second category of the first object and the conditional probability of the second category. The output of the second category of the first object includes: If the conditional probability of the second category meets the conditional probability requirement, output the second category of the first object.

3. The method according to claim 2, characterized in that, The step of inputting the image and mask of the first object into the third model to obtain the fusion vector of the first object includes: The image of the first object is converted into a first image vector based on a first image encoder, the mask of the first object is converted into a first mask vector based on a mask encoder, and a preset prompt word is converted into a first prompt word vector based on a first prompt word encoder; wherein, the preset prompt word is used to guide the third model to fuse the first image vector and the first mask vector; the third model includes the first image encoder, the mask encoder, and the first prompt word encoder; The first image vector and the first mask vector are fused based on the mask query vector, content query vector, pattern query vector, and the first prompt word vector to obtain the fused vector of the first object. The mask query vector, the content query vector, and the pattern query vector are all preset vectors. The mask query vector is used to set the weight of the region where the mask of the first object is located. The content query vector is used to set the weight of the context region of the first object. The pattern query vector is used to set the weight of the mask query vector and the weight of the content query vector. The context region of the first object corresponds to the remaining area in the border of the first object after removing the region where the mask of the first object is located.

4. The method according to claim 1, characterized in that, The first model includes a first sub-model and a second sub-model; the first sub-model is used to output the category of an object based on the image; the second sub-model is used to perform mask segmentation on the image; the step of inputting the first image into the first model to obtain the first category of the first object, the confidence level of the first category, and the mask of the first object includes: Obtain the candidate bounding box information (proposal) of the first object in the first image; the candidate bounding box information of the first object is used to define the position of the first object in the first image; The candidate bounding box information of the first image and the first object are input into the first sub-model to obtain the bounding box of the first object, the first category of the first object, and the confidence of the first category; the matching degree between the bounding box of the first object and the first object is higher than the matching degree between the candidate bounding box of the first object and the first object. The bounding box of the first object and the first image are input into the second sub-model to obtain the mask of the first object.

5. The method according to claim 4, characterized in that, The step of inputting the bounding box of the first object and the first image into the second sub-model to obtain the mask of the first object includes: The first image is converted into a second image vector based on the second image encoder, and the bounding box of the first object is converted into a second cue word vector based on the second cue word encoder; the second sub-model includes the second image encoder, the second cue word encoder, and the mask decoder. The second image vector and the second cue word vector are input into the mask decoder to obtain the mask of the first object.

6. The method according to claim 1, characterized in that, Before outputting the second category of the first object, the following is also included: The confidence score of the second category in the first model is fused with the conditional probability of the second category in the second model to obtain the target probability of the second category; the weights of the confidence score of the second category in the first model and the weights of the conditional probability of the second category in the second model are different. If the conditional probability of the second category meets the conditional probability requirement, output the second category of the first object, including: If the target probability of the second category satisfies the target probability condition, output the second category of the first object.

7. The method according to claim 6, characterized in that, The target probability Satisfy the following formula: Where p is the image of the first object; c is the category of the first object; The first object is identified by the first model, and any one of the multiple categories of the first object is obtained. The conditional probability that the category of the first object is c, output by the second model; The confidence score for the category c of the first object output by the first model; The category of the first object output by the second model is The probability value; This is a preset index; The set of categories corresponding to the training samples of the first model; The background category; Not belonging to the above And the set of other categories of the background category.

8. The method according to claim 1, characterized in that, The first image also includes a second object, and after the first image is input into the first model, the following is also included: Obtain the third category of the second object and the confidence level of the third category; The output of the second category of the first object includes: If the confidence level of the third category meets the confidence level requirement, output the second category of the first object and the third category of the second object.

9. The method according to claim 8, characterized in that, After outputting the second category of the first object and the third category of the second object, the method further includes: The first interface is displayed, which includes the first image and first information; wherein the first image displays the first object, the identifier of the first object, the second object, and the identifier of the second object; wherein the identifier of the first object partially or completely overlaps with the first object; the identifier of the second object partially or completely overlaps with the second object; the first information includes at least one of the following: the total calories of the first object and the second object, and / or the total nutritional composition of the first object and the second object; Upon receiving a trigger operation targeting the identifier of the first object, a second interface is displayed; the second interface includes second information, which includes at least one of the following: the second category of the first object, the calories of the first object, and / or the nutritional composition of the first object; wherein the calories of the first object and / or the nutritional composition of the first object are calculated based on the second category of the first object and the volume of the first object.

10. The method according to claim 1, characterized in that, The first model includes the Ground-SAM model.

11. The method according to any one of claims 1-10, characterized in that, Before acquiring the first image, the process also includes: In response to an operation for launching a camera application, a camera application interface is displayed, the camera application interface including a first control; In response to a trigger operation on the first control, a third interface is displayed; the third interface includes multiple controls for identifying items. In response to a selection operation on a control used to identify food, a fourth interface is displayed; the fourth interface includes a second control. The acquisition of the first image includes: In response to a trigger operation on the second control, the first image is obtained.

12. The method according to any one of claims 1-10, characterized in that, Before acquiring the first image, the process also includes: In response to an operation to launch a health management application, the interface of the health management application is displayed, the interface of the health management application including a third control; the third control is used to access the camera; In response to a trigger operation on the third control, a fifth interface is displayed; the fifth interface includes the fourth control. The acquisition of the first image includes: In response to a trigger operation on the fourth control, the first image is obtained.

13. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1-12.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-12.

15. A chip system, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being configured to run a computer program or instructions to perform the method as described in any one of claims 1-12.

16. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1-12.

Images