Article identification method and device and electronic equipment
Through adaptive adjustment of the recognition granularity through tree graph structure and probability values, combined with geographical location and historical information, the problem of inaccurate fine-grained recognition in dish recognition is solved, and more efficient and accurate recognition results are achieved.
Patent Information
- Application Number
- CN202310611813.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is difficult to ensure the accuracy of fine-grained identification results in dishes recognition, and it is not compatible with coarse and fine-grained identification, resulting in inaccurate identification results.
The tree graph structure is used to manage the item data, determine the probability value of the effective node based on the picture information, adaptively adjust the granularity level of the recognition result, and fall back to coarse grain size when the fine-grained recognition result is not reliable, and output the effective recognition result. And combine geographical location, time and historical search records to optimize identification.
It improves the accuracy and user experience of dish recognition, is compatible with coarse and fine-grained item recognition, improves the efficiency of large-scale item data management, and supports efficient model training and recognition.
Smart Images

Figure CN119027925A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of data processing technology, and more specifically, to a method, device, and electronic device for identifying an object. Background Art
[0002] As living standards improve, people's demand for healthy diets is also increasing. Users can use electronic devices to take photos of dishes to identify them, obtain information such as the type of dish, calorie content, and nutritional elements, and obtain relevant dietary recommendations, health management plans, and other information.
[0003] However, currently the prediction output of dish types can only be performed at a certain fixed granularity, and usually fine-grained prediction output is pursued, resulting in only mutually exclusive relationships between the prediction results of dishes, and no support for inclusion relationships, making it difficult to ensure the accuracy of dish recognition. Summary of the invention
[0004] The embodiments of the present application provide a method, device and electronic device for identifying an object, which can improve the accuracy of object identification.
[0005] In a first aspect, a method for identifying an object is provided, characterized in that it includes: acquiring image information of the object; determining a probability value of a valid node in a tree diagram corresponding to the object based on the image information, the tree diagram including different types of objects and corresponding object data, the valid node being a node in the tree diagram including valid data, and the probability value being used to indicate the possibility that the object is the object corresponding to the current node; when the probability value of the current node is less than a preset threshold and the current node is the node with the largest probability value in the node layer to which it belongs, outputting the object corresponding to the first node, the first node being the node with the largest probability value in the node layer above the node layer to which the current node belongs, and the first node belongs to the valid node.
[0006] In the embodiment provided by the present application, the granularity level of the output item recognition result can be adaptively adjusted according to the probability value of the node. When the reliability of the recognition result under fine granularity is low or the probability value is low, it can automatically fall back to the previous coarser granularity level to give the recognition result under coarse granularity, thereby improving the accuracy of the recognition result and optimizing the user experience. In addition, the probability value of the valid node in the tree diagram corresponding to the item is determined according to the acquired image information, and the first node output belongs to the valid node, which can ensure that the output recognition result is a valid result, that is, it is ensured that the output recognition result will not be a result with too coarse granularity and low reference value for the user, which can further optimize the user experience. Furthermore, the tree diagram is used to manage the data of the item, which can be compatible with the coarse and fine granular items at the same time, so that the granularity level of the output recognition result can be adaptively adjusted when the recognition result is output, and the management efficiency of large-scale item data can be improved, supporting efficient model training and item recognition.
[0007] In combination with the first aspect, in certain implementations of the first aspect, before outputting the item corresponding to the first node, the method also includes: searching for N nodes in the valid nodes that meet preset conditions, the preset conditions including that the probability value of the node is greater than or equal to a preset threshold, and N is an integer greater than or equal to 1; when the number M of nodes in the first node layer that meet the preset conditions is greater than or equal to N, the N nodes are the N nodes with the largest probability values in the first node layer, and the first node layer is the layer closest to the last layer and includes nodes that meet the preset conditions; when M is less than N, the N nodes include the M nodes and the NM nodes with the largest probability values in the node layer above the first node layer; the first node is included in the N nodes.
[0008] In the embodiment provided in the present application, N nodes that meet preset conditions are searched from valid nodes, and the N nodes are included in the first node layer, which is the node layer closest to the last layer of the tree diagram, or also include NM nodes with the highest probability in the node layer above the first node layer. This ensures that the output recognition results are valid results during the output of the recognition results, and can adaptively output recognition results of different granularity levels according to the size of the probability value, thereby improving the accuracy of the recognition results.
[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: outputting items corresponding to N-1 nodes among the N nodes except the first node layer.
[0010] In the embodiment provided in the present application, after searching for N nodes that meet the preset conditions, the items corresponding to the N nodes can be output, so that the user can obtain one or more different recognition results and select the target items from them for relevant records, and the one or more recognition results are the results of the granularity output adaptively adjusted according to the size of the probability value, which can be compatible with different granularity levels, and more likely include the actual items recognized by the user, with higher accuracy.
[0011] In combination with the first aspect, in certain implementations of the first aspect, determining the probability value of a valid node in the tree diagram corresponding to the item based on the image information includes: determining a predicted value of each node in the tree diagram based on the image information, the probability value of the valid node being the product of a predicted value of the current node and a predicted value of a previous node on the propagation path to which the current node belongs, and the predicted value being a numerical value output after the image information passes through an artificial intelligence AI model.
[0012] In the embodiment provided in the present application, the probability value of the valid node in the tree diagram corresponding to the object is determined based on the image information, and the probability value of the valid node is the product of the predicted value of the current node and the predicted value of the previous node on the propagation path, that is, the parent node. This can ensure that the recognition result finally output is a valid result acceptable to the user, and can greatly reduce the calculation difficulty and burden of the electronic device or server, and avoid the problem of introducing too many nodes without valid data in the calculation, resulting in a decrease in model performance.
[0013] In combination with the first aspect, in certain implementations of the first aspect, determining the predicted value of each node in the tree diagram based on the picture information includes: determining the predicted value of each node in the tree diagram based on the picture information and at least one of the following information: the time when the picture information was obtained, the geographical location when the picture information was obtained, the climate information when the picture information was obtained, and the historical search records.
[0014] In the embodiments provided in the present application, the types of items identified by the user are very likely to be close to the geographical location, time, climate and historical search records of the user when the user identifies the items. The predicted value of the node is determined based on the geographical location, time, climate and at least one item of the user's historical search records of the acquired image information and the image information, which can further improve the accuracy of item identification and optimize the user experience.
[0015] In combination with the first aspect, in certain implementations of the first aspect, each node in the tree diagram includes one or more data sets, and the one or more data sets include data corresponding to the names of one or more of the items, and the types of items indicated by the one or more names of the items are the same.
[0016] In the embodiments provided in the present application, items of the same type may have different names. The data of items of the same type but with different names are grouped into the same node. This can ensure the uniqueness of the identification results of the items and can efficiently manage the data level of the items, thereby enabling the tree diagram to support efficient model training and item identification.
[0017] In combination with the first aspect, in certain implementations of the first aspect, the item is a dish, and the method further includes: determining the type of meal to which the dish belongs based on at least one of the following information: an identification result of the dish type, a number of identifications on the day, and an identification time.
[0018] In the embodiments provided in the present application, when the item is a dish, the meal type to which the dish belongs can be determined based on at least one of the recognition result of the dish type, the number of recognition times on the day, and the recognition time, thereby improving the accuracy of the classification of meals to which the dish belongs, especially improving the accuracy of the classification of meals in fuzzy time periods such as 10 to 11 a.m. or 4 to 5 p.m., and avoiding incorrect classification of meals due to direct classification of the dish according to the recognition time.
[0019] In combination with the first aspect, in certain implementations of the first aspect, the item is a dish, and the method further includes: determining a measurement unit of the dish based on a recognition result of the dish and / or a geographical location where image information of the dish is obtained.
[0020] In the embodiments provided in the present application, the measurement unit of the dish is determined based on the recognition result of the dish and / or the geographical location where the image information of the dish is obtained. The measurement unit can be selected based on the characteristics of the dish itself and the preferences of users in the region for dining utensils. For example, measurement units such as "spoon" and "chopsticks" can be used for measurement, avoiding the need for users to convert the measurement units of the dish when using conventional measurement units such as "grams", thereby further optimizing the user experience.
[0021] In a second aspect, a device for identifying an object is provided, comprising: a processing module for acquiring image information of an object; a determination module: for determining a probability value of a valid node in a tree diagram corresponding to the object based on the image information, the tree diagram comprising different types of objects and corresponding object data, the valid node being a node in the tree diagram comprising valid data, and the probability value being used to indicate the probability that the object is the object corresponding to a current node; the processing module is also used to output an object corresponding to a first node when the probability value of the current node is less than a preset threshold and the current node is the node with the largest probability value in the node layer to which it belongs, the first node being the node with the largest probability value in the node layer above the node layer to which the current node belongs, and the first node belongs to the valid node.
[0022] In combination with the second aspect, in certain implementations of the second aspect, the processing module is also used to search for N nodes in the valid nodes that meet preset conditions before outputting the item corresponding to the first node, the preset conditions including that the probability value of the node is greater than or equal to a preset threshold, N is an integer greater than or equal to 1; when the number M of nodes in the first node layer that meet the preset conditions is greater than or equal to N, the N nodes are the N nodes with the largest probability values in the first node layer, and the first node layer is the layer closest to the last layer of the tree diagram and includes nodes that meet the preset conditions, or, when M is less than N, the N nodes include the M nodes and the NM nodes with the largest probability values in the node layer above the first node layer; the first node is included in the N nodes.
[0023] In combination with the second aspect, in some implementations of the second aspect, the processing module is further used to output items corresponding to N-1 nodes among the N nodes except the first node.
[0024] In combination with the second aspect, in certain implementations of the second aspect, the determination module is specifically used to determine the predicted value of each node in the tree diagram based on the image information, the probability value of the valid node is the product of the predicted value of the current node and the predicted value of the previous node on the propagation path to which the current node belongs, and the predicted value is the numerical value output after the image information passes through the artificial intelligence AI model.
[0025] In combination with the second aspect, in certain implementations of the second aspect, the determination module is specifically used to determine the predicted value of each node in the tree diagram based on the picture information and at least one of the following information: the time when the picture information was obtained, the geographical location when the picture information was obtained, the climate information when the picture information was obtained, and the historical search records.
[0026] In combination with the second aspect, in certain implementations of the second aspect, the determination module is also used to, when the item is a dish, determine the type of meal to which the dish belongs based on at least one of the following information: the recognition result of the dish, the number of recognitions on the day, and the recognition time.
[0027] In combination with the second aspect, in certain implementations of the second aspect, the determination module is also used to, when the item is a dish, determine the measurement unit of the dish based on the recognition result of the dish and / or the geographical location where the picture information of the dish is obtained.
[0028] In a third aspect, an electronic device is provided, comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device performs the method for identifying an object as described in the first aspect or any possible implementation of the first aspect.
[0029] In a fourth aspect, a chip system is provided, which includes at least one processor. When program instructions are executed in the at least one processor, the method for identifying an object as described in the first aspect or any possible implementation method of the first aspect is implemented.
[0030] In a fifth aspect, a computer program storage medium is provided, wherein the computer readable medium stores program code, and when the program code runs on a computer, the computer executes the method for identifying an object as described in the first aspect or any possible implementation of the first aspect.
[0031] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program code, and when the computer program code is run on a computer, the computer executes the method for identifying an object as described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.
[0033] Figure 2 It is a flowchart of a method for identifying an object provided in an embodiment of the present application.
[0034] Figure 3 This is a schematic diagram of a user interface of an electronic device provided in an embodiment of the present application.
[0035] Figure 4 This is a tree diagram of dish types provided in an embodiment of the present application.
[0036] Figure 5 It is a tree diagram calculation diagram provided in an embodiment of the present application.
[0037] Figure 6 This is a tree graph node search example provided in an embodiment of the present application.
[0038] Figure 7 This is a schematic diagram of a user interface of an electronic device provided in an embodiment of the present application.
[0039] Figure 8 This is a schematic diagram of a user interface of an electronic device provided in an embodiment of the present application.
[0040] Fig. 9 It is a structural block diagram of a device for identifying an object provided in an embodiment of the present application.
[0041] Fig.10 It is a structural block diagram of an electronic device for identifying objects provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0043] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.
[0044] “One or more” refers to the number of an object, and “plurality” may mean a quantity of two or more.
[0045] The first, second, etc. are only used to indicate that multiple objects are different. For example, the first node layer and the first node are only used to indicate a certain node layer or a certain node, and should not impose any restrictions on the number and structure of nodes and node layers in the embodiments of the present application.
[0046] The terms "include", "comprising", "having" and variations thereof all mean "including but not limited to", unless specifically emphasized otherwise.
[0047] To facilitate understanding of the present application, the professional terms involved in the embodiments of the present application are first briefly introduced.
[0048] Loss function: In the process of training neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two. For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, we need to predefine "how to compare the difference between the predicted value and the target value", which is the loss function, which is an important equation used to measure the difference between the predicted value and the target value.
[0049] Granularity: Granularity is the coarseness or fineness of data statistics under the same dimension. Generally speaking, the higher the degree of refinement, the smaller the granularity (or the finer); conversely, the lower the degree of refinement, the larger the granularity (or the coarser).
[0050] Breadth first search (BFS): Also known as breadth first search, it is a blind search method that aims to systematically expand and check all nodes in the graph to find the result. In other words, it does not consider the possible location of the result and thoroughly searches the entire graph until the result is found.
[0051] For example, Figure 1 The figure is a schematic diagram of the hardware structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a compass 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195.
[0052] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0053] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be set in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 may be a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100 in processing data or executing instructions.
[0054] Exemplarily, the processor 110 may be used to determine the type of an item based on image information of the item and output it.
[0055] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device.
[0056] It should be understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely a schematic illustration and does not constitute a structural limitation on the electronic device 100 .
[0057] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the wearable device through the power management module 141.
[0058] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc.
[0059] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0060] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.
[0061] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0062] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (WiFi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0063] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0064] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. The display screen 194 of the electronic device 100 can be a flexible screen.
[0065] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor, etc. The ISP is used to process the data fed back by the camera 193.
[0066] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.
[0067] Exemplarily, the camera 193 can be used to obtain image information of an object.
[0068] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.
[0069] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs.
[0070] NPU is a neural-network (NN) computing processor. It can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.
[0071] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0072] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can enable the electronic device 100 to perform the methods provided in some embodiments of the present application, as well as various applications and data processing, etc. by running the above instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, a flash memory component, a universal flash storage (UFS), etc. In some embodiments, the processor 110 can enable the electronic device 100 to perform the methods provided in the embodiments of the present application, as well as other applications and data processing by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor 110. The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0073] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.
[0074] In the embodiments of the present application, the electronic device may be a large-screen device such as a television, a projector, a smart screen, or a sports device such as a treadmill, or a wearable device such as a smart watch, a smart bracelet, headphones, a head-mounted display device, or a laptop computer, a tablet computer, a mobile phone, or other device. The present application does not limit the type of electronic device.
[0075] Figure 2A method for identifying an object provided in an embodiment of the present application may include steps S210 to S230. The method may be executed by an electronic device or a chip in the electronic device, or may be executed by a server.
[0076] S210, obtaining image information of the item.
[0077] In some embodiments, the method may be executed by an electronic device, and the electronic device may directly enable a camera application to capture a picture of the object in response to a user operation on the interface, or may select a stored picture from a photo application.
[0078] As an example, a user may use a third-party application such as a sports and health application, a shopping application, etc. to identify the type of an item. Figure 3 (a) is a user interface 310 for identifying a dish type provided in an embodiment of the present application, such as Figure 3 As shown, the user can slide the dish identification control 312 to the middle position to enable the identification of the dish type. Then, after focusing on the dish, click the camera control 315 to take a photo of the dish, and the application can automatically obtain the image information of the item and identify it. Alternatively, the user can also select the image of the item to be identified from the gallery application 314 for identification. The user can also slide the food barcode control 311 to the middle position to identify the dish based on the barcode image, or slide the nutrition table control 313 to the middle position to identify the dish and obtain the nutrition information of the dish.
[0079] As another example, users can also use the object recognition function of electronic devices, see Figure 3 In the user interface 320 shown in (b), the user can slide to the object recognition control 321 and then click the photo control 322 to collect pictures of various items such as plants, animals, and cars to obtain recognition results. Similarly, the user can also slide to the object recognition control 321 and select the gallery application 323 to select the picture to be recognized from the album.
[0080] In other embodiments, the method may be executed by a server. After the electronic device collects or selects the image of the object, it may transmit the acquired image to the server, and the server identifies the object corresponding to the image based on the image information and transmits the identification result to the electronic device, which is displayed on the corresponding user interface.
[0081] S220, determining the probability value of a valid node in a tree diagram corresponding to the item based on the image information, the tree diagram including different item types and corresponding item data, the valid node being a node in the tree diagram including valid data, and the probability value being used to indicate the probability that the item is the item corresponding to the current node.
[0082] Here, the tree diagram of the items provided in the embodiments of the present application is first introduced in detail. The tree diagram of the items can be constructed by the developer, and the tree diagrams corresponding to different types of items are different. For example, when the item is a dish, the corresponding tree diagram is a dish tree diagram, and when the item is a plant, the corresponding tree diagram is a plant tree diagram. The tree diagram of the item may include multiple layers of nodes, and the granularity of the items corresponding to the nodes at different levels is different, and from the root node of the tree diagram to the leaf node at a deeper level, the granularity of the items is distributed from coarse to fine, and the granularity of the nodes at the same level may be the same.
[0083] It should be noted that the deeper the level of the leaf node is, the farther the level of the leaf node is from the root node. Each node in the tree diagram can indicate an item, and each node can include one or more data sets of items corresponding to the node. The tree diagram can be used for training artificial intelligence (AI) models. After the training is completed, the tree diagram can be applied to the identification of items to predict the types of items.
[0084] For example, Figure 4 A tree diagram of dish types provided in an embodiment of the present application, the root node of the tree diagram is used to indicate dishes, and each node in the first layer of leaf nodes of the dish can respectively indicate various types of coarse-grained dishes such as "fruit", "seafood", "soy milk egg", "meat dish" and "vegetarian dish". Each node in the first layer of leaf nodes can further include leaf nodes with finer granularity, that is, they can constitute the second layer of leaf nodes. Taking "soy milk egg" as an example, the number of leaf nodes included in the node corresponding to "soy milk egg" can be multiple, which can be used to indicate "soy products", "dairy products" and "egg products" with finer granularity than "soy milk egg". Similarly, the number of leaf nodes included in the node corresponding to "egg product" can also be multiple, which can be used to indicate "fried egg", "steamed egg", "braised egg" and the like with finer granularity than "egg product", and the leaf nodes included in the node corresponding to "fried egg" can be used to indicate "Mexican fried egg", "shrimp fried egg", "Toona sinensis fried egg" and the like with finer granularity than "fried egg", and examples are not given one by one here.
[0085] It should be noted that Figure 4The shown dish tree diagram is only used as an example to illustrate the structure and granularity distribution of the tree diagram of items. The number and types of items such as dishes included in the tree diagram should not be limited. For example, the corresponding nodes of "fruits" and "meat dishes" can include multiple layers of leaf nodes to indicate fruits or meat dishes of different coarse and fine granularity. The tree diagram can include larger-scale nodes, for example, it can include a number of nodes in the order of thousands or more, so as to comprehensively and accurately identify the types of items. The tree diagram is a structure distributed from top to bottom from the root node to the deep-level leaf nodes. The tree diagram can also be a structure distributed from bottom to top, or from left to right, or from right to left, and this application does not limit this.
[0086] In the embodiment of the present application, a multi-level and multi-granular tree diagram is used for model training and item type identification, which can conveniently and efficiently manage a large number of item types to support efficient model training and item prediction; and from the root node of the tree diagram to the deep leaf nodes, the item types are classified from coarse-grained to fine-grained, which can accommodate the coexistence of coarse and fine-grained items, and thus can take into account both the accuracy of identification and the granularity of the item when identifying the item type, and give a more appropriate identification result.
[0087] In some embodiments, each node in the tree diagram may include one or more data sets, and the one or more data sets may include data corresponding to one or more item names, and the item types indicated by the one or more item names are the same. Taking dishes as an example, due to regional differences or different user habits, the same dish may have different names. For example, a braised egg can also be called a tea egg. Putting dishes with different names but actually the same dish into the same node can avoid the problem of repeated recognition results of dishes.
[0088] Exemplarily, the same node may include dish 1 and dish 2, and the names or names of dish 1 and dish 2 are different, but they belong to the same dish, for example, dish 1 is braised egg, and dish 2 is tea egg. The name of dish 1 may include the Chinese name of dish 1, and may also include the English name and / or alias of dish 1, etc. Similarly, the name of dish 2 may also include the Chinese name of dish 2, and the English name and / or alias. The node may include data set 1 (DataSet1) and data set 2 (DataSet2), and the two data sets may be data sets corresponding to dish 1 and dish 2 respectively, and the two data sets may have a mapping relationship with the corresponding dishes. The format of the dish and the data set may be, for example, [(Chinese name 1, alias 1, English name 1), (Chinese name 2, alias 2, English name 2)] to [DataSet1: (dir1, dir2), DataSet2: (dir1, ...)], wherein the data set may include pictures of the dishes corresponding to the node.
[0089] During model training, all data from multiple data sets included in the same node can be collected together and the node can be trained together. Then, when the model is used for prediction, that is, when identifying dish types, dishes with different names but the same types can be mapped to the same node for dish type prediction, thereby ensuring the uniqueness of the output dish names and avoiding duplication of output recognition results.
[0090] For concepts such as valid nodes and valid data, please continue to refer to Figure 4 In the tree diagram shown, if the dish that the user needs to identify is "shrimp fried eggs", for the user, the prediction result of "soy milk egg" or the prediction result of "egg products" is too coarse and has little reference value for the user, while the prediction result of "fried eggs" or the more fine-grained prediction result of "shrimp fried eggs" is more valuable for the user. Therefore, the node layer where "fried eggs" is located and the leaf node layers of "fried eggs" can be called valid layers, each node in the valid layer can be called a valid node, and the data included in the valid node can be called valid data. In other words, a valid node can be a node that includes valid data in the tree diagram, or a valid node can be a node that includes fine-grained valid data in the tree diagram. Figure 4 The node layer where "fried eggs" is located can be the first node layer including valid data, or it can also be called the first valid layer. The first valid layer can refer to the first node layer including valid data when searching from the level where the root node is located to the deep leaf node layer. The recognition calculation of the type of dish can be calculated with the first valid layer as the starting layer. Similarly, the recognition of the types of other items can also be calculated with the first valid layer of the corresponding tree diagram as the starting layer. Each node from the first valid layer to the last layer of the tree diagram can be called a valid node. The last layer is the node layer of the tree diagram that is farthest from the root node.
[0091] In some embodiments, the first effective layer may be predefined. Exemplarily, it may be determined by the developer when constructing the tree diagram. For example, for the identification of dish types, the 4th layer of the dish tree diagram may be predefined as the first effective layer, and the subsequent model training and dish identification processes may be calculated with the 4th layer of the dish tree diagram as the starting layer. Alternatively, the first effective layer may also be the first node layer with the root node as the starting layer, and the probability value of the node with the highest probability in each layer is less than the preset value.
[0092] In some embodiments, the probability value of the valid node in the tree diagram can be determined according to the predicted value of the node in the tree diagram, that is, the predicted value of the valid node in the tree diagram corresponding to the item can be determined according to the acquired image information, and then the probability value of the node can be calculated according to the predicted value. The probability value of the valid node can be the product of the predicted value of the current node and the predicted value of the previous node on the propagation path to which the current node belongs. The predicted value can be the value output after the image information of the item passes through the AI model. The predicted value of the node can also be called the conditional probability value of the node, which can be a real number between 0 and 1. The previous node on the propagation path to which the current node belongs can also be called the parent node of the current node. The propagation path of the node is a path from a certain node layer to the last layer of the tree diagram. The next node is the child node of the current node, or the previous node is the parent node of the current node.
[0093] It should be noted that when calculating probability values based on predicted values, the number of predicted values output by the AI model can be determined in advance based on the number of nodes in the tree diagram. After the AI model outputs a corresponding number of predicted values, the electronic device or server can map the obtained predicted values to each node of the tree diagram according to the structure of the tree diagram to calculate the probability value of each node.
[0094] Combine the following Figure 5 The simplified tree diagram structure shown in the figure introduces an example of calculating the predicted value and probability value of each node on the tree diagram. Figure 5 When the tree diagram shown is used to identify and predict the types of dishes, the Figure 5 The tree structure shown can be compared with Figure 4 The tree structure shown corresponds to that shown, that is, Figure 5 The node distribution of the tree diagram shown can be compared with Figure 4 The nodes in the dendrogram shown are distributed in the same way, and Figure 5 The types of dishes corresponding to the nodes in the tree diagram shown can be Figure 4 The nodes at corresponding positions in the tree diagram shown correspond to the same type of dishes.
[0095] Assume that the tree diagram includes k+1 layers of nodes, the 0th layer is the root node, the kth layer is the last layer of the tree diagram, and the kjth layer is the first valid layer including valid data (k>j, and k and j are integers). The predicted value of each node in the tree diagram can be Among them, l means that the node layer to which the current node belongs is the lth layer, the value of l can be 0~k, and i means Figure 5 In the tree diagram shown, the current node is the i-th node from left to right in the l-th layer, and the predicted value of the node is It can be a real number between 0 and 1.
[0096] The predicted value It is the output result after the sample passes through the AI model, that is, the value output after the image of the object passes through the AI model. Each sample will get a predicted value after passing through the model. During the training process of the AI model, the difference between the predicted value and the true value becomes the loss. The parameters in the model can be updated to minimize the loss and make the predicted value close to the true value. The predicted value used in the object recognition process, or the predicted value used in the probability calculation process of the node It can be the predicted value output by the model for the item image after training.
[0097] The loss of nodes in the tree diagram can be calculated starting from the first valid layer, for example Figure 5 The kjth layer shown is the first node layer including valid data. Figure 5 The loss corresponding to the tree graph structure shown can be expressed as:
[0098]
[0099] Among them, y represents the true value of the node, represents the predicted value of the node, is the loss of each layer, is the sum of the losses of each layer. In other words, the loss of the tree diagram can be the sum of the losses from the first valid layer to the last layer, while the 0th to kj-1th layers do not include valid data, and no model training and loss calculation are required.
[0100] Furthermore, after the prediction model corresponding to the item is trained, the image of the item can be input into the prediction model and the prediction value can be output. The probability of a node is the probability that the item type is the item corresponding to the current node, or it can be the accuracy or confidence when the output recognition result is the item corresponding to the current node.
[0101] For example, the current node is "fried eggs", and the propagation path of "fried eggs" can be "dishes - soy milk eggs - egg products - fried eggs", and the level where "fried eggs" is located is the first valid level that includes valid data. The probability values of the nodes in the level where "fried eggs" is located and the leaf node level after this level can be calculated. The level where "fried eggs" is located is the k-2 level, and it is the first node of the k-2 level. The "egg products" corresponding to the previous level of "fried eggs" is the k-3 level, and it is the second node of the k-3 level. Then the probability corresponding to "fried eggs" can be:
[0102]
[0103] For another example, the current node is "Toona sinensis fried egg", and the propagation path of "Toona sinensis fried egg" can be "dish - soy milk egg - egg product - fried egg - Toona sinensis fried egg". The layer where "Toona sinensis fried egg" is located is the k-1 layer, and it is the third node of the k-1 layer. The probability corresponding to "Toona sinensis fried egg" can be:
[0104]
[0105] It should be noted that in the embodiment of the present application, only the probability values of valid nodes can be calculated to ensure that the given item recognition results are items corresponding to valid nodes, so as to improve the user experience. Therefore, when calculating the predicted values of each node in the tree diagram corresponding to the item, only the predicted values of each node in the valid node layer and the node layer above the first valid layer can be calculated. Figure 5 For example, the predicted values can be calculated for the nodes at the kj-1th layer to the nodes at the kth layer.
[0106] In the embodiment provided by the present application, model training and item type identification are performed using the first node layer including valid data as the starting layer, which can reduce the learning burden of the backbone nodes, reduce the amount of calculation in the item identification process, and improve the item identification efficiency.
[0107] S230, when the probability value of the current node is less than a preset threshold and the current node is the node with the largest probability value in the node layer to which it belongs, output the item corresponding to the first node, the first node is the node with the largest probability value in the previous node layer of the node layer to which the current node belongs, and the first node is a valid node.
[0108] When the probability value of a node is less than a preset threshold, the probability that the item is identified as this node is small, and outputting this node as the identification result is likely to result in an incorrect identification result. If the probability value of the current node is the largest in the node layer and is less than the preset threshold, the item corresponding to the node with the largest probability value in the previous node layer of the current node layer can be output. The previous node layer is the previous node layer close to the root node. In other words, when a fine-grained result cannot guarantee the accuracy of the result, a result with a coarser granularity than the current node can be output to ensure the accuracy of the identification result.
[0109] It should be noted that the node with the largest probability value in the previous node layer of the node layer to which the current node belongs may be a node whose probability value is greater than or equal to a preset threshold. If the probability of the node with the largest probability value in the previous node layer of the node layer to which the current node belongs is still less than the preset threshold, the search for the node with the largest probability value in the even higher node layer may continue until the maximum probability value in a certain node layer is greater than or equal to the preset threshold, and it is then output as the recognition result.
[0110] As an example, the search can be started from the first valid layer as the starting layer, and the nodes with the largest probability values in the first valid layer and the leaf node layer of the first valid layer are selected in turn until the probability value of the current node is less than a preset threshold, and then the item corresponding to the node with the largest probability value in the node layer above the node layer where the current node is located is output.
[0111] by Figure 6 Take the tree diagram shown as an example, assuming that the preset threshold is 0.15, the k-2th layer is the first effective layer, and the search starts from the k-2th layer to the last layer of the tree diagram. The probability values of the nodes from left to right in the first effective layer are 0.5, 0.05, ..., 0.3, 0.1 respectively. The node with the largest probability value in the first effective layer is the node corresponding to the probability value of 0.5. 0.5 is greater than the preset threshold, and the node with the largest probability value in the next node layer can be selected. The probability values of the nodes from left to right in the k-1th layer are 0.15, 0.3, 0.05, ..., 0.2, 0.1 respectively. The node with the largest probability value in the node layer is the node corresponding to the probability value of 0.3. 0.3 is greater than the preset threshold, and the node with the largest probability value in the next node layer can be selected. Assuming that the probability of the node with the largest probability value in the kth layer is 0.09, which is less than the preset threshold, it can be retreated to the k-1th layer and the node with the largest probability value in the k-1th layer is output. If the node with the largest probability value in the kth layer is the node corresponding to the probability value of 0.15 shown in the figure, and the probability value 0.15 is equal to the preset threshold, the item of the node corresponding to the probability value 0.15 can be directly output.
[0112] As another example, you can also start searching from the last layer of the tree diagram toward the root node, select the node with the largest probability value in each layer in turn, compare the probability value with the preset threshold, and if the probability value is less than the preset threshold, continue searching for the node with the largest probability value in the previous node layer until the probability value of the node with the largest probability value in the current node layer is greater than or equal to the preset threshold.
[0113] For example, the kth layer is the last layer, and the maximum probability value in the kth layer is 0.15. 0.15 is greater than the preset threshold, so the item of the node corresponding to 0.15 can be directly output. Assuming that the maximum probability value in the kth layer is 0.09, you can continue to search for the node with the largest probability value in the k-1th layer. The maximum probability value in the k-1th layer shown in the figure is 0.3, so you can output the item of the node corresponding to the probability value of 0.3. If the maximum probability value in the k-1th layer is still less than the preset threshold, you can continue to search in the direction of the root node until the maximum probability value in the current node layer is greater than or equal to the preset threshold, and output the item corresponding to the node with the largest probability value.
[0114] In the embodiments provided in the present application, when the recognition accuracy of an object at a certain granularity is insufficient, for example, when the recognized object is a new category that the AI model has never been trained at the granularity, the recognition result given at the granularity must be an erroneous recognition result. The solution provided in the present application can automatically roll back the recognition result to the previous coarse-grained level while outputting valid objects, thereby improving the accuracy of the object recognition results and optimizing the user experience.
[0115] In some embodiments, before outputting the item corresponding to the node with the largest probability value in the previous node layer of the node layer to which the current node belongs, the method also includes: searching for N nodes in the tree diagram that meet a preset condition, the preset condition being that the probability value of the node is greater than or equal to a preset threshold, and N is an integer greater than or equal to 1; and, when the number M of nodes in the first node layer that meet the preset condition is greater than N, the N nodes may be the N nodes with the largest probability values in the first node layer, and the first node layer may be the layer closest to the last layer of the tree diagram and including the nodes that meet the preset condition, and when M is less than N, the N nodes include the M nodes and the NM nodes with the largest probability values in the previous node layer of the first node layer, and M is an integer; the above-mentioned first node is included in the N nodes here.
[0116] The method may further include: outputting N-1 nodes among the N nodes except the first node.
[0117] That is, when identifying the corresponding object in the acquired image, one recognition result or multiple recognition results may be given. For example, one object name or multiple object names may be displayed on the corresponding recognition result interface in response to the user's settings or selections. When outputting recognition results of multiple objects, the output recognition results need to satisfy the requirement that the granularity of the recognition results is as fine as possible, that is, the granularity of the recognition results is as close as possible to the granularity of the actual object identified by the user, and the probability value of the node corresponding to the recognition result needs to meet the requirement of the preset threshold value, and the probability value is as large as possible, that is, the accuracy of the output recognition result is ensured as much as possible, so that the recognition result can achieve a balance between granularity and accuracy.
[0118] As an example, see Figure 6 In the tree diagram shown, assuming that the preset threshold is 0.15 and the k-2th layer is the first valid layer, all nodes included in the k-2th layer to the kth layer can be valid nodes, and the first valid layer can be used as the starting layer to search for nodes that meet the preset conditions in the valid nodes. During the search process, stop bits can be added between different node layers as flag bits to separate different node layers. Every time a flag bit is encountered, the nodes in the next node layer whose probability values are greater than or equal to the preset threshold can be sorted in order of probability values from small to large. Figure 6 As shown, the probability values of nodes in the k-2th layer that are greater than or equal to the preset threshold include 0.5 and 0.3. The nodes corresponding to these two probability values are selected and sorted in order of probability value from small to large. Furthermore, when performing BFS output, nodes with smaller probability values can be output first, and nodes with larger probability values can be output later, that is, nodes with probability values of 0.3 can be output first, and then nodes with probability values of 0.5 can be output. Nodes with probability values less than the preset threshold can be not sorted and output, for example Figure 6 The nodes with probability values of 0.05 and 0.1 in the k-2th layer do not need to be sorted and output, and the possibility that the identified items are the items corresponding to these two nodes is small.
[0119] Similarly, the nodes in the k-1th layer whose probability values are greater than or equal to the preset threshold include nodes corresponding to 0.15, 0.3, and 0.2. These three values are sorted in ascending order and output in descending order of probability value. The first node output is the node corresponding to the probability value of 0.15, and the last node output is the node corresponding to the probability value of 0.3. The nodes in the kth layer whose probability values are greater than the preset threshold include the node corresponding to the probability value of 0.15, and the output node is the node corresponding to the probability value of 0.15.
[0120] Since in the k-2 layer, the probability values of 0.05 and 0.1 are less than or equal to the preset threshold, the nodes with probability values of 0.05 and 0.1 do not need to be added to the queue for sorting and output. Furthermore, the probability values of the leaf nodes corresponding to the nodes with probability values of 0.05 and 0.1 are also less than the preset threshold, and do not need to be sorted and output. For example, in the k-1 layer, only the nodes with probability values of 0.5 and 0.3 can be sorted and output, and the leaf nodes of the nodes with probability values of 0.05 and 0.1 do not need to be sorted and output. Similarly, in the k layer, only the leaf nodes of the nodes with probability values of 0.3, 0.2 and 0.15 in the k-1 layer can be sorted and output, and the leaf nodes of the nodes with probability values of 0.05, 0.1, etc. do not need to be sorted and output.
[0121] That is to say, the BFS output order of the tree diagram nodes is, from first to last, the nodes corresponding to probability values 0.3, 0.5, 0.15, 0.2, 0.3 and 0.15. If N is 3, the output recognition result can be the items corresponding to the last 3 nodes output by BFS, that is, the items corresponding to the nodes corresponding to the output probability values 0.2, 0.3 and 0.15. Among them, the level where the node corresponding to the probability value 0.15 output last is located, that is, the kth layer includes 1 node that meets the preset conditions, and the kth layer is the last layer. The kth layer can also be called the first node layer, and M is equal to 1. The output recognition result can include 1 node in the kth layer, that is, the node corresponding to the probability value 0.15, and the 2 nodes with the largest probability values in the k-1th layer, that is, the two nodes corresponding to the probability values 0.3 and 0.2.
[0122] It should be noted that when the tree diagram corresponding to the item includes k layers, and each layer from the first effective layer to the kth layer includes nodes whose probability values are greater than or equal to the preset threshold, the search can be carried out from the first effective layer to the last layer, that is, the kth layer, and the nodes in each layer are sorted and output in the order of probability values from small to large. If the probability values of each node in the mth layer (the mth layer is a valid layer and m<k) are less than or equal to the preset threshold, the search, sorting and output process ends at the mth layer, and the nodes from the first effective layer to the m-1th layer are output according to the above method, and there is no need to search, sort and output the nodes from the mth layer to the kth layer.
[0123] When sorting the nodes of each layer according to the probability value, you can first select the nodes whose probability values are greater than the preset threshold, and then sort these nodes in order from small to large probability values, and then output them; you can also first sort all the nodes in the current level in order from small to large probability values, and then eliminate the nodes whose probability values are less than or equal to the preset threshold, and then output them. This application does not limit this.
[0124] The above-mentioned search, sorting and output method can provide the most likely multiple identification results of the item, or the most likely identification result of the item, that is, the item corresponding to the last node output in the BFS output process can be the most likely item type.
[0125] As another example, you can start from the last layer of the tree graph and search towards the root node for nodes that meet the preset conditions. Figure 6In the tree diagram shown, assuming that N is 3, the kth layer is the last layer, and the nodes in the kth layer whose probability values are greater than or equal to the preset threshold include the nodes corresponding to the probability value of 0.15, then the node can be included in the N recognition results for output, and the kth layer can be called the first node layer. Continue to search for nodes that meet the preset conditions in the k-1th layer until 3 nodes that meet the preset conditions are searched out. The number of nodes in the k-1th layer that are greater than or equal to the preset threshold is 3, then the first two nodes with the largest probability values can be taken out as the recognition result output. Similarly, if the number of nodes in the k-1th layer that are greater than the preset threshold is less than 2, you can continue to search the k-2th layer, and select the nodes with larger probability values in the k-2th layer until the first valid layer is searched or 3 nodes that meet the conditions are selected.
[0126] If the kth layer includes 3 or more nodes whose probability values are greater than or equal to the preset threshold, for example, in addition to the node corresponding to the probability value of 0.15, the kth layer also includes nodes with probability values of 0.16 and 0.17, then the nodes corresponding to these three probability values can be used as recognition results, and there is no need to search the node layers above the k-1th layer.
[0127] If the kth layer does not include nodes with probability values greater than or equal to the preset threshold, there is no need to output the nodes of the kth layer, and continue to search for nodes that meet the preset conditions in the k-1th layer. When the k-1th layer includes nodes with probability values greater than or equal to the preset threshold, output the nodes in the k-1th layer, or when the number of nodes greater than or equal to the preset threshold in the k-1th layer is less than 3, continue to output nodes with higher probabilities in the k-2th layer. In this case, the k-1th layer can be called the first node layer.
[0128] In the embodiments provided in the present application, it is possible to adaptively output at different granularities based on the size of the probability value of the node, on the premise that the output recognition result is a valid item, and the output recognition result can be compatible with coarse-grained recognition results and fine-grained recognition results, thereby improving the accuracy of item recognition and optimizing the user experience.
[0129] In some embodiments, the method further includes: determining the predicted value of the node based on the picture information and at least one of the following information: the geographical location where the picture information of the object is obtained, the time when the object is identified, the climate information when the object is identified, and the historical search records.
[0130] Users in different geographical locations have different preferences for different items. For example, with respect to dishes, Chinese users prefer Chinese food, while Western users prefer Western food. Users in different provinces in China may also prefer local cuisines. Therefore, the types of dishes identified by users in different geographical locations may be more inclined towards the types of dishes preferred by local users.
[0131] Different times of identifying items may result in different types of items. For example, in the morning, for example, from 4 to 10 a.m., the types of dishes identified are more likely to be bread, porridge, and steamed buns. In the lunch time period, for example, from 11 to 14 p.m., the types of dishes identified may include meat, vegetarian dishes, and seafood. In the dinner time period, for example, from 17 to 20 p.m., the types of dishes identified may include snacks and baked goods.
[0132] Climate information, such as weather conditions and temperature, may affect the user's choice of dishes such as hot soup and cold drinks. The user's historical search records, such as search and purchase records in third-party applications such as shopping applications, food delivery applications, and sports and health applications, are related to the user's preferences and needs for items. The types of items identified by the user may be items in the historical search records.
[0133] By determining the probability value of an object based on the geographical location where the image information of the object is obtained, or in other words, based on the geographical location where the image of the object is collected, the time of object recognition, the climate when the object is recognized, historical search records and other information, the range of valid nodes can be further narrowed, the computational burden can be reduced, the recognition efficiency can be improved, and the recognition accuracy can be improved.
[0134] When training the model of an object, factors such as the above-mentioned image information of the object, the time of object recognition, the climate when the object was recognized, and historical search records can be added into the model for training to improve the model performance, thereby improving the efficiency and accuracy of object recognition.
[0135] In some embodiments, the method further includes: displaying the identification result of the object. After the identification result of the object is output, the identification result can be displayed on the user interface 410 of the electronic device. For example, the user uses Figure 3 After the user interface 310 or 320 shown in the figure takes a picture of the object to obtain the picture information, the electronic device or the server can use the above-mentioned method for identifying the type of the object to identify the type of the object, and Figure 7The user interface shown shows the recognition results to the user. The user can choose the number of recognition results displayed on the user interface, for example, the number of display results can be selected in the setting item of the application corresponding to the user interface (not shown in the figure). The number of displayed results can be the size of the above-mentioned N. When N is 1, only one most likely recognition result can be displayed on the user interface, that is, the item corresponding to the node with the largest probability value in the first node layer that meets the preset conditions; when N is an integer greater than 1, the corresponding number of recognition results can be displayed on the user interface, and the nodes corresponding to the N items meet the above-mentioned preset conditions. The N displayed recognition results may include item names of different granularity levels, such as the illustrated "steamed egg with shrimp", "steamed egg with minced meat" and "steamed egg", where "steamed egg with shrimp" and "steamed egg with minced meat" are dishes of the same granularity level, and the granularity level is finer than that of "steamed egg".
[0136] If the dish name displayed in the current display box is inaccurate, and the number of dishes displayed is less than the number of dishes selected by the user, the user can click on the expand more control 415 to view more recognition results. The user can select the closest recognition result in the check box 411 of the user interface according to the actual dish type and record the dish. For example, the user can first select the check box 411 corresponding to "shrimp steamed egg", and then click on the record this dish control 413 to enter the dish recording interface.
[0137] If the recognition result is inaccurate, or there is no recognition result the user wants, the user can also choose to retake 412 and return to Figure 3 The user interface 310 or 320 shown reacquires the image information and performs re-recognition.
[0138] Figure 8 Another user interface diagram provided in the embodiment of the present application, the user can click Figure 7 The record this dish control 413 in the user interface shown enters the user interface 510 or 610.
[0139] In some embodiments, when the item is a dish, the above method for identifying the type of item further includes: determining the type of meal to which the dish belongs based on at least one of the following information: the identification result of the dish type, the number of identifications on the day, and the identification time.
[0140] As an example, the time when the user takes a photo to identify a dish is 10:30 in the morning, and it is the first time to identify and record food that day. According to the identification results of the dish types, if the identified dishes are bread, oatmeal, porridge, biscuits, fried dough sticks, steamed buns, etc., then it can be determined that the meal type of the dish is breakfast; if the identified dishes are pickled fish, sweet and sour spare ribs, etc., then it can be determined that the meal type of the dish is lunch.
[0141] As another example, if the user takes a photo to identify a dish at 4 p.m. and it is the second time the user records a meal that day, it can be set as lunch. If it is the third time the user records a meal that day, it can be set as dinner.
[0142] When the user clicks the control "Record this dish 413" to enter the recording interface 510 or 610, the meal type option 511 or 611 of the user interface 510 or 610 can automatically display the meal type of the dish according to the above information, for example, directly displaying "dinner" as shown in the figure. The user can also manually adjust the meal type of the dish, for example, click "lunch" to switch from the currently displayed "dinner" to "lunch".
[0143] In the embodiments provided by the present application, the meal type to which the identified dish belongs is determined based on information such as the recognition result of the dish type, the number of recognitions on the day, and the recognition time. This can avoid the problem of incorrect identification of the meal type to which the dish belongs due to distinguishing the meal type to which the dish belongs based on the fixed time period for dish identification, and improve the accuracy of identifying the meal type to which the dish belongs during the ambiguous time periods between the three meals, such as from 10 to 11 a.m. and from 4 to 5 p.m.
[0144] In some embodiments, when the identified object is a dish, the method may further include: determining a measurement unit of the dish according to the identification result of the dish type and / or the geographical location where the picture information of the dish is obtained.
[0145] Continue to see Figure 8 The user interface schematic diagram shown in the figure shows that on the dish record page, the calorie reference of the identified dish can also be given, for example Figure 8 As shown in (a) of the figure, the calorie content of "shrimp steamed egg" is about 100 kcal per 100 g. The user can also select other measurement units, such as clicking "portion" or "bowl", and the electronic device can then display the calorie content of one portion or one bowl of shrimp steamed egg. Figure 8 As shown in (b) in the figure, the calories in the steamed egg with shrimp can also be measured using units such as "spoon" or "mouth".
[0146] As an example, if the identified dish is a food that is more suitable for eating with a spoon, such as soup or porridge, the spoon can be used as the unit of measurement for the dish. For example, when recording the calories of pork rib soup, the calories of pork rib soup can be recorded as 20 kcal per spoon. If the identified dish is a staple food such as rice, the mouthful can be used as the unit of measurement for the dish. For example, the calories of pineapple rice can be recorded as 100 kcal per 10 mouthfuls.
[0147] As another example, if the geographical location of the dish to be identified or the geographical location of the dish image information to be obtained is detected as China, a unit of measurement may be given according to the preference of Chinese people for dining utensils, and "chopsticks" may be used as the unit of measurement. For example, when recording the calories of pickled fish, the calories of pickled fish may be recorded as 80 kcal per 10 chopsticks.
[0148] When the user clicks on the record dish control 413 to enter the record page, the electronic device can directly display the calorie information of the dish in the units of "chopsticks", "mouth", and "spoon" based on information such as the type of dish and geographical location. The user can also switch to other units of measurement to view the calorie information of the dish according to personal preference.
[0149] In the embodiments provided in the present application, the identified dishes are recorded, for example, references to parameters such as calories are given, which can facilitate the provision of dietary recommendations or health management plans to users, and the measurement units of the dishes are determined based on the recognition results of the dish types and / or the geographical location where the dish image information is obtained. The user does not need to convert the measurement units of the dishes, which is more in line with the usage habits of dining utensils by users in the current geographical location, further optimizing the user experience.
[0150] Combination of the above Figures 2 to 8 The method for identifying an object provided by the embodiment of the present application is described below. Fig. 9 The device for identifying an object provided in an embodiment of the present application is introduced.
[0151] like Fig. 9 As shown, the device 900 may include a processing module 910 and a determination module 920. The determination module 920 and the processing module 910 cooperate with each other so that the device 900 can perform the above Figure 2 For example, the processing module 910 can be used to perform Figure 2 In the described embodiments, steps S210 and S230 and other operations, the determination module 920 can be used to perform Figure 2 The step S220 and other operations in the described embodiment. The determination module 920 can also be used to perform Figure 8 Corresponding to the operations of determining the meal type to which a dish belongs and determining the measurement unit of the dish in the embodiment, the processing module 910 can also be used to perform operations such as searching for nodes that meet preset conditions and sorting the nodes according to probability values.
[0152] Optionally, the device 900 may further include a display module 930, which may be used to display the recognition result on the user interface in response to the user's operation, or to display the meal type to which the dish belongs and the measurement unit of the dish, etc.
[0153] The embodiment of the present application also provides an electronic device 1000 for executing the above-mentioned method for identifying an object. The electronic device 1000 may include a processor 1010 and a memory 1020. The processor may be used to execute the following steps: Figure 2 In the illustrated embodiment, various processing operations are performed by the electronic device, such as obtaining image information of an object, determining the probability value of a node, and outputting the object corresponding to the first node. The memory stores one or more computer programs, and the one or more computer programs include instructions. When the instructions are executed by one or more processors, any of the above methods for identifying an object is executed.
[0154] An embodiment of the present application also provides a chip system, which includes at least one processor. When program instructions are executed in the at least one processor, the method for identifying an object in the above embodiment is implemented.
[0155] An embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above-mentioned related method steps to implement the method for identifying an object in the above-mentioned embodiment.
[0156] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the method for identifying an object in the above-mentioned embodiment.
[0157] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the method of identifying objects in the above-mentioned method embodiments.
[0158] Among them, the electronic device, computer storage medium, computer program product or chip system provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.
[0159] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0160] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0161] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0164] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for identifying an object, characterized in that: include: Get the image information of the item; Determine, according to the image information, a probability value of a valid node in a tree diagram corresponding to the item, wherein the tree diagram includes different item types and corresponding item data, the valid node is a node in the tree diagram including valid data, and the probability value is used to indicate the possibility that the item is the item corresponding to the current node; When the probability value of the current node is less than a preset threshold and the current node is the node with the largest probability value in the node layer to which it belongs, the item corresponding to the first node is output, the first node is the node with the largest probability value in the previous node layer of the node layer to which the current node belongs, and the first node belongs to the valid node.
2. The method according to claim 1, characterized in that Before outputting the item corresponding to the first node, the method further includes: Searching for N nodes that meet a preset condition among the valid nodes, wherein the preset condition includes that a probability value of the node is greater than or equal to a preset threshold, and N is an integer greater than or equal to 1; When the number M of nodes satisfying the preset condition in the first node layer is greater than or equal to N, the N nodes are the N nodes with the largest probability values in the first node layer, and the first node layer is the layer closest to the last layer of the tree diagram and includes the nodes satisfying the preset condition, or, When the number M of nodes satisfying the preset condition in the first node layer is less than N, the N nodes include the M nodes and NM nodes with the largest probability values in the previous node layer of the first node layer, where M is an integer greater than or equal to 1; The first node is included in the N nodes.
3. The method according to claim 2, characterized in that The method further comprises: Output items corresponding to N-1 nodes among the N nodes except the first node.
4. The method according to any one of claims 1 to 3, characterized in that The determining, according to the image information, the probability value of the valid node in the tree diagram corresponding to the item includes: The predicted value of each node in the tree diagram is determined according to the image information, the probability value of the valid node is the product of the predicted value of the current node and the predicted value of the previous node on the propagation path to which the current node belongs, and the predicted value is the numerical value output after the image information passes through the artificial intelligence AI model.
5. The method according to claim 4, characterized in that The step of determining the predicted value of each node in the tree diagram according to the picture information includes: Determine the predicted value of each node in the tree diagram according to the image information and at least one of the following information: The time when the picture information is obtained, the geographical location when the picture information is obtained, the climate information when the picture information is obtained, and the historical search records.
6. The method according to any one of claims 1 to 5, characterized in that Each node in the tree diagram includes one or more data sets, and the one or more data sets include data corresponding to one or more names of the items, and the item types indicated by the one or more names of the items are the same.
7. The method according to any one of claims 1 to 6, characterized in that The item is a dish, and the method further comprises: The meal category to which the dish belongs is determined based on at least one of the following information: The recognition result of the dish, the number of recognitions on the day and the recognition time.
8. The method according to any one of claims 1 to 7, characterized in that The item is a dish, and the method further comprises: The measurement unit of the dish is determined according to the recognition result of the dish and / or the geographical location where the picture information of the dish is obtained.
9. A device for identifying an object, characterized in that: The method comprises modules or units for executing each step in the method according to any one of claims 1 to 8.
10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method for identifying an object as described in any one of claims 1 to 8.
11. A chip system, characterized in that: The chip system includes at least one processor, and when program instructions are executed in the at least one processor, the method for identifying an object as claimed in any one of claims 1 to 8 is implemented.
12. A computer program storage medium, characterized in that: The computer-readable medium stores a program code, and when the program code is executed on a computer, the computer is caused to execute the method for identifying an object according to any one of claims 1 to 8.
Citation Information
Patent Citations
Method and device for outputting information
CN111046170A
Hierarchical classification method and device
CN113536851A
