Information processing methods and apparatus
By identifying and generating mask images from food photographs and combining them with three-dimensional spatial information to calculate the physical scale of food objects, the problem of inaccurate analysis results in existing technologies is solved, enabling accurate nutritional and weight analysis of food objects.
Patent Information
- Application Number
- CN202611539122.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-30
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies struggle to accurately match user-specified food objects with analysis results when analyzing food photos, resulting in the final displayed nutrition and weight information not meeting the user's actual needs.
By acquiring images and responding to user input, the set of food objects to be analyzed is determined, a mask image of each food object is generated, and the physical scale information of each food object is determined using three-dimensional spatial information and mask images, thereby calculating its weight and nutritional analysis results.
This ensures that the analysis results accurately reflect the food objects specified by the user, improves the accuracy of nutritional and weight information, and avoids interference from background or adjacent foods.
Smart Images

Figure CN123072005A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of information processing technology, and specifically relates to an information processing method and apparatus. Background Technology
[0002] With the development of mobile terminal camera and food recognition technology, users can take pictures of meals using mobile phones, tablets, or smart glasses to obtain information such as dish name, weight, calories, or nutrition. However, a single food photo often contains main dishes, beverages, side dishes, tableware, and packaging, and multiple instances of the same food may appear. But users may only be interested in one or a group of items; for example, a meal may contain two hamburgers and a beverage, and the user may only want to know the weight and calories of the two hamburgers on the left. Existing solutions typically analyze the entire image first and then select one item from the results for display. This can easily lead to the final analysis result targeting an object that is not what the user intended. Summary of the Invention
[0003] The purpose of this application is to provide an information processing method and apparatus that enables the final displayed nutritional analysis results and weight information to correspond to the food object actually specified by the user.
[0004] In a first aspect, embodiments of this application provide an information processing method, the method comprising: Get the first image; Receive the user's first input; In response to the first input, a set of food objects to be analyzed is determined from the first image, the set of food objects including N food objects to be analyzed, where N is a positive integer; Determine the mask image corresponding to each food object in the set of food objects; Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, the physical scale information corresponding to each food object is determined. Based on the physical scale information corresponding to each of the food objects, the weight information and nutritional analysis results corresponding to each of the food objects are determined. The target analysis results information is displayed, which includes the weight information and nutritional analysis results information corresponding to each food object.
[0005] Secondly, embodiments of this application provide an information processing apparatus, which includes: The acquisition module is used to acquire the first image; The receiving module is used to receive the user's initial input; A first determining module is configured to, in response to the first input, determine a set of food objects to be analyzed from the first image, wherein the set of food objects includes N food objects to be analyzed, where N is a positive integer; The second determining module is used to determine the mask image corresponding to each food object in the food object set; The third determining module is used to determine the physical scale information corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object. The fourth determining module is used to determine the weight information and nutritional analysis results of each food object based on the physical scale information corresponding to each food object. The display module is used to display target analysis results information, which includes weight information and nutritional analysis results information for each food object.
[0006] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can run on the processor, and the programs or instructions, when executed by the processor, implement the method as described in the first aspect.
[0007] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the method described in the first aspect.
[0008] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0009] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.
[0010] In this embodiment, in response to a user's first input, a set of food objects to be analyzed is determined from the acquired first image. Furthermore, a mask image corresponding to each food object in the set is determined, ensuring that subsequent processing is not performed on the entire image but rather the calculation range is continuously constrained by the user-specified food objects. Based on this, according to the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, the physical scale information corresponding to each food object is determined. The weight information and nutritional analysis results of the food object are then determined from this physical scale information. Finally, the target analysis results information, including the nutritional analysis results and weight information corresponding to each food object, is displayed. Thus, the user-specified food object corresponds to the object involved in mask generation, physical scale calculation, weight information determination, nutritional inference, and result display. This improves the problem of inconsistency between the query object and the calculation object caused by first analyzing the entire image and then selecting one result to display, making the final displayed nutritional analysis results and weight information more consistent with the food object actually specified by the user. Furthermore, in this embodiment, the physical scale information extracted from the three-dimensional spatial information of the first image based on the mask image of the current food object itself is used as an explicit condition for determining the weight information and nutritional analysis results, rather than simply multiplying the volume by a fixed density or querying the nutritional value per unit weight by category. This allows food objects of the same category but with different actual portions to obtain nutritional analysis results that match their true size, thereby improving the accuracy of the nutritional analysis results of the food objects. Attached Figure Description
[0011] Figure 1 This is a flowchart illustrating the information processing method provided in some embodiments of this application; Figure 2 These are schematic diagrams showing the target analysis results provided in some embodiments of this application; Figure 3 This is a flowchart illustrating the information processing method provided in some embodiments of this application; Figure 4 This is a flowchart illustrating the information processing method provided in some embodiments of this application; Figure 5 These are schematic diagrams illustrating the structure of an information processing apparatus according to some embodiments of this application; Figure 6 These are schematic diagrams illustrating the structure of an electronic device according to some embodiments of this application; Figure 7 These are schematic diagrams illustrating the hardware structure of an electronic device according to some embodiments of this application. Detailed Implementation
[0012] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or N objects. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0014] The terminology used in the embodiments of this application will be explained below.
[0015] The identifiers in this application are text, symbols, images, etc. used to indicate information, and can be displayed in the form of controls or other containers, including but not limited to text identifiers, symbol identifiers, and image identifiers.
[0016] Interface: Refers to the graphical interactive layer seen by users through the screen of an electronic device. Also known as the "user interface" (UI), it is the medium through which applications or operating systems interact and exchange information with users, converting the internal form of information into a form acceptable to the user. The user interface is source code written in specific computer languages such as Java and XML. This source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can include visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and web widgets displayed on the screen of an electronic device.
[0017] Artificial Intelligence (AI) is a new technological science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. It is a branch of computer science. It attempts to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.
[0018] The technical solutions of this application can be applied to scenarios requiring personalized nutritional analysis of food objects in meal images. For example, a user may want to analyze a photo of a meal they took, which contains two hamburgers, a drink, and fries. The user only wants to know the weight and calories of the two hamburgers on the left, and does not want the drink and fries to be included in the analysis. As another example, a user may want to analyze only the nutritional composition of a specific piece of steak in a picture of a steak meal, and does not want the side dishes or tableware next to it to be included in the analysis.
[0019] The information processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0020] Figure 1 This is a flowchart illustrating an information processing method provided in an embodiment of this application. The subject executing this information processing method can be an electronic device, which may be, but is not limited to, a personal computer (PC), a smartphone, a tablet computer, or a personal digital assistant (PDA).
[0021] like Figure 1 As shown, the information processing method provided in this application embodiment may include steps 110-170.
[0022] Step 110: Obtain the first image.
[0023] The first image can be a scene image containing one or more food objects taken by the user through an electronic device, or it can be an image containing food objects selected from a photo album. For example, in the first example above, the first image could be a photo of a meal containing two hamburgers, a drink, and fries. In the second example above, the first image could be a photo of a meal containing steak and side dishes.
[0024] Step 120: Receive the user's first input.
[0025] The first input can be a user instruction indicating the food to be analyzed, which triggers the determination of a set of food objects to be analyzed from the first image. Exemplarily, the first input includes, but is not limited to: touch input by the user on the captured food image using a finger or stylus, a voice command input by the user, a specific gesture input by the user, or other feasible inputs, which can be determined according to actual usage needs and are not limited in this embodiment. The specific gesture in this embodiment can be any one of a single-click gesture, a swipe gesture, a drag gesture, a pressure-recognition gesture, a long-press gesture, an area-change gesture, a double-press gesture, a double-tap gesture, or a shaking gesture; the click input in this embodiment can be a single-click input, a double-tap input, or any number of clicks, and can also be a long-press input or a short-press input. For example, the first input can be a user touch input on the captured food image, such as a selection input by the user on the captured food image.
[0026] In some embodiments of this application, the first input may include at least one of food semantics, spatial reference, quantity constraint, or exclusion constraint. For example, in the first example above, if the user inputs the voice "How many calories are in the two hamburgers on the left?", it contains the food semantics "hamburger", the quantity constraint "2", and the spatial reference "left". As another example, in the second example above, if the user inputs the voice "Only count the steak, not the side dishes", it contains the food semantics "steak" and the exclusion constraint "not counting the side dishes".
[0027] Step 130: In response to the first input, determine the set of food objects to be analyzed from the first image.
[0028] The food object set can be a specific set of food objects to be analyzed, that is, the food object set can include N food objects to be analyzed, where N is a positive integer.
[0029] In some embodiments of this application, the electronic device can respond to a first input, detect candidate food objects in a first image, and determine a set of food objects to be analyzed from the candidate food objects based on the first input. The selection of the food objects to be analyzed can combine at least one of the following: food category, detection confidence, touch position, overlapping area of bounding boxes, distance between candidate centers, and quantity constraints. For example, if a user voice inputs "how many calories are in these two hamburgers on the left?", the electronic device can detect two hamburger candidate objects, a beverage candidate object, and a serving of fries candidate objects. Based on the food semantics "hamburger", the quantity constraint "2", and the spatial reference "left", it can determine that the set of food objects consists of the two hamburgers on the left, while excluding the beverage and fries from the set of food objects.
[0030] The food categories mentioned above refer to what food a candidate food object is identified as, such as "hamburger," "drink," "fries," and "steak." Electronic devices assign a category label to each candidate object through object detection or open-vocabulary detection; the user's initial input usually also contains food semantics, and the electronic device retains candidate objects that match the category accordingly.
[0031] Detection confidence is the degree of confidence a target detection model has in believing that a candidate bounding box is indeed a certain type of food. It is typically a score between 0 and 1. A higher confidence score indicates that the model is more certain that the candidate object belongs to that food category. For example, suppose an electronic device detects three "hamburger" candidates: Hamburger A has a confidence score of 0.95, Hamburger B has a confidence score of 0.93, and Candidate C has a confidence score of 0.35 (possibly a false positive due to a desktop texture). The user wants two hamburgers, with a quantity constraint of 2. Among the candidates in the category of "hamburger," the electronic device prioritizes the two with higher confidence scores, namely Hamburger A and Hamburger B, and excludes the low-confidence candidate C.
[0032] Touch location refers to the coordinates of the position where a user taps or touches on the screen of an electronic device. Based on this location, the electronic device determines which candidate food object the user is referring to, such as which candidate box or mask the touch point falls within. For example, if the user doesn't say "two hamburgers," but instead taps on hamburger A and then hamburger B, and then says "how many calories are in these two?", the electronic device will determine that the touch point falls within the corresponding candidate box or mask for hamburger A and hamburger B, respectively, thus selecting hamburger A and hamburger B. If the user only taps hamburger A and says "how many calories are in this hamburger?", only hamburger A will be selected.
[0033] The overlap area between the user-drawn selection box and the candidate boxes or masks of the candidate objects is the area of overlap. A larger overlap area indicates that the candidate object is more likely to be the user's desired selection. For example, a user draws a box on the screen, including both hamburgers on the left, but excluding the drink and fries. The electronic device calculates the overlap area between the user's selection area and each candidate object: the overlap area with hamburger A and hamburger B is large, while the overlap area with the drink and fries is small or zero. Therefore, hamburger A and hamburger B are selected, while the drink and fries are excluded. If the user only selected hamburger A, then only hamburger A is selected.
[0034] The candidate center distance is the distance between the center point of a candidate object and the user-specified location (touch point, selection center, gaze point, etc.). The smaller the distance, the closer the candidate object is to the user-specified location, and the more likely it is to be the target the user wants to select. For example, if the user taps the left side of the screen near hamburger A, the electronic device calculates the distance between the center of each candidate object and the touch point: hamburger A is the closest, followed by hamburger B, while the drink and fries are farther away. If the user says "this hamburger," the electronic device selects the closest hamburger A. If the user says "these two hamburgers," the electronic device, combined with the quantity constraint "2," selects the two closest hamburger candidates, namely hamburger A and hamburger B.
[0035] Quantity constraints are explicit quantity requirements given in the user's initial input, such as "one," "two," "three," or "count only one." Based on these constraints, the electronic device selects a specified number of target objects from the candidate list. For example, if a user says, "How many calories are in these two hamburgers?", the initial input includes the food semantic "hamburger" and the quantity constraint "2." The electronic device first filters out candidate objects categorized as "hamburger," and then, combining detection confidence, touch location, and candidate center distance, selects two hamburger candidates to form a food object set. If the electronic device detects three hamburger candidates, the quantity constraint "2" helps it select only two, rather than including all three in the analysis.
[0036] In some embodiments of this application, when the food object set includes multiple food objects, a common set identifier can be assigned to the multiple food objects, and an object identifier can be assigned to each food object separately, so as to maintain the consistency of the identity of each food object in subsequent processing.
[0037] Step 140: Determine the mask image corresponding to each food object in the food object set.
[0038] In this context, the mask image corresponding to a food object can be a pixel-level binary region image used to define the pixel range occupied by the food object in the image. One food object corresponds to one mask image, and different food objects correspond to their own separate mask images.
[0039] In some embodiments of this application, an open vocabulary object detector or a food object detector can be used to detect the first image to obtain multiple candidate boxes. Each candidate box corresponds to a candidate region that may contain a food object. Then, based on the first input, the target box corresponding to the food object to be analyzed is determined from the multiple candidate boxes. The target box and the first image are then sent to the segmentation model to generate a mask image corresponding to each food object.
[0040] In other embodiments, an instance segmentation model can be directly used to process the first image once, outputting multiple candidate boxes and instance masks. Each candidate box corresponds to a candidate region that may contain a food object. Then, based on the first input, the candidate box and instance mask corresponding to the food object to be analyzed are determined from the candidate boxes and instance masks, serving as the mask image corresponding to each food object. If the user provides a touch point or a selected area, the electronic device can use the overlap between the image coordinates of the user's touch and the candidate boxes or instance masks as selection criteria to determine the candidate box corresponding to the food object to be analyzed from the candidate boxes.
[0041] Continuing with the first example above, for two hamburgers, the electronic device can generate separate mask images for hamburger A and hamburger B. These two mask images are retained separately and not merged into a single large mask before the subsequent extraction of physical scale information. This avoids the background, tabletop, and adjacent drinks and fries between the two hamburgers from being included in the scale calculation.
[0042] Step 150: Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, determine the physical scale information corresponding to each food object.
[0043] The three-dimensional spatial information can be physical scale-based metric spatial information corresponding to the first image. For example, the three-dimensional spatial information can be a metric depth map or a three-dimensional point map, which is not limited in this embodiment. Each pixel in the metric depth map can correspond to a depth value in meters; each pixel in the three-dimensional point map can correspond to a three-dimensional point in the camera coordinate system.
[0044] The physical scale information corresponding to a food object can refer to information used to represent the actual size of the food object in real physical space and / or the distance between it and the camera. It is determined by the three-dimensional spatial information corresponding to the first image and the mask image corresponding to the food object, and is used to determine the weight information and nutritional analysis results of the food object.
[0045] In some embodiments of this application, three-dimensional spatial information with physical scale can be generated first from a first image. Then, based on the mask image of the current food object, the set of pixels corresponding to the food object can be obtained from the three-dimensional spatial information, and the physical scale information of the food object can be obtained from the set of pixels. That is, the mask image can be used to limit the range of data selected from the three-dimensional spatial information, so that the physical scale information comes from the pixels of the current food object itself, rather than the entire image or adjacent food objects.
[0046] In some embodiments of this application, step 150 may specifically include: Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, determine the set of pixels corresponding to each food object. Based on the set of pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
[0047] Here, the set of pixels corresponding to a food object can refer to the set of pixels belonging to the food object, determined based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to the food object. That is, the pixels in this set come from the range defined by the mask image corresponding to the food object, rather than from the entire first image.
[0048] In some embodiments of this application, for each food object in the food object set, the electronic device determines the set of pixels corresponding to that food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to that food object. That is, the electronic device does not directly determine a uniform set of pixels for the entire first image, but rather determines the set of pixels corresponding to each food object separately. For different food objects, their respective mask images are used to determine the pixels belonging to that food object from the three-dimensional spatial information corresponding to the first image, thereby obtaining the set of pixels corresponding to each food object.
[0049] Furthermore, for each food object in the food object set, the electronic device obtains the physical scale information corresponding to that food object based on the set of pixels corresponding to that food object. In other words, the physical scale information is not obtained based on the entire first image, nor is it based on the pixel set of multiple food objects combined, but rather on the individual pixel set of each food object. Different food objects obtain their own physical scale information based on their respective pixel sets.
[0050] Continuing with the first example above, the first image is a photograph of the meal. The food object set includes the two hamburgers on the left, namely Hamburger A and Hamburger B. The electronic device determines the mask image corresponding to Hamburger A and the mask image corresponding to Hamburger B, respectively. Then, based on the 3D spatial information corresponding to the first image and the mask image corresponding to Hamburger A, the electronic device determines the pixel set corresponding to Hamburger A; based on the 3D spatial information corresponding to the first image and the mask image corresponding to Hamburger B, it determines the pixel set corresponding to Hamburger B. Since Hamburger A and Hamburger B use their own mask images, the pixel set corresponding to Hamburger A mainly comes from the pixel range where Hamburger A is located, and the pixel set corresponding to Hamburger B mainly comes from the pixel range where Hamburger B is located. The beverage and fries do not have corresponding mask images, therefore they are not involved in determining the pixel sets of Hamburger A and Hamburger B.
[0051] Then, the electronic device obtains the physical scale information corresponding to Hamburger A based on the pixel set corresponding to Hamburger A; and obtains the physical scale information corresponding to Hamburger B based on the pixel set corresponding to Hamburger B. That is, the physical scale information of Hamburger A comes from its own pixel set, and the physical scale information of Hamburger B comes from its own pixel set; the two are obtained separately and are not used interchangeably.
[0052] In the embodiments of this application, based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, the set of pixels corresponding to each food object is first determined. Then, based on the set of pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained. This ensures that the physical scale information of each food object comes from its own set of pixels, rather than from the entire first image or the combined set of pixels of multiple food objects. Therefore, the user-specified food object corresponds to the pixels involved in determining the physical scale information. This reduces the possibility of including tableware, background, adjacent food, or non-target food in the physical scale information due to uniform processing of the entire image. The resulting physical scale information is more consistent with the food object actually specified by the user, improving the accuracy of the food object analysis results.
[0053] In some embodiments of this application, determining the set of pixels corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object may specifically include: For each food object, the optical axis direction depth information corresponding to each pixel in the mask image is determined from the three-dimensional spatial information corresponding to the first image. Based on the coordinate information of each pixel in the mask image, the optical axis direction depth information corresponding to each pixel, and the camera parameters when the first image was captured, the three-dimensional coordinate information of each pixel in the mask image is obtained. Based on the three-dimensional coordinate information of each pixel in the mask image corresponding to each food object, determine the set of pixels corresponding to each food object.
[0054] The optical axis direction depth information refers to depth information with physical units along the camera's optical axis direction when the first image was captured. That is, this optical axis direction depth information does not merely represent relative depth in terms of distance, but rather corresponds to a depth value in real physical space. For each pixel in the mask image, the optical axis direction depth information corresponding to that pixel can be determined from the three-dimensional spatial information corresponding to the first image.
[0055] The coordinate information of each pixel in the mask image can refer to the coordinate information of that pixel in the first image, such as the coordinates of the pixel in the image coordinate system. This coordinate information is used in conjunction with the optical axis depth information and camera parameters to calculate the three-dimensional coordinate information of that pixel.
[0056] Camera parameters can refer to the camera's intrinsic parameters when capturing the first image, such as focal length and principal point. These camera parameters are used to convert the coordinates of pixels in the mask image and the corresponding optical axis depth information into three-dimensional coordinates.
[0057] The three-dimensional coordinate information of a pixel can refer to the three-dimensional coordinate information of that pixel in the camera coordinate system. For each pixel in the mask image, the three-dimensional coordinate information of that pixel can be obtained based on the pixel's coordinate information, the corresponding optical axis depth information, and the camera parameters when the first image was captured.
[0058] In some embodiments of this application, for each food object in the food object set, the electronic device uses the mask image corresponding to the food object to determine the optical axis direction depth information corresponding to each pixel in the mask image from the three-dimensional spatial information corresponding to the first image. That is, the electronic device does not directly determine the optical axis direction depth information for the entire first image, but rather determines the optical axis direction depth information corresponding to each pixel in the mask image for each food object from the three-dimensional spatial information corresponding to the first image. For different food objects, their respective mask images are used to determine the optical axis direction depth information belonging to that food object from the three-dimensional spatial information corresponding to the first image.
[0059] For each food object's corresponding mask image, the electronic device obtains the three-dimensional coordinate information of each pixel in the mask image based on the coordinate information of each pixel, the optical axis depth information corresponding to each pixel, and the camera parameters at the time the first image was captured. In other words, for each pixel in the mask image, the electronic device uses the pixel's coordinate information, the corresponding optical axis depth information, and the camera parameters at the time the first image was captured to obtain the pixel's three-dimensional coordinate information. Thus, each pixel in the mask image can obtain its corresponding three-dimensional coordinate information.
[0060] For each food object, the electronic device determines the set of pixels corresponding to that food object based on the 3D coordinates of each pixel in the mask image corresponding to that food object. In other words, the 3D coordinates of each pixel in the mask image corresponding to a food object collectively constitute the set of pixels corresponding to that food object. Different food objects determine their own set of pixels based on the 3D coordinates of each pixel in their respective mask images.
[0061] Specifically, for each pixel in the mask image, the three-dimensional coordinates of that pixel can be obtained according to the pinhole camera model based on its coordinates, the depth along the optical axis corresponding to that pixel, and the camera parameters at the time the first image was captured. For example, let the coordinates of the pixel be... The camera parameter matrix is Then the depth along the optical axis corresponding to that pixel is Then the three-dimensional coordinate information of the pixel in the camera coordinate system can be expressed as shown in the following formula (1).
[0062] In the above formula (1), Let be a three-dimensional row vector, where, For pixels The horizontal coordinate in the image, in pixels; For pixels The vertical coordinates in the image are in pixels; 1 is a third, artificially added component used to express the two-dimensional pixel coordinates in homogeneous coordinate form. This is a transpose.
[0063] In the above formula (1), It can be represented by the following formula (2): In the above formula (2), The equivalent focal length of the camera in the horizontal (X) direction of the image, expressed in pixels; This is the equivalent focal length of the camera in the vertical direction (Y direction) of the image, expressed in pixels; that is... and The focal length of the camera; The coordinates of the camera principal point in the horizontal direction (X direction) of the image; The coordinates of the camera principal point in the vertical direction (Y direction) of the image.
[0064] The aforementioned key point can be understood as the pixel position on the image where the intersection of the camera's optical axis and the image plane is located.
[0065] After unfolding, the three-dimensional coordinate information of the pixel can be expressed as the following formula (3).
[0066] In the above formula (3), , and These are the X-axis, Y-axis, and Z-axis coordinates of the pixel.
[0067] For mask images By performing the above calculations on each pixel in the image, the mask image can be obtained. The three-dimensional coordinate information of each pixel in the mask image. Target point set generated by pixels within range It can be represented as: The above The meaning is, the first A mask image of a food object at pixels A value of 1 at this location indicates that the pixel belongs to the th position. A food object.
[0068] Continuing with the first example above, for the mask image corresponding to Hamburger A, the electronic device determines the optical axis direction depth information of each pixel in the mask image corresponding to Hamburger A from the three-dimensional spatial information corresponding to the first image. For the mask image corresponding to Hamburger B, the electronic device determines the optical axis direction depth information of each pixel in the mask image corresponding to Hamburger B from the three-dimensional spatial information corresponding to the first image.
[0069] For each pixel in the mask image corresponding to Hamburger A, the electronic device obtains the three-dimensional coordinate information of the pixel based on the pixel's coordinate information, the optical axis depth information corresponding to the pixel, and the camera parameters when the first image was captured. Similarly, for each pixel in the mask image corresponding to Hamburger B, the electronic device obtains the three-dimensional coordinate information of the pixel based on the pixel's coordinate information, the optical axis depth information corresponding to the pixel, and the camera parameters when the first image was captured.
[0070] The electronic device determines the set of pixels corresponding to hamburger A based on the three-dimensional coordinate information of each pixel in the mask image corresponding to hamburger A; and determines the set of pixels corresponding to hamburger B based on the three-dimensional coordinate information of each pixel in the mask image corresponding to hamburger B.
[0071] In the embodiments of this application, for each food object, the optical axis direction depth information corresponding to each pixel in the mask image is determined from the three-dimensional spatial information corresponding to the first image. Then, based on the coordinate information of each pixel in the mask image, the optical axis direction depth information corresponding to each pixel, and the camera parameters when the first image was captured, the three-dimensional coordinate information of each pixel in the mask image is obtained. Furthermore, based on the three-dimensional coordinate information of each pixel in the mask image corresponding to each food object, a pixel set corresponding to each food object is determined. This ensures that the pixel set of each food object comes from the three-dimensional coordinate information of each pixel in its own mask image, rather than from the pixel set of the entire first image or a merged set of multiple food objects. Therefore, the user-specified food object corresponds to the pixels participating in the pixel set determination, reducing the possibility of including tableware, background, adjacent food, or non-target food in the pixel set due to uniform processing of the entire image. This makes the obtained pixel set more consistent with the user-specified food object, thereby making the subsequent physical scale information obtained based on this pixel set more accurate.
[0072] In some embodiments of this application, the aforementioned three-dimensional spatial information may include the Euclidean distance from the camera center to a scene point. The aforementioned scene point can be the scene three-dimensional point corresponding to each pixel in the mask image; that is, for each pixel in the mask image, that pixel corresponds to a scene three-dimensional point in the real scene, and that scene three-dimensional point is the scene point corresponding to that pixel.
[0073] The step of determining the optical axis depth corresponding to each pixel in the mask image from the three-dimensional spatial information corresponding to the first image may specifically include: Based on the Euclidean distance and camera parameters corresponding to each pixel in the mask image, the Euclidean distance corresponding to each pixel in the mask image is converted into the optical axis depth.
[0074] The Euclidean distance from the camera center to a scene point can be defined as the straight-line distance between the camera center at the time the first image is captured and that 3D point in the scene. This Euclidean distance is a distance value used to represent the distance between the camera center and the 3D point in the scene.
[0075] In some embodiments of this application, for each pixel in the mask image, the electronic device converts the Euclidean distance corresponding to the pixel into the optical axis direction depth corresponding to the pixel based on the Euclidean distance corresponding to the pixel and the camera parameters when the first image was captured.
[0076] Specifically, for each pixel in the mask image, the direction vector corresponding to that pixel can be determined based on its coordinate information and the camera parameters at the time the first image was captured. For example, let the coordinate information of the pixel be... The camera parameter matrix is Then the direction vector corresponding to this pixel can be represented as Normalizing this direction vector yields the unit direction vector. Since the Euclidean distance from the camera center to the scene point is Therefore, the 3D point of the scene corresponding to this pixel can be represented as The component of a 3D point in the scene along the camera's optical axis is the depth along the optical axis. For example, when the optical axis of the camera coordinate system is the Z-axis, the depth along the optical axis... It can be represented as ,in, Direction vector The component in the Z direction.
[0077] The direction vector can then be written as the following formula (4): Its modulus is given by the following formula (5): Therefore, the depth along the optical axis can be expressed as the following formula (6): Continuing with the first example above, the 3D spatial information corresponding to the first image includes the Euclidean distance from the camera center to the scene point. For each pixel in the mask image corresponding to Hamburger A, this pixel corresponds to a 3D scene point in the real scene, which is the scene point corresponding to that pixel. The Euclidean distance from the camera center to this scene point is the Euclidean distance corresponding to that pixel. Similarly, for each pixel in the mask image corresponding to Hamburger B, it also corresponds to a 3D scene point and the Euclidean distance from the camera center to that 3D scene point.
[0078] For each pixel in the mask image corresponding to Hamburger A, the electronic device converts the Euclidean distance corresponding to that pixel into the optical axis depth based on the Euclidean distance of that pixel and the camera parameters when the first image was captured. Similarly, for each pixel in the mask image corresponding to Hamburger B, the electronic device converts the Euclidean distance corresponding to that pixel into the optical axis depth based on the Euclidean distance of that pixel and the camera parameters when the first image was captured.
[0079] In the embodiments of this application, when the three-dimensional spatial information includes the Euclidean distance from the camera center to the scene point, the Euclidean distance corresponding to each pixel in the mask image is converted into optical axis depth based on the Euclidean distance corresponding to each pixel in the mask image and the camera parameters. This ensures that even when the three-dimensional spatial information provides Euclidean distance instead of optical axis depth, the optical axis depth corresponding to each pixel in the mask image can still be obtained. This provides a foundation for subsequently obtaining three-dimensional coordinate information based on the coordinate information of each pixel in the mask image, the optical axis depth, and the camera parameters. This ensures that the pixel set of each food object still comes from the range defined by the mask image of that food object itself, reducing the impact of different forms of three-dimensional spatial information on the determination of the pixel set, and making the obtained pixel set more consistent with the food object actually specified by the user.
[0080] In some embodiments of this application, after determining the set of pixels corresponding to each food object, the method described above may further include: For each set of pixels corresponding to a food object, perform point set cleanup processing on the set of pixels corresponding to the food object to obtain the target set of pixels corresponding to the food object. The process of obtaining the physical scale information for each food object based on the set of pixels corresponding to each food object can specifically include: Based on the set of target pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
[0081] Point set cleaning can refer to cleaning the set of pixels corresponding to a food object to remove pixels that do not belong to the food object or have low reliability, thereby obtaining the target set of pixels corresponding to the food object. The aforementioned point set cleaning process can include at least one of edge shrinkage, outlier removal, and connectivity checking.
[0082] The aforementioned edge shrinkage can refer to shrinking the edges of the mask image corresponding to the food object inward by a certain number of pixels, and then redetermining the pixel set based on the shrunken mask image, or removing pixels located near the edges of the mask image from the pixel set. Since pixels from plates, tabletops, or adjacent food items can easily be mixed in at the edges of the mask image, edge shrinkage can reduce the risk of these edge pixels entering the target pixel set.
[0083] Outlier removal refers to removing pixels that significantly deviate from the main distribution based on the depth distribution of pixels in a pixel set. For example, outliers can be removed based on the median depth, quartile intervals, local neighborhood consistency, or clustering results. Outlier removal can reduce the impact of a small number of erroneous pixels caused by depth anomalies or segmentation errors on the target pixel set.
[0084] Connectivity checking refers to examining the connectivity relationships between pixels in a pixel set. When a food object exists in an image with multiple unconnected regions, corresponding point clusters can be selected based on candidate box centers, the largest connected region, or target semantics, eliminating connected regions that do not belong to the food object. Connectivity checking avoids including falsely detected connected regions from adjacent food objects or the background in the target pixel set.
[0085] The target pixel set can refer to the pixel set obtained after performing point set cleanup on the pixel set corresponding to the food object. The pixels in this target pixel set come from the pixel set corresponding to the food object, but have undergone at least one of the following processes: edge shrinkage, outlier removal, and connectivity check.
[0086] In some embodiments of this application, for each food object in the food object set, the electronic device performs point set cleaning processing on the pixel set corresponding to the food object to obtain the target pixel set corresponding to the food object. The aforementioned point set cleaning processing includes at least one of edge shrinkage, outlier removal, and connectivity checking. For example, only edge shrinkage, outlier removal, or connectivity checking can be performed; or a combination of two or three of these processes can be performed.
[0087] Then, for each food object in the food object set, the electronic device obtains the physical scale information corresponding to the food object based on the target pixel set corresponding to that food object.
[0088] Continuing with the first example above, for the pixel set corresponding to Hamburger A, the electronic device performs point set cleaning processing to obtain the target pixel set corresponding to Hamburger A. For example, a small number of desktop pixels may be mixed in at the edge of the mask image corresponding to Hamburger A. The electronic device can shrink the edge of the mask image inward by a few pixels through edge shrinkage, thereby reducing the risk of desktop pixels entering the target pixel set corresponding to Hamburger A. Another example is that there may be a small number of points with depth anomalous in the pixel set corresponding to Hamburger A. The electronic device can remove these points with depth anomalous through outlier removal. Yet another example is that there may be multiple unconnected regions in the image for Hamburger A. The electronic device can select the connected regions corresponding to Hamburger A through connectivity checks and remove other connected regions. Similarly, for the pixel set corresponding to Hamburger B, the electronic device also performs point set cleaning processing to obtain the target pixel set corresponding to Hamburger B. Then, based on the target pixel set corresponding to Hamburger A, the electronic device obtains the physical scale information corresponding to Hamburger A; based on the target pixel set corresponding to Hamburger B, it obtains the physical scale information corresponding to Hamburger B.
[0089] In the embodiments of this application, after determining the pixel set corresponding to each food object, a pixel set cleaning process is performed on the pixel set corresponding to each food object to obtain the target pixel set corresponding to that food object. Based on the target pixel set corresponding to each food object, the physical scale information corresponding to each food object is obtained. This ensures that the pixel set used to determine the physical scale information is the cleaned target pixel set, not the pixel set before cleaning. This reduces the possibility of outliers caused by mask image edges being mixed in with plates, tabletops, or adjacent food, outliers due to depth anomalies, or non-target regions being included in the pixel set due to the existence of multiple unconnected regions. This makes the obtained physical scale information more consistent with the food object actually specified by the user. Furthermore, since the pixel set cleaning process is performed separately for each food object, different food objects obtain their own target pixel set and their own physical scale information, reducing the possibility of pixel mixing between multiple food objects. This leads to more accurate weight information and nutritional analysis results obtained subsequently based on the physical scale information.
[0090] In some embodiments of this application, the physical scale information corresponding to a food object may include the distance information from the camera to the food object and the size information of the food object; The process of obtaining the physical scale information for each food object based on the set of pixels corresponding to each food object can specifically include: For each set of pixels corresponding to a food object, the distance information from the camera to the food object is determined based on the set of pixels corresponding to the food object. Construct the bounding box of the food object based on the set of pixels corresponding to the food object; Determine the dimensions of the bounding box of the food object; The size information of the bounding box of the food object is determined as the size information of the food object.
[0091] The distance information from the camera to the food object refers to information used to represent the distance between the camera and the food object. This distance information is determined based on the set of pixels corresponding to the food object; that is, the distance information comes from the food object's own set of pixels and is used to reflect the proximity between the food object and the camera.
[0092] The bounding box of a food object can be a cuboid used to enclose the set of pixels corresponding to that food object. This bounding box can be an axis-aligned bounding box, meaning its three sides are parallel to the three axes of the coordinate system; or it can be a directed bounding box, meaning its orientation can be rotated to fit as closely as possible to the main direction of the set of pixels corresponding to the food object.
[0093] The dimensions of the bounding box mentioned above can refer to the span of the bounding box in three directions, such as length, width, and height. For axis-aligned bounding boxes, the difference between the maximum and minimum values of the pixel set in the three coordinate axes can be taken as the length, width, and height; for directed bounding boxes, the three side lengths of the bounding box can be taken along the target main direction as the length, width, and height.
[0094] The size information of a food object can refer to the information used to represent the size of the food object in real physical space, determined based on the size information of its bounding box. In other words, the size information of the food object directly uses the size information of its bounding box, rather than other separately calculated dimensions.
[0095] In some embodiments of this application, for each food object in the food object set, the electronic device determines the distance information from the camera to the food object based on the set of pixels corresponding to the food object, constructs the bounding box of the food object, then determines the size information of the bounding box of the food object, and then determines the size information of the bounding box of the food object as the size information of the food object.
[0096] Continuing with the first example above, for the set of pixels corresponding to hamburger A, the electronic device determines the distance information from the camera to hamburger A based on the set of pixels corresponding to hamburger A. For the set of pixels corresponding to hamburger B, the electronic device determines the distance information from the camera to hamburger B based on the set of pixels corresponding to hamburger B.
[0097] Furthermore, the electronic device constructs the bounding box of hamburger A based on the pixel set corresponding to hamburger A, and constructs the bounding box of hamburger B based on the pixel set corresponding to hamburger B. For example, the bounding box can be an axis-aligned bounding box, with its three sides parallel to the three axes of the coordinate system; it can also be a directed bounding box, whose direction can be rotated to fit as closely as possible to the main direction of the target point set. Then, the electronic device determines the size information of the bounding box of hamburger A, such as the span of the bounding box of hamburger A in the three directions, i.e., length, width, and height. Similarly, the electronic device determines the size information of the bounding box of hamburger B. Finally, the size information of the bounding box of hamburger A is determined as the size information of hamburger A, and the size information of the bounding box of hamburger B is determined as the size information of hamburger B.
[0098] In the embodiments of this application, the distance information from the camera to the food object is determined based on the set of pixels corresponding to the food object. A bounding box for the food object is constructed based on the set of pixels corresponding to the food object, and the size information of the bounding box is determined. This size information is then used as the size information of the food object, ensuring that both the distance information from the camera to the food object and the size information of the food object originate from the food object's own pixel set, rather than from the entire first image or the combined pixel set of multiple food objects. This maintains a correspondence between the user-specified food object and the pixels involved in determining the physical scale information. It reduces the possibility of including tableware, background, adjacent food, or non-target food in the physical scale information due to uniform processing of the entire image. This makes the obtained distance information from the camera to the food object and the size information of the food object more consistent with the user's actual specified food object, thereby making the subsequent weight information and nutritional analysis results obtained based on the physical scale information more accurate.
[0099] In some embodiments of this application, determining the distance information from the camera to the food object based on the set of pixels corresponding to the food object may specifically include: Calculate the distance information from the camera to each pixel in the set of pixels corresponding to the food object; The median value of the distance information from each pixel in the set of pixels corresponding to the food object to the camera is determined as the distance information from the camera to the food object.
[0100] The median value can be the middle distance among all the distances from each pixel in the set of pixels corresponding to the food object to the camera, arranged in ascending order. If the number of distances is odd, the median value is the middle distance; if the number of distances is even, the median value can be the average of the two middle distances.
[0101] In some embodiments of this application, for each food object in the set of food objects, the electronic device calculates the distance information from each pixel in the set of pixels corresponding to the food object to the camera, and then determines the median value of the distance information from each pixel in the set of pixels corresponding to the food object to the camera as the distance information from the camera to the food object.
[0102] In other words, the electronic device does not directly use the distance information of a certain pixel from the camera as the distance information of the camera from the food object. Instead, it uses the median value of the distance information of all pixels in the set of pixels corresponding to the food object from the camera as the distance information of the camera from the food object.
[0103] Continuing with the first example above, for the pixel set corresponding to hamburger A, the electronic device calculates the distance information from each pixel in the pixel set corresponding to hamburger A to the camera. For example, if the pixel set corresponding to hamburger A includes multiple pixels, the electronic device calculates the distance from the camera to the 3D point corresponding to each pixel. Similarly, for the pixel set corresponding to hamburger B, the electronic device calculates the distance information from the camera to each pixel in the pixel set corresponding to hamburger B.
[0104] For Hamburger A, the electronic device determines the distance from the camera to the camera by taking the median value of the distance information from each pixel in the pixel set corresponding to Hamburger A. For example, if some pixels in the pixel set corresponding to Hamburger A are 0.60 meters, some are 0.62 meters, and some are 0.65 meters from the camera, the electronic device arranges these distances in ascending order and takes the median value as the distance from the camera to Hamburger A. For Hamburger B, the same method is used to determine the distance from the camera to Hamburger B by taking the median value of the distance information from each pixel in the pixel set corresponding to Hamburger B.
[0105] In the embodiments of this application, the distance information from each pixel in the pixel set corresponding to the food object to the camera is calculated, and the median value of the distance information from each pixel in the pixel set corresponding to the food object to the camera is determined as the distance information from the camera to the food object. This ensures that the distance information from the camera to the food object is not directly determined by the distance information of a single pixel, but by the median value of the distance information from multiple pixels in the pixel set corresponding to the food object to the camera. Therefore, even if there are a few depth anomalies or outliers in the pixel set, these anomalous points will not significantly affect the median value, resulting in more stable distance information that better represents the overall distance of the food object. Furthermore, since this distance information is determined separately for each food object, different food objects obtain their own camera-to-food-object distance information, reducing the possibility of mixing distance information between multiple food objects, thereby making the subsequent weight information and nutritional analysis results obtained based on this distance information more accurate.
[0106] Step 160: Based on the physical scale information corresponding to each food object, determine the weight information and nutritional analysis results corresponding to each food object.
[0107] The nutritional analysis results for a food object can refer to information representing the nutritional components of that food object, determined based on its physical scale information. The nutritional analysis results for a food object may include at least one of the following: the calorie content of the food object; the carbohydrate mass of the food object; the protein mass of the food object; the fat mass of the food object; the total calorie content of the food object, i.e., the estimated total calorie content of the food object directly output by the nutritional inference model; and at least one of sugar, dietary fiber, sodium, cholesterol, or trace elements.
[0108] In some embodiments of this application, the estimated weight information of the food object, as well as the nutritional values of at least one of calories, carbohydrates, protein, and fat, can be obtained based on the physical scale information of the food object.
[0109] For example, in the first example above, for hamburger A, the electronic device can obtain the estimated weight information of hamburger A (180 grams), calories (450 kcal), carbohydrates (40 grams), protein (25 grams), and fat (20 grams) based on the physical dimensions of hamburger A. For hamburger B, the electronic device can obtain the estimated weight information of hamburger B (160 grams), calories (400 kcal), carbohydrates (36 grams), protein (22 grams), and fat (18 grams) based on the physical dimensions of hamburger B.
[0110] In some embodiments of this application, before determining the weight information and nutritional analysis results of each food object based on the physical scale information corresponding to each food object, the scale quality state of each food object in the food object set can also be determined, and the physical scale information can be used as a determination condition for nutritional reasoning based on the scale quality state.
[0111] Specifically, before determining the weight information and nutritional analysis results for each food object based on its corresponding physical scale information, the aforementioned method may further include: Determine the scale quality state of each food object in the food object set; The object identifier, second image, mask image, and scale quality status of each food object in the food object set are associated to obtain the initial analysis data set. Once the scale quality status verification is passed, the physical scale information corresponding to each food object is added to the initial analysis data set to obtain the target analysis data set. The process of determining the weight information and nutritional analysis results for each food object based on its physical scale information may specifically include: Based on the target analysis data set, the weight information and nutritional analysis results for each food object are determined.
[0112] In this context, the scale quality state of a food object refers to the state information used to indicate whether the physical scale information corresponding to that food object is reliable and usable. This scale quality state is determined separately for each food object in the set of food objects; that is, each food object corresponds to its own scale quality state, rather than the entire set of food objects sharing a single scale quality state.
[0113] An object identifier refers to the identifying information used to distinguish different food objects within a set of food objects. Each food object corresponds to a unique object identifier, and different food objects correspond to different object identifiers. Through object identifiers, the second image, mask image, scale quality status, and physical scale information of the same food object can be linked together, preventing data mixing between different food objects.
[0114] The second image can refer to an image segmented from the first image based on a mask image of the food object. In other words, the second image is the image region corresponding to the food object, used in subsequent determination of weight and nutritional analysis results. One second image corresponds to one food object, with different food objects each corresponding to their own specific second image.
[0115] An initial analysis data set for a food object can refer to the data set obtained by associating the object identifier, second image, mask image, and scale quality status of the food object. This initial analysis data set does not yet contain the physical scale information corresponding to the food object.
[0116] The target analysis data set can refer to the data set obtained after adding the physical scale information of the food object to the initial analysis data set, assuming the scale quality status verification has passed. In other words, the target analysis data set includes not only the object identifier, second image, mask image, and scale quality status of the food object, but also the physical scale information of the food object.
[0117] In some embodiments of this application, before determining the weight information and nutritional analysis results of each food object based on its corresponding physical scale information, the electronic device determines the scale quality state of each food object in the food object set. Then, it associates the object identifier, second image, mask image, and scale quality state of the food object to obtain the initial analysis data set corresponding to that food object. If the scale quality state of the food object passes verification, the electronic device adds the physical scale information corresponding to the food object to the initial analysis data set corresponding to that food object, thus obtaining the target analysis data set corresponding to that food object.
[0118] In other words, physical scale information is not added unconditionally to the initial analysis data set, but only after the scale quality status has been verified. For different food objects, it is determined whether to add their respective physical scale information to their respective initial analysis data sets based on their respective scale quality status.
[0119] Furthermore, the electronic device determines the weight information and nutritional analysis results for each food object based on the target analysis data set. Specifically, the target analysis data set for each food object can be input into a food nutrition inference model to obtain the estimated weight information of each food object, as well as at least one nutrient value among calories, carbohydrates, protein, and fat. In other words, the weight information and nutritional analysis results are not determined directly based on physical scale information alone, but rather based on the target analysis data set. This target analysis data set includes the object identifier, second image, mask image, scale quality status, and physical scale information of the food object, all of which contribute to the determination of the weight information and nutritional analysis results of the food object.
[0120] The aforementioned nutritional inference model can first learn fine-grained dish names using large-scale food images, and then learn continuous numerical outputs using data with weight information and nutritional annotations. The same target images and scale fields are used in both the training and usage phases; during training, a portion of the originally valid scale can be randomly hidden, allowing the model to still provide lower-confidence estimates based on the images when the scale is invalid. Training can employ supervised learning, instruction fine-tuning, or numerical error-based optimization methods to enable the model to learn the correspondence between target images, physical scales, weight information, and nutritional values.
[0121] Correspondingly, an example target analysis data set may include the object identifier of the food object, a second image, a mask image, scale quality status, and physical scale information. The aforementioned valid physical scale information can be converted into structured text, such as "Hamburger A, distance 0.62 meters, length 11.8 cm, width 10.6 cm, height 6.1 cm, scale valid"; or it can be converted into a numerical vector or scale embedding, fused with the second image features, and input into the nutrition inference model.
[0122] Continuing with the first example above, the electronic device determines the scale quality state corresponding to Hamburger A and Hamburger B. For example, if the mask image of Hamburger A is complete, has continuous depth, and minimal occlusion, then the scale quality state corresponding to Hamburger A can be considered as verified as passed; if Hamburger B is partially blocked by a hand and has a low proportion of effective depth pixels, then the scale quality state corresponding to Hamburger B can be considered as verified as failed.
[0123] For Hamburger A, the electronic device correlates the object identifier of Hamburger A, the second image of Hamburger A, the mask image of Hamburger A, and the scale quality state of Hamburger A to obtain the initial analysis data set corresponding to Hamburger A. The second image of Hamburger A is segmented from the first image based on the mask image of Hamburger A. For Hamburger B, the electronic device correlates the object identifier of Hamburger B, the second image of Hamburger B, the mask image of Hamburger B, and the scale quality state of Hamburger B to obtain the initial analysis data set corresponding to Hamburger B.
[0124] Since Hamburger A passed the scale quality status verification, the electronic device adds the physical scale information corresponding to Hamburger A to the initial analysis data set corresponding to Hamburger A, thus obtaining the target analysis data set corresponding to Hamburger A. Since Hamburger B failed the scale quality status verification, the electronic device either does not add the physical scale information corresponding to Hamburger B to the initial analysis data set corresponding to Hamburger B, or only writes the failure status and reason.
[0125] The electronic device determines the weight and nutritional analysis results of hamburger A based on the target analysis data set corresponding to hamburger A; it also determines the weight and nutritional analysis results of hamburger B based on the target analysis data set corresponding to hamburger B. Since the target analysis data set for hamburger A includes physical scale information, the weight and nutritional analysis results of hamburger A can be determined based on this physical scale information; since the target analysis data set for hamburger B does not contain physical scale information, the weight and nutritional analysis results of hamburger B can be determined based on a second image of hamburger B.
[0126] In the embodiments of this application, before determining the weight information and nutritional analysis results of each food object based on the physical scale information corresponding to each food object, the scale quality state of each food object in the food object set is determined, and the object identifier, second image, mask image and scale quality state of each food object in the food object set are associated to obtain an initial analysis data set. If the scale quality state verification is passed, the physical scale information corresponding to each food object is added to the initial analysis data set to obtain a target analysis data set. Then, based on the target analysis data set, the weight information and nutritional analysis results of each food object are determined. Therefore, on the one hand, by associating the second image, mask image, scale quality status, and physical scale information of the same food object together through object identification, the data mixing between different food objects can be reduced when multiple food objects are processed in parallel. On the other hand, by verifying the physical scale information through scale quality status, the physical scale information is added to the initial analysis data group only when the verification is passed. This can reduce the inclusion of distorted physical scale information caused by occlusion, reflection, transparent containers, or segmentation deviation in the determination process of weight information and nutritional analysis results, making the obtained weight information and nutritional analysis results more consistent with the food object actually specified by the user, and improving the accuracy of food object analysis results.
[0127] In some embodiments of this application, determining the scale quality state corresponding to each food object in the food object set may specifically include: For each food object in the food object set, determine the scale quality status of the food object based on the inspection information corresponding to the food object; The verification of the scale quality status is passed, which may specifically include: If the inspection information meets the preset conditions, the scale quality status verification is deemed successful.
[0128] The inspection information refers to information used to determine the scale quality state of a food object. Each food object corresponds to one set of inspection information, and different food objects have their own separate sets of inspection information. The inspection information may include at least one of the following: the physical scale information of the food object, the mask coverage of the food object, the effective depth pixel ratio of the food object, the depth continuity of the food object, the outlier ratio of the food object, the size information of the food object, the occlusion ratio of the food object, and the proportion of reflective or transparent areas of the food object.
[0129] The mask coverage of a food object refers to the degree to which the mask image covers the food object. Mask coverage can be used to determine whether the mask image completely covers the food object, for example, whether the mask image only covers a part of the food object, or whether it covers an area other than the food object.
[0130] The effective depth pixel ratio of a food object refers to the proportion of pixels in the mask image corresponding to that food object that successfully generate effective depth information. This effective depth pixel ratio can be used to determine whether the depth information corresponding to the food object is complete, for example, whether there are a large number of pixels with missing depth in the mask image.
[0131] The depth continuity of a food object refers to the degree of continuity between the depth information of pixels in the set of pixels corresponding to the food object. This depth continuity can be used to determine whether there are abrupt changes or breaks in the depth information corresponding to the food object, such as whether the surface depth of the food object is continuous, or whether it is mixed with the depth information of the background or adjacent food.
[0132] The outlier ratio of a food object refers to the proportion of outliers in the set of pixels corresponding to that food object. This outlier ratio can be used to determine whether the set of pixels corresponding to the food object is clean, for example, whether there are pixels in the set that are significantly deviated from the main distribution.
[0133] The size information of a food object can refer to the dimensions of that food object, such as the length, width, and height of its bounding box. This size information can be used to determine whether the size of the food object is within a reasonable range; for example, whether the calculated length, width, height, and volume of the food object significantly exceed a reasonable range.
[0134] The occlusion ratio of a food object refers to the degree to which the food object is obscured. The occlusion ratio can be used to determine whether the food object is obscured by tableware, hands, or other food, for example, whether the food object is obscured by a large proportion.
[0135] The reflective or transparent area ratio of a food object refers to the proportion of that reflective or transparent area within the food object. This ratio can be used to determine the degree to which reflective surfaces or transparent containers affect the depth information of the food object.
[0136] Preset conditions refer to the conditions used to determine whether the scale quality status verification has passed. If the inspection information meets the preset conditions, the scale quality status verification is determined to have passed; if the inspection information does not meet the preset conditions, the scale quality status verification is determined to have failed. These preset conditions can be set according to one or more of the inspection information.
[0137] In some embodiments of this application, for each food object in the food object set, the electronic device determines the scale quality status of the food object based on the inspection information corresponding to the food object. If the inspection information corresponding to the food object meets the preset conditions, the electronic device determines that the scale quality status of the food object has passed the verification.
[0138] Continuing with the first example above, for the hamburger jacket, the inspection information for the hamburger jacket can include at least one of the following: physical scale information of the hamburger jacket, mask coverage of the hamburger jacket, effective depth pixel ratio of the hamburger jacket, depth continuity of the hamburger jacket, outlier ratio of the hamburger jacket, size information of the hamburger jacket, occlusion ratio of the hamburger jacket, and reflective or transparent area ratio of the hamburger jacket. If the inspection reveals that the hamburger jacket has a complete mask image, a high effective depth pixel ratio, depth continuity, a low outlier ratio, reasonable size, minimal occlusion, and minimal reflection, then the scale quality status of the hamburger jacket can be considered verified as passed.
[0139] For Hamburger B, the inspection information for Hamburger B may include at least one of the following: physical scale information of Hamburger B, mask coverage of Hamburger B, effective depth pixel ratio of Hamburger B, depth continuity of Hamburger B, outlier ratio of Hamburger B, size information of Hamburger B, occlusion ratio of Hamburger B, and reflective or transparent area ratio of Hamburger B. If the inspection reveals that Hamburger B is partially blocked by a hand, has an incomplete mask image, a low effective depth pixel ratio, or a high occlusion ratio, then the scale quality status of Hamburger B can be considered as failing the verification.
[0140] In the embodiments of this application, for each food object in the food object set, the scale quality state corresponding to that food object is determined based on the inspection information corresponding to that food object. If the inspection information meets preset conditions, the scale quality state verification is deemed successful. The inspection information includes at least one of the following: physical scale information of the food object, mask coverage of the food object, effective depth pixel ratio of the food object, depth continuity of the food object, outlier ratio of the food object, size information of the food object, occlusion ratio of the food object, and reflective or transparent area ratio of the food object. Therefore, the physical scale information of each food object can be verified from multiple dimensions, rather than determining its usability based on a single condition. This reduces the likelihood of distorted physical scale information due to occlusion, reflection, transparent containers, segmentation bias, or depth anomalies being included in the determination process of weight information and nutritional analysis results. Furthermore, since the scale quality state is determined separately for each food object, different food objects obtain their own scale quality states, reducing the possibility of mixing scale quality states among multiple food objects. This makes the subsequent weight information and nutritional analysis results determined based on the scale quality state more consistent with the food object actually specified by the user.
[0141] In some embodiments of this application, after obtaining the initial analysis data set, the method described above may further include: If the scale quality status verification corresponding to the first food object fails, the weight information and nutritional analysis results corresponding to the first food object are determined based on the initial analysis data set. or, If the scale quality status verification for the first food object fails, a prompt message will be output.
[0142] The first food object can refer to any food object in the set of food objects. In other words, every food object in the set of food objects can be the first food object.
[0143] The prompt message can refer to information used to remind the user to retake the first image. This prompt message can be output in the form of text, voice, icons, or vibration, etc., and this application embodiment does not limit this.
[0144] In some embodiments of this application, after obtaining the initial analysis data set, if the scale quality status verification of the first food object fails, the electronic device determines the weight information and nutritional analysis results of the first food object based on the initial analysis data set. That is, when the scale quality status verification fails, the electronic device does not stop processing the first food object, but still determines the weight information and nutritional analysis results of the first food object based on the initial analysis data set. Since the initial analysis data set does not yet contain the physical scale information of the first food object, the weight information and nutritional analysis results can be obtained without using the physical scale information as a determining condition.
[0145] In another embodiment, if the scale quality status verification of the first food object fails, the electronic device can also directly output a prompt message to remind the user to retake the first image. That is, when the scale quality status verification fails, the electronic device can pause the generation of weight information and nutritional analysis results for the first food object and output a prompt message to remind the user to retake the first image so that the user can obtain a first image that passes the scale quality status verification by retaking the image.
[0146] It should be noted that the above two methods can be selected individually or combined. That is, the electronic device can choose to output a prompt message when the scale quality status verification fails, or it can choose to determine the weight information and nutritional analysis results of the first food object based on the initial analysis data set corresponding to the first food object, or it can both output a prompt message and determine the weight information and nutritional analysis results of the first food object based on the initial analysis data set corresponding to the first food object. In this way, while obtaining the weight information and nutritional analysis results of the first food object, the user is informed that the weight information and nutritional analysis results are obtained based on the initial analysis data set. This allows the user to retake the first image if they are not satisfied with the weight information and nutritional analysis results, so as to obtain the weight information and nutritional analysis results of the first food object again.
[0147] Continuing with the first example above, since the scale quality status verification for Hamburger B failed, the electronic device can determine the weight information and nutritional analysis results of Hamburger B based on the initial analysis data set for Hamburger B. Since the initial analysis data set for Hamburger B does not yet contain the physical scale information of Hamburger B, the weight information and nutritional analysis results of Hamburger B can be obtained without using the physical scale information of Hamburger B as a determining condition.
[0148] In another example, if the scale quality verification for Hamburger B fails, the electronic device outputs a prompt message, asking the user to retake the first image. For example, the electronic device could display "The physical scale of Hamburger B failed the check. Please keep the target intact and adjust the shooting distance before retaking the photo," prompting the user to retake the photo of the meal. Thus, after the user retakes the photo and obtains a new first image, the electronic device can re-determine the scale quality of Hamburger B based on the new first image.
[0149] In the embodiments of this application, after obtaining the initial analysis data set, if the scale quality status verification corresponding to the first food object fails, the electronic device determines the weight information and nutritional analysis results corresponding to the first food object based on the initial analysis data set, or outputs a prompt message to remind the user to retake the first image. Thus, on the one hand, even if the scale quality status verification fails, the electronic device can still determine the weight information and nutritional analysis results of the first food object based on the initial analysis data set, avoiding interruption of the entire analysis process due to the failure of the scale quality status verification of a single food object; on the other hand, the electronic device can also output a prompt message to remind the user to retake the first image, so that the user can obtain a first image that has passed the scale quality status verification by retaking the image, thereby improving the accuracy of the weight information and nutritional analysis results. Simultaneously, since the first food object is any food object in the food object set, each food object in the food object set can be processed in the above manner, reducing the mixing of scale quality status information among multiple food objects, making the obtained weight information and nutritional analysis results more consistent with the food object actually specified by the user.
[0150] Step 170: Display the target analysis results information, which includes the weight information and nutritional analysis results information for each food object.
[0151] The target analysis result information can refer to the analysis results output in response to the user's first input for the set of food objects to be analyzed in the first image. The target analysis result information can include the weight information and nutritional analysis result information corresponding to each food object in the set of food objects.
[0152] In some embodiments of this application, the electronic device can display the target analysis results information of this analysis on the results page. This target analysis results information may include the weight information and nutritional analysis results corresponding to each food object, and may also mark food objects that were not included or have lower confidence levels. This results page may be, for example, the functional interface of an application that implements the solutions of the embodiments of this application based on AI technology, or it may be the desktop of the electronic device; no limitation is made in the embodiments of this application.
[0153] For example, refer to Figure 2 The results page can display the outline of the hamburger (21), its weight information (180 grams), its calories (450 kcal), and also the values for carbohydrates, protein, and fat (in...). Figure 2 (Not shown in the image); This results page can also display the outline of Hamburger B 22, the weight information of Hamburger B 160 grams, the calories of Hamburger B 400 kcal, and the values of carbohydrates, protein and fat of Hamburger B (in... Figure 2 (Not shown in the image); beverages and fries are not shown as targets of this analysis, or may be shown as not participating in this analysis.
[0154] In some embodiments of this application, when N≥2, the method described above may further include the following before displaying the target analysis result information: The weight information and nutritional analysis results of each food object in the food object set are summarized to obtain the summary result; The displayed target analysis result information includes: Display the summary results.
[0155] The summary result can refer to the result obtained by summarizing the weight information and nutritional analysis results of each food object in the food object set. The summary result can be a set-level result, used to represent the total weight information and total nutritional analysis results of the entire food object set.
[0156] In some embodiments of this application, when the food object set includes two or more food objects, before displaying the target analysis result information, the electronic device summarizes the weight information and nutritional analysis result information of each food object in the food object set to obtain a summary result, as shown in the following formula (7), and then displays the summary result.
[0157] In the above formula (7), Indicates the first The weight, calories, or total nutrients of each food item can be added together. This represents a summary of the weight, calories, or total nutrients of a collection of food objects.
[0158] In other words, the electronic device doesn't just display the weight and nutritional analysis results for each individual food item. Instead, it first aggregates the weight and nutritional analysis results for multiple food items before displaying them. For example, it sums the weights of multiple food items to obtain the total weight; it sums the calories to obtain the total calories; it sums the carbohydrate masses to obtain the total carbohydrate mass; it sums the protein masses to obtain the total protein mass; and it sums the fat masses to obtain the total fat mass. Then, it displays these aggregated results. This allows users to view the aggregated results or expand to view the individual weight and nutritional analysis results for each food item.
[0159] Continuing with the first example above, before displaying the target analysis results, the electronic device summarizes the weight and nutritional analysis results of Hamburger A and Hamburger B separately, obtaining a summary result. For example, if Hamburger A weighs 180 grams and Hamburger B weighs 160 grams, the total weight is 340 grams; if Hamburger A has 450 kcal and Hamburger B has 400 kcal, the total calories are 850 kcal; if Hamburger A has 40 grams of carbohydrates and Hamburger B has 36 grams, the total carbohydrate mass is 76 grams; if Hamburger A has 25 grams of protein and Hamburger B has 22 grams, the total protein mass is 47 grams; if Hamburger A has 20 grams of fat and Hamburger B has 18 grams, the total fat mass is 38 grams. Thus, it can be done as follows: Figure 2 As shown, the results page displays the total weight of the two hamburgers: 340 grams; total calories: 850 kcal; total carbohydrates: 76 grams; total protein: 47 grams; and total fat: 38 grams. Users can view this summary result or expand to see the individual weight and nutritional analysis results for hamburger A and hamburger B.
[0160] In the embodiments of this application, when N≥2, before displaying the target analysis result information, the weight information and nutritional analysis result information of each food object in the food object set are summarized separately to obtain a summary result. This summary result is then displayed when the target analysis result information is shown. This allows the electronic device to display not only the weight information and nutritional analysis result information corresponding to each food object individually, but also the summary result of the entire food object set, even when the user specifies multiple food objects at once. Therefore, the user does not need to manually accumulate the weight information and nutritional analysis result information of multiple food objects; they can directly obtain the total weight information and total nutritional analysis result information of the entire food object set, improving the convenience for the user to view the analysis results of multiple food objects. Furthermore, since the summarization is performed separately for each food object in the food object set, the food objects corresponding to the summary result are consistent with the food objects specified in the first input. This reduces the possibility of including non-target foods in the summary result due to uniform processing of the entire image, making the summary result more consistent with the food objects actually specified by the user.
[0161] In some embodiments of this application, nutritional analysis results of a food object include the carbohydrate mass, protein mass, and fat mass of the food object, as well as the first total calories of the food object.
[0162] Before displaying the target analysis results information, the methods described above may further include: For each food object in the food object set, the energy supply of the food object's carbohydrate mass, protein mass, and fat mass is converted to obtain the second total calorie of the food object. Determine the absolute value of the difference between the first total calorie and the second total calorie; Calculate the ratio of the absolute value to the first total calorie. The information displayed as the target analysis results may specifically include: If the ratio is less than a preset threshold, the nutritional composition ring corresponding to the food object is displayed.
[0163] The first total calorie can refer to the total calorie information included in the nutritional analysis results of the food object, such as the estimated total calorie of the food object directly output by the nutritional inference model. The first total calorie is an estimate of the overall calorie content of the food object.
[0164] The second total calorie refers to the total calories obtained after converting the energy supply from the mass of carbohydrates, protein, and fat in the food. In other words, the second total calorie is not directly output by the model, but is calculated based on the mass of these three macronutrients in the food.
[0165] The energy conversion mentioned above refers to converting the mass of carbohydrates, proteins, and fats into corresponding calories according to their respective energy coefficients. For example, carbohydrates and proteins can be converted to 4 kcal per gram, and fats to 9 kcal per gram. The sum of the three types of energy obtained from the energy conversion is the second total calorie.
[0166] A ratio can refer to the absolute value of the difference between the first and second total calories, and the ratio to the first total calories. This ratio indicates the degree of difference between the first and second total calories. The smaller the ratio, the closer the first and second total calories are; the larger the ratio, the greater the difference between the first and second total calories.
[0167] The preset threshold can refer to a threshold used to determine whether the difference between the first total calorie and the second total calorie is within an allowable range. If the ratio is less than the preset threshold, it indicates that the difference between the first total calorie and the second total calorie is within an allowable range; if the ratio is greater than or equal to the preset threshold, it indicates that the difference between the first total calorie and the second total calorie exceeds an allowable range. The preset threshold can range from 0 to 1, for example, it can be 0.1 or 0.2. The specific value of the preset threshold is not limited in this embodiment.
[0168] A nutrient composition ring can be a graph used to represent the percentage of total energy provided by carbohydrates, protein, and fat in a food, such as... Figure 2 Ring 23 in the diagram. This nutrient composition ring is displayed according to the proportion of energy supplied, rather than directly comparing the grams of the three macronutrients.
[0169] In some embodiments of this application, the nutritional analysis results of a food object may include the carbohydrate mass, protein mass, and fat mass of the food object, as well as the first total calorie content of the food object. That is, for each food object in the set of food objects, its nutritional analysis results include the carbohydrate mass, protein mass, fat mass, and first total calorie content of the food object. Different food objects correspond to their respective carbohydrate mass, protein mass, fat mass, and first total calorie content.
[0170] For each food object in the food object set, the energy supply of its carbohydrate, protein, and fat masses is converted to obtain the second total calorie count. Then, the absolute value of the difference between the first and second total calorie counts is determined, and the ratio of this absolute value to the first total calorie count is calculated. If the ratio is less than a preset threshold, the electronic device displays the nutritional composition ring corresponding to that food object.
[0171] In other words, the electronic device does not directly use the first total calorie as the second total calorie. Instead, it calculates the second total calorie based on the mass of carbohydrates, protein, and fat in the food, according to their respective energy coefficients, and then uses the sum of these three energy types as the second total calorie. Furthermore, the electronic device only displays the corresponding nutrient composition ring for the food if the difference between the first and second total calorie is within an acceptable range.
[0172] Continuing with the first example above, the nutritional analysis information for Hamburger A includes the carbohydrate mass, protein mass, and fat mass, as well as its total calories. For example, Hamburger A has a carbohydrate mass of 40 grams, a protein mass of 25 grams, a fat mass of 20 grams, and a total calorie content of 450 kcal. The nutritional analysis information for Hamburger B includes the carbohydrate mass, protein mass, and fat mass, as well as its total calories. For example, Hamburger B has a carbohydrate mass of 36 grams, a protein mass of 22 grams, a fat mass of 18 grams, and a total calorie content of 400 kcal.
[0173] For Hamburger A, the electronic device calculates the energy based on its 40g carbohydrates, 25g protein, and 20g fat. For example, if carbohydrates and protein are converted to 4 kcal per gram, and fat to 9 kcal per gram, then the second total calorie count is 40×4 + 25×4 + 20×9 = 160 + 100 + 180 = 440 kcal. For Hamburger B, the electronic device calculates the energy based on its 36g carbohydrates, 22g protein, and 18g fat, resulting in a second total calorie count of 36×4 + 22×4 + 18×9 = 144 + 88 + 162 = 394 kcal.
[0174] For Hamburger A, the electronic device determines that the absolute value of the difference between the first total calorie intake of 450 kcal and the second total calorie intake of 440 kcal is |450-440|=10 kcal. For Hamburger B, the electronic device determines that the absolute value of the difference between the first total calorie intake of 400 kcal and the second total calorie intake of 394 kcal is |400-394|=6 kcal.
[0175] For Hamburger A, the electronic device calculates the ratio of the absolute value of 10 kcal to the first total calorie of 450 kcal as 10 / 450 ≈ 0.022. For Hamburger B, the electronic device calculates the ratio of the absolute value of 6 kcal to the first total calorie of 400 kcal as 6 / 400 = 0.015.
[0176] Assuming a preset threshold of 0.1, if the ratio of Hamburger A (0.022) is less than the preset threshold of 0.1, the electronic device displays the corresponding nutrient composition ring for Hamburger A. This nutrient composition ring represents the percentage of total energy provided by carbohydrates, protein, and fat in Hamburger A. Similarly, if the ratio of Hamburger B (0.015) is also less than the preset threshold of 0.1, the electronic device displays the corresponding nutrient composition ring for Hamburger B. If the ratio of a food item is greater than or equal to the preset threshold, the nutrient composition ring for that food item is not displayed, or the original weight in grams is retained and an estimated value pending confirmation is displayed.
[0177] In embodiments of this application, the nutritional analysis results of a food object include the carbohydrate mass, protein mass, and fat mass of the food object, as well as the first total calorie of the food object. Before displaying the target analysis results, the energy contribution of the carbohydrate mass, protein mass, and fat mass of the food object is converted to a second total calorie. The absolute value of the difference between the first and second total calories is determined, and the ratio of this absolute value to the first total calorie is calculated. If this ratio is less than a preset threshold, the corresponding nutrient composition ring for the food object is displayed. This ensures that the display of the nutrient composition ring is not unconditional, but only when the difference between the first and second total calories is within an allowable range. This reduces the misleading graphical results caused by significant inconsistencies between the first and second total calories, allowing the nutrient composition ring to more accurately represent the energy contribution percentage of carbohydrates, protein, and fat to the total calorie. Furthermore, since the nutrient composition ring is displayed according to the energy contribution percentage, rather than directly comparing the grams of the three macronutrients, it more intuitively illustrates the contribution of carbohydrates, protein, and fat to the total calorie.
[0178] To better understand the information processing method provided in the embodiments of this application, the information processing method provided in the embodiments of this application will be described below with specific scenarios.
[0179] Figure 3 This is a flowchart illustrating an information processing method according to an exemplary embodiment. Figure 3 As shown, the information processing method may include steps 301-308.
[0180] Step 301: Obtain the first image.
[0181] Step 302: Receive the user's first input.
[0182] The execution order of steps 301 and 302 is not limited. For example, step 301 can be executed first, followed by step 302, or step 302 can be executed first, followed by step 301. Alternatively, steps 301 and 302 can be executed simultaneously. Users can set the order according to their needs.
[0183] Step 303: In response to the first input, determine the set of food objects to be analyzed from the first image. The set of food objects includes N food objects to be analyzed.
[0184] Steps 301-303 described above are consistent with steps 110-130 in the above embodiments, and will not be repeated here.
[0185] Step 304: Assign an object identifier to each food object.
[0186] In step 304, a corresponding object identifier can be assigned to each food object.
[0187] Step 305: Determine the mask image corresponding to each food object in the food object set.
[0188] Step 306: Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, determine the physical scale information corresponding to each food object.
[0189] Steps 305-306 above are consistent with steps 140-150 in the above embodiments, and will not be repeated here.
[0190] Step 307: Associate the object identifier, second image, and physical scale information of each food object in the food object set to obtain the target analysis data set.
[0191] In step 307, the object identifier, second image, and physical scale information of each food object in the food object set are associated to obtain the target analysis data set. Referring to the above embodiment, the scale quality state corresponding to each food object in the food object set is determined. The object identifier, second image, mask image, and scale quality state of each food object in the food object set are associated to obtain the initial analysis data set. The second image of a food object is obtained by segmenting the first image based on the mask image of the food object. If the scale quality state verification is passed, the physical scale information corresponding to each food object is added to the initial analysis data set to obtain the target analysis data set. The process will not be described in detail here.
[0192] Step 308: Based on the target analysis data group corresponding to each food object, determine the weight information and nutritional analysis results information corresponding to each food object.
[0193] In step 308, after obtaining the target analysis data set corresponding to each food object, the weight information and nutritional analysis results of each food object can be determined based on the target analysis data set corresponding to each food object. This process is consistent with the process of determining the weight information and nutritional analysis results of each food object based on the target analysis data set in the above embodiment, and will not be described again here.
[0194] In this way, the user's voice, point selection, or bounding selection directly determines the set of food objects and continuously determines which masks, 3D points, and nutritional results participate in the calculation. When the user changes the food object, the food category, the corresponding pixel set, physical scale information, weight information, and nutritional analysis results all change accordingly, reducing inconsistencies between the queried object and the calculated object in multi-food scenarios. Furthermore, the object identifier and set identifier map the second image, mask image, physical scale information, weight information, and nutritional analysis results to the same food object or food object set, reducing cross-object scale misuse, omissions, duplicate calculations, and erroneous aggregations during multi-instance parallel processing. In addition, the physical scale information comes from the pixels and 3D points corresponding to the mask image of the current food object, excluding tableware, background, and adjacent foods that are clearly not part of the target. Each processing module exchanges results through the object identifier, food object set identifier, and target analysis data set; after the user changes the target, the food object outline, food object quantity, physical scale information, weight information, and nutritional analysis results are updated synchronously.
[0195] Figure 4 This is a flowchart illustrating an information processing method according to an exemplary embodiment. Specifically, the process involves determining the physical scale information and scale quality state of each food object, such as... Figure 4 As shown, the information processing method may include steps 401-409.
[0196] Step 401: Obtain the mask image corresponding to each food object.
[0197] Step 402: Obtain the first image.
[0198] Step 403: Determine the three-dimensional spatial information corresponding to the first image.
[0199] In step 403, the method for determining the three-dimensional spatial information can refer to the method for determining the three-dimensional spatial information in the above embodiments, and will not be repeated here.
[0200] Step 404: Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, determine the set of pixels corresponding to each food object.
[0201] Step 404 is consistent with the process in the above embodiment of determining the set of pixels corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, and will not be described again here.
[0202] Step 405: For each set of pixels corresponding to each food object, perform pixel set cleanup processing on the set of pixels corresponding to the food object to obtain the target pixel set corresponding to the food object.
[0203] In step 405 above, the point set cleaning process includes at least one of edge shrinking, outlier removal, and connectivity checking.
[0204] Step 406: Determine the scale quality state corresponding to each food object in the food object set.
[0205] In step 406, the process of determining the scale quality state corresponding to each food object in the food object set is the same as in the above embodiment, and will not be described again here.
[0206] Step 407: Determine whether the dimensional quality status verification has passed. If it has passed, proceed to step 408; if it has not passed, proceed to step 409.
[0207] In step 407, determining whether the scale quality status verification has passed is equivalent to determining whether the inspection information meets the preset conditions in the above embodiment. If it does, the scale quality status verification passes; otherwise, the scale quality status verification fails.
[0208] Step 408: Add the physical scale information corresponding to each food object to the initial analysis data group to obtain the target analysis data group.
[0209] Step 408 is consistent with the process in the above embodiment of adding the physical scale information corresponding to each food object to the initial analysis data group to obtain the target analysis data group, and will not be described again here.
[0210] Step 409: Based on the initial analysis data set, determine the weight information and nutritional analysis results of the first food object, or output a prompt message.
[0211] In step 409, the prompt message is used to prompt the user to retake the first image. This step 409 is consistent with the process in the above embodiment, where, if the scale quality status verification corresponding to the first food object fails, the weight information and nutritional analysis results corresponding to the first food object are determined based on the initial analysis data set, or, if the scale quality status verification corresponding to the first food object fails, a prompt message is output. Therefore, it will not be described in detail here.
[0212] Thus, the same mask image defines both the second image and the 3D points selected from the metric space information. Therefore, the appearance and physical scale information of food objects come from the same instance, reducing the probability of tableware, background, adjacent dishes, or other similar instances being mixed into scale calculations. Without the need for known size references, weighing sensors, or dedicated depth sensors, a single ordinary color image can provide target distance, length, width, height, or volume, making it suitable for everyday use with mobile phones and smart glasses. The scale quality status clearly distinguishes between valid physical scales and unavailable scales. When a check fails, no incorrect values are entered; instead, the system can choose to estimate the image only and reduce its confidence level, or prompt the user to adjust the distance, angle, and occlusion and retake the image, thus preventing obvious scale errors from continuing to affect nutritional results. Physical scale information provides size information for foods of the same category but with different actual portions, reducing the dependence of nutritional inference on typical portions in the training data.
[0213] The information processing method provided in this application can be executed by an information processing device. This application uses an information processing device executing the information processing method as an example to illustrate the information processing device provided in this application.
[0214] Figure 5 This is a schematic diagram illustrating the structure of an information processing apparatus according to an exemplary embodiment. For example... Figure 5 As shown, the information processing device 500 may include: The acquisition module 510 is used to acquire the first image; Receiver module 520 is used to receive the user's first input; The first determining module 530 is configured to, in response to the first input, determine a set of food objects to be analyzed from the first image, wherein the set of food objects includes N food objects to be analyzed, where N is a positive integer; The second determining module 540 is used to determine the mask image corresponding to each food object in the food object set; The third determining module 550 is used to determine the physical scale information corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object. The fourth determining module 560 is used to determine the weight information and nutritional analysis results information corresponding to each of the food objects based on the physical scale information corresponding to each of the food objects. Display module 570 is used to display target analysis result information, which includes weight information and nutritional analysis result information corresponding to each food object.
[0215] In this embodiment, in response to a user's first input, a set of food objects to be analyzed is determined from the acquired first image. Furthermore, a mask image corresponding to each food object in the set is determined, so that subsequent processing is no longer performed on the entire image in a general way, but rather the calculation range is continuously constrained by the food objects specified by the user. Based on this, according to the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, the physical scale information corresponding to each food object is determined. The weight information and nutritional analysis results of the food object are then determined from this physical scale information. Finally, the target analysis results information, including the nutritional analysis results and weight information corresponding to each food object, is displayed. Therefore, the food object specified by the user corresponds to the object involved in mask generation, physical scale calculation, weight information determination, nutritional inference, and result display. This improves the problem of inconsistency between the query object and the calculation object caused by first analyzing the entire image and then selecting one item from the results for display. This makes the final displayed nutritional analysis results and weight information more consistent with the food object actually specified by the user. Furthermore, in this embodiment, the physical scale information extracted from the three-dimensional spatial information of the first image based on the mask image of the current food object is used as an explicit condition for determining the weight information and nutritional analysis results, rather than simply multiplying the volume by a fixed density or querying the nutritional value per unit weight by category. This allows food objects of the same category but different actual portions to obtain nutritional analysis results that match their actual size, thereby improving the accuracy of the nutritional analysis results of the food objects.
[0216] In some embodiments of this application, the third determining module is specifically used for: Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, determine the set of pixels corresponding to each food object; Based on the set of pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
[0217] In some embodiments of this application, the third determining module is specifically used for: For each of the food objects corresponding to the mask image, the optical axis direction depth information corresponding to each pixel in the mask image is determined from the three-dimensional spatial information corresponding to the first image; Based on the coordinate information of each pixel in the mask image, the optical axis direction depth information corresponding to each pixel, and the camera parameters when the first image was captured, the three-dimensional coordinate information of each pixel in the mask image is obtained. Based on the three-dimensional coordinate information of each pixel in the mask image corresponding to each food object, determine the set of pixels corresponding to each food object.
[0218] In some embodiments of this application, after determining the set of pixels corresponding to each of the food objects, the apparatus further includes: The pixel cleaning module is used to perform point set cleaning processing on the pixel set corresponding to each food object to obtain the target pixel set corresponding to the food object. The point set cleaning processing includes at least one of edge shrinkage, outlier removal, and connectivity check. The third determining module is specifically used for: Based on the set of target pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
[0219] In some embodiments of this application, the physical scale information corresponding to a food object includes distance information from the camera to the food object and size information of the food object; The third determining module is specifically used for: For each set of pixels corresponding to the food object, the distance information from the camera to the food object is determined based on the set of pixels corresponding to the food object. Construct the bounding box of the food object based on the set of pixels corresponding to the food object; Determine the size information of the bounding box of the food object; The size information of the bounding box of the food object is determined as the size information of the food object.
[0220] In some embodiments of this application, before determining the weight information and nutritional analysis results for each food object based on the physical scale information corresponding to each food object, the device further includes: The fifth determining module is used to determine the scale quality state corresponding to each food object in the food object set; The association module is used to associate the object identifier, second image, mask image and scale quality state of each food object in the food object set to obtain an initial analysis data set. The second image of a food object is obtained by segmenting the first image based on the mask image of the food object. The information addition module is used to add the physical scale information corresponding to each food object to the initial analysis data group after the scale quality status verification is passed, so as to obtain the target analysis data group. The fourth determining module is specifically used for: Based on the target analysis data set, the weight information and nutritional analysis results corresponding to each food object are determined.
[0221] The information processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.
[0222] The information processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.
[0223] The information processing device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0224] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described information processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.
[0225] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0226] Figure 7A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.
[0227] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.
[0228] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.
[0229] The processor 710 is used to acquire the first image; User input unit 707 is used to receive the user's first input; The processor 710 is further configured to, in response to the first input, determine a set of food objects to be analyzed from the first image, the set of food objects including N food objects to be analyzed, where N is a positive integer; determine a mask image corresponding to each food object in the set of food objects; determine physical scale information corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object; and determine weight information and nutritional analysis result information corresponding to each food object based on the physical scale information corresponding to each food object. Display unit 706 is used to display target analysis result information, which includes weight information and nutritional analysis result information corresponding to each food object.
[0230] Thus, by responding to the user's first input, the set of food objects to be analyzed is determined from the acquired first image, and further, the mask image corresponding to each food object in the set is determined. This ensures that subsequent processing is no longer performed on the entire image in a general way, but rather the calculation range is continuously constrained by the food objects specified by the user. Based on this, according to the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, the physical scale information corresponding to each food object is determined, and the weight information and nutritional analysis results of the food object are determined from the physical scale information. Finally, the target analysis result information, including the nutritional analysis result information and weight information corresponding to each food object, is displayed. Therefore, the food objects specified by the user correspond to the objects involved in mask generation, physical scale calculation, weight information determination, nutritional inference, and result display. This can improve the problem of inconsistency between the query object and the calculation object caused by first analyzing the entire image and then selecting one item from the results for display, making the final displayed nutritional analysis result information and weight information more consistent with the food objects actually specified by the user. Furthermore, in this embodiment, the physical scale information extracted from the three-dimensional spatial information of the first image based on the mask image of the current food object itself is used as an explicit condition for determining the weight information and nutritional analysis results, rather than simply multiplying the volume by a fixed density or querying the nutritional value per unit weight by category. This allows food objects of the same category but with different actual portions to obtain nutritional analysis results that match their true size, thereby improving the accuracy of the nutritional analysis results of the food objects.
[0231] Optionally, the processor 710 is further configured to determine a set of pixels corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object; and to obtain physical scale information corresponding to each food object based on the set of pixels corresponding to each food object.
[0232] Optionally, the processor 710 is further configured to, for each mask image corresponding to the food object, determine the optical axis direction depth information corresponding to each pixel in the mask image from the three-dimensional spatial information corresponding to the first image; obtain the three-dimensional coordinate information of each pixel in the mask image based on the coordinate information of each pixel in the mask image, the optical axis direction depth information corresponding to each pixel, and the camera parameters when the first image was captured; and determine the set of pixels corresponding to each food object based on the three-dimensional coordinate information of each pixel in the mask image corresponding to each food object.
[0233] Optionally, after determining the set of pixels corresponding to each of the food objects, the processor 710 is further configured to perform point set cleaning processing on the set of pixels corresponding to each food object to obtain a target set of pixels corresponding to the food object. The point set cleaning processing includes at least one of edge shrinkage, outlier removal, and connectivity check. Based on the target set of pixels corresponding to each of the food objects, the processor 710 obtains physical scale information corresponding to each of the food objects.
[0234] Optionally, the physical scale information corresponding to a food object includes distance information from the camera to the food object and size information of the food object; the processor 710 is further configured to, for each set of pixels corresponding to the food object, determine the distance information from the camera to the food object based on the set of pixels corresponding to the food object; construct the bounding box of the food object based on the set of pixels corresponding to the food object; determine the size information of the bounding box of the food object; and determine the size information of the bounding box of the food object as the size information of the food object.
[0235] Optionally, the processor 710 is further configured to: determine the scale quality state of each food object in the food object set before determining the weight information and nutritional analysis result information corresponding to each food object based on the physical scale information corresponding to each food object; associate the object identifier, second image, mask image, and scale quality state of each food object in the food object set to obtain an initial analysis data set, wherein the second image of a food object is obtained by segmenting from the first image based on the mask image of the food object; if the scale quality state verification is passed, add the physical scale information corresponding to each food object to the initial analysis data set to obtain a target analysis data set; and determine the weight information and nutritional analysis result information corresponding to each food object based on the target analysis data set.
[0236] It should be understood that, in this embodiment, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a color camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.
[0237] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 709 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.
[0238] Processor 710 may include one or more processing units; optionally, processor 710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 710.
[0239] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described information processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0240] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0241] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above information processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0242] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0243] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the information processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0244] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0245] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0246] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. An information processing method, characterized in that, The method includes: Get the first image; Receive the user's first input; In response to the first input, a set of food objects to be analyzed is determined from the first image, the set of food objects including N food objects to be analyzed, where N is a positive integer; Determine the mask image corresponding to each food object in the set of food objects; Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, the physical scale information corresponding to each food object is determined. Based on the physical scale information corresponding to each of the food objects, the weight information and nutritional analysis results corresponding to each of the food objects are determined. The target analysis results information is displayed, which includes the weight information and nutritional analysis results information corresponding to each food object.
2. The method according to claim 1, characterized in that, The step of determining the physical scale information corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object includes: Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, determine the set of pixels corresponding to each food object; Based on the set of pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
3. The method according to claim 2, characterized in that, The step of determining the set of pixels corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object includes: For each of the food objects corresponding to the mask image, the optical axis direction depth information corresponding to each pixel in the mask image is determined from the three-dimensional spatial information corresponding to the first image; Based on the coordinate information of each pixel in the mask image, the optical axis direction depth information corresponding to each pixel, and the camera parameters when the first image was captured, the three-dimensional coordinate information of each pixel in the mask image is obtained. Based on the three-dimensional coordinate information of each pixel in the mask image corresponding to each food object, determine the set of pixels corresponding to each food object.
4. The method according to claim 2, characterized in that, After determining the set of pixels corresponding to each of the food objects, the method further includes: For each set of pixels corresponding to the food object, a point set cleaning process is performed on the set of pixels corresponding to the food object to obtain the target set of pixels corresponding to the food object. The point set cleaning process includes at least one of edge shrinkage, outlier removal, and connectivity check. The process of obtaining physical scale information for each food object based on the set of pixels corresponding to each food object includes: Based on the set of target pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
5. The method according to claim 2, characterized in that, The physical scale information corresponding to a food object includes the distance information from the camera to the food object and the size information of the food object; The process of obtaining physical scale information for each food object based on the set of pixels corresponding to each food object includes: For each set of pixels corresponding to the food object, the distance information from the camera to the food object is determined based on the set of pixels corresponding to the food object. Construct the bounding box of the food object based on the set of pixels corresponding to the food object; Determine the size information of the bounding box of the food object; The size information of the bounding box of the food object is determined as the size information of the food object.
6. The method according to claim 1, characterized in that, Before determining the weight information and nutritional analysis results for each food object based on its corresponding physical scale information, the method further includes: Determine the scale quality state corresponding to each food object in the set of food objects; The object identifier, second image, mask image, and scale quality state of each food object in the food object set are associated to obtain an initial analysis data set. The second image of a food object is obtained by segmenting the first image based on the mask image of the food object. If the scale quality status verification is passed, the physical scale information corresponding to each food object is added to the initial analysis data group to obtain the target analysis data group; The step of determining the weight information and nutritional analysis results for each food object based on its corresponding physical scale information includes: Based on the target analysis data set, the weight information and nutritional analysis results corresponding to each food object are determined.
7. An information processing device, characterized in that, The device includes: The acquisition module is used to acquire the first image; The receiving module is used to receive the user's initial input; A first determining module is configured to, in response to the first input, determine a set of food objects to be analyzed from the first image, wherein the set of food objects includes N food objects to be analyzed, where N is a positive integer; The second determining module is used to determine the mask image corresponding to each food object in the food object set; The third determining module is used to determine the physical scale information corresponding to each food object based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object. The fourth determining module is used to determine the weight information and nutritional analysis results of each food object based on the physical scale information corresponding to each food object. The display module is used to display target analysis results information, which includes weight information and nutritional analysis results information for each food object.
8. The apparatus according to claim 7, characterized in that, The third determining module is specifically used for: Based on the three-dimensional spatial information corresponding to the first image and the mask image corresponding to each food object, determine the set of pixels corresponding to each food object; Based on the set of pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
9. The apparatus according to claim 8, characterized in that, The third determining module is specifically used for: For each of the food objects corresponding to the mask image, the optical axis direction depth information corresponding to each pixel in the mask image is determined from the three-dimensional spatial information corresponding to the first image; Based on the coordinate information of each pixel in the mask image, the optical axis direction depth information corresponding to each pixel, and the camera parameters when the first image was captured, the three-dimensional coordinate information of each pixel in the mask image is obtained. Based on the three-dimensional coordinate information of each pixel in the mask image corresponding to each food object, determine the set of pixels corresponding to each food object.
10. The apparatus according to claim 8, characterized in that, After determining the set of pixels corresponding to each of the food objects, the device further includes: The pixel cleaning module is used to perform point set cleaning processing on the pixel set corresponding to each food object to obtain the target pixel set corresponding to the food object. The point set cleaning processing includes at least one of edge shrinkage, outlier removal, and connectivity check. The third determining module is specifically used for: Based on the set of target pixels corresponding to each food object, the physical scale information corresponding to each food object is obtained.
11. The apparatus according to claim 8, characterized in that, The physical scale information corresponding to a food object includes the distance information from the camera to the food object and the size information of the food object; The third determining module is specifically used for: For each set of pixels corresponding to the food object, the distance information from the camera to the food object is determined based on the set of pixels corresponding to the food object. Construct the bounding box of the food object based on the set of pixels corresponding to the food object; Determine the size information of the bounding box of the food object; The size information of the bounding box of the food object is determined as the size information of the food object.
12. The apparatus according to claim 7, characterized in that, Before determining the weight information and nutritional analysis results for each food object based on its corresponding physical scale information, the device further includes: The fifth determining module is used to determine the scale quality state corresponding to each food object in the food object set; The association module is used to associate the object identifier, second image, mask image and scale quality state of each food object in the food object set to obtain an initial analysis data set. The second image of a food object is obtained by segmenting the first image based on the mask image of the food object. The information addition module is used to add the physical scale information corresponding to each food object to the initial analysis data group after the scale quality status verification is passed, so as to obtain the target analysis data group. The fourth determining module is specifically used for: Based on the target analysis data set, the weight information and nutritional analysis results corresponding to each food object are determined.
