Food image labeling method and system based on multi-information fusion

Abstract

The application provides a food image labeling method and system based on multi-information fusion, and relates to the technical field of computer vision, the method comprises the following steps: step 1, obtaining a food image to be labeled, and synchronously collecting a shooting angle, a distance between a terminal and the food and a shooting direction through a mobile terminal sensor; step 2, inputting the food image into an improved mask region convolutional neural network to obtain a preliminary target detection result; and step 3, performing a graph cut cycle processing on a target region of the preliminary target detection result, and combining morphological erosion and expansion to generate food region binary mask data. The application labels the food image by fusing multi-dimensional information, recognizes and distinguishes food instances, provides comprehensive and accurate labeling results for food-related analysis applications, and improves labeling quality and efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a food image annotation method and system based on multi-information fusion. Background Technology

[0002] Traditional food image annotation techniques have some limitations in family gatherings. When processing food images, traditional techniques mostly rely on the visual features of the image itself, ignoring the spatial information during the shooting process. This makes the annotation results prone to deviation. For example, when a plate of braised pork and a plate of sweet and sour pork ribs are placed on the table at the same time, because the two are similar in color and texture, traditional methods may misidentify the braised pork at the edge as sweet and sour pork ribs because they do not take into account the shooting angle. When the shooting angle makes the edge areas of the two foods present a similar visual effect in the image, it is difficult to distinguish them accurately.

[0003] In addition, traditional techniques are not accurate enough in segmenting food areas and have difficulty handling situations where food is obscured. For example, when steamed fish is on a plate during a meal, it may be obscured by stir-fried vegetables. Traditional methods may incorrectly classify the part of the fish that is obscured by the vegetables into the vegetable area, or fail to accurately identify the complete outline of the fish after it is obscured, thus making it difficult to accurately segment the area of ​​the obscured food. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a food image annotation method and system based on multi-information fusion. By fusing food image features and shooting spatial information, the method identifies food regions and clarifies the hierarchical relationship between food instances, thereby improving the accuracy and semantic integrity of food image annotation.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, a food image annotation method based on multi-information fusion, the method comprising:

[0007] Step 1: Acquire an image of the food to be labeled, and simultaneously collect the shooting angle, distance between the terminal and the food, and shooting orientation through the mobile terminal's sensor;

[0008] Step 2: Input the food image into the improved mask region convolutional neural network to obtain preliminary target detection results;

[0009] Step 3: Perform graph cut loop processing on the target area of ​​the preliminary target detection results, and combine morphological erosion and dilation to generate binary mask data of the food area;

[0010] Step 4: On the binary mask data of the food region, select spatial reference elements based on preset rules, divide the reference elements into uniform units, analyze the distribution characteristics of mask pixels in each unit, and generate region accuracy calibration parameters.

[0011] Step 5: Adjust the binary mask data using the regional accuracy calibration parameters, extract the planar coverage features, centroid coordinates and contour boundary information of the food area from the adjusted binary mask data, and integrate the extracted feature information with the shooting angle, the distance between the terminal and the food and the shooting orientation for multi-dimensional data integration.

[0012] Step 6: Based on the integrated multi-dimensional data, assign a unique color value code to different food instances according to the preset color value mapping index rules, and generate annotation results including hierarchical information.

[0013] Furthermore, the food image is input into an improved masked region convolutional neural network to obtain preliminary object detection results, including:

[0014] The food image to be labeled is input into the feature extraction network of the improved mask region convolutional neural network to generate multi-scale feature maps. Based on the multi-scale feature maps, candidate food target regions are generated through the region proposal network.

[0015] The system performs region of interest alignment on the candidate food target regions, extracts region features, and identifies food categories through the classification branch of the neural network based on the region features. The original category scores generated by the branch are transformed into exponential form and normalized by summation to calculate the category confidence. At the same time, the bounding box coordinates of the candidate regions are fine-tuned through the regression branch.

[0016] By integrating the category confidence score with the bounding box coordinates, preliminary target detection results are generated, including the precise bounding box location of the target, the predicted category, and the corresponding confidence score.

[0017] Furthermore, graph cut loop processing is performed on the target region of the preliminary target detection results, and combined with morphological erosion and dilation, binary mask data of the food region is generated, including:

[0018] The bounding box containing food targets in the preliminary target detection results is set as the initial region for graph cut processing;

[0019] Starting from the initial region, a graph cut segmentation process is performed, pixel-level calculations are carried out, and finally the segmented region is generated.

[0020] Morphological erosion is applied to the segmented region, and morphological dilation is applied to the result of the erosion to obtain the morphologically adjusted region.

[0021] Based on the morphologically adjusted region, binary mask data that accurately represents the spatial distribution of food pixels in the image is generated, with food pixels labeled as a first preset value and non-food pixels labeled as a second preset value.

[0022] Furthermore, based on the binary mask data of the food region, spatial reference elements are selected according to preset rules, the reference elements are uniformly divided into units, and the distribution characteristics of mask pixels within each unit are analyzed to generate region accuracy calibration parameters, including:

[0023] Based on the binary mask data of the food region, at least one spatial reference element is selected from the image region defined by the binary mask data according to the preset spatial location rules;

[0024] For each selected spatial reference element, a uniform grid is divided into several units of uniform size and shape.

[0025] For each unit that has been divided, based on the binary mask data of the food region, the number of pixels in the unit region that are marked as food region is counted, and the proportion of the number to the total number of pixels in the unit is calculated to obtain the mask pixel ratio of the unit.

[0026] Based on the mask pixel ratio of the unit, the distribution characteristics of the mask pixel ratio within the spatial reference element are analyzed to identify the spatial distribution pattern of the mask pixel ratio within the spatial reference element.

[0027] Based on the spatial distribution pattern, calculate and generate regional accuracy calibration parameters for calibrating the accuracy of the binary mask in the food region.

[0028] Furthermore, the binary mask data is adjusted using regional precision calibration parameters. From the adjusted binary mask data, the planar coverage features, centroid coordinates, and contour boundary information of the food area are extracted. The extracted feature information is then integrated with multi-dimensional data such as the shooting angle, the distance between the terminal and the food, and the shooting orientation, including:

[0029] Based on the region accuracy calibration parameters, pixel-level adjustment operations are performed on the binary mask data of the food region to obtain calibrated binary mask data;

[0030] Based on the calibrated binary mask data, the total number of pixels marked as the first preset value is calculated, and combined with the image resolution parameters, it is converted into physical area to generate the planar coverage area feature of the food region.

[0031] Based on the calibrated binary mask data, the mean spatial position of all pixels marked with the first preset value in the image coordinate system is calculated to generate the geometric centroid coordinates of the food region.

[0032] Based on the calibrated binary mask data, the connected domain outer boundary of the pixel region marked with the first preset value is detected, and an ordered sequence of contour boundary points is generated.

[0033] The planar coverage area features, geometric centroid coordinates, and contour boundary point sequence are spatiotemporally correlated and structurally integrated with the synchronously collected shooting angle, distance between the terminal and the food, and shooting orientation information to form multi-dimensional fused data.

[0034] Furthermore, based on the integrated multi-dimensional data and according to the preset color value mapping index rules, a unique color value code is assigned to different food instances, generating annotation results including hierarchical information, including:

[0035] Based on multi-dimensional fusion data, different food instances in an image are identified and distinguished to obtain a set of mutually independent food instance identifiers;

[0036] For each food instance identifier in the food instance identifier set, a globally unique identifier color value is assigned to the food instance identifier according to the preset color value mapping index rules, so as to obtain the unique identifier color value corresponding to each food instance identifier;

[0037] Based on the food instance identifier set and the spatial location relationship information of food instances, the hierarchical relationship between each food instance identifier in the food instance identifier set is determined by calculating the relative spatial relationship between each food instance identifier, and the hierarchical structure information of food instances is generated.

[0038] The unique identifier color value corresponding to each food instance is integrated with the food instance hierarchical structure information to generate the final image annotation result, which includes the unique identifier color value and hierarchical information.

[0039] Furthermore, based on the food instance identifier set and the spatial location relationship information of food instances, the hierarchical relationship between each food instance identifier in the food instance identifier set is determined by calculating the relative spatial relationship between each food instance identifier, generating food instance hierarchical structure information, including:

[0040] Based on the geometric centroid coordinates and the sequence of contour boundary points, the three-dimensional Euclidean distance and contour overlap between food instances are calculated to obtain the distance and overlap data of food instance pairs.

[0041] Using distance and overlap data of food instance pairs, a dataset representing the spatial relationships of all food instance pairs is constructed;

[0042] Based on the spatial relationship dataset, food instance pairs with overlapping areas are screened, and the vertical projection relationship of the overlapping areas is calculated and analyzed to determine the hierarchical affiliation and obtain the hierarchical affiliation determination result.

[0043] Based on the hierarchical attribution determination results, food instance identifiers are organized in a tree structure to generate a structured food instance hierarchy tree as the food instance hierarchy structure information.

[0044] Secondly, a food image annotation system based on multi-information fusion includes:

[0045] The information acquisition module is used to acquire images of the food to be labeled, and to simultaneously acquire the shooting angle, the distance between the terminal and the food, and the shooting direction through the mobile terminal's sensor.

[0046] The target detection module is used to input food images into an improved mask region convolutional neural network to obtain preliminary target detection results;

[0047] The mask processing module is used to perform graph cut loop processing on the target area of ​​the preliminary target detection results, and combine morphological erosion and dilation to generate binary mask data of the food area;

[0048] The calibration generation module is used to select spatial reference elements based on preset rules on the binary mask data of the food region, divide the reference elements into uniform units, analyze the distribution characteristics of mask pixels in each unit, and generate regional accuracy calibration parameters.

[0049] The extraction and integration module is used to adjust the binary mask data using regional accuracy calibration parameters. It extracts the planar coverage features, centroid coordinates, and contour boundary information of the food area from the adjusted binary mask data. The extracted feature information is then integrated with the shooting angle, the distance between the terminal and the food, and the shooting orientation for multi-dimensional data integration.

[0050] The annotation generation module is used to assign unique color value codes to different food instances based on the integrated multi-dimensional data and according to the preset color value mapping index rules, and generate annotation results including hierarchical information.

[0051] Thirdly, a computing device includes:

[0052] One or more processors;

[0053] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0054] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0055] The above-described solution of the present invention has at least the following beneficial effects:

[0056] When relying solely on visual image information for annotation, misjudgments can easily occur due to issues such as similar food appearances or cooking distortions. However, by fusing multiple information sources, such as ingredient composition analysis and cooking process characteristics, easily confused foods can be accurately distinguished. This allows the annotation results to not only include food names but also cover deeper information such as nutritional components and dietary restrictions, meeting more detailed application needs. In complex scenarios, such as when food is partially occluded or the image quality is poor due to dim lighting, the reliability of single visual information decreases. Multi-information fusion can supplement and verify the results with other dimensions of information, such as combining physical information like the weight and volume of food, or related data such as user dietary records and regional dietary habits. This reduces the impact of external interference on the annotation results, enabling the method to function stably in diverse scenarios such as home kitchens, restaurants, and food delivery platforms.

[0057] Traditional manual annotation is not only time-consuming and labor-intensive, but also prone to inconsistencies due to differences in the subjective perception of the annotators. This method, through the automated fusion and analysis of multiple information sources, enables rapid batch annotation of large numbers of food images while reducing manual intervention. For example, it automatically matches a pre-set food database with an image feature library, and then combines nutritional information algorithms to generate annotation content, improving the automation level of the annotation process and providing an efficient solution for large-scale food image data processing. In the field of health management, accurate food annotation can help users record their dietary intake more accurately and assist nutritionists in developing personalized dietary plans. In the field of smart kitchens, it can help smart kitchen appliances achieve automatic identification of ingredients and cooking guidance. In the catering industry, it can provide technical support for the classification of dishes and the generation of nutritional labels for food delivery platforms, promoting the development of the entire food-related industry towards intelligence and refinement. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating a food image annotation method based on multi-information fusion provided in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of a food image annotation system based on multi-information fusion provided in an embodiment of the present invention. Detailed Implementation

[0060] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0061] like Figure 1 As shown, embodiments of the present invention propose a food image annotation method based on multi-information fusion, the method comprising the following steps:

[0062] Step 1: Acquire an image of the food to be labeled, and simultaneously collect the shooting angle, distance between the terminal and the food, and shooting orientation through the mobile terminal's sensor;

[0063] Step 2: Input the food image into the improved mask region convolutional neural network to obtain preliminary target detection results;

[0064] Step 3: Perform graph cut loop processing on the target area of ​​the preliminary target detection results, and combine morphological erosion and dilation to generate binary mask data of the food area;

[0065] Step 4: On the binary mask data of the food region, select spatial reference elements based on preset rules, divide the reference elements into uniform units, analyze the distribution characteristics of mask pixels in each unit, and generate region accuracy calibration parameters.

[0066] Step 5: Adjust the binary mask data using the regional accuracy calibration parameters, extract the planar coverage features, centroid coordinates and contour boundary information of the food area from the adjusted binary mask data, and integrate the extracted feature information with the shooting angle, the distance between the terminal and the food and the shooting orientation for multi-dimensional data integration.

[0067] Step 6: Based on the integrated multi-dimensional data, assign a unique color value code to different food instances according to the preset color value mapping index rules, and generate annotation results including hierarchical information.

[0068] In this embodiment of the invention, during the data acquisition stage, the shooting angle, the distance between the terminal and the food, and the shooting orientation are acquired simultaneously. This allows the analysis of the food image to go beyond the image itself, laying the foundation for accurate labeling. By using an improved mask region convolutional neural network to obtain preliminary target detection results, the approximate range of the food target can be quickly locked, improving the efficiency of target detection and the accuracy of preliminary positioning. The preliminary detection results are processed by graph cut loop, and combined with morphological erosion and dilation to generate binary mask data of the food region. This can effectively separate the food region from the background, remove redundant information, and make the outline of the food region clearer. By selecting spatial reference elements, dividing uniform units, and analyzing pixel distribution characteristics to generate region accuracy calibration parameters, possible deviations in the mask data can be corrected in a targeted manner, improving the accuracy of food region positioning and avoiding misjudgment of regions due to image noise or complex backgrounds.

[0069] After adjusting the binary mask data using regional precision calibration parameters, the extracted planar coverage features, centroid coordinates, and contour boundary information become more accurate. This is then integrated with information such as shooting angle, distance, and orientation in multiple dimensions, achieving a deep fusion of image features and spatial information. This results in a more comprehensive and three-dimensional description of the food, reducing the limitations of a single data dimension. Based on the integrated data and assigning unique color value codes to food instances according to color value mapping index rules, annotation results containing hierarchical information are generated. This not only clearly distinguishes different food instances and avoids confusion, but the hierarchical information also presents the inherent relationships between foods, making the annotation results more organized and readable.

[0070] In a preferred embodiment of the present invention, in step 1 above, an image of the food to be labeled is acquired, and the shooting angle, the distance between the terminal and the food, and the shooting orientation are simultaneously collected by the mobile terminal sensor.

[0071] In this embodiment of the invention, the user opens the camera application on a mobile terminal (such as a smartphone or tablet). At this time, the terminal automatically activates the camera hardware, including the image sensor and lens drive components. The system will first perform a brief initialization, such as checking whether the lens is clean and adjusting the sensitivity of the image sensor to adapt to the surrounding light environment. If the light is too dim, the camera will automatically turn on the flash or prompt the user to turn it on to ensure that the food can be clearly captured. If the light is too strong, the aperture size will be automatically adjusted or HDR mode will be enabled to avoid overexposure of the image. The user points the camera lens at the food to be labeled and previews the image effect of the food in real time through the terminal screen. At this time, the system will analyze the preview image in real time and automatically perform focusing processing. The motor in the lens will drive the lens group to move back and forth until the food presents a clear outline in the preview image. For example, the texture and edges of the food can be clearly distinguished. The user can also manually adjust the framing range and change the composition of the shot through the mobile terminal to ensure that the food is in a suitable position in the picture. It will not be too close and cause some areas to be out of the picture, nor will it be too far away and cause the food to occupy too small a proportion in the picture.

[0072] Once the user confirms that the preview image meets the requirements, they press the shutter button. Instantly, the image sensor quickly captures the optical signals of the food under the current lighting conditions and converts them into electrical signals. After analog-to-digital conversion, digital image data is generated. This data is immediately transmitted to the terminal's memory. At the same time, the system calls the storage mode to save the digital image in a common format (such as JPEG or PNG) to the terminal's local storage space (such as the built-in memory card) and automatically generates a unique identifier for the image (such as a timestamp or file number).

[0073] When the camera app is launched, the mobile device's built-in gyroscope and accelerometer are also activated simultaneously. The system first calibrates these two sensors. For example, the gyroscope uses the initial placement of the device (such as when it is placed horizontally on a table, with the rotation angle of each axis being 0) as the reference value to eliminate zero-point drift caused by minor errors in the device itself. The accelerometer detects the gravitational acceleration experienced by the device when it is stationary to determine the vertical reference. When the user presses the shutter button, the system accurately records the real-time readings of the gyroscope and accelerometer. The gyroscope can sense the changes in the rotation angle of the device around three coordinate axes (corresponding to rotation in the forward, backward, left, right, and up / down directions, respectively). For example, when the device tilts to the left, the rotation angle around the left / right axis will change; when it tilts forward, the rotation angle around the forward / backward axis will change. The accelerometer can detect the linear acceleration of the device in the three directions, and, combined with the component of gravitational acceleration, determine the degree of tilt of the device relative to the horizontal plane.

[0074] The system compares and analyzes the sensor readings at the moment of shooting with the benchmark values ​​determined during the calibration phase. For example, if the accelerometer detects a certain proportional relationship between the vertical component of gravitational acceleration and the horizontal component, the tilt angle of the terminal relative to the horizontal plane can be calculated using this ratio. The larger the proportion of the vertical component, the closer the terminal is to being perpendicular to the horizontal plane; conversely, the larger the tilt. The gyroscope readings can supplement the rotation angle of the terminal around each axis. Finally, the system comprehensively determines the specific angle of the camera lens relative to each benchmark direction in three-dimensional space, i.e., the shooting angle. For example, if the terminal tilts forward by 20 degrees and rotates to the left by 10 degrees, this angle information will be accurately recorded.

[0075] If the mobile terminal is equipped with an infrared distance sensor, the sensor will be automatically activated when the camera is turned on and pointed at the food. The sensor will first perform an internal self-test to ensure that the infrared emitter and receiver are working properly. Then, the infrared emitter will emit an infrared signal of a specific wavelength in the direction of the food. The signal propagates in a straight line, covering the area where the food is located. If the terminal is equipped with a depth camera (such as a camera using structured light technology), it will emit a series of coded light spot patterns onto the food surface. The distribution and position information of these light spots are preset. When the infrared signal encounters the food surface, it will be reflected, and some of the reflected signal will be captured by the terminal's infrared receiver. For a depth camera, the light spot pattern reflected from the food surface will be received by the camera's image sensor. The system will accurately record the time interval from signal transmission to reception. For an infrared distance sensor, since the speed of infrared signal propagation in the air is known (close to the speed of light), this time interval includes the distance information between the terminal and the food. For a structured light depth camera, the system will compare the positional deviation between the emitted and received light spot patterns. The larger the deviation, the farther the food is from the terminal.

[0076] The system calculates distances based on received signals. For infrared distance sensors, the distance the signal travels from the terminal to the food and back is twice the actual distance between them. Therefore, multiplying the signal propagation speed by the time interval and dividing by 2 yields the straight-line distance between the terminal and the food. For example, if the round trip takes 0.000002 seconds, the calculated distance is 0.3 meters. The system also combines multiple measurements (continuously transmitting multiple signals in a very short time) to remove outliers (such as excessively strong or weak signals due to food surface reflection) for a more accurate distance value. The mobile terminal's compass (magnetometer) also starts simultaneously with the camera and performs a calibration. This usually requires the user to perform simple actions, such as rotating the terminal around a specific axis, to eliminate magnetic field interference from the surrounding environment (such as magnetic fields generated by nearby metal objects) and ensure the magnetometer accurately senses the direction of the Earth's magnetic field. After calibration, the magnetometer determines the direction of the magnetic north pole as a reference.

[0077] At the moment of shooting, the magnetometer continuously detects the angle between the terminal's current pointing direction and the geomagnetic north pole. For example, if the terminal's lens is pointing in the same direction as the geomagnetic north pole, the angle is 0 degrees; if it's pointing towards the geomagnetic south pole, the angle is 180 degrees. Simultaneously, the gyroscope and accelerometer provide additional attitude information, such as whether the terminal is level or tilted. This is because tilting the terminal can cause a deviation between the actual direction the lens is pointing and the horizontal direction detected by the magnetometer alone. The magnetometer combines the angle information detected by the magnetometer with the attitude information from the gyroscope and accelerometer for comprehensive calculation. For instance, if the magnetometer shows the terminal pointing towards the geomagnetic north pole... The polar angle is 30 degrees, which is northeast. The gyroscope and accelerometer show that the terminal is tilted forward by 15 degrees. Based on this information, the shooting direction is further determined, and the specific directional position of the food relative to the terminal is clarified. For example, if the food is northeast of the terminal, and the terminal lens is slightly tilted downward to aim at the food, the final shooting direction information will be accurate to a specific angle range. Throughout the process, from the moment the user presses the shooting button, the time point of image generation is precisely matched with the data collected by the gyroscope, accelerometer, distance sensor, and magnetometer at the same moment. They are associated with each other through timestamps and stored in the terminal's database.

[0078] In a preferred embodiment of the present invention, step 2 above, which involves inputting the food image into an improved mask region convolutional neural network to obtain preliminary target detection results, may include:

[0079] Step 220: Input the food image to be labeled into the feature extraction network of the improved mask region convolutional neural network to generate a multi-scale feature map, and generate candidate food target regions based on the multi-scale feature map through the region proposal network.

[0080] Step 221: Perform region of interest alignment on the candidate food target region, extract region features, and identify food categories based on the region features through the classification branch of the neural network; perform exponential form transformation and sum-normalization on the original category scores generated by the branch, calculate the category confidence, and fine-tune the bounding box coordinates of the candidate region through the regression branch.

[0081] Step 222: Integrate the category confidence score with the bounding box coordinates to generate preliminary target detection results, including the precise bounding box location of the target, the predicted category, and the corresponding confidence score.

[0082] In this embodiment of the invention, after the food image to be labeled enters the feature extraction network of the improved mask region convolutional neural network, it first undergoes a first convolution operation. This layer has multiple different convolutional kernels, each with a fixed size, such as 3×3. When the convolutional kernel slides across the image, it covers a 3×3 pixel area in the image each time. The weight of each position of the convolutional kernel is multiplied by the value of the corresponding pixel. For example, the weight of the upper left corner of the convolutional kernel is multiplied by the pixel value of the corresponding position in the image, the weight of the upper right corner is multiplied by the pixel value of the corresponding upper right corner of the image, and so on. Then, all nine product results are added together to obtain a new value, which is the value of the image. The feature value corresponding to the sliding position is used to complete one sliding operation. Then, the convolution kernel moves to the next position according to the preset stride (e.g., 2 pixels). The above calculation is repeated until the entire image is covered, generating a feature map. Then, the image enters the next layer of convolution. The number of convolution kernels in this layer may be more than that in the previous layer, and the above sliding calculation is performed in the same way. However, the size of the generated feature map will be smaller due to the stride. After multiple layers of convolution, multiple feature maps of different sizes will be obtained. For example, some feature maps can clearly show the texture of the food surface (small-scale features), while others can present the overall shape of the food (large-scale features).

[0083] Based on multi-scale feature maps, the region proposal network begins its work. First, multiple different anchor boxes are pre-defined for each pixel in each feature map. These anchor boxes have different sizes (e.g., 128×128 pixels, 256×256 pixels, etc.) and aspect ratios (e.g., 1:1, 1:2, 2:1, etc.). Then, for each anchor box, the network performs convolution calculations on the feature map region it covers using a specific convolution kernel to obtain the feature vector corresponding to that anchor box. Afterward, through a simple classification calculation, the feature vector is converted into two values: one representing the number of elements within the anchor box. The probability of a food target is calculated, and the probability of no food target is calculated. These two values ​​are added together to get 1. At the same time, four adjustment parameters are calculated to adjust the center x-coordinate, center y-coordinate, width, and height of the anchor box. For example, if the center x-coordinate of the anchor box on the feature map is 50 and the calculated adjustment parameter is +5, it means that the center x-coordinate of the anchor box needs to be moved 5 pixels to the right. Next, anchor boxes with a high probability of having a food target (e.g., a probability greater than 0.7) are selected, and the position and size of these anchor boxes are corrected according to the adjustment parameters to obtain the candidate food target region.

[0084] For each candidate food target region, first determine its coordinate range in the original image, such as the top left corner ( , ) and bottom right corner ( , Then, based on the scaling ratio between the feature map and the original image, this coordinate range is mapped onto the corresponding feature map to obtain the coordinate range of the region on the feature map. Since the mapped coordinates may not be integers, for example, the coordinates corresponding to a certain boundary might be 3.2, bilinear interpolation is required. Specifically, the four nearest integer coordinates (3,3), (3,4), (4,3), and (4,4) around this coordinate are found, and the feature values ​​of these four points are obtained. Then, the weights are calculated based on the distance between 3.2 and these four points, with closer distances resulting in larger weights. Finally, these four feature values ​​are multiplied by their respective weights and then summed. The feature value at coordinate 3.2 is obtained. In this way, the region corresponding to the candidate region on the feature map is accurately extracted. The extracted feature region is then adjusted to a fixed size (e.g., 7×7). During the adjustment process, bilinear interpolation is also used to ensure the accuracy of the feature. Then, this fixed-size feature region is converted into a one-dimensional feature vector. For example, a 7×7 feature region will be converted into a vector with 49 elements. Then, through a fully connected layer, this vector is calculated with the weight matrix of the fully connected layer (each element is multiplied by the corresponding weight and then added together) to obtain a longer feature vector, that is, the region feature.

[0085] After the regional features enter the classification branch, they are processed by multiple fully connected layers. In each fully connected layer, each element of the feature vector is multiplied by the corresponding weight in the weight matrix of that layer. Then, all the product results are added together and a bias value is added to obtain a new feature vector. For example, a 100-element feature vector may become a 200-element feature vector after passing through a fully connected layer. Each new element is calculated by the original element and the corresponding weight. After multiple layers of processing, the original category score is finally obtained, which is the same as the preset number of food categories. For example, if there are 100 food categories, 100 scores will be obtained. Next, each original category score is transformed into an exponential form, that is, each score is used as the exponent of the natural constant to obtain a new value. For example, if the original score is 2, the transformed value is the square of the natural constant. Then, all the transformed values ​​are added together to obtain a sum. Then, each transformed value is divided by this sum to obtain the category confidence. For example, if the transformed value of "apple" is 10 and the sum is 100, then the category confidence of "apple" is 0.1.

[0086] After receiving the region features, the regression branch also processes them through multiple fully connected layers, ultimately outputting four adjustment parameters, corresponding to the center x-coordinate offset, center y-coordinate offset, width scaling factor, and height scaling factor of the bounding box. These parameters are calculated based on the difference between the candidate region's bounding box and the ideal bounding box. For example, if the center x-coordinate of the candidate region is 100, while the ideal center x-coordinate is 110, then the center x-coordinate offset is +10; if the candidate region's width is 50, and the ideal width is 60, then the width scaling factor is 1.2 (60 ÷ 50). After obtaining these parameters, the original bounding box coordinates of the candidate region are added to or multiplied by these parameters to obtain the fine-tuned bounding box coordinates. For example, adding an offset of 10 to the original center x-coordinate of 100 results in a new center x-coordinate of 110; multiplying the original width of 50 by the scaling factor of 1.2 results in a new width of 60.

[0087] For each candidate food target region, find the highest category confidence score and its corresponding food category. For example, if the category confidence score for "hamburger" in a certain region is 0.8, which is the highest among all categories, then the predicted category for that region is determined to be "hamburger," with a corresponding confidence score of 0.8. Then, record the fine-tuned bounding box coordinates for that region, including the top left corner. , ) and bottom right corner ( , These two coordinates are accurate to the pixel level, which can accurately frame the food target in the image. Finally, the predicted category, corresponding confidence score and bounding box coordinates of each region are integrated to form a list containing all this information. This list is the preliminary object detection result. For example, the list may contain a record like this: predicted category "hamburger", confidence score 0.8, bounding box (50, 60)-(200, 180), which means that the region from (50, 60) to (200, 180) in the image is a hamburger, with a confidence score of 80%.

[0088] In the process of generating multi-scale feature maps, convolutional operations at different levels can capture various features of food from details to the whole. Small-scale feature maps can clearly reflect details such as texture and color changes of food, while large-scale feature maps can reflect the overall shape of food. This allows features to be effectively captured, whether it's a small strawberry or a whole pizza, improving the detection capability for food of different sizes. The region proposal network can quickly filter out regions that may contain food by pre-setting multiple anchor boxes and calculating the probability that each anchor box contains food, reducing the amount of computation and making the detection process more efficient. In the region of interest alignment operation, the use of bilinear interpolation can accurately calculate the feature values ​​of non-integer coordinates, avoiding the problems caused by coordinate rounding. The loss of feature information makes the extracted regional features more accurate. In the calculation of category confidence, the exponential transformation amplifies the difference in scores between different categories, while normalization makes the results more intuitive, clearly showing the probability of each category and reducing the probability of misclassifying food. The fine-tuning of the bounding box by the regression branch, through precise calculation of offset and scaling factors, allows the bounding box to more tightly surround the food target, avoiding the situation where the bounding box is too large and contains too much background or too small and misses some food, thus improving the accuracy of target localization. Integrating the predicted category, confidence, and bounding box coordinates together forms a complete and clear preliminary target detection result, simplifying the annotation process. At the same time, because the information is accurate, the possibility of annotation errors is reduced.

[0089] In a preferred embodiment of the present invention, step 3 above, which involves performing graph cut loop processing on the target area of ​​the preliminary target detection result and combining morphological erosion and dilation to generate binary mask data for the food region, may include:

[0090] Step 330: Set the bounding box of the food target in the preliminary target detection result as the initial region for graph cut processing;

[0091] Step 331: Starting from the initial region, perform the graph cut segmentation process, perform pixel-level calculations, and finally generate the segmented region;

[0092] Step 332: Apply morphological erosion to the segmented region, and apply morphological dilation to the result of the erosion operation to obtain the morphologically adjusted region.

[0093] Step 333: Based on the morphologically adjusted region, generate binary mask data that accurately represents the spatial distribution of food pixels in the image, and label the food pixels as the first preset value and the non-food pixels as the second preset value.

[0094] In this embodiment of the invention, the precise bounding box position information corresponding to each food target is extracted from the preliminary target detection results. These bounding boxes are determined by the pixel coordinates of the upper left and lower right corners. For example, the upper left corner coordinate of a certain bounding box is (…). , The coordinates of the lower right corner are ( , The region bounded by the coordinate range is directly set as the initial region for graph cut processing. This means that the graph cut operation will start from all pixels contained in this bounding box, and pixels outside this region will not participate in the initial graph cut segmentation process for the time being. All pixels in the initial region are used as nodes of the graph. For each pixel, the similarity between it and its surrounding neighboring pixels (usually pixels in the four directions of up, down, left, and right) is calculated. The similarity is mainly based on the pixel's color value (such as RGB value) and brightness value. For example, the closer the RGB values ​​of two pixels are and the smaller the difference in brightness, the higher their similarity is, and the greater the weight of the edge connecting these two pixels in the graph structure. Conversely, the weight is smaller. At the same time, two special nodes are set, namely the source node (representing the food region) and the sink node (representing the non-food region). Pixels in the initial region are connected to the source node by default. The weight of the edge is set according to the probability that the pixel belongs to the food region. Initially, it can be assumed that the pixels in the initial region are more likely to belong to the food region, so the connection weight with the source node is larger.

[0095] The region is segmented by calculating the minimum cut of the graph. The minimum cut is a set of edges that separates the source node and the sink node with the minimum total weight. After each minimum cut calculation, the pixels are divided into regions connected to the source node (temporarily designated as food regions) and regions connected to the sink node (temporarily designated as non-food regions). Then, the connection weights of each pixel with the source and sink nodes are re-evaluated. For pixels classified as food regions, if their color and brightness differ significantly from the surrounding known food pixels, their connection weight with the source node is reduced. For pixels classified as non-food regions, if their color and brightness differ slightly from the surrounding food pixels, their connection weight with the source node is increased. The process of constructing the graph structure and calculating the minimum cut is repeated until the change in the segmented region obtained from two iterations is less than a preset threshold, for example, the number of pixel segmentation changes does not exceed 5% of the total number of pixels in the initial region. At this point, the iteration stops, and the segmented region is obtained.

[0096] Choose a structuring element of a fixed shape and size, such as a 3×3 square structuring element (containing 9 pixels). Align the center of the structuring element sequentially with each pixel in the segmented region. Check if all pixels covered by the structuring element belong to the segmented region (i.e., are all pixels identified as food). If any pixel does not belong to the segmented region, the currently aligned center pixel is removed from the region. Only when all pixels covered by the structuring element belong to the segmented region is the center pixel retained. Through this pixel-by-pixel checking and processing, part of the edge of the segmented region is "eroded," especially removing protruding small burrs or isolated small pixel clusters. Use the same structuring element as in the erosion operation. Similarly, align the center of the structuring element sequentially with each pixel in the erosion result region. As long as at least one pixel covered by the structuring element belongs to the eroded region, the currently aligned center pixel is included in the region. In this way, the edge of the eroded region expands outward, filling small holes within the region and smoothing the overall contour of the region, ultimately resulting in a morphologically adjusted region.

[0097] Based on the morphologically adjusted region, each pixel in the image is examined one by one. If a pixel belongs to the morphologically adjusted region (i.e., is determined to be a food pixel), the pixel is marked with a first preset value, such as 255. If a pixel does not belong to the morphologically adjusted region (i.e., is determined to be a non-food pixel), the pixel is marked with a second preset value, such as 0. In this way, the entire image is converted into a binary image containing only these two preset values, which is a binary mask data that accurately represents the spatial distribution of food pixels in the image.

[0098] Using the target bounding box obtained from preliminary detection as the initial region for graph cut, the scope of graph cut processing can be narrowed using existing detection results, avoiding indiscriminate processing of the entire image and reducing computational load. At the same time, the graph cut operation starts from the region most likely to contain food, improving the targeting and efficiency of segmentation. By constructing a graph structure based on pixel similarity and iteratively calculating the minimum cut, pixel-level fine segmentation can be achieved, accurately distinguishing food and background pixels. Even if the food and background colors are similar, the segmentation results can be gradually optimized by adjusting the weights through multiple iterations, making the segmented region more closely match the actual outline of the food. The binary mask data clearly distinguishes food and non-food pixels with two simple preset values, intuitively and accurately reflecting the spatial distribution of food in the image, providing a concise and clear data foundation for food image annotation, recognition, and other processing.

[0099] In a preferred embodiment of the present invention, step 4 above, which involves selecting spatial reference elements based on preset rules on the binary mask data of the food region, uniformly dividing the reference elements into units, analyzing the distribution characteristics of mask pixels within each unit, and generating region accuracy calibration parameters, may include:

[0100] Step 440: Based on the binary mask data of the food region, select at least one spatial reference element from the image region defined by the binary mask data according to the preset spatial location rules;

[0101] Step 441: For each selected spatial reference element, perform uniform grid division on each spatial reference element to divide the spatial reference element into several units of uniform size and shape.

[0102] Step 442: For each unit obtained by division, based on the binary mask data of the food region, count the number of pixels in the unit region that are marked as food region, and calculate the proportion of the number to the total number of pixels in the unit to obtain the mask pixel ratio of the unit.

[0103] Step 443: Based on the mask pixel ratio of the unit, analyze the distribution characteristics of the mask pixel ratio within the spatial reference element, and identify the spatial distribution pattern of the mask pixel ratio within the spatial reference element.

[0104] Step 444: Calculate and generate regional accuracy calibration parameters for calibrating the accuracy of the binary mask in the food region based on the spatial distribution pattern.

[0105] In this embodiment of the invention, based on the acquired binary mask data of food regions (where a pixel value of 1 in the mask represents a food region and 0 represents a non-food region), according to preset spatial location rules (e.g., selecting the largest connected region in the mask region, selecting a fixed-size region at the center of the mask, or selecting a feature region on the edge contour of the mask, etc.), at least one region conforming to the rules is selected and determined as a spatial reference element from the entire image region defined by the binary mask data. The specific process is as follows: First, all connected regions in the binary mask data are traversed, and the area of ​​each connected region is calculated (the total number of pixels with a pixel value of 1 in the region is counted). If the preset rule is "select the largest connected region", then the area values ​​of all connected regions are compared, and the connected region with the largest area value is determined as the spatial reference element. If the preset rule is "select the center region", then the geometric center coordinates of the entire mask region are calculated (by calculating the average of the coordinates of all pixels with a pixel value of 1), and then a square region with a fixed side length (e.g., a square with a side length of 1 / 5 of the longest side of the mask region) is delineated with the center coordinates as the origin, and the mask portion within the square region is used as the spatial reference element.

[0106] For each spatial reference element selected in step 440, the spatial reference element is uniformly divided according to a preset grid size (such as a 10×10 pixel square unit). The specific process is as follows: First, determine the boundary range of the spatial reference element (i.e., the pixel coordinates of the upper left and lower right corners of the element), and calculate the pixel length of the element in the horizontal and vertical directions; then, according to the preset unit size, calculate the number of units that need to be divided in the horizontal and vertical directions; finally, starting from the upper left corner of the spatial reference element, divide it into several units of uniform size and shape in the horizontal and vertical directions, and the size of each unit is strictly consistent (if the remaining pixels at the edge of the element are less than the size of a unit, it is still divided into a complete unit, and the pixel value of the part that exceeds the boundary of the element is recorded as 0).

[0107] For each unit obtained in step 441, based on the original binary mask data of the food region, count the number of pixels marked as food region (i.e., the total number of pixels with a value of 1) in all pixels within the unit region. The specific process is as follows: traverse each pixel in the unit, check its pixel value in the binary mask data one by one, and increment the count by 1 if it is 1, and finally obtain the number of mask pixels in the unit; at the same time, calculate the total number of pixels in the unit (i.e., unit width × unit height); then divide the number of mask pixels by the total number of pixels in the unit, and the quotient is the proportion of mask pixels in the unit (the result is rounded to 4 decimal places, for example, if there are 30 mask pixels in the unit and the total number of pixels is 100, then the proportion is 0.3000).

[0108] Based on the mask pixel proportion values ​​of each unit obtained in step 442, the distribution characteristics of these proportion values ​​within the entire spatial reference element are analyzed to identify its spatial distribution pattern. The specific process is as follows: First, record the position coordinates of each unit (such as the coordinates of the upper left corner of the unit) and its corresponding mask pixel proportion value to form a "position-proportion value" dataset; then, by comparing the differences in proportion values ​​of adjacent units, determine whether there are continuous high proportion regions (such as proportion values ​​of more than 3 consecutive units all greater than 0.8) or low proportion regions (such as proportion values ​​of more than 3 consecutive units all less than 0.2); at the same time, calculate the average and standard deviation of the mask pixel proportion values ​​within the entire spatial reference element. If the standard deviation is small (such as less than 0.1), it indicates that the proportion values ​​are evenly distributed; if the standard deviation is large (such as greater than 0.3), it indicates that there are differences in the distribution; finally, based on the position of the high / low proportion regions, the number of consecutive units, and the overall statistical characteristics, determine the spatial distribution pattern (such as "high in the center and low at the edge", "uniform distribution", "local clustering", etc.).

[0109] Based on the spatial distribution pattern identified in step 443, regional accuracy calibration parameters for calibrating the accuracy of the binary mask in the food region are calculated and generated. The specific process is as follows: If the distribution pattern is "high in the center and low at the edges", it indicates that the mask may be inaccurate in the edge region (such as over-segmentation or under-segmentation). Then, a calibration coefficient is calculated for the edge unit (for example, the calibration coefficient increases by 0.05 for every 0.1 lower than the average value of the edge unit, which is used to enhance the mask accuracy of the edge region in the future). If the distribution pattern is "locally clustered", higher weight parameters are assigned to the units in the clustered region (such as a weight of 1.2 for the clustered region and 0.8 for the non-clustered region), which is used to focus on optimizing the mask in the clustered region during calibration. If the distribution pattern is "uniformly distributed", an overall offset parameter is generated (such as calculating a global compensation value based on the difference between the average value and the ideal value, so that the overall mask proportion is closer to the proportion of the real food region). All parameters need to clearly define the specific location or region in the corresponding spatial reference element to form a parameter set that can be directly used for calibration calculation.

[0110] By selecting spatial reference elements and analyzing their internal mask pixel distribution, areas where errors may exist in the mask data can be accurately located, making the generation of calibration parameters more targeted. By calculating the proportion and distribution characteristics of unit mask pixels, the originally abstract mask quality assessment is transformed into specific numerical indicators, providing a quantifiable calculation basis for calibration parameters. This makes the calibration process more objective and repeatable. After identifying the spatial distribution pattern of mask pixels, the generated calibration parameters can specifically adjust the mask characteristics of local areas, reducing distribution deviations caused by shooting angles, uneven lighting, etc. This makes the mask data of the food area more spatially consistent with the actual food shape distribution. The mask data after regional precision calibration is closer to the actual food area, providing more accurate basic data for steps such as food volume calculation and nutritional analysis, and reducing the deviation of analysis results caused by mask errors.

[0111] In a preferred embodiment of the present invention, step 5 above, which involves adjusting the binary mask data using region precision calibration parameters, extracting the planar coverage features, centroid coordinates, and contour boundary information of the food area from the adjusted binary mask data, and integrating the extracted feature information with multi-dimensional data such as the shooting angle, the distance between the terminal and the food, and the shooting orientation, may include:

[0112] Step 550: Based on the region accuracy calibration parameters, perform pixel-level adjustment operations on the binary mask data of the food region to obtain calibrated binary mask data;

[0113] Step 551: Based on the calibrated binary mask data, calculate the total number of pixels marked as the first preset value, and convert it into physical area by combining the image resolution parameters to generate the planar coverage area feature of the food area.

[0114] Step 552: Based on the calibrated binary mask data, calculate the average spatial position of all pixels marked with the first preset value in the image coordinate system to generate the geometric centroid coordinates of the food region.

[0115] Step 553: Based on the calibrated binary mask data, detect the outer boundary of the connected region of the pixel region marked with the first preset value, and generate an ordered sequence of contour boundary points;

[0116] Step 554 involves spatiotemporally and structurally integrating the planar coverage area features, geometric centroid coordinates, and contour boundary point sequence with the synchronously acquired shooting angle, distance between the terminal and the food, and shooting orientation information to form multi-dimensional fused data.

[0117] In this embodiment of the invention, regional accuracy calibration parameters are first obtained. These parameters are predetermined based on factors such as lens distortion characteristics of the image acquisition device, light interference during shooting, and hardware errors of the device. They include the deviation values ​​that each pixel in the image needs to compensate for in the horizontal and vertical directions (i.e., the x-direction correction amount and y-direction correction amount corresponding to each pixel). All pixels in the binary mask data are traversed row by row and column by column. The label value of each pixel is checked one by one to determine whether the pixel is labeled with the first preset value (usually the first preset value represents pixels belonging to the food area, and the second preset value represents pixels in the non-food area). When a pixel labeled with the first preset value is encountered, the original coordinates of the pixel in the mask data are recorded (i.e., the row number and column number of the pixel, corresponding to the x and y coordinates in the image coordinate system). Based on the original coordinates of the pixel, the corresponding horizontal and vertical deviation values ​​are found from the regional accuracy calibration parameters.

[0118] The new x-coordinate is equal to the original x-coordinate plus the horizontal deviation value, and the new y-coordinate is equal to the original y-coordinate plus the vertical deviation value. If the calculated new coordinate exceeds the boundary of the mask data (i.e., less than 0 or greater than the maximum width / height of the mask), it is adjusted to the closest boundary coordinate (for example, if the new x-coordinate is less than 0, it is set to 0). The first preset value at the original coordinate is transferred to the calculated new coordinate, and the pixel value at the original coordinate is changed to the second preset value. This ensures that each pixel with the first preset value is retained only in the calibrated position. The above operation is repeated for all pixels marked with the first preset value. After all pixels are adjusted, the resulting mask data is the calibrated binary mask data.

[0119] Each pixel in the calibrated binary mask data is traversed, and its marker value is checked one by one to see if it is the first preset value. Each time a pixel with the first preset value is encountered, it is counted once until all pixels are traversed. The total number of times is obtained is the total number of pixels marked with the first preset value (referred to as the total number of pixels). The image resolution parameters are obtained, including the total number of pixels in the horizontal direction (i.e., the number of pixels in the image width), the total number of pixels in the vertical direction (i.e., the number of pixels in the image height), and the "pixel physical size parameter" corresponding to the image acquisition device (i.e., the length represented by each pixel in the actual physical space, for example, 0.1 mm per pixel, which is determined by the hardware specifications of the device). The actual physical area corresponding to a single pixel is calculated by multiplying the physical length of each pixel by itself (for example, if each pixel corresponds to 0.1 mm, then the physical area of ​​a single pixel is 0.1 mm × 0.1 mm). The actual planar coverage area of ​​the food area is calculated by multiplying the total number of pixels with the first preset value obtained in step 551 by the physical area of ​​a single pixel. The result is the planar coverage area feature of the food area.

[0120] The rules for setting the image coordinate system are determined by taking the top left corner of the image as the origin, the positive x-axis as the horizontal direction to the right, and the positive y-axis as the vertical direction downward. The coordinates of each pixel are represented by (x, y) (x is the index of the pixel in the horizontal direction, and y is the index in the vertical direction). All pixels marked with the first preset value in the calibrated binary mask data are traversed, and the coordinates of each pixel (i.e., the x and y values ​​of each pixel) are recorded one by one. The total number of these pixels is then counted.

[0121] Calculate the sum of the x-coordinates of all pixels with the first preset value: add the x-values ​​of each pixel one by one, and the cumulative result is the sum of the x-coordinates.

[0122] Calculate the sum of the y coordinates of all pixels with the first preset value: add the y values ​​of each pixel one by one, and the cumulative result is the sum of the y coordinates.

[0123] Calculate the mean value in the x-direction: Divide the sum of the x-coordinates by the total number of pixels in the first preset value. The result is the average value of the x-coordinates of all pixels (denoted as the average x-value).

[0124] Calculate the mean value in the y-direction: Divide the sum of the y-coordinates by the total number of pixels in the first preset value. The result is the average value of the y-coordinates of all pixels (denoted as the average y-value).

[0125] The coordinates (average x value, average y value) composed of the average x value and average y value are the geometric centroid coordinates of the food region in the image coordinate system.

[0126] In the calibrated binary mask data, the region consisting of all pixels marked with the first preset value is the food region. "Connectivity" refers to pixels being adjacent in the horizontal, vertical, or diagonal directions (i.e., two pixels sharing an edge or a corner). Pixels belonging to the same connected region must be mutually connected. A region growing method is used to determine the complete connected region. Starting from any unmarked pixel with the first preset value in the food region, its eight adjacent pixels (up, down, left, right, upper left, upper right, lower left, lower right) are checked. If an adjacent pixel is also a first preset value and is unmarked, it is added to the current connected region, and the process of checking its adjacent pixels continues. This process is repeated until no new pixels can be added, ultimately obtaining the complete connected region of the food region. The outer boundary pixels of the connected region are then detected, starting from the upper left corner pixel of the connected region. Moving clockwise along the edge of the connected region, the system checks the eight adjacent pixels of the current pixel in each direction. If an adjacent pixel in a certain direction is a second preset value (non-food area) or exceeds the image boundary, the current pixel is a boundary pixel, and its coordinates are recorded. Boundary pixels are recorded in sequence. After each boundary pixel is determined, the system moves clockwise to the next possible boundary pixel (for example, moving to the right from the current pixel; if the pixel on the right is a boundary pixel, it is recorded; otherwise, it moves down to check). This process is repeated until the system returns to the starting boundary pixel, forming a closed boundary. All the recorded boundary pixel coordinates are arranged in clockwise order to form an ordered sequence (for example, starting from the starting point, the (x, y) coordinates of each boundary pixel are recorded in sequence until the system returns to the starting point), which is the contour boundary point sequence.

[0127] Acquire spatiotemporal identification information for data acquisition, including the precise time of image acquisition (e.g., year, month, day, hour, minute, second, millisecond) and the unique identifier of the acquisition device (e.g., device serial number), ensuring all data is traceable to the same acquisition time and device. Associate the planar coverage area features, geometric centroid coordinates, and contour boundary point sequence with the aforementioned time identifiers, and label the corresponding acquisition time next to each feature data to ensure these visual feature data correspond to the same shooting time. Acquire synchronously acquired shooting parameters, including shooting angle (divided into horizontal rotation angle, i.e., the angle of device rotation around the vertical axis; and vertical pitch angle, i.e., the angle of device up and down rotation around the horizontal axis), the actual distance between the terminal and the food (measured by the device's distance sensor, in units such as centimeters), and shooting azimuth (based on due north, the horizontal azimuth angle pointed to by the device's lens, in units such as degrees). The shooting parameters are associated with time markers. The same acquisition time as the visual feature data is marked next to the shooting angle, distance, and orientation parameters to ensure that the shooting parameters and visual feature data come from the same acquisition time. A structured data format is constructed, and fixed data fields are set, including "acquisition time", "device identifier", "planar coverage area", "centroid x coordinate", "centroid y coordinate", "contour boundary point sequence" (stored in the form of coordinate array), "horizontal rotation angle", "vertical pitch angle", "distance between terminal and food", and "shooting azimuth angle". All the above data are filled into the structured format. For example, the "planar coverage area" field is filled with the calculation result of step 551, the "centroid x coordinate" field is filled with the average x value of step 552, and the "horizontal rotation angle" field is filled with the angle value acquired synchronously. The final complete data is the multi-dimensional fused data.

[0128] Pixel-level calibration compensates for device errors and environmental interference, correcting pixel position deviations in food areas of the mask data. This ensures that subsequently extracted features such as area, centroid, and contour are closer to the actual physical form, reducing system errors caused by device characteristics. It converts the number of pixels into actual physical area, transforming abstract features (pixel count) that originally only existed in the image into physical parameters with real-world significance (such as square centimeters). This facilitates direct association with the actual size of the food and provides a quantifiable basis for food quantity estimation (such as weight and volume). The calculated centroid coordinates reflect the spatial distribution center of the food area and can serve as a core reference for the overall position of the food. The contour boundary point sequence accurately describes the shape contour of the food. The combination of these two features fully presents the geometric characteristics of the food, providing a comprehensive geometric basis for food type identification and morphological analysis.

[0129] In a preferred embodiment of the present invention, step 6 above, based on the integrated multi-dimensional data and according to a preset color value mapping index rule, assigns a unique color value code to different food instances to generate a labeling result including hierarchical information, and may include:

[0130] Step 660: Based on multi-dimensional fusion data, identify and distinguish different food instances in the image to obtain a set of mutually independent and distinguishable food instance identifiers;

[0131] Step 661: For each food instance identifier in the food instance identifier set, assign a globally unique identifier color value to the food instance identifier according to the preset color value mapping index rule, so as to obtain the unique identifier color value corresponding to each food instance identifier.

[0132] Step 662: Based on the food instance identifier set and the spatial relationship information of food instances, the hierarchical relationship between each food instance identifier in the food instance identifier set is determined by calculating the relative spatial relationship between each food instance identifier, and food instance hierarchical structure information is generated. Specifically, this includes: calculating the three-dimensional spatial Euclidean distance and contour overlap between food instances based on the geometric centroid coordinates and contour boundary point sequence to obtain the distance and overlap data of food instance pairs; using the distance and overlap data of food instance pairs, constructing a dataset representing the spatial relationship of all food instance pairs; based on the spatial relationship dataset, filtering food instance pairs with overlapping areas, calculating and analyzing the vertical projection relationship of the overlapping areas, determining the upper and lower level affiliation, and obtaining the hierarchical affiliation determination result; based on the hierarchical affiliation determination result, organizing the food instance identifiers in a tree structure to generate a structured food instance hierarchical tree as food instance hierarchical structure information.

[0133] Step 663: Integrate the unique identifier color value corresponding to each food instance identifier with the food instance hierarchical structure information to generate the final image annotation result including the unique identifier color value and hierarchical information.

[0134] In this embodiment of the invention, firstly, multi-dimensional fusion data is acquired, which includes the color features, texture features, shape features, spatial coordinates, and ingredient information of the food. Next, the color features are processed by comparing the RGB value of each pixel in the image with the typical color range of known foods, such as the white range of rice and the green range of green peppers, to initially filter out pixel regions that may belong to the same food category. For texture features, the frequency and pattern of pixel grayscale value changes in different regions are calculated to distinguish differences such as the rough texture of bread and the smooth texture of tofu. Regarding shape features, the contour of each candidate region is extracted, and the perimeter, area, and bounding moment of the contour are calculated. Parameters such as aspect ratio are used to determine whether a circular pizza or a long noodle will show significant differences. Combined with spatial coordinates, the continuity of pixel regions is determined. If multiple adjacent pixels are consistent in color, texture, and shape and are spatially connected, they are grouped into the same preliminary food region. Then, these preliminary regions are further verified to check whether there are obvious feature differences between different regions, such as color differences exceeding a preset threshold or completely different texture types, to determine that they are different food instances. Finally, each determined different food instance is assigned a unique identifier, such as number 1, number 2, etc., thus forming a set of mutually independent food instance identifiers.

[0135] First, define the preset color value mapping index rules. These rules specify the correspondence between different identifiers and color values, and ensure that each color value is unique globally. For example, the rules might set identifier 1 to correspond to red with an RGB value of (255, 0, 0), identifier 2 to green with an RGB value of (0, 255, 0), etc. At the same time, ensure that these color values ​​do not appear in the mapping of any other food instance identifiers. Then, iterate through each identifier in the set of food instance identifiers, matching each identifier with the color value mapping index rules. For each identifier, find the corresponding color value from the rules. For example, if the color value corresponding to identifier 3 is found to be blue with an RGB value of (0, 0, 255), assign this color value to identifier 3. During the assignment process, a global uniqueness check is performed to compare whether the color value is duplicated with the color values ​​already assigned to other identifiers. If there is no duplication, the color value is determined to be the unique identifier color value for the current identifier. In this way, the corresponding color values ​​are assigned to all identifiers in the set, and finally, the unique identifier color value corresponding to each food instance identifier is obtained.

[0136] Extract the spatial coordinate range corresponding to each food instance identifier from the spatial location relationship information of food instances, such as the coordinates of the top left and bottom right corners of each food instance in the image, to determine their position and occupied area in the image. Then, calculate the relative spatial relationship between each food instance identifier. For any two food instance identifiers A and B, determine whether the spatial region of A completely contains the spatial region of B. If the x-coordinate of the top left corner of A is less than the x-coordinate of the top left corner of B, the y-coordinate of the top left corner of A is less than the y-coordinate of the top left corner of B, and the x-coordinate of the bottom right corner of A is greater than the x-coordinate of the bottom right corner of B, and the y-coordinate of the bottom right corner of A is greater than the y-coordinate of the bottom right corner of B, then it means that A contains B. If the spatial regions of the two instances partially overlap but do not completely contain each other, or do not overlap at all... If the food instances are stacked, their vertical position is further determined by comparing their average y-coordinates. The instance with the smaller average y-coordinate is on top, and the instance with the larger average y-coordinate is on the bottom. Based on these relative spatial relationships, the hierarchical relationship is determined. The contained instance is at a lower level than the instance that contains it. For example, if B is contained by A, then B's level is lower than A's. In terms of vertical position, the instance above may be at a higher level than the instance below, or it can be determined based on the actual stacking of food in the scenario. For example, the food in the plate is at a higher level than the plate. In this way, all food instance identifiers are compared pairwise to gradually build the hierarchical relationship between them, and finally generate the food instance hierarchical structure information, such as presenting it in a tree structure, where the root node is the instance with the highest level, and the child nodes are instances with lower levels, etc.

[0137] The unique identifier color value corresponding to each food instance is organized into a table that clearly shows the color corresponding to each identifier. Then, the hierarchical structure information of the food instances is presented in a structured form, such as a hierarchy diagram or hierarchy list, which clarifies the position of each instance in the hierarchy and its hierarchical relationship with other instances. Next, the table of identifiers and color values ​​is associated with the hierarchical structure information so that each instance in the hierarchy can be associated with its unique identifier color value. For example, in the hierarchical structure tree, each node is labeled with the instance identifier corresponding to that node and its unique color value. Finally, the integrated information is checked to ensure that the color value and hierarchical information of each instance are accurate and without omissions or incorrect associations, thereby generating the final image annotation result including unique identifier color values ​​and hierarchical information.

[0138] By identifying and distinguishing different food instances based on multi-dimensional fusion data, multiple characteristics of food can be comprehensively considered, improving the accuracy of identification and avoiding misjudgments caused by the limitations of a single feature. This results in an accurate set of mutually independent food instance identifiers. Each food instance identifier is assigned a globally unique identifier color value according to a preset color value mapping index rule, ensuring that each food instance has a unique color identifier in the image, thus improving the efficiency and convenience of identification. By calculating the relative spatial relationships between food instances to determine the hierarchical relationship, the spatial distribution and hierarchical structure of food instances in the image can be clearly reflected, which helps to understand the positional associations and combinations between foods. By integrating the unique identifier color value with the hierarchical structure information, the final image annotation result contains both the unique color identifier of each food instance and reflects the hierarchical relationship between them, making the annotation result more comprehensive and intuitive.

[0139] like Figure 2 As shown, embodiments of the present invention also provide a food image annotation system based on multi-information fusion, comprising:

[0140] The information acquisition module is used to acquire images of the food to be labeled, and to simultaneously acquire the shooting angle, the distance between the terminal and the food, and the shooting direction through the mobile terminal's sensor.

[0141] The target detection module is used to input food images into an improved mask region convolutional neural network to obtain preliminary target detection results;

[0142] The mask processing module is used to perform graph cut loop processing on the target area of ​​the preliminary target detection results, and combine morphological erosion and dilation to generate binary mask data of the food area;

[0143] The calibration generation module is used to select spatial reference elements based on preset rules on the binary mask data of the food region, divide the reference elements into uniform units, analyze the distribution characteristics of mask pixels in each unit, and generate regional accuracy calibration parameters.

[0144] The extraction and integration module is used to adjust the binary mask data using regional accuracy calibration parameters. It extracts the planar coverage features, centroid coordinates, and contour boundary information of the food area from the adjusted binary mask data. The extracted feature information is then integrated with the shooting angle, the distance between the terminal and the food, and the shooting orientation for multi-dimensional data integration.

[0145] The annotation generation module is used to assign unique color value codes to different food instances based on the integrated multi-dimensional data and according to the preset color value mapping index rules, and generate annotation results including hierarchical information.

[0146] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0147] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0148] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0149] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A food image annotation method based on multi-information fusion, characterized in that, The method includes: Step 1: Acquire an image of the food to be labeled, and simultaneously collect the shooting angle, distance between the terminal and the food, and shooting orientation through the mobile terminal's sensor; Step 2: Input the food image into the improved mask region convolutional neural network to obtain preliminary target detection results; Step 3: Perform graph cut loop processing on the target area of ​​the preliminary target detection results, and combine morphological erosion and dilation to generate binary mask data of the food area; Step 4: On the binary mask data of the food region, spatial reference elements are selected based on preset rules. The reference elements are uniformly divided into units, and the distribution characteristics of mask pixels within each unit are analyzed to generate region accuracy calibration parameters. The analysis of the distribution characteristics of mask pixels within each unit and the generation of region accuracy calibration parameters include: for each divided unit, based on the binary mask data of the food region, counting the number of pixels marked as food within the unit area, and calculating the proportion of this number to the total number of pixels in the unit to obtain the mask pixel proportion value of the unit. The position coordinates of each unit and its corresponding mask pixel proportion value are recorded to form a position-proportion value dataset. By comparing the proportion value differences between adjacent units, continuous high proportion regions and continuous low proportion regions are identified. The system calculates the average and standard deviation of the mask pixel proportions within the entire spatial reference element. If the standard deviation is less than a set threshold, it indicates that the proportions are evenly distributed; if the standard deviation is greater than the set threshold, it indicates that there are differences in the distribution. Based on the location of high-proportion areas, the location of low-proportion areas, the number of consecutive units, and the average and standard deviation, a spatial distribution pattern is determined. The spatial distribution pattern includes a high-proportion-value-at-the-center-low-proportion-value pattern, a local clustering pattern, and a uniform distribution pattern. When the spatial distribution pattern is a high-proportion-value-at-the-center-low-proportion-value pattern, edge units are generated and calibration coefficients are calculated. When the spatial distribution pattern is a local clustering pattern, region weight parameters are generated. When the spatial distribution pattern is a uniform distribution pattern, overall offset parameters are generated. Step 5: Adjust the binary mask data using the regional accuracy calibration parameters, extract the planar coverage features, centroid coordinates and contour boundary information of the food area from the adjusted binary mask data, and integrate the extracted feature information with the shooting angle, the distance between the terminal and the food and the shooting orientation for multi-dimensional data integration. Step 6: Based on the integrated multi-dimensional data, assign a unique color value code to different food instances according to the preset color value mapping index rules, and generate annotation results including hierarchical information.

2. The food image annotation method based on multi-information fusion according to claim 1, characterized in that, The food image is input into an improved masked region convolutional neural network to obtain preliminary object detection results, including: The food image to be labeled is input into the feature extraction network of the improved mask region convolutional neural network to generate multi-scale feature maps. Based on the multi-scale feature maps, candidate food target regions are generated through the region proposal network. The system performs region of interest alignment on the candidate food target regions, extracts region features, and identifies food categories through the classification branch of the neural network based on the region features. The original category scores generated by the branch are transformed into exponential form and normalized by summation to calculate the category confidence. At the same time, the bounding box coordinates of the candidate regions are fine-tuned through the regression branch. By integrating the category confidence score with the bounding box coordinates, preliminary target detection results are generated, including the precise bounding box location of the target, the predicted category, and the corresponding confidence score.

3. The food image annotation method based on multi-information fusion according to claim 2, characterized in that, The target region from the preliminary target detection results undergoes graph cut loop processing, and combined with morphological erosion and dilation, generates binary mask data for the food region, including: The bounding box containing food targets in the preliminary target detection results is set as the initial region for graph cut processing; Starting from the initial region, a graph cut segmentation process is performed, pixel-level calculations are carried out, and finally the segmented region is generated. Morphological erosion is applied to the segmented region, and morphological dilation is applied to the result of the erosion to obtain the morphologically adjusted region. Based on the morphologically adjusted region, binary mask data that accurately represents the spatial distribution of food pixels in the image is generated, with food pixels labeled as a first preset value and non-food pixels labeled as a second preset value.

4. The food image annotation method based on multi-information fusion according to claim 3, characterized in that, On the binary mask data of the food region, spatial reference elements are selected based on preset rules, the reference elements are uniformly divided into units, and the distribution characteristics of mask pixels within each unit are analyzed to generate region accuracy calibration parameters, including: Based on the binary mask data of the food region, at least one spatial reference element is selected from the image region defined by the binary mask data according to the preset spatial location rules; For each selected spatial reference element, a uniform grid is divided into several units of uniform size and shape. For each unit that has been divided, based on the binary mask data of the food region, the number of pixels in the unit region that are marked as food region is counted, and the proportion of the number to the total number of pixels in the unit is calculated to obtain the mask pixel ratio of the unit. Based on the mask pixel ratio of the unit, the distribution characteristics of the mask pixel ratio within the spatial reference element are analyzed to identify the spatial distribution pattern of the mask pixel ratio within the spatial reference element. Based on the spatial distribution pattern, calculate and generate regional accuracy calibration parameters for calibrating the accuracy of the binary mask in the food region.

5. The food image annotation method based on multi-information fusion according to claim 4, characterized in that, The binary mask data is adjusted using region precision calibration parameters. From the adjusted binary mask data, the planar coverage features, centroid coordinates, and contour boundary information of the food area are extracted. The extracted feature information is then integrated with multi-dimensional data such as the shooting angle, the distance between the device and the food, and the shooting orientation, including: Based on the region accuracy calibration parameters, pixel-level adjustment operations are performed on the binary mask data of the food region to obtain calibrated binary mask data; Based on the calibrated binary mask data, the total number of pixels marked as the first preset value is calculated, and combined with the image resolution parameters, it is converted into physical area to generate the planar coverage area feature of the food region. Based on the calibrated binary mask data, the mean spatial position of all pixels marked with the first preset value in the image coordinate system is calculated to generate the geometric centroid coordinates of the food region. Based on the calibrated binary mask data, the connected domain outer boundary of the pixel region marked with the first preset value is detected, and an ordered sequence of contour boundary points is generated. The planar coverage area features, geometric centroid coordinates, and contour boundary point sequence are spatiotemporally correlated and structurally integrated with the synchronously collected shooting angle, distance between the terminal and the food, and shooting orientation information to form multi-dimensional fused data.

6. The food image annotation method based on multi-information fusion according to claim 5, characterized in that, Based on the integrated multi-dimensional data, and according to the preset color value mapping index rules, a unique color value code is assigned to different food instances, generating annotation results including hierarchical information, including: Based on multi-dimensional fusion data, different food instances in an image are identified and distinguished to obtain a set of mutually independent food instance identifiers; For each food instance identifier in the food instance identifier set, a globally unique identifier color value is assigned to the food instance identifier according to the preset color value mapping index rules, so as to obtain the unique identifier color value corresponding to each food instance identifier; Based on the food instance identifier set and the spatial location relationship information of food instances, the hierarchical relationship between each food instance identifier in the food instance identifier set is determined by calculating the relative spatial relationship between each food instance identifier, and the hierarchical structure information of food instances is generated. The unique identifier color value corresponding to each food instance is integrated with the food instance hierarchical structure information to generate the final image annotation result, which includes the unique identifier color value and hierarchical information.

7. The food image annotation method based on multi-information fusion according to claim 6, characterized in that, Based on the set of food instance identifiers and the spatial relationship information of food instances, the hierarchical relationship between each food instance identifier in the set of food instance identifiers is determined by calculating the relative spatial relationship between each food instance identifier, generating food instance hierarchical structure information, including: Based on the geometric centroid coordinates and the sequence of contour boundary points, the three-dimensional Euclidean distance and contour overlap between food instances are calculated to obtain the distance and overlap data of food instance pairs. Using distance and overlap data of food instance pairs, a dataset representing the spatial relationships of all food instance pairs is constructed; Based on the spatial relationship dataset, food instance pairs with overlapping areas are screened, and the vertical projection relationship of the overlapping areas is calculated and analyzed to determine the hierarchical affiliation and obtain the hierarchical affiliation determination result. Based on the hierarchical attribution determination results, food instance identifiers are organized in a tree structure to generate a structured food instance hierarchy tree as the food instance hierarchy structure information.

8. A food image annotation system based on multi-information fusion, wherein the system implements the method as described in any one of claims 1 to 7, characterized in that, include: The information acquisition module is used to acquire images of the food to be labeled, and to simultaneously acquire the shooting angle, the distance between the terminal and the food, and the shooting direction through the mobile terminal's sensor. The target detection module is used to input food images into an improved mask region convolutional neural network to obtain preliminary target detection results; The mask processing module is used to perform graph cut loop processing on the target area of ​​the preliminary target detection results, and combine morphological erosion and dilation to generate binary mask data of the food area; The calibration generation module is used to select spatial reference elements based on preset rules on the binary mask data of the food region, divide the reference elements into uniform units, analyze the distribution characteristics of mask pixels in each unit, and generate regional accuracy calibration parameters. The extraction and integration module is used to adjust the binary mask data using regional accuracy calibration parameters. It extracts the planar coverage features, centroid coordinates, and contour boundary information of the food area from the adjusted binary mask data. The extracted feature information is then integrated with the shooting angle, the distance between the terminal and the food, and the shooting orientation for multi-dimensional data integration. The annotation generation module is used to assign unique color value codes to different food instances based on the integrated multi-dimensional data and according to the preset color value mapping index rules, and generate annotation results including hierarchical information.

9. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images