A Method and Device for Estimating the Monocular Visual Weight of Dishes Based on Multimodal Intelligent Agents
By using multimodal intelligent agents to collaboratively process monocular images and combining 3D shape scoring and natural language interpretation, the problems of large errors, cumbersome operation, and low user trust in traditional food weight estimation methods are solved, achieving high-precision and convenient food weight estimation.
Patent Information
- Application Number
- CN202611335960.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
Traditional methods for estimating the weight of dishes suffer from problems such as large errors, cumbersome operation, lack of interpretability, and low user trust in practical applications. In particular, the estimation accuracy is insufficient for irregularly shaped dishes, and users cannot judge the credibility of the estimation results.
A monocular visual weight estimation method for dishes based on multimodal agents is adopted. By coordinating the visual estimation model and the multimodal agents, the weight of dishes is estimated using a single image. Combined with 3D shape scoring and natural language interpretation, uncertainty quantification and reliability assessment are provided.
It achieves high-precision food weight estimation based on a single image, balancing ease of operation and user trust. The practicality and reliability of the estimation are improved through 3D shape scoring and natural language interpretation.
Smart Images

Figure CN123118540A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of image processing and artificial intelligence technology, and in particular to a method and apparatus for estimating the monocular visual weight of dishes based on multimodal intelligent agents. Background Technology
[0002] Currently, food weight estimation is widely needed in scenarios such as disease treatment, nutritional therapy, and chronic disease management. In recent years, visual weight estimation methods based on image recognition and deep learning technologies have achieved automatic prediction of food weight by analyzing visual features such as contours and areas in food images, providing technical support for diet management and nutritional assessment.
[0003] However, traditional food weight estimation methods still have significant shortcomings in practical applications. First, traditional monocular vision-based weight measurement methods typically assume that the food has a regular geometric shape, estimating its weight by combining contour information with volume formulas. However, the actual shape of food is often highly irregular, leading to significant estimation errors. Second, while multi-view geometry-based weight measurement methods offer higher accuracy, they require users to take photos from multiple angles, making the process cumbersome, resulting in a poor user experience and hindering practical application. Furthermore, traditional deep learning-based monocular weight estimation methods only output a single weight value without providing a reliability assessment, leaving users unable to judge the credibility of the estimate. Simultaneously, the estimation process lacks interpretability, making it difficult for users to understand the rationale behind the estimation and resulting in low trust levels. Summary of the Invention
[0004] This application proposes a method and device for monocular visual weight estimation of dishes based on multimodal intelligent agents. It aims to achieve the weight estimation of dishes based on a single image through the collaboration of a visual estimation model and a multimodal intelligent agent, balancing ease of operation and estimation accuracy. At the same time, it uses three-dimensional shape scoring to specifically correct irregular dishes and provides uncertainty quantification and natural language explanation, enabling users to understand the estimation basis and judge the credibility of the results, thereby improving the practicality of weight estimation and user trust.
[0005] In a first aspect, embodiments of this application provide a method for estimating the monocular visual weight of a dish based on a multimodal intelligent agent, including: Acquire a monocular image of the dish to be tested; The monocular image is input into the visual estimation model to generate an initial weight estimate; the monocular image is input into the multimodal agent to generate a three-dimensional shape score, which includes an irregularity score reflecting the regularity of the contour, a packing density score reflecting the compactness of the packing, and an occlusion score reflecting the degree of occlusion. The corrected weight is generated based on the initial weight estimate and the 3D shape score; Uncertainty quantification results are generated based on the three-dimensional shape score, and natural language explanatory text is generated through a multimodal agent. The natural language explanatory text contains the basis for the estimation of the corrected weight. Output the corrected weight, uncertainty quantification results, and natural language explanation text.
[0006] Secondly, this application provides a monocular visual weight estimation device for dishes based on a multimodal intelligent agent, comprising: The image acquisition unit is used to acquire a monocular image of the dish to be tested; The image processing unit is used to input monocular images into the visual estimation model to generate initial weight estimates; and to input monocular images into a multimodal agent to generate three-dimensional shape scores, wherein the three-dimensional shape scores include irregularity scores reflecting contour regularity, packing density scores reflecting packing tightness, and occlusion scores reflecting occlusion degree. The weight correction unit is used to generate the corrected weight based on the initial weight estimate and the three-dimensional shape score; The result interpretation unit is used to generate uncertainty quantification results based on the three-dimensional shape score, and to generate natural language interpretation text through a multimodal agent. The natural language interpretation text contains the basis for the estimation of the corrected weight. The data output unit is used to output the corrected weight, uncertainty quantification results, and natural language explanation text.
[0007] Thirdly, embodiments of this application provide a server including a processor, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps as described in the first aspect of embodiments of this application.
[0008] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps in the first aspect of embodiments of this application.
[0009] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement some or all of the steps described in the first aspect of embodiments of this application.
[0010] As can be seen, in this embodiment, by collaborating with a visual estimation model and a multimodal agent, food weight estimation based on a single image is achieved. Users only need to take a single photo to obtain the estimation result, balancing ease of operation and estimation accuracy. The multimodal agent performs 3D shape analysis on the food image, generating a 3D shape score including irregularity, packing density, and occlusion degree. Based on this score, the initial weight is specifically corrected, effectively improving the estimation accuracy for irregularly shaped foods. Simultaneously, this application generates uncertainty quantification results based on the 3D shape score, allowing users to intuitively understand the reliability of the estimation results. Furthermore, the multimodal agent generates natural language explanation text containing the estimation basis, enabling users to understand the source and basis of the weight estimation. This solves the problem of traditional methods only outputting a single numerical value, making it impossible for users to judge the reliability of the result, thus improving the practicality and user trust in weight estimation. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart illustrating a method for estimating the monocular visual weight of a dish based on a multimodal intelligent agent, as provided in an embodiment of this application. Figure 2 This is a schematic diagram of the training process for a visual estimation model using geometric perceptual distillation, as provided in an embodiment of this application. Figure 3 This is a schematic diagram of a process for calculating the loss of geometric perception knowledge distillation provided in an embodiment of this application; Figure 4 This is a block diagram of the functional units of a food monocular visual weight estimation device based on a multimodal intelligent agent provided in an embodiment of this application; Figure 5 This is a block diagram of the functional units of another food monocular visual weight estimation device based on a multimodal intelligent agent provided in this application embodiment; Figure 6 This is a structural block diagram of a server provided in an embodiment of this application. Detailed Implementation
[0013] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0014] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0015] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0016] This application provides a method for monocular visual weight estimation of dishes based on a multimodal agent. The following is a detailed description of the embodiments of this application with reference to the accompanying drawings.
[0017] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for estimating the monocular visual weight of a dish based on a multimodal intelligent agent, as provided in an embodiment of this application, including: Step S101: Obtain a monocular image of the dish to be tested.
[0018] In this context, a monocular image refers to a two-dimensional image captured by a single camera from a single viewpoint. This step uses only a single image as input, allowing users to initiate the weight estimation process with just one shot, avoiding the tedious process of shooting from multiple angles.
[0019] Step S102: Input the monocular image into the visual estimation model to generate an initial weight estimate; input the monocular image into the multimodal agent to generate a three-dimensional shape score.
[0020] The three-dimensional shape score includes an irregularity score reflecting the regularity of the contour, a packing density score reflecting the compactness of the packing, and an occlusion score reflecting the degree of occlusion.
[0021] Furthermore, the visual estimation model is a lightweight neural network model trained through geometric perception distillation, used to regress the initial weight estimate from the monocular image; the multimodal agent is an artificial intelligence model containing a visual encoder and an autoregressive language decoder, used to extract visual features from the monocular image and generate structured output; the 3D shape score is a multi-dimensional quantitative representation of the dish's 3D geometric shape, where the irregularity score reflects the degree to which the dish's outline deviates from a regular geometric shape, the packing density score reflects the looseness or compactness of the dish in the container, and the occlusion score reflects the severity of mutual occlusion between different parts of the dish. This step achieves parallel processing of weight estimation and shape analysis by inputting the same monocular image into two different processing models, providing a data foundation for subsequent correction and interpretation generation.
[0022] In one possible embodiment, the multimodal agent is obtained through the following fine-tuning training steps: Obtain a training dataset, which includes sample images of dishes and corresponding shape-aware annotation data; input the sample images into a multimodal agent to obtain predicted shape scores; determine the shape analysis loss based on the difference between the predicted shape scores and the shape-aware annotation data; based on the training dataset, update the network parameters of the multimodal agent according to the gradient of the shape analysis loss, and iteratively execute the above steps until the shape analysis loss converges to a preset condition, and use the converged multimodal agent as the multimodal agent for generating 3D shape scores.
[0023] The shape-aware annotation data includes at least one of the following indicators: sphericity, eccentricity, or concavity / convexity, obtained from 3D scanning of the dish. Sphericity measures how closely the dish's shape approximates a sphere, ranging from 0 to 1; a value closer to 1 indicates a more regular shape. Eccentricity measures the stretching of the dish's shape, characterized by the ratio of the largest to the smallest eigenvalue after calculating the eigenvalues of the covariance matrix using principal component analysis; a larger ratio indicates a more elongated shape. Concavity / convexity measures the degree of surface depression, characterized by the ratio of the dish's actual volume to its convex hull volume; a lower ratio indicates more surface depressions. These indicators provide quantified shape-aware training targets for the multimodal agent, enabling it to learn the ability to recognize 3D shape features from 2D images.
[0024] Among them, shape analysis loss Shape analysis loss is used to measure the degree of deviation between predicted shape scores and shape-aware labeled data. The calculation method is as follows: ; in, For the sample size, This is the sample number, with values 1, 2, 3, ..., N. The predicted shape score output by the multimodal agent. For shape-aware labeled data, To adjust the weight hyperparameter for the shape analysis task, a value of 0.5 is typically used. During the fine-tuning of the multimodal agent, the visual encoder backbone remains frozen, and only the projection layer and language model are updated. The learning rate scheduler employs a cosine annealing strategy. The above training steps are iteratively executed until the shape analysis loss converges to the preset condition, thus obtaining a multimodal agent capable of accurately perceiving 3D shape features from food images.
[0025] For example, during fine-tuning training, the visual encoder backbone remains frozen, and only the projection layer and language model parts are updated. The learning rate scheduler uses a cosine annealing strategy, with a maximum learning rate of 2×10⁻⁶. -4 The minimum learning rate is 1×10. -6 The warm-up rounds consist of two rounds. For rapid deployment scenarios across restaurants or similar cuisines, a low-rank adaptive fine-tuning approach can be used, where the training parameters account for a very small proportion of the total model parameters. For scenarios involving drastic scene migration, a full-parameter fine-tuning approach can be used to reconstruct deep semantic associations.
[0026] Among them, the point cloud or mesh model obtained by 3D scanning of the dish can be used to quantitatively represent the shape of the dish through the following three indicators: sphericity The formula used to measure how closely a dish's shape resembles a sphere is: ; in, The actual volume of the dish. This represents the surface area of the dish. The sphericity value ranges from 0 to 1, with values closer to 1 indicating a more regular shape and values closer to 0 indicating a more irregular shape.
[0027] Eccentricity To measure the stretching of the dish's shape, principal component analysis is performed on the point cloud coordinate set to calculate the eigenvalues of the covariance matrix. Eccentricity is defined as the ratio of the spindle lengths: ; ( The larger the value, the more elongated the shape.
[0028] Concave and convex It is used to measure the degree of indentation on the surface of a dish, and is characterized by calculating the ratio of the volume of the convex bulge to the actual volume of the dish. The calculation formula is as follows: ; in, The convex hull volume of the dot cloud for the dish. This represents the actual volume of the dish. A lower ratio indicates more surface depressions and a more irregular shape.
[0029] Furthermore, during the fine-tuning training of the multimodal agent, the following multiple loss functions are introduced to improve model performance: Uncertainty quantification loss is used to enable the model to output a quantitative evaluation of the reliability of the estimation results. The calculation formula is as follows: ; in, For the sample size, For sample number, The model predicts the first The variance of a sample, The model predicts the first The weight of each sample For the first The true weight of each sample The weight hyperparameter for adjusting the proportion of uncertainty quantification tasks is usually set to 0.2.
[0030] The explanation generation loss is used to optimize the model's ability to generate natural language explanation text, and is calculated using the following formula: ; in, The length of the output sequence. Number the position in the sequence. For the first The word output at each position, For the first All words generated before this position, For the input image, To generate the first image for the model given the generated content and the input image. The probability of each word The weight hyperparameter, which adjusts the proportion of the explanation and generation tasks, is usually set to 1.0.
[0031] The geometrically perceived weight loss is used to assign higher weights to irregularly shaped samples, making the model more focused on the estimation accuracy of irregularly shaped dishes. The calculation formula is as follows: ; in, For the first Irregularity score for each sample This is the irregularity penalty coefficient, used to control the degree to which irregularity amplifies the loss weights; it is usually set to 0.2.
[0032] Geometric perceptual distillation loss is used to constrain the alignment of the visual latent space features of the multimodal agent with the geometric features of the teacher model. The calculation formula is as follows: ; in, The latent space features output by the visual encoder of the multimodal agent. The geometric features extracted for the teacher model include MSE (Mean Squared Error), a statistical indicator used to measure the degree of difference between two values. It is calculated by taking the square of the difference between the predicted value and the true value for each sample, summing the squared differences of all samples, and then taking the average. The smaller the difference, the closer the predicted value is to the true value. The weight hyperparameter for adjusting the proportion of the geometry-aware distillation task is usually set to 0.1.
[0033] The fine-tuning total loss is the weighted sum of the above five losses: ; in, To adjust the weighting hyperparameters for each task, they are usually set to [value]. ; This represents the irregular penalty coefficient.
[0034] As can be seen, in this example, this embodiment constructs shape-aware annotation data including sphericity, eccentricity, and concavity / convexity using 3D scanning technology, and fine-tunes the multimodal agent using shape analysis loss, enabling the multimodal agent to recognize 3D shape features from 2D images. Compared to the untuned general multimodal model, the fine-tuned multimodal agent in this embodiment achieves a significant improvement in shape analysis accuracy, providing more accurate shape scoring input for subsequent shape feature-based weight correction, thereby further improving the accuracy of weight estimation for irregularly shaped dishes.
[0035] Please see Figure 2 , Figure 2 This is a schematic diagram of the training process for a visual estimation model using geometric perceptual distillation, as provided in an embodiment of this application. Figure 2Steps T1 to T5 are shown in the following order: Step T1 involves constructing a training dataset, which includes three-view images (images taken from three different perspectives), depth maps, and true weight values; Step T2 involves training a teacher model, which is a multi-view geometric perception model capable of extracting geometric features by fusing image information from multiple perspectives; Step T3 involves constructing a student model, which is a monocular geometric perception model with a more streamlined structure and faster inference speed; Step T4 involves knowledge distillation training, using the weight estimate output by the teacher model and the intermediate layer geometric features to supervise the training of the student model, which is the core innovation of this application; Step T5 is an optional agent shape analysis fine-tuning training, specifically fine-tuning the visual perception module of the multimodal agent to enable it to recognize three-dimensional shape features from two-dimensional images. Each step is executed sequentially in the direction of the arrows.
[0036] For example, the teacher model can employ a 3D variant of the Transformer architecture, such as a 3D version based on the Swing Transformer, to support multi-view input; the student model can employ a lightweight convolutional neural network, such as EfficientNet, to ensure real-time inference on mobile or edge devices.
[0037] In one possible embodiment, the visual estimation model is obtained through geometrical distillation training, which includes: acquiring a training dataset; training a teacher model based on the training dataset and supervised training according to real weights, enabling the teacher model to learn the mapping relationship between output weights and real weights; inputting multi-view sample images into the teacher model to output the teacher's output weight and teacher's intermediate geometric features; inputting monocular sample images into the student model to output the student's output weight and student's intermediate geometric features; determining the output distillation loss based on the difference between the teacher's output weight and the student's output weight, and determining the feature distillation loss based on the feature difference between the teacher's intermediate geometric features and the student's intermediate geometric features; optimizing the network parameters of the student model based on the weighted sum of the output distillation loss and the feature distillation loss, based on the training dataset, and iteratively executing the above distillation steps until the output distillation loss and the feature distillation loss converge to a preset condition, thereby obtaining the trained student model as the visual estimation model.
[0038] The training dataset includes multi-view sample images of dishes, corresponding monocular sample images, and the actual weight of the dishes. Specifically, it can contain samples from multiple dish categories, such as staple foods, stir-fries, soups, and large pieces of meat. Each category contains thousands to tens of thousands of samples to cover various shapes, including regular, semi-regular, and irregular shapes, enabling the trained model to have good generalization ability. Multi-view sample images refer to images of the same dish taken from multiple different perspectives, providing rich 3D geometric information for the teacher model. Monocular sample images refer to images taken from a single perspective, used to train the student model to estimate weight under conditions with only a single input image. The actual weight is the weight of the dish obtained by weighing with a high-precision electronic scale, serving as the benchmark for supervised training of the teacher model. The teacher model is a multi-view geometric perception model, capable of fusing image features from multiple perspectives and extracting intermediate geometric features. The student model is a lightweight monocular geometric perception model with a more streamlined structure and faster inference speed. By transferring the capabilities of the teacher model to the student model, the student model achieves estimation accuracy close to that of the teacher model while maintaining its lightweight and fast inference advantages.
[0039] The teacher model employs a multi-view geometric perception model. Its input is a four-dimensional tensor formed by stacking three images from different perspectives along the viewpoint dimension. These three perspectives are a top-down view, a frontal view, and a side view. The model has four channels (including three RGB channels and a depth map) and an image size of 224×224 pixels. The intermediate geometric features extracted by the teacher model's multi-view geometric perception network are 1024-dimensional feature vectors. The student model uses a lightweight convolutional neural network. Its input is a single RGB image, and its output is an initial weight estimate.
[0040] In this method, geometrical distillation training simultaneously transfers the output layer knowledge and intermediate geometric feature layer knowledge of the teacher model. The core principle is to optimize the student model by weighting the output distillation loss and the feature distillation loss. The output distillation loss measures the difference between the student's output weight and the teacher's output weight, and its calculation formula is as follows: ; in, The number of training samples. The weight estimate output by the student model. The weight estimate of the teacher model output is calculated by the loss function, which calculates the squared difference between the student output and the teacher output for each sample, sums them over all samples, and then takes the average. The smaller the difference, the closer the student model is to the teacher model in terms of output.
[0041] Furthermore, the feature distillation loss is used to measure the difference between the intermediate geometric features of students and the intermediate geometric features of teachers, and is calculated using the following formula: ; in, For the feature vector dimension, For projection mapping function, Intermediate geometric features extracted for the student model. The intermediate geometric features extracted for the teacher model are denoted by the length of each feature vector.
[0042] The loss function first performs length normalization on the mapped student features and teacher features respectively, and then calculates the mean square error between the two after normalization. This loss forces students to imitate the geometric feature extraction method of the teacher model at the feature level, thereby learning the ability to perceive three-dimensional geometric information from two-dimensional images.
[0043] Please see Figure 3 , Figure 3 This is a schematic diagram of a process for calculating geometric perception knowledge distillation loss according to an embodiment of this application. Figure 3 The system is configured with two processing branches: the left branch takes a three-view image and an optional depth map as input, feeds it into the teacher model (i.e., a multi-view geometric perception model), outputs a weight estimate and intermediate layer geometric features, and calculates the output distillation loss accordingly; the right branch takes a single RGB image (i.e., a color image) as input, feeds it into the student model (i.e., a monocular geometric perception model), outputs a weight estimate and intermediate layer geometric features, and calculates the feature distillation loss accordingly; the output distillation loss and the feature distillation loss are used together to calculate the total loss, which is the sum of the output distillation loss and the feature distillation loss multiplied by a tradeoff factor. The sum of these two values, along with a weighting factor, controls the weight of the feature distillation loss in the total loss. Backpropagation is then performed based on the obtained total loss to update the student model parameters.
[0044] The total loss is the sum of the output distillation loss and the characteristic distillation loss multiplied by the balancing weight hyperparameter λ, i.e. ; Where the optimal range of λ is... .
[0045] The network parameters of the student model are updated by minimizing the total loss, enabling the student to approximate the teacher model in both output prediction and feature extraction. This distillation process is repeated iteratively until both losses converge to preset conditions, resulting in a lightweight, trained student model used as a visual estimation model to generate initial weight estimates from monocular images during the actual inference phase.
[0046] As can be seen, in this example, this embodiment transfers the geometric perception capability of the multi-view teacher model to the monocular student model through geometric perception distillation training. This allows the student model to obtain an estimation accuracy close to that of the teacher model with only a single image input. This solves the problem of insufficient estimation accuracy caused by the lack of geometric information in traditional monocular methods, while maintaining the computational efficiency advantage of the lightweight model and balancing accuracy and real-time performance.
[0047] In one possible embodiment, the feature distillation loss is determined based on the feature difference between the teacher's intermediate geometric features and the student's intermediate geometric features, including: performing global pooling on the student's intermediate geometric features to obtain pooled student features; inputting the pooled student features into a projection mapping layer for dimensional transformation to obtain aligned student features with the same dimensions as the teacher's intermediate geometric features; normalizing the vector lengths of the aligned student features and the teacher's intermediate geometric features respectively, so that the length of each feature vector is 1; calculating the mean square error between the normalized aligned student features and the normalized teacher's intermediate geometric features, and using the mean square error as the feature distillation loss.
[0048] Global pooling refers to the spatial compression of student intermediate geometric features, aiming to convert feature maps containing spatial location information into fixed-length feature vectors for subsequent dimensionality transformation and feature alignment. The projection mapping layer is a network layer that performs dimensionality transformation on the pooled student features. It typically employs a multilayer perceptron structure, including linear transformations and non-linear activation functions, to map the dimensions of student intermediate geometric features to the same dimensional space as the teacher's intermediate geometric features, thus achieving feature dimensionality consistency. Specifically, the projection mapping layer can use a two-layer fully connected network structure: the first layer maps the pooled student features from 1280 dimensions to 1024 dimensions, and the second layer maps from 1024 dimensions to 1024 dimensions, connected by a non-linear activation function. The intermediate geometric features of the student model can be extracted from the last convolutional block of its backbone network, while the intermediate geometric features of the teacher model can be extracted from the last transformer block of its backbone network. Vector length normalization involves dividing the feature vector by its own length to make the processed feature vector have a length of 1. The purpose of normalization is to eliminate the absolute difference in the numerical scale of features between the teacher model and the student model, making the distillation process focus more on the direction and spatial distribution of geometric features rather than the absolute value. Mean squared error is the average of the squares of the element-wise differences between two feature vectors, used to quantify the degree of difference between the aligned student features and the teacher's geometric features.
[0049] The specific calculation method for feature distillation loss is as follows: For the intermediate geometric features extracted by the student model, the spatial dimension is first compressed by global pooling to obtain the pooled feature vector; then the pooled features are input into the projection mapping layer for dimensional transformation to obtain aligned student features with the same dimension as the teacher's intermediate geometric features; the aligned student features and the teacher's intermediate geometric features are respectively normalized to make their respective feature vector lengths 1; finally, the mean square error between the two normalized feature vectors is calculated as the feature distillation loss.
[0050] As can be seen, in this example, this embodiment uses a combination of global pooling, projection mapping, length normalization and mean squared error calculation to gradually align the intermediate geometric features of the student model to the geometric feature space of the teacher model. This enables the feature distillation process to effectively transfer the geometric perception ability of the teacher model, solves the alignment difficulties caused by the inconsistency of feature dimensions and scale differences between the teacher and student models during the distillation process, and improves the effect and stability of distillation training.
[0051] In one possible embodiment, the multimodal agent includes a visual encoder and an autoregressive language decoder; inputting a monocular image into the multimodal agent to generate a three-dimensional shape score includes: extracting features from the monocular image using the visual encoder to obtain image features; inputting the image features into the autoregressive language decoder according to a preset prompt word template, driving the autoregressive language decoder to perform autoregressive inference output, and obtaining an irregularity score, a packing density score, and an occlusion score.
[0052] The visual encoder is a neural network module used to extract visual feature representations from images, converting raw pixel inputs into high-dimensional semantic feature vectors. The autoregressive language decoder is a language model module used to generate output sequences word-by-word based on visual features. Its autoregressive characteristic means that each word generation is conditional on previously generated content, progressively completing the generation of the complete output sequence. The cue word template is a predefined text instruction used to guide the autoregressive language decoder to reason according to the desired output format and content. The 3D shape score is a quantitative evaluation result of the dish's 3D geometric shape by the multimodal agent, including three independent scores: irregularity score to measure the degree to which the dish's outline deviates from a regular geometric shape, packing density score to measure the looseness or compactness of the dish in the container, and occlusion score to measure the severity of mutual occlusion between different parts of the dish. The collaborative mechanism between the visual encoder and the autoregressive language decoder is as follows: after the visual encoder converts the image into visual features, the autoregressive language decoder uses these features and the cue word template as conditions to progressively generate a text description of the 3D shape score, thereby achieving a mapping from 2D images to 3D shape perception.
[0053] The process of generating 3D shape scores using a multimodal intelligent agent can be formally described as follows: Given an input image, the visual encoder encodes the image into image features; the autoregressive language decoder receives a pre-defined prompt word template and image features as input, and generates each word in the output sequence sequentially through an autoregressive approach. That is, each time a word is generated, the previously generated content is used as contextual conditions, and the probability distribution of the next word is predicted jointly by the image features and the prompt word template, until a complete output sequence is generated. The generated output sequence contains three scoring information items. The autoregressive language decoder generates these items according to the output constraints agreed upon in the prompt word template, ensuring that the output text can be parsed into a structured data format. Taking the example prompt word in the prompt word template, the prompt word requires the autoregressive language decoder to evaluate the dish's contour irregularity, packing density, and occlusion, and output scores ranging from 0 to 10 respectively. This inference process utilizes the visual and language joint understanding capabilities obtained by the autoregressive language decoder through pre-training on large-scale data, enabling it to generate 3D shape evaluation results that conform to human cognition based on image features.
[0054] As can be seen, in this example, this embodiment achieves end-to-end generation from monocular images to 3D shape scores through the collaborative cooperation of a visual encoder and an autoregressive language decoder. It does not rely on traditional computer vision tools for explicit shape analysis or geometric calculations. It can directly obtain 3D shape scores in continuous numerical form by utilizing the visual understanding capabilities of a multimodal intelligent agent, providing a quantitative basis for subsequent weight correction, thereby improving the efficiency and accuracy of shape analysis.
[0055] In one possible embodiment, the autoregressive language decoder outputs data in a structured text markup format (JavaScript Object Notation, JSON), which includes shape analysis fields and explanation fields. Based on a preset cue word template, image features are input into the autoregressive language decoder to drive autoregressive inference and output irregularity scores, packing density scores, and occlusion scores. This includes: obtaining a preset cue word template; filling the cue word template with image features and initial weight estimates to generate structured cue words; inputting the structured cue words into the autoregressive language decoder to drive autoregressive inference according to the output constraints of the text markup format to obtain structured text markup data; parsing the irregularity score, packing density score, and occlusion score from the structured text markup data, and parsing natural language explanation text from the explanation fields.
[0056] The prompt word template defines output constraints for the text markup format. Structured text markup format refers to the text format where the autoregressive language decoder output is arranged according to a predefined key-value structure. This format offers good machine readability and structural consistency. The shape analysis field carries the analysis results of the multimodal agent on the three-dimensional shape of the dish, including three specific values: irregularity score, packing density score, and occlusion score. The explanation field carries natural language explanation text for the weight estimation results. This text is intended for ordinary users and includes the basis for the corrected weight estimation, enabling users to understand the source and rationale behind the weight estimation. The output constraints for the text markup format defined in the prompt word template require that the output of the autoregressive language decoder must meet the preset structural specifications, ensuring that the output results can be correctly parsed by the subsequent parsing module.
[0057] In its implementation, the prompt word template comprises two parts: task instructions and output format constraints. The task instructions inform the autoregressive language decoder of the task type to be performed, while the output format constraints require the output results to be organized according to a predefined structure. For example, in a practical application, a preset prompt word template might include the following instructions: informing the decoder that the current scenario is food weighing, providing input image features and an initial weight estimate, and requiring the output of shape analysis and explanatory text in a specified format. During inference, the system fills the corresponding positions in the prompt word template with the image features extracted by the visual encoder and the initial weight estimate generated by the visual estimation model, forming a structured prompt word containing specific image features and numerical information. After receiving this structured prompt word, the autoregressive language decoder performs word-by-word inference according to the output format constraints defined in the prompt word, thus ensuring the output results maintain structural consistency and standardization. The parsing process involves matching and extracting the generated text-marked format data according to predefined field names, parsing three scores from the shape analysis field and natural language explanatory text from the explanation field for use in subsequent weight correction and result output stages.
[0058] As can be seen, in this example, this embodiment defines strict output format constraints through preset prompt word templates, driving the autoregressive language decoder to generate structured text tag format data. This enables the output of the multimodal agent to be automatically parsed and used in a structured manner, solving the technical problem of inconsistent output formats of Large Language Models (LLMs) and their difficulty in being directly used in subsequent calculation stages. At the same time, by incorporating the interpretation field and the shape analysis field into the same structured output, the natural language interpretation of the estimation basis and the shape score are collaboratively generated, providing complete interpretable information for subsequent output stages.
[0059] Step S103: Generate the corrected weight based on the initial weight estimate and the three-dimensional shape score.
[0060] Here, the corrected weight refers to the final weight estimate obtained after compensating and adjusting the initial weight estimate. This step uses a three-dimensional shape score to specifically compensate the initial weight estimate, ensuring that the final estimate reflects the actual three-dimensional shape characteristics of the dish. This solves the problem of traditional methods having large estimation errors for irregularly shaped dishes due to a lack of geometric information.
[0061] In one possible embodiment, generating a corrected weight based on an initial weight estimate and a three-dimensional shape score includes: determining a shape compensation factor based on an irregularity score in the three-dimensional shape score, and determining a density penalty coefficient based on a packing density score in the three-dimensional shape score; adjusting the initial weight estimate for loss based on the density penalty coefficient to obtain a loss-adjusted weight; compensating the loss-adjusted weight based on the shape compensation factor, and using the compensated weight as the corrected weight.
[0062] The shape compensation factor is a numerical coefficient determined based on the irregularity score, used to compensate for the shape of the initial weight estimate. The density penalty coefficient is a numerical coefficient determined based on the bulk density score, used to adjust the density of the initial weight estimate. The adjustment refers to the fact that when the bulk density of the dish is low, it indicates a looser structure with more voids in the volume, and the actual weight should be appropriately reduced compared to a denser state. Therefore, the initial weight estimate is adjusted using the density penalty coefficient. The compensation process refers to the fact that when the irregularity of the dish is high, area estimation based on a two-dimensional image often underestimates the actual three-dimensional volume of the dish. Therefore, the shape compensation factor is used to compensate for the weight after adjustment, making the estimated result closer to the true weight.
[0063] The corrected weight is generated based on the following correction formula: ; in, For the corrected weight, This is the initial weight estimate. The density and looseness penalty coefficient was selected experimentally. , Standard compact density (default is 10). The stacking density score output for the multimodal agent (with a value ranging from 0 to 10). This is the shape compensation factor determined based on the irregularity score. The formula indicates that the corrected weight equals the initial weight estimate multiplied by the density loss factor, and then multiplied by the shape compensation factor. In other words, the initial weight is first adjusted for loss based on the bulk density score, and then the weight after loss adjustment is compensated for shape based on the irregularity score.
[0064] Among them, density loss coefficient Used to reflect the effect of bulk density on weight: when predicting bulk density Equal to standard compact density When the predicted bulk density is lower than the standard compact density, the loss factor is less than 1, and the initial weight is adjusted downwards. When the predicted bulk density is higher than the standard compact density, the loss factor is greater than 1, and the initial weight is adjusted upwards. However, the bulk density usually does not exceed the standard compact density, so the loss factor mainly plays a downward adjustment role. Loose density penalty factor. The extent to which density differences affect the degree of damage. The higher the value, the greater the loss caused by the same density difference.
[0065] As can be seen, in this example, this embodiment uses irregularity score and bulk density score, combined with correction formula to determine density loss coefficient and shape compensation factor respectively, and performs loss adjustment and compensation processing on the initial weight estimate in sequence, so that the corrected weight can reflect the actual three-dimensional geometric shape and looseness of the dish. In particular, for irregularly shaped dishes, the shape compensation factor effectively makes up for the underestimation deviation of the three-dimensional volume in the two-dimensional image estimation, and significantly improves the weight estimation accuracy of irregularly shaped dishes.
[0066] In one possible embodiment, determining the shape compensation factor based on the irregularity score includes: obtaining a preset regularity threshold and a preset base compensation value; comparing the irregularity score with the regularity threshold; when the irregularity score is not greater than the regularity threshold, using the base compensation value as the shape compensation factor; when the irregularity score is greater than the regularity threshold, calculating the shape compensation factor incrementally on the base compensation value in a manner that is positively correlated with and continuously changes the magnitude of the irregularity score exceeding the regularity threshold.
[0067] The regularity threshold is a preset critical value used to determine whether the shape of a dish is regular. When the irregularity score is less than or equal to this threshold, the dish is considered to have a regular or basically regular shape, and no shape compensation is required. The base compensation value is the baseline value of the shape compensation factor used when the dish has a regular shape; in this case, no additional weight adjustment is made after the weight loss. The incremental calculation refers to gradually increasing the compensation amount based on the base compensation value according to the extent to which the irregularity score exceeds the regularity threshold. The greater the extent to which it exceeds the threshold, the greater the compensation amount. Positive correlation and continuous change are important design features of this embodiment, meaning that the shape compensation factor monotonically increases with the increase of the irregularity score, and there will be no numerical jumps due to threshold switching, ensuring a smooth transition of correction results under different scores.
[0068] Among them, shape compensation factor Based on the irregularity score, to prevent over-correction under regular patterns, this factor employs a piecewise linear compensation function: ; in, The score represents the irregularity (ranging from 0 to 10), with 5 being a preset threshold for regularity. This is an adjustment factor, taken as a positive value, calculated based on the scale of the reference object; a value generally chosen based on experience is used. This is used to control the rate at which the compensation level increases as the irregularity exceeds a threshold. The formula above indicates that when the irregularity score is no greater than 5, the dish is judged to be of a regular or basically regular shape, and the shape compensation factor takes the base compensation value of 1. In this case, no additional compensation is made for the weight after adjustment for losses. When the irregularity score is greater than 5, the dish is judged to be of an irregular shape. The shape compensation factor, based on the base compensation value of 1, increases by multiplying the amount exceeding the regular shape threshold by an adjustment coefficient. That is, for every 1 point of irregularity exceeding the threshold, the shape compensation factor increases. In practical applications, the adjustment coefficient This can be calculated by combining the proportion of a reference object. For example, an empirical value of 0.1 means that when the irregularity score is 8, the deviation from the threshold is 3. The shape compensation factor is 1 plus 0.1 multiplied by 3, which equals 1.3. This indicates that for this irregularly shaped dish, due to the underestimation of the three-dimensional volume by the two-dimensional image estimation, the weight needs to be compensated by about 30%. The positive correlation and continuous variation of this piecewise linear function ensures a smooth transition of the correction amplitude between different irregularity scores, avoiding abrupt changes in the correction results caused by hard threshold switching.
[0069] As can be seen, in this example, this embodiment uses a preset regular shape threshold and a basic compensation value, and a piecewise linear compensation function to determine the shape compensation factor. This makes the shape compensation factor stable under regular shapes and continuously increases with the irregularity score under irregular shapes. This achieves adaptive weight compensation based on the regularity of the shape, which avoids over-correction of regular shaped dishes and ensures a smooth transition of the correction amplitude of irregular shaped dishes under different degrees of irregularity, thus improving the rationality and stability of weight correction.
[0070] Step S104: Generate uncertainty quantification results based on the three-dimensional shape score, and generate natural language explanation text through a multimodal agent.
[0071] The natural language explanation text includes the basis for the corrected weight estimation; the uncertainty quantification result is a quantitative evaluation of the reliability of the corrected weight, including weight range estimation and confidence score, used to allow users to intuitively understand the credibility of the estimation results; the natural language explanation text is readable text described in natural language, used to help users understand the source and basis of the weight estimation. This step, by simultaneously evaluating the reliability and interpreting the understandability of the estimation results, solves the problems of traditional methods that only output a single numerical value, making it difficult for users to judge the reliability of the results and understand the estimation basis.
[0072] In one possible embodiment, the natural language explanation text is generated through the following steps: obtaining the thought chain reasoning results of the multimodal agent in the process of generating the corrected weight; extracting shape analysis findings and correction reasons from the thought chain reasoning results; and combining the shape analysis findings, correction reasons, and weight range estimation into natural language explanation text according to a preset explanation template.
[0073] The reasoning chain result refers to the sequence of intermediate reasoning steps generated by the multimodal agent during the reasoning process of generating the corrected weight. This sequence records the complete reasoning chain from initial image analysis to the final correction decision, including the multimodal agent's findings on the shape of the dish, the reasons for choosing the correction strategy, and the basis for determining the correction magnitude. Shape analysis findings refer to the multimodal agent's observations of the dish's three-dimensional geometric shape, including conclusions regarding the regularity of the contours, the density of stacking, and occlusion. Correction reasons refer to the reasoning basis for the multimodal agent's decision to adopt a specific correction strategy, such as why the weight is increased or decreased, or why a specific correction magnitude is chosen. The explanation template is a pre-defined natural language text framework used to transform structured reasoning information into coherent explanatory text for ordinary users.
[0074] The generation process of the natural language explanation text utilizes the thought chain information generated by the multimodal agent during reasoning. In generating the corrected weight, the multimodal agent does not directly output the final correction result, but first performs step-by-step reasoning, forming a thought chain containing multiple reasoning steps. This thought chain records the complete reasoning trajectory of the multimodal agent, starting from the input image, through a series of intermediate steps such as shape analysis, strategy selection, and correction amount determination, ultimately obtaining the corrected weight. The natural language explanation text is generated as follows: Findings related to shape analysis and reasons related to the correction decision are extracted from the thought chain reasoning results to form structured explanation elements; then, according to a preset explanation template, the extracted shape analysis findings, correction reasons, and weight range estimates from the uncertainty quantification results are filled into the corresponding positions in the template, combining them into a natural language explanation text for ordinary users that includes the estimation basis. Taking a practical application scenario as an example, for a bowl of rice, the thought chain might record the following reasoning process: It is recognized that the rice is densely packed in the bowl, with a standard hemispherical shape and no obstructions, therefore, it is determined that no significant correction is needed. Shape analysis extracted from this thought process revealed it to be "densely packed and regularly shaped," with the correction reason being "regular shape, no correction needed." The weight range was estimated at "plus or minus ten grams." After filling these elements into the explanation template, the generated natural language explanation text was: "Current estimated weight is 150 grams, plus or minus ten grams. Basis: The rice is densely packed in the bowl, with a standard hemispherical shape, no obstructions, and the initial prediction of the visual model is highly reliable, requiring no significant correction." As can be seen, in this example, this embodiment obtains the reasoning results of the multimodal intelligent agent's thought chain, extracts shape analysis findings and correction reasons from them, and generates natural language explanation text according to the preset explanation template combination, so that the weight estimation results are accompanied by understandable estimation basis, which solves the problem that the traditional method only outputs a single value and users cannot understand the source and basis of the estimation, and improves the user's trust and understanding of the estimation results.
[0075] Step S105: Output the corrected weight, uncertainty quantification results, and natural language explanation text.
[0076] The corrected weight, uncertainty quantification results, and natural language explanation text together constitute the complete estimation result output. This step presents the corrected weight estimate, the reliability evaluation of the estimate, and the natural language explanation of the estimate to the user, enabling the user to not only obtain the weight value but also understand the credibility of the value and its source basis, thereby improving the user's understanding and trust in the estimation results.
[0077] The following two typical application scenarios are given to further illustrate the complete workflow of the technical solution of this application.
[0078] Scenario 1: Weight estimation of braised pork. A user uploads a photo of braised pork. The visual estimation model outputs an initial weight estimate of 250 grams. A multimodal agent analyzes the image and outputs a 3D shape score, with an irregularity score of 8, a packing density score of 5, and an occlusion score of 0. After obtaining the initial weight estimate and the 3D shape score, the system generates a corrected weight based on a correction formula. Specifically, based on the packing density score of 5, a density reduction factor of 1 is determined, minus the density penalty factor multiplied by the difference between the standard compact density and the predicted packing density. Based on the irregularity score of 8, a shape compensation factor of 1 is determined, plus an adjustment coefficient multiplied by the extent to which the irregularity score exceeds the regularity threshold. Combining the above correction calculations, the initial weight estimate of 250 grams, after reduction adjustment and compensation, results in a corrected weight of 330 grams. Meanwhile, the system generates uncertainty quantification results based on the three-dimensional shape score, outputting a weight range of 330 grams plus or minus 50 grams with a confidence level of 0.7, and generates natural language explanation text through a multimodal agent: "The initial estimate is based on the average model, but it was observed that the meat pieces are irregular and occupy a large area of the bowl. After conversion based on the 15 cm diameter of the bowl, it was determined that the actual volume was underestimated, so the weight was adjusted upward and a reasonable range was given." Scenario 2: Rice Weight Estimation. The user uploads a photo of rice, and the visual estimation model outputs an initial weight estimate of 180 grams. The multimodal agent analyzes the image and outputs a 3D shape score, with an irregularity score of 2, a packing density score of 8, and an occlusion score of 0. After obtaining the initial weight estimate and the 3D shape score, the system generates a corrected weight based on a correction formula. Since the irregularity score of 2 is no greater than the regularity threshold of 5, the shape compensation factor is set to a base compensation value of 1, and no additional compensation is applied to the weight after adjustment for folded shape. The corrected weight is 180 grams. Simultaneously, the system generates an uncertainty quantification result based on the 3D shape score, outputting a weight range of 180 grams plus or minus 15 grams with a confidence level of 0.95. The multimodal agent then generates a natural language explanation: "The rice has a regular shape and is densely packed; the initial prediction is reliable." As can be seen, in this embodiment, by acquiring a monocular image and inputting it into a visual estimation model, the user only needs to take one photo to obtain an initial weight estimate. Simultaneously, through geometric perception distillation training, the lightweight student model achieves estimation accuracy close to that of a multi-view teacher model, balancing operational convenience and estimation accuracy. Existing monocular methods, lacking geometric information, suffer from significant estimation bias for irregularly shaped dishes, and high-precision methods require multi-angle shooting, making operation cumbersome. This application, by inputting the same monocular image into a multimodal agent, generates a three-dimensional shape score containing three dimensions: irregularity, packing density, and occlusion. Based on this score, the initial weight is adjusted for loss and compensated for shape, ensuring that the corrected weight reflects the actual three-dimensional shape and looseness of the dish. This effectively compensates for the underestimation bias in three-dimensional volume caused by two-dimensional image estimation, significantly improving the weight estimation accuracy for irregularly shaped dishes. Meanwhile, existing methods only output a single value, making it impossible for users to judge whether the result is reliable or understand the basis for the estimation. This application generates uncertainty quantification results based on three-dimensional shape scoring, enabling users to intuitively understand the reliability of the estimation results. Furthermore, it generates natural language explanation text containing the basis for the estimation through a multimodal intelligent agent, allowing users to understand the source and reasoning of the weight estimation. This solves the problem of low trust caused by the "black box" output of traditional methods.
[0079] In summary, this application achieves high-precision, adaptively correctable, interpretable, and reliable food weight estimation by relying on only a single image, and can be widely applied to various scenarios such as canteen self-service checkout, family health management, and food industry metering.
[0080] The following are embodiments of the apparatus of this application. These embodiments of the apparatus and the embodiments of the method of this application belong to the same concept and are used to execute the methods described in the embodiments of this application. For ease of explanation, only the parts related to the apparatus embodiments of this application are shown in the embodiments of this application. For specific technical details not disclosed, please refer to the description of the embodiments of the method of this application, which will not be repeated here.
[0081] This application provides a monocular visual weight estimation device for dishes based on multimodal intelligent agents. Specifically, the monocular visual weight estimation device for dishes based on multimodal intelligent agents is used to execute the steps in the above-described monocular visual weight estimation method for dishes based on multimodal intelligent agents. The monocular visual weight estimation device for dishes based on multimodal intelligent agents in this application may include modules corresponding to the respective steps.
[0082] This application embodiment can divide the food monocular visual weight estimation device based on multimodal intelligent agents into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. The module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.
[0083] When dividing each function into modules according to its corresponding function. Figure 4 This is a functional unit block diagram of a monocular visual weight estimation device for dishes based on a multimodal intelligent agent, provided in an embodiment of this application. The device 40 includes: an image acquisition unit 401 for acquiring a monocular image of the dish to be tested; an image processing unit 402 for inputting the monocular image into a visual estimation model to generate an initial weight estimate; inputting the monocular image into a multimodal intelligent agent to generate a three-dimensional shape score, wherein the three-dimensional shape score includes an irregularity score reflecting contour regularity, a packing density score reflecting packing compactness, and an occlusion score reflecting occlusion degree; a weight correction unit 403 for generating a corrected weight based on the initial weight estimate and the three-dimensional shape score; a result interpretation unit 404 for generating an uncertainty quantification result based on the three-dimensional shape score, and generating a natural language explanation text through the multimodal intelligent agent, the natural language explanation text containing the estimation basis for the corrected weight; and a data output unit 405 for outputting the corrected weight, the uncertainty quantification result, and the natural language explanation text.
[0084] In one possible embodiment, the visual estimation model is obtained through geometrical distillation training. Geometrical distillation training includes: acquiring a training dataset comprising multi-view sample images of the dish, monocular sample images corresponding to the multi-view sample images, and the actual weight of the dish; training a teacher model based on the training dataset and supervised training according to the actual weight, enabling the teacher model to learn the mapping relationship between the output weight and the actual weight; inputting the multi-view sample images into the teacher model to output the teacher's output weight and teacher's intermediate geometric features; inputting the monocular sample images into a student model to output the student's output weight and student's intermediate geometric features; determining the output distillation loss based on the difference between the teacher's output weight and the student's output weight, and determining the feature distillation loss based on the feature difference between the teacher's intermediate geometric features and the student's intermediate geometric features; optimizing the network parameters of the student model based on the weighted sum of the output distillation loss and the feature distillation loss, iteratively executing the above distillation steps until the output distillation loss and the feature distillation loss converge to a preset condition, thereby obtaining the trained student model as the visual estimation model.
[0085] In one possible embodiment, in determining the feature distillation loss based on the feature difference between the teacher's intermediate geometric features and the student's intermediate geometric features, the image processing unit 402 is specifically configured to: perform global pooling on the student's intermediate geometric features to obtain pooled student features; input the pooled student features into the projection mapping layer for dimensional transformation to obtain aligned student features with the same dimension as the teacher's intermediate geometric features; perform vector length normalization on the aligned student features and the teacher's intermediate geometric features respectively, so that the length of each feature vector is 1; calculate the mean square error between the normalized aligned student features and the normalized teacher's intermediate geometric features, and use the mean square error as the feature distillation loss.
[0086] In one possible embodiment, the multimodal agent includes a visual encoder and an autoregressive language decoder; in terms of inputting a monocular image into the multimodal agent to generate a three-dimensional shape score, the image processing unit 402 is specifically used to: extract features from the monocular image through the visual encoder to obtain image features; input the image features into the autoregressive language decoder according to a preset prompt word template, drive the autoregressive language decoder to perform autoregressive inference output, and obtain an irregularity score, a packing density score, and an occlusion score.
[0087] In one possible embodiment, the data output by the autoregressive language decoder is structured text tag format data, which includes shape analysis fields and interpretation fields. Specifically, the image processing unit 402 is used to: obtain a preset prompt word template, which defines output constraints for the text tag format; fill the prompt word template with image features and an initial weight estimate to generate structured prompt words; input the structured prompt words into the autoregressive language decoder to drive it to perform autoregressive inference according to the output constraints of the text tag format, obtaining structured text tag format data; parse the irregularity score, packing density score, and occlusion score from the structured text tag format data, and parse natural language interpretation text from the interpretation fields.
[0088] In one possible embodiment, in generating the corrected weight based on the initial weight estimate and the three-dimensional shape score, the weight correction unit 403 is specifically configured to: determine a shape compensation factor based on the irregularity score in the three-dimensional shape score, and determine a density penalty coefficient based on the packing density score in the three-dimensional shape score; perform a depreciation adjustment on the initial weight estimate based on the density penalty coefficient to obtain a depreciation-adjusted weight; perform compensation processing on the depreciation-adjusted weight based on the shape compensation factor, and use the compensated weight as the corrected weight.
[0089] In one possible embodiment, in determining the shape compensation factor based on the irregularity score, the weight correction unit 403 is specifically configured to: obtain a preset regularity threshold and a preset basic compensation value; compare the irregularity score with the regularity threshold; when the irregularity score is not greater than the regularity threshold, use the basic compensation value as the shape compensation factor; when the irregularity score is greater than the regularity threshold, calculate the shape compensation factor incrementally on the basic compensation value in a manner that is positively correlated with and continuously changes the magnitude of the irregularity score exceeding the regularity threshold.
[0090] When using integrated units, such as Figure 5 As shown, Figure 5 This is a block diagram of the functional units of another food monocular visual weight estimation device based on a multimodal intelligent agent provided in this application embodiment. Figure 5 The food monocular visual weight estimation device 40 based on multimodal intelligent agents includes a processing module 502 and a communication module 501. The processing module 502 controls and manages the actions of the food monocular visual weight estimation device 40, such as the steps of the image acquisition unit 401, image processing unit 402, weight correction unit 403, result interpretation unit 404, and data output unit 405, and / or other processes for executing the techniques described herein. The communication module 501 supports interaction between the food monocular visual weight estimation device and other devices. Figure 5 As shown, the food monocular visual weight estimation device based on multimodal intelligent agents may include a storage module 503, which is used to store the program code and data of the food monocular visual weight estimation device based on multimodal intelligent agents.
[0091] The processing module 502 can be a processor or processing module, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 501 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 503 can be a memory.
[0092] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. The above-mentioned food monocular visual weight estimation device 40 based on multimodal intelligent agents can all perform the above-mentioned... Figure 1 The method shown is a monocular visual weight estimation method for dishes based on multimodal intelligent agents.
[0093] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.
[0094] Figure 6 This is a structural block diagram of a server provided in an embodiment of this application. For example... Figure 6 As shown, server 60 may include one or more of the following components: processor 601, memory 602 coupled to processor 601, wherein memory 602 may store one or more computer programs 603, and the one or more computer programs 603 may be configured to implement the methods described in the above embodiments when executed by one or more processors 601.
[0095] Processor 601 may include one or more processing cores. Processor 601 connects to various parts of the server 60 using various interfaces and lines, and performs various functions and processes data of the server 60 by running or executing instructions, programs, code sets, or instruction sets stored in memory 602, and by calling data stored in memory 602. Optionally, processor 601 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 601 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 601 and may be implemented separately using a communication chip.
[0096] The memory 602 may include random access memory (RAM) or read-only memory (ROM). The memory 602 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 602 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the server 60 during use.
[0097] It is understood that server 60 may include more or fewer structural elements than those shown in the above block diagram, and this is not limited here.
[0098] This application also provides a computer storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements some or all of the steps of any of the methods described in the above method embodiments.
[0099] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.
[0100] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0101] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0103] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.
[0104] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.
[0105] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.
Claims
1. A method for estimating the monocular visual weight of food dishes based on multimodal intelligent agents, characterized in that, include: Acquire a monocular image of the dish to be tested; The monocular image is input into the visual estimation model to generate an initial weight estimate. The monocular image is input into a multimodal agent to generate a three-dimensional shape score, wherein the three-dimensional shape score includes an irregularity score reflecting the regularity of the contour, a packing density score reflecting the compactness of the packing, and an occlusion score reflecting the degree of occlusion. Based on the initial weight estimate and the three-dimensional shape score, a corrected weight is generated; Uncertainty quantification results are generated based on the three-dimensional shape score, and natural language explanation text is generated through the multimodal agent, the natural language explanation text containing the basis for the estimation of the corrected weight; Output the corrected weight, the uncertainty quantification result, and the natural language explanation text.
2. The method according to claim 1, characterized in that, The visual estimation model is obtained through geometric perceptual distillation training, which includes: Obtain a training dataset, which includes multi-view sample images of the dish, monocular sample images corresponding to the multi-view sample images, and the actual weight of the dish. Based on the training dataset, a teacher model is trained under supervision according to the real weight, so that the teacher model learns the mapping relationship between the output weight and the real weight; The multi-view sample image is input into the teacher model to output the teacher output weight and teacher intermediate geometric features; the monocular sample image is input into the student model to output the student output weight and student intermediate geometric features. The output distillation loss is determined based on the difference between the teacher's output weight and the student's output weight, and the feature distillation loss is determined based on the feature difference between the teacher's intermediate geometric feature and the student's intermediate geometric feature. Based on the training dataset, the network parameters of the student model are optimized according to the weighted sum of the output distillation loss and the feature distillation loss. The above distillation steps are iteratively executed until the output distillation loss and the feature distillation loss converge to a preset condition, so as to obtain the trained student model as the visual estimation model.
3. The method according to claim 2, characterized in that, The step of determining the feature distillation loss based on the feature difference between the teacher's intermediate geometric features and the student's intermediate geometric features includes: The student's intermediate geometric features are subjected to global pooling to obtain the pooled student features. The pooled student features are input into the projection mapping layer for dimensional transformation to obtain aligned student features that are consistent with the dimensions of the teacher's intermediate geometric features. The aligned student features and the teacher intermediate geometric features are respectively processed by vector length normalization so that the length of each feature vector is 1; Calculate the mean square error between the normalized aligned student features and the normalized intermediate geometric features of the teachers, and use the mean square error as the feature distillation loss.
4. The method according to any one of claims 1-3, characterized in that, The multimodal agent includes a visual encoder and an autoregressive language decoder; the step of inputting the monocular image into the multimodal agent to generate a 3D shape score includes: The visual encoder is used to extract features from the monocular image to obtain image features; According to the preset prompt word template, the image features are input into the autoregressive language decoder, which drives the autoregressive language decoder to perform autoregressive inference and output the irregularity score, the packing density score and the occlusion score.
5. The method according to claim 4, characterized in that, The autoregressive language decoder outputs structured text tag format data, which includes shape analysis fields and interpretation fields. The image features are input into the autoregressive language decoder according to a preset prompt word template, driving the decoder to perform autoregressive inference and output the irregularity score, the packing density score, and the occlusion score, including: Obtain a preset prompt word template, wherein the prompt word template defines output constraints for text markup format; The image features and the initial weight estimate are filled into the prompt word template to generate structured prompt words; The structured cue words are input into the autoregressive language decoder, which is then driven to perform autoregressive inference according to the output constraints of the text tag format to obtain the structured text tag format data. The irregularity score, the stacking density score, and the occlusion score are parsed from the structured text markup format data, and the natural language explanation text is parsed from the explanation field.
6. The method according to any one of claims 1-3, characterized in that, The step of generating the corrected weight based on the initial weight estimate and the three-dimensional shape score includes: The shape compensation factor is determined based on the irregularity score in the three-dimensional shape score, and the density penalty coefficient is determined based on the packing density score in the three-dimensional shape score. The initial weight estimate is adjusted for loss based on the density penalty coefficient to obtain the weight after loss adjustment. The weight after the folding adjustment is compensated according to the shape compensation factor, and the weight obtained after compensation is used as the corrected weight.
7. The method according to claim 6, characterized in that, The step of determining the shape compensation factor based on the irregularity score includes: Obtain the preset rule shape threshold and the preset basic compensation value; The irregularity score is compared with the regularity threshold. When the irregularity score is not greater than the regularity threshold, the basic compensation value is used as the shape compensation factor; When the irregularity score is greater than the regularity threshold, the shape compensation factor is calculated incrementally on the basic compensation value in a manner that is positively correlated with and continuously changes the magnitude of the irregularity score exceeding the regularity threshold.
8. A monocular visual weight estimation device for dishes based on multimodal intelligent agents, characterized in that, include: The image acquisition unit is used to acquire a monocular image of the dish to be tested; The image processing unit is used to input the monocular image into the visual estimation model and generate an initial weight estimate. The monocular image is input into a multimodal agent to generate a three-dimensional shape score, wherein the three-dimensional shape score includes an irregularity score reflecting the regularity of the contour, a packing density score reflecting the compactness of the packing, and an occlusion score reflecting the degree of occlusion. A weight correction unit is used to generate a corrected weight based on the initial weight estimate and the three-dimensional shape score; The result interpretation unit is used to generate uncertainty quantification results based on the three-dimensional shape score, and to generate natural language interpretation text through the multimodal agent, wherein the natural language interpretation text contains the basis for the estimation of the corrected weight; The data output unit is used to output the corrected weight, the uncertainty quantification result, and the natural language interpretation text.
9. A server, characterized in that, The method includes a processor, a memory, a communication interface, and one or more programs, said programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, A computer program for storing electronic data interchange is provided, wherein the computer program causes a computer to perform the method as described in any one of claims 1-7.
