A method for estimating food volume or weight in a container based on artificial intelligence deep learning and computer vision

Abstract

A method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision mainly involves acquiring color and depth images of the food in the container using an RGB-D sensor, then using a convolutional neural network (CNN) to identify the type of food, mapping the coordinates of the container's center position to the depth image to obtain a set of feature vectors, and then using the feature vectors as input values ​​for multiple multilayer perceptron (MLP) models. The computational weights of each MLP model are determined based on the test accuracy of each MLP model, and then the volume or weight of the food in the container is estimated.

Claims

1. A method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision, comprising the following steps: a. Obtaining RGB color images and depth information images of the food in the container using an RGB-D sensor; b. Distinguishing the type of food in the container from the RGB color images using a convolutional neural network, then obtaining the outermost contour of the container from the distinguished RGB color images and calculating the center position of the container, then mapping the coordinates of the center position to the depth information image, extracting depth values ​​of multiple sampling points from the center of the depth information image outward, and storing the depth values ​​of each sampling point into a set of feature vectors; c. Using the set of feature vectors as input values ​​for multiple MLP multilayer perceptron models with different network architectures after deep learning, and determining the computational weight of each multilayer perceptron model based on the test accuracy of each multilayer perceptron model, and then estimating the volume or weight of the food in the container.

2. The method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision as described in claim 1, wherein, When capturing depth values ​​from multiple sampling points, the depth values ​​of 25 or 30 sampling points are captured by expanding outward along a spiral path from the center of the depth information image.

3. The method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision, as described in claim 1 or 2, wherein, When capturing depth values ​​from multiple sampling points, if a null or zero value appears during the sampling process, it is replaced by a valid depth sampling point within 2 to 8 pixels, or by the average of its surrounding non-zero values.

4. The method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision as described in claim 1, wherein, The system randomly places ingredients into containers according to different quantities, uses an RGB-D sensor to acquire RGB color images and depth information images of the food inside the container, and measures and records its actual volume, weight, or density information to collect actual cases of various types of food. This leads to the establishment of a case database covering ingredients in various containers. The actual cases in the case database are then randomly divided into training, validation, and test sets according to proportions for training, validation, and testing of the convolutional neural network and various multilayer perceptron models. The predicted volume can be converted into weight based on the density information of specific foods.

5. The method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision as described in claim 4, wherein, By repeatedly training and validating the convolutional neural network, once the model achieves the expected accuracy on the training dataset, it is evaluated using test data to confirm its final accuracy. After completing the model construction process, heat maps are randomly plotted to examine the model's attention focus areas and further verify the rationality of its judgment criteria.

6. The method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision as described in claim 4, wherein, Each multilayer perceptron uses the aforementioned feature vector as the input value of the multilayer perceptron model and the corresponding known volume or weight as the output value to perform regression training in order to predict the volume or weight of the food.

7. The method for estimating the volume or weight of food in a container based on artificial intelligence deep learning and computer vision as described in claim 1, wherein, Each multilayer perceptron system calculates the final volume or weight of the food using an average weighted voting formula, which is: where: p represents the number of MLP models selected after ranking them according to their prediction errors in the test dataset; is the weighting value corresponding to the k-th high-accuracy MLP model; and is the prediction result output by the k-th high-accuracy MLP model.

Images