Food image recognition and calorie recognition method based on convolutional neural network

Abstract

The invention discloses a food image recognition and calorie recognition method based on a convolutional neural network, and belongs to the field of image recognition. A food image lacks a fixed semantic mode, and local detail features of the food image present a multi-scale characteristic. Limited by fixed damage size and receptive field, DCL cannot extract multi-scale discriminative features. The method comprises the following steps: preprocessing image data in a selected data set; training a convolutional neural network by using the fused multi-stage convolutional feature pyramid; an ECA attention module is added, and different feature maps are reweighted to enhance a discriminative visual mode; capturing local cross-channel interaction; extracting features by using a convolutional neural network which is trained and is added with an attention mechanism; carrying out feature fusion; a target detection module and a heat prediction module are added; and performing target detection and calorie calculation on the input to-be-detected food image. The multi-scale feature learning ability of the network is enhanced by using the feature pyramid, and the food identification precision is improved.

Description

Technical Field

[0001] This invention relates to image recognition methods, and particularly to a food image recognition and calorie recognition method based on convolutional neural networks. Background Technology

[0002] In recent years, computer vision methods have become very popular in the field of food recognition. Generally, there are two types of research methods: traditional machine learning methods, including image preprocessing, feature selection, feature extraction, and classification; and deep learning, which has recently become very popular in the field of image recognition.

[0003] Using deep learning for food recognition is challenging because food is deformable and exhibits significant variations in appearance. Convolutional Neural Networks (CNNs) are a state-of-the-art deep learning method, and observations of trained CNNs have revealed that color and texture features play crucial roles in food recognition. Methods exist to develop food image recognition approaches that consider more features. These methods first locate the food region in an image using image segmentation, then extract and fuse the color and texture features of that region for food image recognition. Ravi et al. designed a food recognition platform that extracts three types of features from the image: color, texture (LBP), and HOG. They construct a feature hierarchy based on combinations of different feature types and finally achieve food image recognition using Fisher Vector-based hierarchical features.

[0004] Food identification often requires consideration of its spatial structure. One method proposes a novel food identification approach that utilizes the spatial relationships of semantic texture features between different components (such as meat and bread in a sandwich) to identify food, achieving a TOP-1 accuracy of nearly 80%. Another method argues that co-occurrence relationships are more important than relative positions, and proposes a multi-food image recognition method that considers food co-occurrence statistics for some food combinations. To handle the vertical structure of food, another method introduces sliced ​​convolutional blocks into a residual network to capture this specific information, achieving a TOP-1 accuracy close to 90%.

[0005] To address health issues such as obesity and diabetes, methods based on computer vision have been developed to study the calorie content of different foods. One method utilizes offline algorithms to design a webcam that directly identifies food from restaurant food videos; however, this method only estimates calories based on the appearance of the food and cannot calculate portion sizes. Anthimopoulos et al. proposed an automatic food identification and classification system based on the Bag-of-Features (BoF) model, introducing descriptors to determine food size and providing dietary recommendations for diabetic patients. Ming et al. proposed a diet tracking system covering 60 foods by pre-editing weight photos covering the target food type. This algorithm requires users to select photos of different portion sizes from the same food type to determine food size. Manoj et al. proposed GrabCut, an image segmentation method based on graph cutting optimization. For each bounding box, the GrabCut algorithm is applied to obtain accurate contours to identify different food images, and the calories of each food component are calculated by effectively cropping the area of ​​the food portion.

[0006] Transfer learning in deep learning is a machine learning method that transfers knowledge learned from one task to other related tasks. Its core idea is to leverage knowledge from the source domain to solve problems in the target domain. Yanai et al. created the UECFOOD100 dataset containing 100 food categories and the UEC-FOOD256 dataset containing 256 food categories, and pre-trained the model using large-scale ImageNet data, fine-tuning the pre-trained model to better recognize food images. Pan et al. employed transfer learning, integrating ResNet deep feature sets, Information Gain (IG) feature selection, and an SMO classifier into their model, enabling them to extract rich high-level features from food images, demonstrating its advantages in food component recognition compared to several existing works in this field. Web crawlers can not only automatically collect large amounts of data from the internet, providing the datasets needed for training subsequent deep learning models, but also perform information retrieval. Nupur et al. used transfer learning, employing a pre-trained Start v3 CNN model to stimulate the original custom CNN framework. The predicted food labels generated by the model were forwarded to a web crawler to retrieve the food's origin, nutritional details, recipe, and the restaurant that served the dish.

[0007] Overfitting can occur when the number of samples for certain classes in the training set is small. To address this issue, Pereira et al. devised a novel solution by embedding partially imbalanced domain adaptation techniques (tree-adapted networks) into deep learning models, leveraging the hierarchical structure of food ingredients. Rare classes can avoid overfitting by sharing some parameters with their sibling nodes through their common parent node in the tree. While deeper models offer higher prediction accuracy, they also introduce overfitting and consume significant computational resources. Pan et al. proposed a subnet merging technique to construct a state-of-the-art combined convolutional neural network (CBNet), which helps alleviate the overfitting problem in almost all large networks due to the limited number of training samples.

[0008] Considering that while there are tens of thousands of dish types, they are actually composed of far fewer ingredients, ingredient identification in real-world scenarios is easier to handle. Chaitanya et al. constructed a single-ingredient image dataset, where a single ingredient image indicates that there is only one ingredient in the image. The dataset collects various single-ingredient images for each ingredient category, which are cooked in different ways and have different visual appearances. Using a model based on an ingredient dataset, researchers were able to predict the ingredient list of a given image, even if the model had never seen the recipe corresponding to the image. Zhang et al. proposed a multi-task system based on a deep convolutional neural network (DCNN) that can identify dish type, food ingredients, and cooking methods from food images. Xiao et al. considered leveraging the interrelationships between food categories and ingredients—for example, the ingredient "cherry tomatoes" is more likely to appear in "mushroom salad" than "pork"—and designed a method that simultaneously classifies food and ingredients based on global image features. Ciocca et al., recognizing that existing methods primarily focus on food visual information for recognition, modeled visual information, textual content (e.g., ingredients), and attributes (e.g., dish) together to address extended recipe-oriented problems. Experiments were conducted on three different recognition tasks using the most common convolutional neural network (CNN) architecture: food category, food state, and food category and state. Deep features were compared with some handcrafted features, showing that deep features outperformed handcrafted features.

[0009] Fine-grained images refer to images where the differences between categories are small, but the differences within the same category are large. Given a food image, identifying which restaurant the food belongs to is a challenging task, as it requires distinguishing subtle differences between different categories while dealing with scarce training data. For example, cars with the same brand, model, but different years are considered similar. Zhang et al. seamlessly embedded hierarchical or shared attribute label structures into the framework to effectively learn fine-grained feature representations, significantly improving image retrieval accuracy across different levels of label structures. Ultra-fine granularity naturally leads to rich relationships between object classes. Zhou et al. proposed a novel method using binary iconography (BGL) to mine these rich relationships. Due to its binary structure, it has high computational efficiency in inference. Since many foods with similar appearances and ingredients are classified into different categories in the dataset, to more closely resemble how people classify food, Yang et al. designed a two-layer classifier to classify Korean food. The first layer divides food into four main categories, and the second layer further refines the classification. This hierarchical structure improves recognition accuracy compared to traditional single-layer classifiers. Attention mechanisms play a crucial role in deep learning. In recent years, attention models have been widely used in fields such as object recognition, image caption generation, and image quality assessment. Visual attention in images refers to the process by which the human eye quickly scans the entire image and identifies the target region requiring attention. This target region receives more attentional resources (i.e., weight distribution) to acquire more necessary target details while suppressing other irrelevant information. Adding attention mechanisms to food recognition algorithms can significantly improve their performance.

[0010] Currently, in the field of food recognition, food image recognition and classification still suffer from problems such as low accuracy and poor generalization. To improve the accuracy of food image recognition and classification, it is crucial to fully utilize the global and local detail information of food images. Liang et al. proposed a fine-grained food image recognition model based on a multi-level convolutional feature pyramid. Addressing the characteristic of large scale variations in food images, their model incorporates a feature pyramid structure into each level of the food feature extraction network, improving the model's robustness to target size. Liu et al., considering the irregular and multi-scale characteristics of local detail features in food images, introduced feature pyramids into the food image recognition task and used auxiliary training to optimize the backbone network, aiming to overcome the problem that jigsaw puzzle mechanisms cannot fully extract multi-scale features. Matsuda et al. detected food objects by searching image pyramids in multiple candidate regions of an image. Because there are many types of dishes, even foods of the same type can have very different appearances. To achieve accurate food recognition, they explored methods to mine discriminative regions in food images. Inspired by adversarial erasure strategies, Qiu et al. proposed a novel network architecture where the main network maintains basic accuracy in classifying the input image, an auxiliary network adversarially mines distinguishable food regions, and a region network classifies the mined regions. Then, global and local representations are combined for final prediction, achieving a TOP-1 test accuracy of over 90%.

[0011] In summary, most existing food recognition methods directly use visual features extracted by CNNs for food image recognition without considering the characteristics of the food images themselves. Food images are fine-grained images without fixed semantic patterns. This invention introduces feature pyramids into the food image recognition task and uses auxiliary training to optimize the backbone network, fully extracting and learning the multi-scale and irregular detailed features of food images.

[0012] Food images are non-rigid, lack a uniform spatial layout, and contain a lot of noise. This noise can severely impair the feature learning ability of the network and further exacerbate the confusion between similar categories. This invention also considers introducing a channel attention mechanism into the food recognition framework, which enhances discriminative visual patterns and reduces noise interference by reweighting different feature maps. Summary of the Invention

[0013] The purpose of this invention is to address the lack of fixed semantic patterns in existing food images, whose local details exhibit multi-scale characteristics. Limited by fixed destruction size and receptive field, Discrete Collision Graph (DCL) cannot extract multi-scale discriminative features. Therefore, this invention utilizes feature pyramids to enhance the multi-scale feature learning ability of the network, proposing a food image recognition and calorie identification method based on convolutional neural networks.

[0014] The above objectives are achieved through the following technical solutions:

[0015] A method for food image recognition and calorie recognition based on convolutional neural networks, the method being implemented through the following steps:

[0016] Step 1: Preprocess the image data in the selected dataset using the OpenCV and NumPy framework; the preprocessing includes rotation, width and height settings, horizontal and vertical flipping, scaling ratio and scaling range, and ZCA whitening transformation operations on the image data.

[0017] The datasets are static datasets, which are divided into Chinese cuisine datasets, Western cuisine datasets, and hybrid datasets according to cuisine. The Chinese cuisine dataset is Vireo Food-172, the Western cuisine datasets include ETHZ Food-101, PFID, and UPMC Food-101, and the hybrid datasets include Food11, ISIA Food-200, and ISIAFood-500.

[0018] Step 2: Fuse mid-level features of different depths in the convolutional neural network, and use the fused multi-level convolutional feature pyramid to train the convolutional neural network;

[0019] The mid-level features contain local fine-grained visual patterns at different scales.

[0020] Step 3: Add an ECA attention module to the convolutional neural network to reweight different feature maps to enhance discriminative visual patterns; avoid network dimensionality reduction and capture local cross-channel interactions;

[0021] Step 4: Extract features from the preprocessed image using the trained convolutional neural network; extract features from the preprocessed image using a convolutional neural network with an added attention mechanism; and fuse the features extracted in Step 2 and Step 3 using the ADD feature fusion network.

[0022] Step 5: Add an object detection module and a calorie prediction module to the convolutional neural network; then, perform object detection and calorie calculation on the input food image to be detected; among which,

[0023] The target detection module is implemented through a selective search algorithm to perform target detection on food images;

[0024] The heat prediction module is used to calculate the volume of the food and obtain its heat value by using two sets of images, a side view and a top view of the food target, after obtaining the category and candidate box of the target food.

[0025] Furthermore, the step of training the convolutional neural network using the fused multi-level convolutional feature pyramid described in step two specifically involves:

[0026] The feature pyramid includes bottom-up paths, bridging paths, and top-down paths; the bottom-up paths represent the feedforward computation process of the backbone network; assuming the backbone network has k stages, the feature output from the last convolutional layer of each stage is defined as a mid-layer feature, thus obtaining the feature set C = {c1, c2, ..., c...}. k-1 c k};

[0027] The bridging path is used to compress the dimensionality of the feature maps to facilitate feature fusion. Each connection is a convolutional layer with a 1×1 kernel and a stride of 1. After passing through the bridging path, each feature in C is compressed to the same dimension. The compressed features are defined as S = {s1, s2, ..., s}. k-1 s k};

[0028] The top-down path represents the feature fusion process of the feature pyramid; let P = {p1, ..., p2} k If} is a set of pyramid features after fusion, then p i The calculation formula is as follows:

[0029] p i =S i (U(p i+1 )+s i )

[0030] p k =S k (s k )

[0031] Where U(·) is the upsampling function, used to unify the size of features from different network layers; S is a convolutional layer with a kernel size of 3×3, used to smooth the fused features to prevent redundancy; this process is iterated repeatedly to obtain the final pyramid feature p. fin .

[0032] Furthermore, the step of training the convolutional neural network using the fused multi-level convolutional feature pyramid described in step two is as follows: the convolutional neural network is a deep residual network, abbreviated as ResNet, with H(x) representing the optimal mapping. The stacked nonlinear layers are used to fit another mapping F(x) = H(x) - x, at which point the optimal mapping is represented as H(x) = F(x) + x. Assuming that the residual mapping is easier to optimize than the original mapping, it is easy to calculate the residual to 0 in extreme cases, which is much simpler than approximating another mapping. F(x) + x is represented by adding a skip connection to a feedforward network. The skip connection skips one or more layers to perform a simple identity mapping, and the entire network is trained using SGD and backpropagation.

[0033] The residual network is composed of a convolutional network (CNN). The CNN uses convolutional layers and sampling layers alternately, which is the intuition behind the CNN. That is, one convolutional layer is followed by one sampling layer, and then another convolutional layer is followed by another sampling layer. This process is repeated so that the convolutional layers extract features and then combine them to form more abstract features, and finally form a feature description of the image object. The down-pooling layer, also known as the pooling layer, includes average pooling and max pooling. Max pooling is a non-linear downsampling method. Its value in computer vision is mainly reflected in two aspects: (1) it reduces the computational complexity from the upper hidden layers; (2) these pooling units have translation invariance.

[0034] Furthermore, step three, which involves adding an ECA attention module to the convolutional neural network to reweight different feature maps to enhance discriminative visual patterns, avoid network dimensionality reduction, and capture local cross-channel interactions, specifically includes:

[0035] This is effectively achieved through fast one-dimensional convolution with a kernel size of k, where the kernel size k represents the coverage of local cross-channel interactions, i.e. how many neighbors participate in the attention prediction of a channel.

[0036] The ECA attention module reweights different feature maps to enhance discriminative visual patterns as follows:

[0037] (1) The input feature map is transformed into a vector of [1, 1, c] by global average pooling.

[0038] (2) Calculate the adaptive one-dimensional convolution kernel size kernel_size based on the number of channels in the feature map;

[0039] (3) Use kernel_size in one-dimensional convolution to obtain the weights for each channel of the feature map;

[0040] (4) Multiply the normalized weights and the original input feature map channel by channel to generate the weighted feature map.

[0041] Furthermore, the step four, which involves fusing the features extracted in steps two and three using an ADD feature fusion network, specifically involves:

[0042] Adding the corresponding feature maps together before performing the next convolution operation is equivalent to adding a prior: the feature maps of the corresponding channels are semantically similar, so the corresponding feature maps share a convolution kernel;

[0043] Since the convolution kernel for each output channel is independent, we can consider only the output of a single channel; assuming the two input channels are X1, X2, ..., X... c and Y1, Y2, ..., Y c ;

[0044] The single output channel of add is:

[0045]

[0046] Furthermore, the steps described in step five, which involve adding a target detection module and a calorie prediction module to the convolutional neural network, followed by target detection and calorie calculation on the input food image to be detected, specifically involve:

[0047] First, the ECUSTED dataset was defined, in which each image contains a corresponding food category label, volume, mass record, and specific calibration reference.

[0048] Next, identify all food objects of interest in the image, determine the category and location of the food objects, use image segmentation and region growing techniques to determine object candidate boxes, and build a model using the selection search method to obtain the food category and candidate boxes;

[0049] Next, input two sets of images: a side view and a top view of the food, and calculate the volume of the food. A one-yuan coin is used as a reference object, and this reference object is always positioned in the lower left corner of all input images. A metric, pixels_per_metric, is defined to measure the number of pixels in the food image; the specific definition is shown below:

[0050] pixels_per_metric=object_width / know_width

[0051] Here, `object_width` represents the width of the reference target in pixels, i.e., how many pixels the reference target occupies in the image; `know_width` represents the width of the reference object, i.e., the actual size of the reference target; calculate the size of the food in the image; that is, in actual measurement, the volume of the food is deduced from the pixels; using the two parameters of the food volume and mass in the dataset, the density of this food can be calculated; using the obtained food volume and food density, the calories contained in the food can be easily calculated.

[0052] The beneficial effects of this invention are as follows:

[0053] This invention uses feature pyramids to enhance the multi-scale feature learning ability of networks, specifically:

[0054] 1. Unlike conventional fine-grained recognition tasks, food images lack fixed semantic patterns, and their discriminative detail features exhibit multi-scale and irregular characteristics. Existing fine-grained recognition methods treat local detail features as fixed semantics, extracting these semantic features for recognition. However, food images lack fixed semantic patterns, making it difficult to extract common semantic features across different foods. This invention introduces feature pyramids into food image recognition tasks and uses auxiliary training to optimize the backbone network, better adapting to the multi-scale and irregular characteristics of discriminative detail features in food images.

[0055] 2. Food images are non-rigid, lacking a uniform spatial layout and containing significant noise. This noise severely impairs the network's feature learning ability and further exacerbates confusion between similar categories. Different visual patterns in food images exist in different channels of the feature map; some channels contain discriminative visual patterns, while others represent noise. Therefore, this invention also considers introducing a channel attention mechanism into the food recognition framework, reweighting different feature maps to enhance discriminative visual patterns and reduce noise interference.

[0056] 3. Food image recognition is a fundamental and primary step in various food-related applications and has wide applications in many fields. This invention incorporates a selective search algorithm into the model to perform target detection on food images. It also uses reference objects to calculate the volume of food in the image, thereby calculating the calories ingested by the user. Attached Figure Description

[0057] Figure 1 This is a flowchart of the method of the present invention;

[0058] Figure 2 This invention relates to a feature pyramid network structure diagram;

[0059] Figure 3 This is a schematic diagram of the residual network involved in the present invention;

[0060] Figure 4 This is the ECA attention network structure diagram involved in the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific implementation method one:

[0063] This embodiment of the invention provides a food image recognition and calorie recognition method based on a convolutional neural network, such as... Figure 1 As shown, the method is implemented through the following steps:

[0064] Step 1: Preprocess the image data in the selected dataset using the OpenCV and NumPy frameworks to remove all unwanted and distorted details and reduce the image resolution; among these steps...

[0065] The preprocessing includes transformation operations such as rotation, width and height setting, horizontal and vertical flipping, scaling ratio and scaling range, and ZCA whitening of the image data;

[0066] The datasets described are static datasets, categorized by cuisine into Chinese cuisine datasets, Western cuisine datasets, and hybrid datasets. Representative Chinese cuisine datasets include Vireo Food-172, while representative Western cuisine datasets include ETHZ Food-101, PFID, and UPMC Food-101. Hybrid datasets include Food11, ISIA Food-200, and ISIA Food-500.

[0067] Step 2: Fuse mid-level features of different depths in the convolutional neural network, and use the fused multi-level convolutional feature pyramid to train the convolutional neural network to enhance its multi-scale feature learning ability.

[0068] The mid-level features contain local fine-grained visual patterns at different scales.

[0069] Step 3: Add an ECA attention module to the convolutional neural network. By reweighting different feature maps, the discriminative visual patterns are enhanced and noise interference is reduced. Avoid network dimensionality reduction. After global average pooling of channels without dimensionality reduction, the ECA attention module captures local cross-channel interactions in an efficient way by considering each channel and its k neighbors.

[0070] Step 4: Extract features from the preprocessed image using the trained convolutional neural network; extract features from the preprocessed image using a convolutional neural network with an added attention mechanism; and fuse the features extracted in Step 2 and Step 3 using the ADD feature fusion network.

[0071] Step 5: Add an object detection module and a calorie prediction module to the convolutional neural network; then, perform object detection and calorie calculation on the input food image to be detected; among which,

[0072] The target detection module is implemented through a selective search algorithm to perform target detection on food images;

[0073] The heat prediction module is used to calculate the volume of the food and obtain its heat value by using two sets of images, a side view and a top view of the food target, after obtaining the category and candidate box of the target food.

[0074] The method of the present invention can identify foods including bread, dairy products, desserts, eggs, fried foods, meat, noodles, rice, seafood, and soups. Specific Implementation Method Two:

[0076] This embodiment of the food image recognition and calorie recognition method based on convolutional neural networks differs from specific embodiment one in that the step two, which involves training the convolutional neural network using a fused multi-level convolutional feature pyramid, specifically involves:

[0077] The feature pyramid includes bottom-up paths, bridging paths, and top-down paths; the bottom-up paths represent the feedforward computation process of the backbone network; assuming the backbone network has k stages, the feature output from the last convolutional layer of each stage is defined as a mid-layer feature, thus obtaining the feature set C = {c1, c2, ..., c...}. k-1 c k};

[0078] The bridging path is used to compress the dimensionality of the feature maps to facilitate feature fusion. Each connection is a convolutional layer with a 1×1 kernel and a stride of 1. After passing through the bridging path, each feature in C is compressed to the same dimension. The compressed features are defined as S = {s1, s2, ..., s}. k-1 s k};

[0079] The top-down path represents the feature fusion process of the feature pyramid; let P = {p1, ..., p2} k If} is a set of pyramid features after fusion, then p i The calculation formula is as follows:

[0080] p i =S i (U(p i+1 )+s i )

[0081] p k =S k (s k )

[0082] Where U(·) is the upsampling function, used to unify the size of features from different network layers; S is a convolutional layer with a kernel size of 3×3, used to smooth the fused features to prevent redundancy; this process is iterated repeatedly to obtain the final pyramid feature p. fin ; Figure 2 This is a diagram of the feature pyramid network structure. Specific implementation method three:

[0084] This embodiment of the food image recognition and calorie recognition method based on convolutional neural networks differs from Specific Embodiment Two in that the step two, which involves training the convolutional neural network using a fused multi-level convolutional feature pyramid, specifically uses a Deep Residual Network (ResNet). ResNet is a deep learning network proposed by Kainming. As the network deepens, it acquires more information and richer features. However, increasing the network depth leads to poorer optimization, and the accuracy of test data decreases, mainly due to gradient explosion or vanishing gradients. To address this issue, methods such as weight regularization, image preprocessing, or stochastic gradient descent are used to help the network converge. ResNet addresses this issue through residuals and skip connections. Formally, H(x) represents the optimal mapping, and stacked nonlinear layers fit another mapping F(x) = H(x) - x. The optimal mapping is then expressed as H(x) = F(x) + x. Assuming the residual mapping is easier to optimize than the original mapping, in extreme cases, it's easy to calculate the residual to zero, which is much simpler than approximating one mapping to another. The residual network structure is as follows: Figure 3As shown, F(x)+x is represented by adding a skip connection to a feedforward network. The skip connection skips one or more layers to perform a simple identity mapping, which neither adds extra parameters nor increases computational complexity, and the entire network can still be trained using SGD and backpropagation;

[0085] The residual network is composed of a convolutional network (CNN). The CNN uses alternating convolutional layers and sampling layers, which is the intuition behind the CNN. That is, one convolutional layer is followed by one sampling layer, and then another convolutional layer is followed by another sampling layer. This process is repeated so that the convolutional layers extract features and then combine them to form more abstract features, and finally form a feature description of the image object. The down-pooling layer, also known as the pooling layer, generally includes average pooling and max pooling. Max pooling is a non-linear downsampling method. Its value in computer vision is mainly reflected in two aspects: (1) it reduces the computational complexity from the upper hidden layers; (2) these pooling units have translation invariance. Because it enhances the robustness to displacement, it is an efficient sampling method to reduce the data dimensionality. Specific implementation method four:

[0087] This embodiment of the food image recognition and calorie recognition method based on convolutional neural networks differs from specific embodiment three in that step three involves adding an ECA attention module to the convolutional neural network to reweight different feature maps to enhance discriminative visual patterns; avoiding network dimensionality reduction; and capturing local cross-channel interactions. Specifically, the steps described in step three are:

[0088] This is effectively achieved through fast one-dimensional convolution with a kernel size of k, where the kernel size k represents the coverage of local cross-channel interactions, i.e. how many neighbors participate in the attention prediction of a channel.

[0089] The ECA attention module reweights different feature maps to enhance discriminative visual patterns as follows:

[0090] (1) The input feature map is transformed into a vector of [1, 1, c] by global average pooling.

[0091] (2) Calculate the adaptive one-dimensional convolution kernel size kernel_size based on the number of channels in the feature map;

[0092] (3) Use kernel_size in one-dimensional convolution to obtain the weights for each channel of the feature map;

[0093] (4) Multiply the normalized weights and the original input feature map channel by channel to generate the weighted feature map. Specific implementation method five:

[0095] This embodiment of the food image recognition and calorie recognition method based on convolutional neural networks differs from specific embodiment four in that the step four, which involves fusing the features extracted in steps two and three using an ADD feature fusion network, increases the amount of information described by the image features. However, the dimensions describing the image themselves do not increase; only the amount of information in each dimension increases. This is clearly beneficial for the final image classification. Specifically:

[0096] Adding the corresponding feature maps together before performing the next convolution operation is equivalent to adding a priori assumption: the feature maps of the corresponding channels are semantically similar, so the corresponding feature maps share a convolution kernel.

[0097] Since the convolution kernel for each output channel is independent, we can consider only the output of a single channel. Assume the two input channels are X1, X2, ..., X... c and Y1, Y2, ..., Y c ;

[0098] The single output channel of add is:

[0099] Specific implementation method six:

[0101] This embodiment of the food image recognition and calorie recognition method based on convolutional neural networks differs from specific embodiment five in that step five adds a target detection module and a calorie prediction module to the convolutional neural network; subsequently, the steps of target detection and calorie calculation on the input food image to be detected are as follows:

[0102] First, the ECUSTED dataset was defined, in which each image contains a corresponding food category label, volume, mass record, and specific calibration reference.

[0103] Subsequently, object detection aims to identify all food objects of interest in an image, determining their category and location. This is one of the core problems in computer vision. Due to the different appearances, shapes, and poses of various objects, coupled with interference from factors such as lighting and occlusion during imaging, object detection remains one of the most challenging problems in computer vision. This study uses image segmentation and region growing techniques to determine candidate bounding boxes for objects, and then builds a model using a selection search method to obtain the food categories and candidate bounding boxes.

[0104] Next, to measure the volume of the input food image, two sets of images are needed: a side view and a top view of the food. The volume of the food is then calculated. When measuring the food, a reference object is set to determine its actual size in the image. This reference object typically needs to meet two conditions: 1. The dimensions of the reference object need to be known. 2. The required reference object can be easily found in the input image. A one-yuan coin is used as the reference object, and it is ensured that the reference object is always in the lower left corner in all input images. A metric, pixels_per_metric, is defined to measure the number of pixels of the food in the image; the specific definition is as follows:

[0105] pixels_per_metric=object_width / know_width

[0106] Here, `object_width` represents the width of the reference target in pixels, i.e., how many pixels the reference target occupies in the image; `object_width` represents the width of the reference object, i.e., the actual size of the reference target. A one-yuan coin is 30mm. Assuming `object_width` is 150 pixels wide, then `pixels_per_metric` = 150px / 30mm = 5px / mm, meaning there are approximately 5 pixels per millimeter in the image being measured. Using this ratio, the size of food in the image can be calculated; that is, in actual measurement, the volume of food can be deduced from the pixels. Using the volume and mass of food in the dataset, the density of this food can be calculated. Using the obtained food volume and density, the calories contained in the food can be easily calculated.

[0107] The embodiments disclosed herein are preferred embodiments, but are not limited thereto. Those skilled in the art can readily grasp the spirit of the present invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for food image recognition and calorie recognition based on convolutional neural networks, characterized in that: The method is implemented through the following steps: Step 1: Preprocess the image data in the selected dataset using the OpenCV and NumPy frameworks; among them, The preprocessing includes transformation operations such as rotation, width and height setting, horizontal and vertical flipping, scaling ratio and scaling range, and ZCA whitening of the image data; The datasets are static datasets, which are divided into Chinese cuisine datasets, Western cuisine datasets, and hybrid datasets according to cuisine. The Chinese cuisine dataset is Vireo Food-172, the Western cuisine datasets include ETHZFood-101, PFID, and UPMC Food-101, and the hybrid datasets include Food11, ISIA Food-200, and ISIA Food-500. Step 2: Fuse mid-level features of different depths in the convolutional neural network, and use the fused multi-level convolutional feature pyramid to train the convolutional neural network; The mid-level features contain local fine-grained visual patterns at different scales. Step 3: Add an ECA attention module to the convolutional neural network to reweight different feature maps to enhance discriminative visual patterns; avoid network dimensionality reduction and capture local cross-channel interactions; Step 4: Extract features from the preprocessed image using the trained convolutional neural network; extract features from the preprocessed image using a convolutional neural network with an added attention mechanism; and fuse the features extracted in Step 2 and Step 3 using the ADD feature fusion network. Step 5: Add an object detection module and a calorie prediction module to the convolutional neural network; then, perform object detection and calorie calculation on the input food image to be detected; among which, The target detection module is implemented through a selective search algorithm to perform target detection on food images; The heat prediction module is used to calculate the volume of the food and obtain its heat value by using two sets of images, a side view and a top view of the food target, after obtaining the category and candidate box of the target food.

2. The food image recognition and calorie recognition method based on convolutional neural networks according to claim 1, characterized in that: Step two, which describes the process of training the convolutional neural network using the fused multi-level convolutional feature pyramid, specifically involves: The feature pyramid includes bottom-up paths, bridging paths, and top-down paths; the bottom-up paths represent the feedforward computation process of the backbone network; assuming the backbone network has k stages, the feature output from the last convolutional layer of each stage is defined as a mid-layer feature, thus obtaining the feature set C = {c1, c2, ..., c...} k-1 c k }; The bridging path is used to compress the dimension of the feature map to facilitate feature fusion. Each connection is a convolutional layer with a kernel of 1×1 and a stride of 1. Through the bridging path, each feature in C is compressed to the same dimension, and the compressed features are defined as S = {s1, s2, ..., s}. k-1 s k }; The top-down path represents the feature fusion process of the feature pyramid; Let P = {p1, ..., p2} k If} is a set of pyramid features after fusion, then p i The calculation formula is as follows: p i =S i (U(p i+1 )+s i ) p k =S k (s k ) Where U(·) is the upsampling function, used to unify the size of features from different network layers; S is a convolutional layer with a kernel size of 3×3, used to smooth the fused features to prevent redundancy; this process is iterated repeatedly to obtain the final pyramid feature p. fin .

3. A food image recognition and calorie recognition method based on a convolutional neural network according to claim 1 or 2, characterized in that: Step two, which describes training the convolutional neural network using the fused multi-level convolutional feature pyramid, specifically involves using a deep residual network, or ResNet for short, where H(x) represents the optimal mapping. Stacked nonlinear layers are used to fit another mapping F(x) = H(x) - x, resulting in the optimal mapping H(x) = F(x) + x. Assuming the residual mapping is easier to optimize than the original mapping, it's easy to calculate the residual to 0 in extreme cases, which is much simpler than approximating another mapping. F(x) + x is represented by adding a skip connection to a feedforward network. This skip connection skips one or more layers to perform a simple identity mapping, and the entire network is trained using SGD and backpropagation. The residual network is composed of a convolutional network (CNN). The CNN uses convolutional layers and sampling layers alternately, which is the intuition behind the CNN. That is, one convolutional layer is followed by one sampling layer, and then another convolutional layer is followed by another sampling layer. This process is repeated so that the convolutional layers extract features and then combine them to form more abstract features, and finally form a feature description of the image object. The down-pooling layer, also known as the pooling layer, includes average pooling and max pooling. Max pooling is a non-linear downsampling method. Its value in computer vision is mainly reflected in two aspects: (1) it reduces the computational complexity from the upper hidden layers; (2) these pooling units have translation invariance.

4. The food image recognition and calorie recognition method based on convolutional neural networks according to claim 3, characterized in that: Step three, which involves adding an ECA attention module to the convolutional neural network to reweight different feature maps to enhance discriminative visual patterns, avoid network dimensionality reduction, and capture local cross-channel interactions, specifically includes: This is effectively achieved through fast one-dimensional convolution with a kernel size of k, where the kernel size k represents the coverage of local cross-channel interactions, i.e. how many neighbors participate in the attention prediction of a channel. The ECA attention module reweights different feature maps to enhance discriminative visual patterns as follows: (1) The input feature map is transformed into a vector of [1, 1, c] by global average pooling. (2) Calculate the adaptive one-dimensional convolution kernel size kernel_size based on the number of channels in the feature map; (3) Use kernel_size in one-dimensional convolution to obtain the weights for each channel of the feature map; (4) Multiply the normalized weights and the original input feature map channel by channel to generate the weighted feature map.

5. The food image recognition and calorie recognition method based on convolutional neural networks according to claim 4, characterized in that: Step four, which involves fusing the features extracted in steps two and three using an ADD feature fusion network, specifically involves: Adding the corresponding feature maps together before performing the next convolution operation is equivalent to adding a prior: the feature maps of the corresponding channels are semantically similar, so the corresponding feature maps share a convolution kernel; Since the convolution kernel for each output channel is independent, we can consider only the output of a single channel; assuming the two input channels are X1, X2, ..., X... c and Y1, Y2, ..., Y c ; The single output channel of add is:

6. The food image recognition and calorie recognition method based on convolutional neural networks according to claim 5, characterized in that: Step five involves adding a target detection module and a calorie prediction module to the convolutional neural network; subsequently, the steps of performing target detection and calorie calculation on the input food image to be detected are as follows: First, the ECUSTED dataset was defined, in which each image contains a corresponding food category label, volume, mass record, and specific calibration reference. Next, identify all food objects of interest in the image, determine the category and location of the food objects, use image segmentation and region growing techniques to determine object candidate boxes, and build a model using the selection search method to obtain the food category and candidate boxes; Next, input two sets of images: a side view and a top view of the food, and calculate the volume of the food. Using a one-yuan coin as a reference object, and ensuring that the reference object is always located in the lower left corner in all input images, a metric `pixels_per_metric` is defined to measure the number of pixels of food in the image; the specific definition is as follows: pixels_per_metric=object_width / know_width Here, `object_width` represents the width of the reference target in pixels, i.e., how many pixels the reference target occupies in the image; `know_width` represents the width of the reference object, i.e., the actual size of the reference target; calculate the size of the food in the image; that is, in actual measurement, the volume of the food is deduced from the pixels; using the two parameters of the food volume and mass in the dataset, the density of this food can be calculated; using the obtained food volume and food density, the calories contained in the food can be easily calculated.

Images