Food ingredient identification method and system based on progressive multi-scale feature aggregation

Abstract

The invention provides a food ingredient identification method and system based on progressive multi-scale feature aggregation, and the method comprises the steps: constructing an ingredient analysis neural network model which comprises a global feature extraction module, a progressive local feature learning module and an ingredient identification classifier; an input food image passes through a global feature extraction module and a progressive local feature learning module in sequence, and multi-scale global semantic features and fine-grained local features are obtained respectively; an end-to-end progressive training strategy is adopted, network layers with different depths are activated in stages, and the difference of multi-scale features is enhanced in combination with KL divergence constraint; and finally, outputting a component identification result through the classifier. According to the invention, the technical problem of insufficient food identification local feature representation is solved.

Description

Technical Field

[0001] This invention relates to the field of food image recognition technology, specifically to a method and system for food component recognition based on progressive multi-scale feature aggregation. Background Technology

[0002] Food computation, as an emerging field, has received widespread attention in recent years for supporting food-related issues such as food selection and healthy eating. Food recognition, a fundamental task in food computation, plays a crucial role not only in meeting human survival needs but also in many health applications, such as nutritional analysis and diet management. Furthermore, food image recognition, as an important branch of fine-grained visual classification, has profound theoretical research significance. Therefore, food recognition has attracted increasing attention in computer vision and related fields.

[0003] Despite the abundance of cooking recipes online, accurately matching food images to corresponding recipes remains an unresolved challenge. At the heart of this challenge lies the accurate identification of food categories and their components. Since the advent of Convolutional Neural Networks (CNNs), numerous studies have focused on identifying food types from images to infer their ingredients and nutritional content. However, a major problem with existing methods is the lack of a dataset that covers a vast range of food types globally. While dishes are diverse, they are actually composed of a relatively small number of ingredients. Approaching the problem from the perspective of ingredient identification not only simplifies the task but also allows for the processing of unseen data, thus enabling the prediction of effective outputs.

[0004] The existing invention patent application document CN116434224A, entitled "A Fine-Grained Image Recognition Method and System," describes a method comprising: acquiring an original image and preprocessing it; performing fine-grained image recognition based on the preprocessed image using a pre-trained fine-grained learning network model; specifically, the fine-grained image recognition using the pre-trained fine-grained learning network model involves: extracting global features from the preprocessed image using a global feature learning network; extracting features at different stages from the preprocessed image using a progressive region learning network and enhancing them through a self-attention mechanism to obtain local features at different scales; fusing the local features and global features at different scales to obtain fused features; and classifying based on the fused features to complete fine-grained image recognition. While the aforementioned prior art proposes a progressive region learning network and global feature fusion scheme, its local feature extraction relies on fixed convolutional kernels, making it difficult to adaptively capture the complex texture changes of food components. Especially in multi-scale feature fusion, the self-attention mechanism only merges contextual information, failing to address the redundancy problem caused by features at different stages potentially focusing on similar regions, thus affecting the discriminative power of fine-grained component recognition.

[0005] The existing invention patent application document CN114049303A, entitled "A Progressive Bone Age Assessment Method Based on Multi-Granularity Feature Fusion," describes a method that includes: constructing a granularity grading module based on random jigsaw puzzles to grade the granularity information contained in the input image from fine to coarse, enabling the network to learn rich local features of various parts of the hand bones; constructing a progressive multi-scale feature fusion module to enable the network to learn not only global features and the most distinguishable local features through multiple iterations, but also to fuse other local features, ultimately learning features containing information of different granularities; and performing backpropagation based on the KL divergence between the output and the label to update parameters, completing four iterations. This prior art employs a random jigsaw puzzle grading strategy, which disrupts the spatial continuity of food structures, such as the complete shape of food ingredients, by segmenting image blocks. Its progressive fusion relies on fixed-level output (step 4.2), failing to consider the semantic correlation between features of different granularities, resulting in a disconnect between local features and global semantics in bone age assessment. When applied to food scenarios, this leads to the loss of topological relationships of key components.

[0006] The existing invention patent application document CN119380059A, entitled "Self-Supervised Clustering Method for Hyperspectral Images Based on Local-Global Dual-Branch Networks," describes a method that includes: constructing 3D pixel blocks; obtaining shallow depth features through a ResNet module; injecting the shallow features into a dual-path network module; calculating the similarity between local features and cluster centers to obtain a local semantic probability distribution, and then obtaining the target distribution; feeding global features into a feedforward neural network to obtain the probability of each pixel and obtain a global semantic probability distribution; and constructing a network loss function through a dual self-supervised mechanism to guide the update of the entire network model. This prior art uses KL divergence for hyperspectral image clustering, but only constrains the similarity between local features and cluster centers, without integrating global features. Its dual-branch network is insufficiently adaptable to RGB food images, and the KL divergence has a single objective, only optimizing feature distribution alignment, failing to address the class imbalance problem in multi-label component recognition.

[0007] The prior art patent application document CN117911795A, entitled "Food Image Recognition Method, Apparatus, Electronic Device, and Computer-Readable Medium," describes a food classification network in the prior art method. This network comprises a food category feature extraction network, a fully connected layer, and a global average pooling layer. The method involves inputting the image of the food to be recognized into the food classification network to obtain a food category recognition information set. This includes: inputting the image of the food to be recognized into the food category feature extraction network to obtain a first global food feature vector; inputting the first global food feature vector into the fully connected layer to obtain a second global food feature vector; and inputting the second global food feature vector into the fully connected layer to obtain a second global food feature vector. The global feature vector of the food is input into the global average pooling layer to obtain the third global feature vector of the food. Progressive stage feature extraction processing is performed on the image of the food to be identified to obtain a set of local stage semantic feature vectors of the food. Feature fusion processing is performed on the third global feature vector and the set of local stage semantic feature vectors of the food to obtain a fused feature vector of the food. Classification prediction is performed on the fused feature vector and the set of local stage semantic feature vectors of the food to obtain first food category information and second food category information. The first food category information and the second food category information are weighted and summed to obtain food category recognition information. The aforementioned existing technology relies on comparison with a pre-stored food feature library, requiring additional collection of similar images to update the feature library, and cannot handle newly added food categories that have not been trained. Its progressive training is only used for stage feature extraction and does not establish a correlation mechanism between feature differences and model generalization ability, leading to failure of component recognition in zero-sample scenarios.

[0008] In summary, existing technologies suffer from insufficient representation of local features in food identification. Summary of the Invention

[0009] The technical problem to be solved by this invention is: how to solve the problem of insufficient local feature representation in food identification in the prior art.

[0010] This invention solves the above-mentioned technical problems by employing the following technical solution: A food component identification method based on progressive multi-scale feature aggregation includes:

[0011] S1. Construct a neural network model for component analysis, including: a global feature extraction module, a progressive local feature learning module, and a component recognition classifier; extract multi-scale global semantic features of food images through the global feature learning module;

[0012] S2. Through the progressive local feature learning module, multi-scale local fine-grained features are gradually extracted, and KL divergence is combined to perform multi-scale feature diversity enhancement operations.

[0013] S3. The global semantic features and local fine-grained features are fused to generate a unified feature representation. The component identification results are output through the classifier for multi-label prediction of food components.

[0014] This invention constructs a multi-task progressive feature aggregation network capable of achieving high-precision component recognition. By fusing multi-granularity visual features through a progressive feature aggregation strategy, this invention significantly improves the fine-grained recognition capability of complex food images; it employs KL divergence constraints to enhance feature diversity, effectively solving the problem of multi-scale feature convergence; and it supports zero-shot component inference, maintaining high component recognition accuracy in cross-dataset testing, providing reliable technical support for intelligent nutritional analysis. The trained network model can output food component recognition results, maintaining high classification accuracy while possessing zero-shot inference capability.

[0015] In a more specific technical solution, in S1, the global feature learning module adopts the ViT architecture, which extracts global features based on a backbone network pre-trained by self-supervised learning.

[0016] Global average pooling (GAP) is applied to the output of the last Transformer layer to obtain the global feature representation.

[0017] In a more specific technical solution, the global feature representation is expressed using the following logic:

[0018] f Glo =GAP(f out )

[0019] In the formula, f out This represents the output feature of the last layer of ViT.

[0020] This invention preserves spatial integrity based on the ViT native block segmentation mechanism and directly extracts global features from image block sequences. Furthermore, an end-to-end feature aggregation module concatenates multi-scale local features (fLoci) and global semantics (fGlo) into a unified representation (formula fing), enabling the model to simultaneously learn the local details and overall compatibility of ingredients. For example, when recognizing "Mapo Tofu," it simultaneously associates the texture of the fermented bean paste with the overall sauce distribution, avoiding semantic gaps caused by manual block segmentation.

[0021] In a more specific technical solution, S2 adopts an end-to-end progressive training strategy, activating network layers of different depths in stages.

[0022] By incorporating KL divergence constraints, the differences between local fine-grained features and global semantic features at multiple scales are enhanced.

[0023] In the progressive local feature learning module, the training process is divided into no less than two steps according to the preset number of steps, and deep Transformer blocks are gradually introduced to learn local features of different granularities from shallow to deep layers.

[0024] In each stage i, local feature vectors are extracted using global max pooling (GMP):

[0025]

[0026] Maximizing the KL divergence between features at different stages allows the neural network model of component analysis to focus on different regions.

[0027] The progressive local feature learning module of this invention adopts a phased activation mechanism, gradually unlocking deep network layers during forward propagation, extracting multi-scale local features by combining global max pooling, and constraining the distribution differences of features at different stages by KL divergence regularization.

[0028] This invention employs an end-to-end joint optimization strategy, using a multi-dimensional loss function to balance component identification loss and feature difference constraint loss, thereby achieving overall updating of model parameters.

[0029] This invention introduces KL divergence diversity constraints to force Transformer blocks of different depths to focus on differentiated regions during progressive training. Specifically, it uses the KL divergence (formula LKL) that maximizes the multi-stage feature distribution as a regularization term, compelling the network to learn complementary local features. For example, shallow layers focus on food textures, while deeper layers capture structural combinations, fundamentally avoiding feature convergence and significantly improving component recognition accuracy.

[0030] This invention enhances feature diversity through KL divergence-driven features, enabling the model to learn component-independent general feature representations. For example, in the third stage of progressive training, when jointly optimizing all network layers, KL constraints force features from different stages to cover a differentiated semantic space, generating features with greater generalization ability.

[0031] In a more specific technical solution, during the KL divergence optimization process of S2, local fine-grained features at different stages are mapped to probability distributions, and feature convergence suppression is performed by maximizing the KL divergence differences between the distributions.

[0032] A multi-loss fusion optimization strategy is adopted to determine the improved multi-label cross-entropy loss function and the KL divergence loss. Based on the KL divergence loss and the improved multi-label cross-entropy loss function, the total loss function is obtained by weighted combination.

[0033] In a more specific technical solution, an improved multi-label cross-entropy loss function is used to perform the component identification task:

[0034]

[0035] In a more specific technical solution, the KL divergence loss is calculated using the following logic:

[0036] L KL (P||Q)=∑ j P(i)·(log(P(i)-logQ(i)).

[0037] In a more specific technical solution, the total loss function is obtained using the following logic:

[0038] L=αL ing +βL KL

[0039] In the formula, α and β are equilibrium parameters.

[0040] This invention designs a multi-loss joint optimization framework, extending the KL divergence to a feature diversity constraint (LKL), and dynamically weighting it with the multi-label classification loss (Ling) (formula L = αLing + βLKL). The improved multi-label loss alleviates label imbalance caused by component co-occurrence by decoupling positive and negative samples.

[0041] In a more specific technical solution, S3 utilizes a feature aggregation module to concatenate local fine-grained features and global semantic features from different stages through a fully connected layer:

[0042] f in g = concat(f Glo ,f U-S+1 ,...,f U )

[0043] In the formula, U is the total number of network stages, and S is the number of progressive training steps.

[0044] In more specific technical solutions, food component recognition systems based on progressive multi-scale feature aggregation include:

[0045] The model building and global feature acquisition module is used to build a neural network model for component analysis, including: a global feature extraction module, a progressive local feature learning module, and a component recognition classifier; through the global feature learning module, multi-scale global semantic features of food images are extracted;

[0046] The progressive local feature learning module is used to progressively extract multi-scale local fine-grained features and combine them with KL divergence to perform multi-scale feature diversity enhancement operations.

[0047] The component recognition classifier is used to fuse global features with local fine-grained features to generate a unified feature representation. The classifier outputs the component recognition results for multi-label prediction of food components. The component recognition classifier is connected to the progressive local feature learning module and the model building and global feature acquisition module.

[0048] The present invention has the following advantages over the prior art:

[0049] This invention constructs a multi-task progressive feature aggregation network capable of achieving high-precision component recognition. By fusing multi-granularity visual features through a progressive feature aggregation strategy, this invention significantly improves the fine-grained recognition capability of complex food images; it employs KL divergence constraints to enhance feature diversity, effectively solving the problem of multi-scale feature convergence; and it supports zero-shot component inference, maintaining high component recognition accuracy in cross-dataset testing, providing reliable technical support for intelligent nutritional analysis. The trained network model can output food component recognition results, maintaining high classification accuracy while possessing zero-shot inference capability.

[0050] The progressive local feature learning module of this invention adopts a phased activation mechanism, gradually unlocking deep network layers during forward propagation, extracting multi-scale local features by combining global max pooling, and constraining the distribution differences of features at different stages by KL divergence regularization.

[0051] This invention employs an end-to-end joint optimization strategy, using a multi-dimensional loss function to balance component identification loss and feature difference constraint loss, thereby achieving overall updating of model parameters.

[0052] This invention solves the technical problem of insufficient local feature representation in food identification in the prior art. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the food component identification method based on progressive multi-scale feature aggregation according to Embodiment 1 of the present invention;

[0054] Figure 2 This is a schematic diagram of the food recognition model in Embodiment 1 of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] like Figure 1 As shown, the food component identification method based on progressive multi-scale feature aggregation provided by this invention includes the following basic steps:

[0058] S1. Construct a neural network model for component analysis, which includes a global feature extraction module, a progressive local feature learning module, and a component recognition classifier; extract multi-scale global semantic features of food images through the global feature learning module;

[0059] The global feature extraction module in this embodiment is based on the Vision Transformer architecture. It captures the overall semantic information of the image through a self-attention mechanism and performs global average pooling on the output of the last layer to obtain the global feature vector.

[0060] In this embodiment, the global feature learning module adopts the Vision Transformer (ViT) architecture, extracts global features based on the backbone network pre-trained by self-supervised learning, and performs global average pooling (GAP) on the output of the last Transformer layer to obtain the global feature representation:

[0061] f Glo =GAP(f out )

[0062] Where fout is the output feature of the last layer of ViT;

[0063] S2. Multi-scale local fine-grained features are gradually extracted through the progressive local feature learning module, and the diversity of features is enhanced by combining KL divergence optimization. An end-to-end progressive training strategy is adopted to activate network layers of different depths in stages, and the difference of multi-scale features is enhanced by combining KL divergence constraints.

[0064] In the progressive local feature learning module of this embodiment, the training process is divided into S steps, and deep Transformer blocks are introduced step by step to learn local features of different granularities from shallow to deep layers.

[0065] In each stage i, local feature vectors are extracted using global max pooling (GMP):

[0066]

[0067] Maximizing the KL divergence between features at different stages forces the model to focus on different regions, enhancing feature diversity; the KL divergence optimization is implemented as follows:

[0068] In this embodiment, local features at different stages are mapped to probability distributions, and feature convergence is suppressed by maximizing the KL divergence difference between the distributions.

[0069] In this embodiment, a multi-loss fusion optimization strategy is adopted; specifically, the component recognition task uses an improved multi-label cross-entropy loss function to solve the label imbalance problem:

[0070]

[0071] Calculate the KL divergence loss:

[0072] L KL (P||Q)=∑ i P(i)·(log(P(i)-logQ(i))

[0073] The total loss function is a weighted combination:

[0074] L=αL ing +βL KL

[0075] Where α and β are equilibrium parameters.

[0076] S3. The global and local features are fused to generate a unified feature representation. The component identification results are output through a classifier for multi-label prediction of food components.

[0077] In this embodiment, the feature aggregation module concatenates local features from different stages with global features through a fully connected layer, as shown in the formula:

[0078] f in g = concat(f Glo ,f U-S+1 ,...,f U )

[0079] Where U is the total number of network stages and S is the number of progressive training steps.

[0080] Example 2

[0081] In this embodiment, a multi-task progressive feature aggregation network model is constructed;

[0082] In the progressive local feature learning module of this embodiment, multi-scale local features are extracted at layers 6, 9, and 12 of the Transformer encoder, and a progressive training strategy is adopted, including three stages of optimization:

[0083] Phase 1: Freeze the last 6 layers and train the first 6 Transformers to extract the underlying texture features;

[0084] Phase 2: Unfreeze layers 7-9 and constrain feature diversity using KL divergence loss;

[0085] Phase 3: Jointly optimize all 12 layers, integrating global and local features;

[0086] In the task classifier of this embodiment, component identification is performed; specifically, the multi-label classifier outputs the probability of the existence of N-dimensional components (N≥100), and the LogSumExp loss function is used to handle label imbalance.

[0087] In the model training and optimization operation of this embodiment, data preprocessing is performed; specifically, the input image is uniformly scaled to 256×256 pixels, a 224×224 region is randomly cropped, and horizontal flipping, ±15° rotation and HSV color gamut enhancement are applied.

[0088] The component label is encoded as an N-dimensional binary vector, with 1 indicating the presence of a component and 0 otherwise.

[0089] The training strategy in this embodiment includes, but is not limited to:

[0090] Initialization: Fine-tuning based on ImageNet-21K pre-trained weights;

[0091] Optimizer: SGD is used, with an initial learning rate of 1e-3, momentum of 0.9, and weight decay of 1e-4.

[0092] Progressive training: divided into 3 stages (10 epochs per stage), with a total of 30 epochs of training, and the learning rate decays by 0.8 times in each stage;

[0093] Loss function: L = 0.8L ing +0.2L KL .

[0094] In summary, this invention constructs a multi-task progressive feature aggregation network capable of achieving high-precision component recognition. By fusing multi-granularity visual features through a progressive feature aggregation strategy, this invention significantly improves the fine-grained recognition capability of complex food images; it employs KL divergence constraints to enhance feature diversity, effectively solving the problem of multi-scale feature convergence; and it supports zero-shot component inference, maintaining high component recognition accuracy in cross-dataset testing, providing reliable technical support for intelligent nutritional analysis. The trained network model can output food component recognition results, maintaining high classification accuracy while possessing zero-shot inference capability.

[0095] The progressive local feature learning module of this invention adopts a phased activation mechanism, gradually unlocking deep network layers during forward propagation, extracting multi-scale local features by combining global max pooling, and constraining the distribution differences of features at different stages by KL divergence regularization.

[0096] This invention employs an end-to-end joint optimization strategy, using a multi-dimensional loss function to balance component identification loss and feature difference constraint loss, thereby achieving overall updating of model parameters.

[0097] This invention solves the technical problem of insufficient local feature representation in food identification in the prior art.

[0098] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A food component identification method based on progressive multi-scale feature aggregation, characterized in that, The method includes: S1. Construct a neural network model for component analysis, including: a global feature extraction module, a progressive local feature learning module, and a component recognition classifier; extract multi-scale global semantic features of food images through the global feature learning module; S2. Through the progressive local feature learning module, multi-scale local fine-grained features are extracted step by step, and combined with KL divergence, multi-scale feature diversity enhancement operation is performed. S3. The global semantic features and the local fine-grained features are fused to generate a unified feature representation. The component identification results are output through a classifier for multi-label prediction of food components.

2. The food component identification method based on progressive multi-scale feature aggregation according to claim 1, characterized in that, In S1, the global feature learning module adopts the ViT architecture and extracts global features based on the backbone network pre-trained by self-supervised learning. Global average pooling (GAP) is applied to the output of the last Transformer layer to obtain the global feature representation.

3. The food component identification method based on progressive multi-scale feature aggregation according to claim 2, characterized in that, The global feature representation is expressed using the following logic: f Glo =GAP(f out ) In the formula, f out This represents the output feature of the last layer of ViT.

4. The food component identification method based on progressive multi-scale feature aggregation according to claim 1, characterized in that, In S2, an end-to-end progressive training strategy is adopted to activate network layers of different depths in stages. By incorporating the constraints of the KL divergence, the differences between the local fine-grained features and the global semantic features at multiple scales are enhanced. In the progressive local feature learning module, the training process is divided into no less than two steps according to the preset number of steps, and deep Transformer blocks are gradually introduced to learn local features of different granularities from shallow to deep layers. In each stage i, local feature vectors are extracted using global max pooling (GMP): Maximizing the KL divergence between features at different stages allows the neural network model of the component analysis to focus on different regions.

5. The food component identification method based on progressive multi-scale feature aggregation according to claim 1, characterized in that, In the KL divergence optimization process of S2, local fine-grained features at different stages are mapped to probability distributions, and feature convergence suppression is performed by maximizing the KL divergence differences between the distributions. A multi-loss fusion optimization strategy is adopted to determine the improved multi-label cross-entropy loss function and the KL divergence loss; based on the KL divergence loss and the improved multi-label cross-entropy loss function, the total loss function is obtained by weighted combination.

6. The food component identification method based on progressive multi-scale feature aggregation according to claim 5, characterized in that, Using the improved multi-label cross-entropy loss function, perform the component identification task:

7. The food component identification method based on progressive multi-scale feature aggregation according to claim 5, characterized in that, The KL divergence loss is calculated using the following logic: L KL (P||Q)=∑ i P(i)·(log(P(i)-logQ(i))。 8. The food component identification method based on progressive multi-scale feature aggregation according to claim 5, characterized in that, The total loss function is obtained using the following logic: L=αL ing +βL KL In the formula, α and β are equilibrium parameters.

9. The food component identification method based on progressive multi-scale feature aggregation according to claim 1, characterized in that, In step S3, the feature aggregation module uses a fully connected layer to concatenate the local fine-grained features and global semantic features from different stages. f ing =concat(f Glo ,f U-S+1 ,...,f U ) In the formula, U is the total number of network stages, and S is the number of progressive training steps.

10. A food component recognition system based on progressive multi-scale feature aggregation, characterized in that, The system includes: The model building and global feature acquisition module is used to build a neural network model for component analysis, including: a global feature extraction module, a progressive local feature learning module, and a component recognition classifier; through the global feature learning module, multi-scale global semantic features of food images are extracted; The progressive local feature learning module is used to progressively extract multi-scale local fine-grained features and combine them with KL divergence to perform multi-scale feature diversity enhancement operations. The component identification classifier is used to fuse the global features with the local fine-grained features to generate a unified feature representation. The classifier outputs the component identification results for multi-label prediction of food components. The component identification classifier is connected to the progressive local feature learning module and the model construction and global feature acquisition module.

Images