Fine-grained food image classification method and system

Abstract

The invention belongs to the technical field of food image classification, and discloses a fine-grained food image classification method and system. According to the invention, a fine-grained food image classification model is established. In the aspect of the model structure, the local feature extraction module makes up the congenital deficiency of the ViT model in the aspect of local detail capture by embedding depth separable convolution into the transformation process of a mark sequence, and the perception ability of the model to fine-grained features such as food texture and edge is improved. The multi-scale dilatation attention module combines sparse local attention with a multi-scale cavity sampling strategy, and gives a flexible multi-scale context modeling capability to the model while reducing self-attention calculation complexity, so that food materials of different sizes can be effectively focused, and background noise interference is inhibited. And the efficient feed-forward network module reduces the parameter quantity and the calculation quantity of the feed-forward network. According to the method, food images with complex backgrounds and fine granularity differences can be efficiently processed, and the calculation efficiency and the classification precision are obviously improved.

Description

Technical Field

[0001] This invention belongs to the field of food image classification technology, and relates to a fine-grained food image classification method and system. Background Technology

[0002] Fine-grained food image classification is a crucial application in computer vision, widely used in intelligent diet management, restaurant recommendation systems, and health monitoring. Existing fine-grained food image classification methods primarily rely on Convolutional Neural Networks (CNNs) and Visual Transformers (ViT) architectures. However, these methods still face some technical bottlenecks in practical applications, leading to a trade-off between classification accuracy and computational efficiency.

[0003] Traditional CNN methods classify images by extracting local features, making them suitable for general food image classification tasks. Lightweight CNN models like MobileNetV2 and MobileNetV3 exemplify this. These methods efficiently extract local image features (such as food texture and edges) through the local connectivity and weight sharing of convolutional kernels. However, their inherent local receptive field limits their ability to model global semantic information and long-distance dependencies (such as relationships between scattered ingredients), leading to bottlenecks in fine-grained food classification with complex compositions. In fine-grained food image classification, food appearances are often highly similar, and traditional CNN models cannot effectively capture these subtle differences. Especially when processing images containing multiple ingredients or dishes, CNNs tend to overlook global semantic relationships between ingredients, resulting in lower classification accuracy. Furthermore, CNNs perform poorly when handling complex backgrounds (such as restaurant backgrounds, cluttered tabletops, etc.), struggling to effectively suppress background noise.

[0004] ViT, through its self-attention mechanism, can effectively model long-range dependencies in images, compensating for the shortcomings of CNNs in global information modeling. However, the computational complexity of ViT increases quadratically with the number of image segments, making it computationally expensive and time-consuming when processing high-resolution food images. This results in low inference efficiency, making it unsuitable for real-time scenarios requiring high computational efficiency and for resource-constrained devices (such as mobile devices or embedded systems). Furthermore, in fine-grained food classification tasks, ViT is weak in capturing local detail features, particularly subtle texture differences in food images. This makes it difficult to identify minute variations in ingredients, impacting its performance in fine-grained classification tasks and resulting in lower recognition accuracy compared to CNNs.

[0005] Furthermore, in practical applications, food images are often affected by complex backgrounds, such as restaurant lighting, clutter on the table, or other ingredients. This noise further reduces the classification accuracy of existing methods. CNNs, lacking global context modeling capabilities, are easily affected by background noise, while ViT, although capable of capturing global information, has shortcomings in extracting local features. Therefore, existing image classification methods present a significant contradiction between accuracy and computational efficiency, necessitating a new approach to improve accuracy and reduce computational overhead. To address this issue, Sheng et al. designed a parallel architecture of ViT and CNN, aiming to simultaneously consider global context and local features. However, while this method enhances feature extraction capabilities, its complex architecture results in a large number of model parameters and computational cost (FLOPs), failing to effectively overcome the bottleneck of computational efficiency. Song et al. introduced metric learning to strengthen local feature representation, which improved fine-grained classification accuracy, but computational efficiency still falls short of real-time requirements, especially performing poorly on high-resolution images. Furthermore, the multi-task learning framework proposed by Liu et al. jointly optimizes food category and ingredient recognition. Although it improves the functionality of the model, it further increases the complexity of the model, leading to a heavier computational burden in the training and inference process, and fails to solve the problem of computational efficiency. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention proposes a lightweight and high-precision fine-grained food image classification method. This method proposes a hybrid model, DKTransformer, that combines the advantages of ViT and CNN. It aims to achieve powerful local feature extraction capabilities, efficient global context modeling capabilities, and extremely low parameter and computational complexity through an innovative hybrid architecture design, enabling efficient processing of food images with complex backgrounds and fine-grained differences.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A fine-grained food image classification method includes the following steps: Step 1. Obtain fine-grained food images, perform preprocessing and annotation operations on the images, and divide the dataset; Step 2. Build a fine-grained food image classification model. This model takes sequential patches as input, performs feature transformation through multiple cascaded DKBlock basic units, and finally outputs the image category prediction. Each DKBlock basic unit includes a Local Feature Extraction (LDE) module, a Multi-Scale Expanded Attention (MSDA) module, and an EfficientKAN module for Transformer. The image feature processing flow in each DKBlock basic unit is as follows: First, the image features are processed by the LDE module, which uses depthwise separable convolution to enhance the ability to capture local texture and edge features; The output features of the LDE module are then input into the MSDA module. The MSDA module uses a sparse attention mechanism with multiple porosities to reduce computation while covering a larger receptive field and suppressing noise interference. Finally, the output features of the MSDA module are processed by the efficient Kolmogorov-Arnold network of the EfficientKAN module. Through parameter sharing and sparse activation function optimization, the number of parameters is reduced while ensuring the expressive power of the model. Step 3. Based on the dataset divided in Step 1, train the fine-grained food image classification model built in Step 2; after the model is trained, use the trained model to classify fine-grained food images.

[0008] Furthermore, based on the aforementioned fine-grained food image classification method, this invention also proposes a fine-grained food image classification system corresponding to this method, which adopts the following technical solution: A fine-grained food image classification system includes a computer device; the computer device includes a memory and a processor; wherein, a computer program is stored in the memory; when the processor executes the computer program, it implements the steps of the fine-grained food image classification method as described above.

[0009] The present invention has the following advantages: As described above, this invention relates to a fine-grained food image classification method and system. The method constructs a hybrid model, DKTransformer, combining the advantages of ViT and CNN. Specifically, regarding model structure, this invention proposes a Local Feature Extraction (LDE) module. By embedding depthwise separable convolutions into the transformation process of the labeled sequence, it effectively compensates for the inherent limitations of the ViT model in capturing local details, significantly improving the model's ability to perceive fine-grained features such as food texture and edges, with almost no increase in computational cost. Furthermore, this invention proposes a Multi-Scale Expanded Attention (MSDA) module. By combining sparse local attention with a multi-scale dilated sampling strategy, it significantly reduces the computational complexity of self-attention while endowing the model with flexible multi-scale context modeling capabilities, enabling it to effectively focus on food of different sizes and suppress background noise interference. Furthermore, this invention also designs an efficient feedforward network module for Transformers, EfficientKAN. Based on the Kolmogorov-Arnold theorem, it replaces the fixed activation function and linear layers in traditional MLPs with a parameterized set of learnable univariate functions. Through strategies such as parameter sharing, group learning, and sparsification, it significantly reduces the number of parameters and computational cost of the feedforward network without sacrificing the model's expressive power. Thanks to the LDE module's enhancement of local details and the MSDA module's effective modeling of multi-scale global context, the classification accuracy of this invention is significantly improved. This invention achieves leading classification accuracy on several publicly available fine-grained food datasets, such as Food-101 and Vireo Food-172 (e.g., a Top-1 accuracy of 90.16% on Food-101). In addition, due to the sparse attention mechanism of MSDA and the downsampling strategy in the model, the computational complexity (FLOPs) is significantly reduced, and the computational efficiency is significantly improved. The FLOPs of this invention are only 621.2M, which is about 65.3% lower than the MobileViTv2 model with equivalent accuracy, and the inference speed is faster. Furthermore, thanks to the efficient parameterization of the feedforward network by the EfficientKAN module, the total number of parameters in this invention is only 4.7M. This small parameter count makes the model lightweight and easy to deploy on mobile and embedded devices. In addition, the multi-scale characteristics of the MSDA module enable it to better focus on the core ingredient region, effectively suppressing background noise from tableware, side dishes, etc., resulting in more focused and accurate attention and stronger anti-interference capabilities. Moreover, the proposed DKTransformer model performs excellently on multiple datasets of different sizes and styles (Western and Chinese cuisine), demonstrating its good generalization ability. Attached Figure Description

[0010] Figure 1This is a flowchart illustrating the fine-grained food image classification method in an embodiment of the present invention. Figure 2 This is an overall architecture diagram of the DKTransformer model in an embodiment of the present invention; Figure 3 This is a flowchart of the image processing of the DKTransformer model in an embodiment of the present invention; Figure 4 This is a flowchart of the image processing steps before inputting the image into the DKTransformer model in this embodiment of the invention. Figure 5 This is a flowchart illustrating the internal processing of the DKBlock basic unit in an embodiment of the present invention. Figure 6 This is a flowchart illustrating the depth-separable convolution processing in an embodiment of the present invention. Figure 7 This is a structural diagram of the Multiscale Extended Attention Module (MSDA) in an embodiment of the present invention. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 This embodiment 1 describes a fine-grained food image classification method, in which a hybrid neural network model called DKTransformer is constructed. The DKTransformer model takes a sequenced patch as input, performs feature transformation through multiple cascaded DKBlock basic units, and finally outputs the image category prediction.

[0012] Each DKBlock basic unit contains a Local Feature Enhancement (LDE) module, a Multi-Scale Expanded Attention (MSDA) module, and a parameter-efficient feedforward network module, EfficientKAN.

[0013] LDE enhances the model's ability to capture fine-grained local features of food images without significantly increasing computational cost. MSDA reduces the computational complexity of attention through sparsification and multi-scale strategies, expands the receptive field, and effectively suppresses background noise interference. EfficientKAN replaces traditional MLP, significantly reducing the number of parameters while maintaining the model's expressive power.

[0014] By integrating the aforementioned modules such as LDE, MSDA, and EfficientKAN, it is possible to construct an end-to-end lightweight hybrid network model, achieving simultaneous improvement in classification accuracy and computational efficiency on publicly available food image datasets.

[0015] like Figure 1As shown, the fine-grained food image classification method in this embodiment includes the following steps: Step 1. Obtain fine-grained food images, perform preprocessing and annotation operations on the images, and divide the dataset.

[0016] The datasets used in this embodiment include, for example, the ETHZ Food–101, Vireo Food–172, and ETHZFood–500 datasets (which are just examples), covering multiple fine-grained food classification scenarios such as Western and Chinese cuisine.

[0017] The following provides a detailed introduction to each dataset, image preprocessing, annotation process, and dataset partitioning: I. Introduction and Construction of Datasets

[0018] ETHZ Food–101 dataset: This dataset is a large-scale image dataset of Western cuisine, containing 101 food categories and a total of 101,000 images. Each category contains approximately 1,000 images, covering mainstream Western cuisine dishes. The images were sourced from web crawling and manual selection, exhibiting high intra-class diversity and inter-class similarity, making it suitable for fine-grained classification tasks.

[0019] Vireo Food–172 dataset: This dataset, released in 2016, contains images of Chinese dishes across 172 categories, totaling 110,241 images. The images primarily originated from user uploads on food delivery platforms, covering various cuisines and shooting environments, exhibiting significant real-world scene complexity and background interference.

[0020] ETHZ Food–500 dataset: This dataset is an expanded version of ETHZ Food–101, containing 500 food categories and a total of 500,000 images. With broader category coverage and a larger number of images, it is suitable for validating the model's generalization ability and stability on large-scale data.

[0021] II. Image preprocessing and annotation process.

[0022] All input images undergo the following preprocessing operations: Data augmentation: During the training phase, enhancement strategies such as random horizontal flipping, random rotation (±10°), and color jitter (brightness, contrast, and saturation adjustment) are employed to improve the robustness and generalization ability of the model.

[0023] Standardized dimensions: The original image was scaled and centered to a fixed size of 224×224 pixels to fit the model input requirements.

[0024] Normalization process: Image pixel values ​​are standardized using the mean and standard deviation of the ImageNet dataset (mean: [0.485, 0.456, 0.406], standard deviation: [0.229, 0.224, 0.225]) to accelerate model convergence.

[0025] III. Data labeling.

[0026] Data annotation was completed by a professional annotation team, and each image underwent manual review to ensure accurate category labels. The annotation process followed unified standards, and images with ambiguity underwent cross-validation by multiple researchers to guarantee annotation quality.

[0027] Taking Food-101 as an example, there are 101 categories, each of which is a fine-grained label, including but not limited to: Apple_Pie, Baby_Back_Ribs, Beef_Carpaccio, etc. Vireo Food-172 also covers additional Chinese food subcategories, such as Braised Pork Belly, Kung Pao Chicken, and Mapo Tofu.

[0028] Labeling is done in units of "whole dish," with only one label assigned to each image. If an image contains multiple dishes, the most prominent one is selected for labeling. Label names are identical to those in the official category table of the public dataset in terms of capitalization and underscores, eliminating the need for semantic mapping.

[0029] IV. Dataset partitioning.

[0030] To ensure the fairness and reproducibility of the experiment, all datasets were divided into training, testing, and validation sets according to publicly released standard partitioning methods (used for model training, testing, and validation, respectively): ETHZ Food–101: Training set 80,800 images, validation set 10,100 images, test set 10,100 images.

[0031] Vireo Food–172: Training set 88094 images, validation set 11012 images, test set 11012 images.

[0032] ETHZ Food-500: Training set 400,000 images, validation set 50,000 images, test set 50,000 images.

[0033] Step 2. Build a fine-grained food image classification model, DKTransformer, taking sequential patches as input and performing feature transformation through multiple cascaded DKBlock basic units, ultimately outputting the image's category prediction, such as... Figure 2 As shown.

[0034] like Figure 3 As shown, the general processing procedure of the DKTransformer model is as follows: Given an RGB food image (size H×W×3), segment it into non-overlapping patches of size P×P, and map each patch to a D-dimensional embedding vector through a linear projection layer.

[0035] After adding a learnable [CLS] classification label, an initial label sequence is formed. The formula is expressed as follows: ;in .

[0036] Initial label sequence It passes through N identical DKBlock basic units in sequence, each preceded by a downsampling step.

[0037] After processing by L DKBlock basic units, the output vector corresponding to the [CLS] label is taken as the representation of the entire image and fed into a lightweight classification head (usually a linear layer + Softmax) for final classification.

[0038] like Figure 4 As shown, the image undergoes the following preprocessing before being input into multiple cascaded DKBlock basic units: The input image is an RGB color image with dimensions of [size missing]. .

[0039] First, it was split into A fixed-size patch is then projected as Each patch has 1 marker and a size of 1. These markers serve as input to the model, and the patches are then reshaped from a two-dimensional grid arrangement into a one-dimensional sequence.

[0040] in , 3 represents height, width, and number of channels, respectively. .

[0041] At this point, the image is represented as Each patch is a [number] patch. A dimensional vector.

[0042] Taking an input RGB image of size 224×224×3 and a patch size of 16×16 as an example, the image segmentation process is as follows: First, the 224×224 image is evenly divided into 16-pixel steps (without overlap). Horizontally, there are 224 / 16 = 14 grids, and vertically, there are also 14 grids. Therefore, the total number of patches N = 14×14 = 196.

[0043] Then, the area is flattened. Each 16×16×3 patch is stretched into a vector in RGB order, with a length of 16×16×3 = 768.

[0044] Finally, the projection operation is performed, which feeds these 196 768-dimensional vectors into a learnable projection matrix at once, resulting in 196 768-dimensional position embeddings. Position encoding is then added before feeding them into the model.

[0045] After flattening each segmented patch, the model maps each flattened patch to a lower-dimensional embedding space through a linear projection (usually a fully connected layer).

[0046] In this way, each The patch will be represented as an embedding vector called "patchembedding".

[0047] Using matrices This represents the linear projection weight of each patch; in This is the embedding dimension (usually the hidden dimension of the model); the embedding of each patch is represented as: .

[0048] in, It is a bias term. It is the first The flattened vector of each patch, This is the embedded representation of the patch.

[0049] Since the model itself does not have the ability to process the order of elements in the sequence (CNN can naturally capture positional relationships through local receptive fields), the model needs to add positional encoding to preserve the spatial position information of each patch in the original image.

[0050] The location code is a trainable vector that represents the spatial location of each patch.

[0051] Location encoding is For the first Each image patch has a positional encoding vector determined by its spatial location in the image; the positional encoding is added to the embedding representation of the image patch to form an input sequence with spatial location information.

[0052] The formula for calculating the position code is as follows: .

[0053] in It is the first The embedded representation of a patch, This is the location information of the corresponding patch.

[0054] like Figure 2 As shown, image features undergo a downsampling operation before entering each DKBlock basic unit. To further reduce computational cost, the model introduces downsampling layers (such as convolutions with a stride of 2) between stacked DKBlocks as needed. By reducing the sequence length (resolution), the overhead of subsequent attention and FFN computations is significantly reduced.

[0055] The computational complexity of the self-attention mechanism in ViT is the square of the sequence length (number of tokens). The computational complexity of a feedforward network (FFN) is also linearly proportional to the sequence length. .

[0056] Downsampling directly reduces sequence length This drastically reduces the computational and memory footprint of all subsequent Transformer Blocks. This is the most direct and important purpose of incorporating downsampling, and it is crucial for enabling real-time applications and edge deployments.

[0057] Furthermore, hierarchical downsampling can mimic the backbone of a CNN, gradually expanding the receptive field of each token.

[0058] Early blocks handle low-level, high-resolution detail features (such as texture and edges), while later blocks handle high-level, low-resolution semantic features (such as object parts and overall shape). This hierarchical representation is highly beneficial for understanding complex food images composed of multiple ingredients. Furthermore, gradually reducing the resolution helps the network filter out unnecessary detail noise and background interference, forcing the model to learn more robust and essential feature representations.

[0059] Therefore, a downsampling ratio of 2 was chosen (i.e., the sequence length is reduced to 1 / 4, because the height and width are each halved). After each downsampling, in a 12-layer model, downsampling layers are inserted after layers 4 and 8.

[0060] Downsampling is achieved through a convolutional layer with a stride of 2 and a kernel size of 3x3. This operation reduces the spatial resolution (height and width) of the feature map to half, thereby reducing the sequence length to 1 / 4 of the original.

[0061] Meanwhile, to maintain model capacity, the channel dimension (D) of the feature map is doubled accordingly. This strategy significantly reduces the computational overhead of the attention and feedforward networks in subsequent DKBlock operations, achieving a balance between accuracy and efficiency.

[0062] After processing by layers of DKBlock basic units, the output (i.e., the representation of all patches) is computed through several layers of self-attention, and the final output is fed into the classification head for image classification. The model uses a pooling operation to aggregate all blocks into a global image representation, and then performs classification through a fully connected layer.

[0063] Final output Represented as: ; in It's the number of patches. It is the embedded dimension.

[0064] A global image representation is obtained through pooling operations (such as average pooling or taking the representation of a specific block). Then input it into the classification header: ; in It is the final category prediction for the image. The weight matrix represents the classification head, with each row corresponding to a linear discriminant vector for a class. The score for that category is obtained by performing the inner product. This represents the bias vector, which provides a learnable scalar bias for each category and is directly added to the corresponding score.

[0065] like Figure 2 As shown, the DKBlock basic unit is the fundamental module that constitutes the backbone of the entire DKTransformer model. The internal structure of the DKBlock basic unit is as follows: Figure 5 As shown.

[0066] Each DKBlock basic unit includes a Local Feature Extraction (LDE) module, a Multi-Scale Expanded Attention (MSDA) module, and an EfficientKAN module for Transformers.

[0067] The image feature processing flow in each DKBlock basic unit is as follows: First, the image features are processed by the LDE module. Within the LDE module, depthwise separable convolution is used to enhance the capture of local texture and edge features. The output features of the LDE module are then input into the MSDA module. The MSDA module uses a sparse attention mechanism with multiple porosities to reduce computation while covering a larger receptive field and suppressing noise interference. Finally, the output features of the MSDA module are processed by the efficient Kolmogorov-Arnold network of the EfficientKAN module. Through parameter sharing and sparse activation function optimization, the number of parameters is reduced while ensuring the expressive power of the model.

[0068] like Figure 5 As shown, the LDE module is used to solve the problem of local detail loss caused by linear projection in ViT.

[0069] In step 2, the processing flow of the LDE module is as follows: I. Sequence rearrangement: The input tag sequence X, containing the [CLS] tag and patch tag, is restructured into a two-dimensional feature map based on the spatial structure of the original image, in addition to the batch dimension (B) and sequence length dimension (N+1). .

[0070] II. Depthwise Convolution: Because food image classification requires a greater focus on fine-grained features, this invention introduces depthwise separable convolution to compensate for the loss of local details within the model, applying a depthwise separable convolution operation to the aforementioned feature map F.

[0071] Standard ViT segments the image into blocks and feeds them directly into the Transformer. After linear projection, the local spatial relationships (such as edges, corners, and textures) of the pixels within a patch are largely lost. The Transformer's self-attention mechanism is inherently global and permutation invariant, but it lacks the inductive bias of locality and translation equivariance inherent in CNNs.

[0072] Depthwise separable convolutions operate specifically in the spatial dimension, making them exceptionally adept at capturing local spatial patterns. Embedding them into early versions of ViT allows it to perceive subtle textures and edges of food like a CNN, and then the powerful capabilities of the Transformer are used to establish global relationships between these local features.

[0073] This depthwise separable convolution operation first uses a depthwise convolution kernel to extract features in the spatial dimension, capturing local spatial features within each channel; then it uses a pointwise convolution to fuse the channel information.

[0074] As attached Figure 6 As shown, depthwise separable convolution is a form of decomposition convolution that breaks down a complete standard convolution operation into two independent, consecutive steps: depthwise convolution and pointwise convolution.

[0075] The first step, depthwise convolution, processes the spatial features of each input channel separately.

[0076] Each channel has its own convolutional kernel, performing independent spatial filtering to extract local features. indivual The convolution kernels are configured such that each kernel convolves with only one channel of the input feature map, ultimately outputting a... The feature map.

[0077] This step only performs spatial filtering, does not change the number of channels, and there is no information exchange between channels.

[0078] The second step, pointwise convolution, mixes information from all channels of the depthwise convolution output to generate a new feature representation. Using... indivual The convolutional kernel is essentially a simple linear combination, equivalent to performing a fully connected operation at each pixel location, thus... dimensional channel vectors mapped to Dimension, final output feature map .

[0079] The standard convolution parameter count is The computational cost (FLOPs) is approximately equal to The number of parameters for depthwise separable convolution is: The computational workload is approximately equal to: Depthwise separable convolutions typically have only 1 / 3 the number of parameters and computational cost of standard convolutions. +1 / 9 or so. when When the values ​​are large (e.g., 256, 512), the savings are extremely significant.

[0080] III. Sequence Restoration and Assembly: The two-dimensional feature map F' obtained after depthwise convolution is flattened back into a one-dimensional label sequence; this one-dimensional label sequence is then concatenated with the unconvolution-processed [CLS] label to form the output sequence.

[0081] The LDE module can inject local feature extraction capabilities into ViT at extremely low cost. Because it processes 2D feature maps while the model processes 1D sequences, it needs to go through the process of "sequence reconstruction → convolution → sequence restoration" to convert between 2D image space and 1D sequence space, enabling CNN and Transformer to work together seamlessly.

[0082] like Figure 5 As shown, the LDE module embeds depthwise separable convolutions into the labeled sequence, first splitting the sequence into patch labels. and category tags In a two-dimensional image, .

[0083] and It can be represented as , .

[0084] Then, a depthwise separable convolution of depth k is introduced to extract local features around the pixel. , can be represented as: , .

[0085] Finally Flatten it into a one-dimensional sequence space, and then combine it with... Connect them: .

[0086] The LDE module effectively compensates for the inherent shortcomings of the ViT model in capturing local details by embedding depthwise separable convolutions into the transformation process of the labeled sequence, without increasing the computational cost. This significantly improves the model's ability to perceive fine-grained features such as food texture and edges.

[0087] like Figure 5 and Figure 7 As shown, the Multi-Scale Extended Attention Module (MSDA) reduces the computational complexity of standard self-attention and introduces multi-scale receptive fields to better handle ingredients of different sizes and suppress noise. MSDA is an improved version of the standard multi-head self-attention module (MSA), and its core features are sparse key-value sampling and multi-scale dilated sampling.

[0088] Sparse local attention is as follows: For each query vector It does not perform calculations with all keys globally. Instead, it selects a set of key-value pairs centered at the corresponding spatial position (i, j) within a pre-defined local window (e.g., ω×ω) with a fixed step size (sparse factor r). , ) to perform calculations.

[0089] The formula for calculating attention is: .

[0090] The set of sampling locations is defined as follows: , .

[0091] Multi-scale extended perception is as follows: To capture multi-scale information, the channel dimensions of the attention heads are grouped. Different dilation rates r (e.g., r=1, 2, 3) are used in different head groups. The dilation rate r essentially controls the sampling step size; a larger r results in a larger receptive field and a more macroscopic region of focus. In this way, different head groups focus on semantic information at different scales.

[0092] The head group output fusion is as follows: After each head group independently performs sparse attention computation, their outputs are concatenated along the channel dimension, and finally, the information is fused through a linear projection layer to obtain the final output of the MSDA module. Traditional MHSA in ViT consumes high computational and storage resources due to its multi-head computation and high-dimensional feature representation, and dense attention is susceptible to noise interference.

[0093] To address this, MSDA proposes a strategy that combines sparse local attention with multi-scale receptive fields.

[0094] The improved formula for calculating the attention module is as follows: ; In the formula , and These represent the query, key, and value matrices, respectively.

[0095] For location The query at the location is defined. To represent the sparsity, in Centered on, with dimensions of In a sliding window within a window, sparsely select key and value for self-attention computation.

[0096] The following formula represents the set of positions The key and value at this location are used for self-attention calculation: , .

[0097] To explore the sparsity at different scales in the self-attention mechanism, the MSDA module is used to extract semantic information at different scales.

[0098] In the MSDA module, the feature map is divided into multiple heads in the channel dimension. Different dilation coefficients are used in different heads to sample the patch around the query for self-attention computation.

[0099] Finally, the calculation results from different heads are concatenated, and the concatenated result is fed into the Linear Layer for feature calculation. ; The sliding window has a size of 3×3 and a hole coefficient r = 1, 2, 3; the formula is as follows: ; ; In the above formula This represents the void coefficient of the i-th head. , and This represents the feature of the i-th head.

[0100] Define a window around each query, controlled by... The size of the value is set, and the key and value are selected within this window. This ensures that the attention mechanism focuses primarily on information within the local region, reducing the computational load.

[0101] A higher dilation rate means a sparser selection that can cover a larger receptive field, but may sacrifice some local details. Setting different dilation rates in different heads allows each head to focus on features at different scales. Each head is then computed independently, and all outputs are then merged and the final result is obtained through linear layer feature aggregation.

[0102] The MSDA module combines sparse local attention with a multi-scale dilated sampling strategy, significantly reducing the computational complexity of self-attention while endowing the model with flexible multi-scale context modeling capabilities. This allows the model to effectively focus on ingredients of different sizes and suppress background noise interference. In the modified attention module, the Dropout algorithm is introduced as a regularizer. Before calculating the attention weights, a subset of elements in the Attention Score Matrix are randomly set to negative infinity, effectively setting the Key as a drop object. These Keys are randomly discarded on the attention weight graph. Keys set to negative infinity are converted to 0 probability in the subsequent Softmax function calculation, meaning that when calculating the association between a query and all keys, the model completely "ignores" these randomly discarded keys.

[0103] Specifically, because the applied methods emphasize the ability to focus on local features of the target to complete a specified task, they are prone to overfitting. To overcome this problem, ViT typically introduces the Dropout algorithm, commonly used in CNNs, as a regularizer. It randomly drops the attention weights and sets a uniform drop probability for attention layers of different depths. Randomly dropping after softmax normalization disrupts the probability distribution of attention weights and fails to penalize weight peaks, leading to overfitting to specific local information. Secondly, a higher drop probability in deeper layers leads to a loss of high-level semantic information, while a lower drop probability in shallower layers leads to overfitting to low-level detail features; therefore, a constant drop probability results in instability during training. Finally, the effectiveness of the structured drop method commonly used in CNNs on ViT is unclear. (See attached image) Figure 5 and Figure 7 As shown, DropKey sets the Key as the drop object and verifies theoretically and experimentally that this change can penalize high attention value parts while encouraging the model to pay more attention to other patches related to the target, which helps capture globally robust features. Furthermore, the progressively deeper attention layers set decreasing drop probabilities, which avoids overfitting low-level features while ensuring sufficient high-level features for stable training. By advancing the drop operation to before the attention matrix calculation and setting the Key as the drop unit, this scheme preserves both the regularity and probabilistic features of the attention weights, mitigating overfitting to specific patterns and enhancing the model's ability to capture important information globally. Second, instead of using a constant drop rate across all layers, a new decreasing schedule is used, which gradually reduces the drop rate along the stack of self-attention layers, thereby improving the robustness and stability of model training. This method uses the Key as the drop unit and utilizes decreasing scheduling to reduce the drop rate, improving the overall resistance to vit's fit.

[0104] like Figure 5 As shown, the EfficientKAN module replaces the parameter-redundant traditional MLP, reducing the number of model parameters. This module is based on the Kolmogorov-Arnold representation theorem. Its core idea is to replace the fixed activation function in the MLP with a learnable unary function. Basic structure: The input to the EfficientKAN layer is x (a vector), and the output y is calculated as follows: ,in It is a set of learnable unary functions. This indicates the number of input variables, each... The function itself is a linear combination of a set of learnable basis functions (such as piecewise polynomials, rational functions, B-spline bases, etc.).

[0105] Compared to the original KAN, the EfficientKAN used in this invention has undergone several optimizations to improve efficiency: (i) Parameter sharing: The activation functions of adjacent neurons share basis function parameters (such as segmentation points); (ii) Group learning: Neurons are grouped together and the basis functions are shared within the group, which greatly reduces the number of parameters; (iii) Use simpler basis functions: such as rational functions Φ(x) = ω * (P(x) / Q(x)); where P(x) and Q(x) are low-order polynomials rather than complex B-spline bases; (iv) Sparsification and pruning: Apply L1 regularization to the coefficients of the basis functions to induce sparsity and prune coefficients close to zero; Replace MLP: In DKBlock, replace the standard feedforward network (FFN) layer directly with the EfficientKAN layer.

[0106] To reduce the complexity of feedforward networks, EfficientKAN, based on the Kolmogorov-Arnold representation theorem, replaces the fixed activation function of traditional MLPs with learnable adaptive basis functions.

[0107] KAN is inspired by the Kolmogorov-Arnold representation theorem, which states that any multivariate continuous function can be decomposed into a finite number of univariate functions. Each neuron in KAN achieves this decomposition through a learnable activation function.

[0108] Specifically, KAN's first The layer output is: .

[0109] in, It is a learnable activation function, i.e., a parameterized B-spline basis function. It is the first The number of neurons in a layer. No. Layer The input value of each neuron, No. Layer The input value of each neuron.

[0110] The core improvement of EfficientKAN lies in simplifying the parameterization of the activation function and optimizing the computation process. The original KAN uses highly complex basis functions (such as B splines): .

[0111] in It is a B-spline basis. It is the number of basis functions. This represents a piecewise linear basis function.

[0112] EfficientKAN uses lower-order polynomials or piecewise linear basis functions, as shown in the following formula: .

[0113] in , and For learnable parameters, Let m be the segmentation point and m be the number of segments.

[0114] The parameters of EfficientKAN are also shared and grouped, with some parameter segmentation points shared for the activation functions of adjacent neurons. Neurons are grouped, with basic functions shared within groups and independent learning between groups, balancing expressive power and efficiency while reducing redundancy.

[0115] Also, the basis function coefficients in the activation function , L1 regularization is applied to promote sparsity; further, coefficients close to zero are pruned to compress the model. The optimization results can be seen from the complex comparisons. Assuming each floor has There are neurons, and each activation function contains _____ neurons. There are basis functions, and the original KAN parameter count is... (Each pair of neurons needs to be) (parameters), while the number of parameters in EfficientKAN is (Shared basis functions + sparse coefficients), so the computational cost of forward propagation also increases from... Down to .

[0116] The MLP layer in BLOCK is replaced with EfficientKAN, and the improved formula is as follows: ; in Given the input sequence, This represents the input sequence after the attention mechanism has been applied. Presentation layer normalization operation.

[0117] Rational functions are used as the basis functions for the EfficientKAN layers, instead of the original B-spline functions, i.e., the functions on each edge. Parameterization polynomial of order , Rational numbers in the domain.

[0118] .

[0119] in This represents a learnable scaling factor used to adjust the output amplitude of the function. , , The coefficients are set in the numerator polynomial. , , These are the pre-defined coefficients in the denominator polynomial.

[0120] In summary, this improvement has the following main advantages: (1) More efficient computation: Low-order basis functions and parameter sharing reduce floating-point operations; (2) Fewer parameters: Sparsification and grouping strategies reduce memory usage; (3) Maintaining expressive power: Approximating complex nonlinearities through flexible piecewise functions. These improvements significantly enhance the computational efficiency and memory usage of KAN, enabling it to be applied to food image classification tasks on larger datasets.

[0121] The EfficientKAN module is based on the Kolmogorov-Arnold theorem. It replaces the fixed activation function and linear layers in traditional MLPs with a parameterized set of learnable univariate functions. Through strategies such as parameter sharing, group learning, and sparsification, it significantly reduces the number of parameters and computational cost of the feedforward network without losing much of the model's expressive power.

[0122] Step 3. Based on the dataset divided in Step 1, train the fine-grained food image classification model built in Step 2; after the model is trained, use the trained model to classify fine-grained food images.

[0123] In this embodiment, the model training uses the following unified hyperparameter settings: The input resolution is 224×224, the batch size is 32, the optimizer is AdamW, the weight decay is 0.005, the initial learning rate is 0.001, and the training epochs are 500.

[0124] The learning rate is dynamically adjusted using a cosine annealing strategy, the loss function is cross-entropy loss, and the evaluation metrics are Top-1 and Top-5 classification accuracy.

[0125] Through the systematic data preprocessing, annotation management, dataset partitioning, and model training settings described above, the standardization of the training process and the comparability of the results in this embodiment are ensured, providing a reliable basis for the objective evaluation of model performance.

[0126] After the model is trained, for any fine-grained food image to be tested, the image is uniformly sized (to 224×224) and standardized using ImageNet statistics (mean [0.485, 0.456, 0.406], variance [0.229, 0.224, 0.225]) and then fed into the model. After a single forward inference, the probability distribution of the corresponding food category is directly output, and the final classification result is obtained by taking argmax.

[0127] This invention can simultaneously achieve powerful local feature extraction capabilities, efficient global context modeling capabilities, and extremely low parameter and computational complexity, enabling efficient processing of food images with complex backgrounds and fine-grained differences.

[0128] Example 2 This embodiment 2 describes a fine-grained food image classification system, which is based on the same inventive concept as the fine-grained food image classification method in embodiment 1 above.

[0129] In this embodiment, the fine-grained food image classification system includes a computer device; the computer device includes a memory and a processor; wherein, the memory stores a computer program; characterized in that, when the processor executes the computer program, it is used to implement the steps of the fine-grained food image classification method described in Embodiment 1 above.

[0130] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A fine-grained food image classification method, characterized in that, The method comprises the following steps: Step 1. Obtain a fine-grained food image, pre-process and label the image, and divide the data set; Step 2. Build a fine-grained food image classification model, which takes a serialized patch as input, and performs feature transformation through multiple cascaded DKBlock basic units, and finally outputs the category prediction of the image; Each DKBlock basic unit comprises a local feature extraction module LDE, a multi-scale dilated attention module MSDA, and an efficient feedforward network module EfficientKAN for Transformer; The processing flow of image features in each DKBlock basic unit is as follows: First, the image features pass through the LDE module, which uses a depth separable convolution to enhance the ability to capture local texture and edge features; Then, the output features of the LDE module are input into the MSDA module, which uses a multi-hole sparse attention mechanism to cover a larger receptive field while reducing computational complexity and suppressing noise region interference; Finally, the output features of the MSDA module pass through the EfficientKAN module of the high-efficiency Kolmogorov-Arnold network, which optimizes through parameter sharing and sparse activation functions, reducing the parameter quantity while ensuring the expression ability of the model; Step 3. Based on the data set divided in step 1, train the fine-grained food image classification model built in step 2; After the model is trained, the trained model is used for fine-grained food image classification.

2. The fine-grained food image classification method according to claim 1, wherein In step 1, the image pre-processing process includes data augmentation, size unification, and normalization processing.

3. The fine-grained food image classification method according to claim 1, wherein In step 2, before the image is input into multiple cascaded DKBlock basic units, the following pre-processing is performed: The input image is an RGB color image with the size of ; First split into patches of fixed size, then projected as markers, each patch of size ; these markers serve as input to the model, which subsequently reshapes the patches from a two-dimensional grid arrangement into a one-dimensional sequence; wherein , , 3 represent height, width, channel number, respectively, ; At this point, the image is represented as a patch, each patch being a dimensional vector; After each cut-out patch is flattened, the model maps each flattened patch through a linear projection to a lower-dimensional embedding space; each patch will be represented as an embedding vector; with matrix denotes the linear projection weights of each patch; wherein is the embedding dimension; the embedding representation of each patch is denoted as: ; wherein, is the bias term, is the flattened vector of the th patch, is the embedding representation of that patch. Position encoding is a trainable vector that represents the spatial position of each patch. Position encoding is For the first patch, its position encoding vector is determined according to its spatial position in the image; the position encoding is added to the patch's embedding representation to form an input sequence with spatial position information; The calculation formula of position encoding is as follows: ； wherein is an embedding representation of the th patch, is position information of the corresponding patch.

4. The fine-grained food image classification method according to claim 1, wherein In step 2, before the image features enter each DKBlock basic unit, a downsampling operation is performed.

5. The fine-grained food image classification method according to claim 1, wherein The fine-grained food image classification model further comprises a classification head; After the image features are processed by multiple cascaded DKBlock basic units, the classification head is used to perform category prediction of the image.

6. The fine-grained food image classification method according to claim 1, wherein In step 2, the processing flow of the LDE module is as follows: I. Sequence reorganization: The input token sequence X containing the [CLS] token and the patch token is reorganized into a two-dimensional feature map F outside the batch dimension and the sequence length dimension according to the spatial structure of the original image; II. Deep convolution: a depth separable convolution operation is applied to the above feature map F; The depth separable convolution operation first uses a depth convolution kernel to extract features in the spatial dimension, capturing local spatial features within each channel; Then use a pointwise convolution to fuse channel information; III. Sequence restoration and splicing: flatten the two-dimensional feature map F' obtained after depth convolution processing into a one-dimensional label sequence; splice the one-dimensional label sequence with the [CLS] label without convolution processing to form the output sequence.

7. The fine-grained food image classification method according to claim 1, wherein, In step 3, the processing process of the multi-hole rate sparse attention mechanism is as follows: The improved attention module calculation formula is as follows: ； wherein , and represent the query, key and value matrices, respectively; For location The query at the location is defined. To represent the sparsity, in Centered on, with dimensions of Sparse selection of key and value in a sliding window within a window for self-attention computation; A set of positions represented by the following formula key and value at the position are used for self-attention computation: ， ； In order to mine the sparsity of different scales in the self-attention mechanism, the MSDA module is used to extract semantic information of different scales; In the MSDA module, the feature map is divided into multiple heads in the channel dimension, and different patch sampling is used around the query in different heads for self-attention calculation; Finally, the results of the different heads are spliced, and the spliced results enter the Linear Layer for calculation to obtain features ; The size of the sliding window is 3x3, and the hole coefficient r=1, 2, 3; The formula is as follows: ； ； In the above formulae represents the hole coefficients of the i-th head, , and represents the features of the i-th head.

8. The fine-grained food image classification method according to claim 1, wherein, In the attention module, the Dropout algorithm is introduced as a regularizer; Before calculating the attention weight, a part of the elements in the attention score matrix are randomly set to negative infinity, which means that the Key is set as a drop object, and it is randomly discarded in the attention weight graph; The keys set to negative infinity are converted to 0 probability in the subsequent Softmax function calculation, which means that the model completely ignores these randomly discarded keys when calculating the correlation between a query and all keys.

9. The fine-grained food image classification method according to claim 1, wherein, In step 2, EfficientKAN replaces the fixed activation function of the traditional MLP with a learnable adaptive basis function based on the Kolmogorov-Arnold representation theorem; Each neuron of a KAN implements a decomposition by a learnable activation function; the output of the first layer of a KAN is: layer output is: ； wherein, is a learnable activation function, i.e. a parametrized B-spline basis function, is the number of neurons of the layer; is the input value of the neuron of the layer, is the input value of the neuron of the layer; EfficientKAN uses low-order polynomials or piecewise linear basis functions, as follows: ； wherein , and are learnable parameters, is a segmentation point, m represents the number of segments; denotes a piecewise linear basis function; The parameters of EfficientKAN are also shared and grouped, and the activation functions of adjacent neurons share part of the parameters, and the neurons are grouped, and the basis functions are shared within the group and learned independently between groups; Coefficient of basis function in activation function 、 L1 regularization is applied to promote sparsity; EfficientKAN replaces the MLP layer in BLOCK, and the improved formula is as follows: ； wherein denotes the input sequence, denotes the input sequence through an attention mechanism, denotes a layer normalization operation; uses a rational function as the base function of the EfficientKAN layer, i.e. a function on each edge is parameterized as a polynomial of degree , rational numbers on, is expressed as follows: ； wherein denotes a learnable scaling factor to adjust the output amplitude of the function, , , are set coefficients in the numerator polynomial, , , are set coefficients in the denominator polynomial.

10. A fine-grained food image classification system comprising a computer device; the computer device comprising a memory and a processor; wherein, The computer program is stored in the memory; characterized in that when the processor executes the computer program, the steps of the fine-grained food image classification method according to any one of claims 1 to 9 are implemented.

Images