A multi-class feature fusion-based food material image classification method
By employing a multi-category feature fusion method for food image classification, combining LBP, Gabor, and HSV features with deep features, the problem of manual dependence and model bias in food classification is solved, achieving efficient and accurate food classification and rapid data entry and exit.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI SUN CREATE ELECTRONICS
- Filing Date
- 2022-07-15
- Publication Date
- 2026-06-09
AI Technical Summary
In existing technologies, food classification relies on manual identification, and the classification results of a single model are prone to bias. Furthermore, when the image dataset is insufficient, it is difficult to avoid model overfitting, resulting in insufficient classification reliability and accuracy.
A multi-class feature fusion method is adopted, which combines LBP local binary features, Gabor wavelet features, and HSV color features with deep features, transfers weights from VGG16 and VGG19 pre-trained models, performs feature fusion and decision fusion, uses Euclidean distance to calculate the matching degree, and assigns weights to form a classification model.
It enables intelligent classification of ingredients, improving the reliability and accuracy of classification. It is suitable for situations where there are insufficient samples in the image dataset, and performs particularly well in cases with few samples and multiple categories, enabling rapid entry and exit of ingredients.
Smart Images
Figure CN115393844B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of food classification technology, specifically a food image classification method based on multi-category feature fusion. Background Technology
[0002] In existing technologies, the classification of goods and food items during transportation and warehousing mostly relies on human visual identification. This method depends on manpower, and staff must be familiar with all types of food. When classifying food items using image algorithms, it often depends on the classification results of a single model. Such methods cannot avoid the bias caused by insufficient training of a single model, and it is difficult to avoid the problem of model overfitting when the collected image dataset is insufficient. Summary of the Invention
[0003] The purpose of this invention is to provide a food image classification method based on multi-category feature fusion. While improving the reliability and accuracy of classification, it is suitable for situations where the image dataset has insufficient samples. This method does not have high requirements for the data to be processed and can realize the rapid entry and exit of food products. It has a good classification effect, especially in the case of few samples and multiple categories, so as to solve the problems mentioned in the background art.
[0004] The objective of this invention can be achieved through the following technical solutions:
[0005] A food image classification method based on multi-class feature fusion, the specific steps of which include:
[0006] Step 1: Obtain food images with similar backgrounds that do not overlap;
[0007] Step 2: Obtain three low-level features for each food image: LBP local binary features, Gabor wavelet features, and HSV color features.
[0008] Step 3: Transfer the weight parameters from two different pre-trained convolutional neural network models to the defined new model, and extract the deep features from the fully connected layer outputs of the two new models after training for feature fusion;
[0009] Step 4: By calculating the matching degree between the image to be retrieved and the sample images in the dataset, the deep feature classification results and low-level feature classification results are obtained. The results are then fused for decision-making, and a classification model suitable for the actual dataset is obtained by assigning different weights.
[0010] Furthermore, the number of duplicate food images required in step one should be less than 10 as needed.
[0011] Furthermore, the low-level features of the food images obtained in step two are optional. When there are no more than 20 types of food, only two low-level features can be obtained.
[0012] Furthermore, the pre-trained models used in step three are VGG16 and VGG19, and the feature fusion method is the add method.
[0013] Furthermore, the method for decision fusion of the results in step four is as follows:
[0014] Two pre-trained models, VGG16 and VGG19, were trained on similar image datasets. Their weight parameters were then transferred to the new models for training and fine-tuning. The two fully connected layers after the convolutional layers of the two new models were extracted. The two-dimensional feature maps calculated by the fully connected layers were converted into one-dimensional vectors. The output values of the corresponding layers were converted into one-dimensional feature vectors, thereby storing the features extracted from the image samples in the dataset in the feature library.
[0015] Furthermore, before making decisions and fusing them, the feature vectors of the represented image are first normalized using the L2 norm.
[0016] Furthermore, the matching degree in step four is calculated using Euclidean distance.
[0017] Compared with the prior art, the beneficial effects of the present invention are: the present invention realizes intelligent classification of food ingredients, solves the shortcomings of the current method of manually classifying food ingredients, and improves the corresponding automation process; while improving the reliability and accuracy of classification, it can be applied to situations where there are insufficient samples in the image dataset, and realizes rapid entry and exit of food ingredients, especially with a small number of samples and multiple categories, it also has a good classification effect. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is the flowchart of the method of the present invention;
[0020] Figure 2 This is a flowchart of the image feature extraction and classification process of the present invention. Detailed Implementation
[0021] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0022] like Figures 1 to 2 As shown, a food image classification method based on multi-class feature fusion includes the following steps:
[0023] Step 1: Obtain food images with similar backgrounds that do not overlap;
[0024] Step 2: Obtain three low-level features for each food image: LBP local binary features, Gabor wavelet features, and HSV color features.
[0025] Step 3: Transfer the weight parameters from two different pre-trained convolutional neural network models to the defined new model, and extract the deep features from the fully connected layer outputs of the two new models after training for feature fusion;
[0026] Step 4: By calculating the matching degree between the image to be retrieved and the sample images in the dataset, the deep feature classification results and low-level feature classification results are obtained. The results are then fused for decision-making, and a classification model suitable for the actual dataset is obtained by assigning different weights.
[0027] The different weights were determined through manual experiments, such as starting with 10%, and then experimenting with 10%, 20%, and so on, to obtain a classification model suitable for real-world datasets.
[0028] A dataset is a collection of all image samples. These image samples have undergone preprocessing, which includes image segmentation, image denoising, image enhancement, and grayscale transformation.
[0029] The number of images required in step one, after deduplication, can be less than 10.
[0030] In step two, the low-level features of the food images can be selected. When there are no more than 20 types of food, only two low-level features need to be obtained.
[0031] The pre-trained models used in step three are VGG16 and VGG19, and the feature fusion method is the add method, which ensures that the number of dimensions of image features does not increase, but the amount of information in each dimension increases. Therefore, compared with a single model, it does not increase the amount of computation generated by dimensionality reduction of feature vectors.
[0032] like Figure 2As shown, the method for decision fusion of the results in step four is as follows:
[0033] First, two pre-trained models, VGG16 and VGG19, are trained on similar image datasets. Then, their weight parameters are transferred to the new model, followed by further training and fine-tuning. The specific training and fine-tuning are common knowledge in this field, so they will not be described in detail. The two fully connected layers (FC1_16, FC2_16 and FC1_19, FC2_19) after the convolutional layers of the two new models are extracted respectively. After the fully connected layers are calculated, the two-dimensional feature maps are converted into one-dimensional vectors. The output values of the corresponding layers are converted into one-dimensional feature vectors, thereby storing the features extracted from the image samples in the dataset in the feature library.
[0034] Before decision fusion, the feature vectors of the represented image are first normalized using the L2 norm.
[0035] In step four, the matching degree is calculated using Euclidean distance, which refers to the Euclidean distance between the image to be retrieved and the feature vectors represented by the corresponding sample images in the dataset.
[0036] In one embodiment, the method for acquiring food images with similar and non-overlapping backgrounds in step one can be achieved using existing image acquisition methods. The goal is to ensure that the backgrounds of the acquired food images are similar and non-overlapping. Users can select the corresponding existing image acquisition methods and acquisition devices for image acquisition according to their actual needs. In certain scenarios, such as when the acquisition location is not fixed or the amount of food is small, flexibly applying existing acquisition methods will be more convenient and faster, and will not require high installation costs for acquisition equipment.
[0037] In another embodiment, the method for obtaining food images with similar backgrounds and no overlap in step one is as follows:
[0038] Set up a food collection area and install image acquisition devices within it. Use existing image acquisition devices as needed. When food arrives in the food collection area, acquire the corresponding initial image using the set image acquisition devices. Judge whether the initial image meets the requirements of similar background and no overlap. If it meets the requirements, use the initial image as the food image and proceed to acquire the image of the next food. If it does not meet the requirements, adjust the corresponding food and re-acquire the image until it meets the requirements.
[0039] The method for setting up the food collection area is as follows:
[0040] The system acquires historical ingredient entry and exit routes and creates ingredient route maps, including corresponding ingredient routes and surrounding building maps. It analyzes these maps to identify potential areas and marks them on the map. The current ingredient route map is then sent to the relevant management personnel for confirmation. Confirmed areas are designated as ingredient collection zones. Multiple potential areas or large regions can be selected. Management personnel determine the corresponding ingredient collection zones based on the recommended potential areas. Delineating collection zones solely through system analysis is somewhat limited and fails to account for subsequent user plans within each area. Confirmation by management personnel avoids frequent changes to collection zones, preventing resource waste and increased collection costs.
[0041] The method for analyzing food route maps is to build a corresponding analysis model based on a CNN or DNN network, manually set up a corresponding training set for training, and then analyze the food route map using the successfully trained analysis model to obtain the corresponding candidate range. The specific building and training process is common knowledge in this field, so it will not be described in detail.
[0042] The method for judging the initial image is as follows: a corresponding judgment model is established based on a CNN network or a DNN network, and the corresponding training set is set manually for training. The successfully trained analysis model is used to judge the initial image and obtain the judgment result, including whether it conforms or does not conform, and the reason for non-conformity when it does not conform.
[0043] The method for adjusting the corresponding ingredients is as follows:
[0044] Based on the corresponding judgment results, the reasons for non-compliance are obtained, and adjustments are made accordingly. Alternatively, manual adjustments can be made to the corresponding ingredients to reduce adjustment costs. Preferably, existing equipment capable of performing the corresponding functions can be set up, and the ingredients can be adjusted by controlling the equipment, which is more intelligent, but requires higher equipment costs. Users can choose the adjustment method according to their actual situation.
[0045] The working principle of this invention is as follows: Images of food ingredients with similar backgrounds and no overlap are acquired; three low-level features—LBP local binary features, Gabor wavelet features, and HSV color features—are obtained from each image; weight parameters are transferred from two different pre-trained convolutional neural network models to a newly defined model, and deep features from the fully connected layer outputs of the two trained new models are extracted and fused; the classification results using deep features are combined with the classification results using low-level features, and different weights are assigned experimentally to derive a classification model suitable for real-world datasets.
[0046] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.
Claims
1. A food image classification method based on multi-class feature fusion, characterized in that, The specific steps include: Step 1: Obtain food images with similar backgrounds that do not overlap; Step 2: Obtain three low-level features for each food image: LBP local binary features, Gabor wavelet features, and HSV color features. Step 3: Transfer the weight parameters from two different pre-trained convolutional neural network models to the defined new model, and extract the deep features from the fully connected layer outputs of the two new models after training for feature fusion; the pre-trained models are VGG16 and VGG19, and the feature fusion method is the add method; Step 4: By calculating the matching degree between the image to be retrieved and the sample images in the dataset, the deep feature classification result and the low-level feature classification result are obtained; the matching degree is calculated using Euclidean distance; The results are fused for decision-making, and a classification model suitable for real-world datasets is derived by assigning different weights. The method for integrating the results into a decision is as follows: Before performing decision fusion, the feature vectors of the image are first normalized using the L2 norm. Two pre-trained models, VGG16 and VGG19, were trained on similar image datasets. Their weight parameters were transferred to the new model for training and fine-tuning. The two fully connected layers after the convolutional layers of the two new models were extracted respectively. After computation by the fully connected layer, the two-dimensional feature map is converted into a one-dimensional vector. The output values of the corresponding layers are all converted into one-dimensional feature vectors, thereby storing the features extracted from the image samples in the dataset in the feature library. The method for obtaining food images with similar backgrounds and no overlap in step one is as follows: Set up a food collection area and set up an image acquisition device in the food collection area; when the food arrives in the food collection area, the corresponding initial image is obtained through the set image acquisition device; The initial image is evaluated to determine if it meets the requirements of similar backgrounds and no overlap. If it meets the requirements, the initial image is used as the food image and the image of the next food is acquired. If it does not meet the requirements, the corresponding food image is adjusted and the image is acquired again until it meets the requirements. The method for setting up the food collection area is as follows: Obtain historical food ingredient entry and exit routes, draw food ingredient route maps, including corresponding food ingredient routes and surrounding building maps, analyze the food ingredient route maps to obtain corresponding candidate areas, and mark the obtained candidate areas on the food ingredient route maps; send the current food ingredient route map to the corresponding management personnel for confirmation, and mark the confirmed areas as food ingredient collection areas. The method for analyzing the food route map is to build a corresponding analysis model based on a CNN network or a DNN network, manually set the corresponding training set for training, and analyze the food route map through the successfully trained analysis model to obtain the corresponding candidate range. The method for judging the initial image is as follows: a corresponding judgment model is established based on a CNN network or a DNN network, and the corresponding training set is set manually for training. The initial image is judged by the successfully trained analysis model to obtain the judgment result, including whether it conforms or does not conform, and the reason for non-conformity when it does not conform. The method for adjusting the corresponding ingredients is as follows: obtain the reasons for non-compliance based on the corresponding judgment results, and make adjustments based on the obtained reasons for non-compliance.
2. The food image classification method based on multi-class feature fusion according to claim 1, characterized in that, The number of duplicate food images required in step one should be less than 10, as needed.
3. The food image classification method based on multi-class feature fusion according to claim 1, characterized in that, In step two, the low-level features of the food images can be selected. When there are no more than 20 types of food, only two low-level features can be obtained.