Method, device, equipment and medium for determining food classification topological structure

Through the extraction and clustering analysis of food image image features, the topology of food classification is determined, which solves the problem of insufficient reliability of food classification in the existing technology, and improves the accuracy and reliability of food identification in smart kitchen appliances.

CN118691913BActive Publication Date: 2024-11-22HANGZHOU ROBAM APPLIANCES CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411196373.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-22
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The existing technology lacks reliability in food classification, especially the recognition accuracy of similar food ingredients and semi-finished products needs to be improved, resulting in poor application of image recognition technology in smart kitchen appliances.

Method used

By performing feature extraction, processing and clustering analysis of food ingredient images, the condensation hierarchical clustering algorithm and full link criteria are used to determine the food classification topology, and provide classification basis that conforms to machine vision logic, thereby improving the accuracy and reliability of food classification. .

Benefits of technology

It improves the accuracy and reliability of food classification based on machine vision, provides a theoretical basis for the classification model, and is suitable for food identification and processing in smart kitchen appliances.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691913B_ABST
    Figure CN118691913B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device, equipment and medium for determining a food classification topology structure, which relates to the field of data processing technology. The method comprises extracting features from target food images of corresponding food in each category to obtain feature vectors of the corresponding food, processing the feature vectors to obtain corresponding target feature vectors, and using a clustering algorithm to perform cluster analysis on each target feature vector to determine a food classification topology structure based on data clusters obtained by clustering and the hierarchical relationship between data clusters. The food classification topology structure provides a basis for food classification based on machine vision, thereby improving the reliability of food classification based on machine vision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for determining a food classification topology structure. Background Art

[0002] With the advancement of science and technology, smart kitchen appliances are increasingly entering people's lives and work. In the application process of smart kitchen appliances, the intelligent identification and classification of ingredients is one of the key tasks. For example, reliable identification of ingredients is a key factor to ensure the adaptability of smart cooking of cooking utensils, smart storage of refrigerators, and smart cleaning of cleaning equipment.

[0003] Currently, the category of food ingredients can be determined through image recognition; however, its reliability needs to be improved. Summary of the invention

[0004] The purpose of the present invention includes, for example, providing a method, device, equipment and medium for determining a food classification topology structure, so as to at least partially improve the reliability of food classification.

[0005] The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, a method for determining a food classification topology structure is provided, comprising:

[0007] Feature extraction is performed on target food images of corresponding food in each category to obtain feature vectors of corresponding food;

[0008] Processing the feature vector by a preset processing method to obtain a corresponding target feature vector;

[0009] A clustering algorithm is used to perform cluster analysis on each of the target feature vectors to determine the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters.

[0010] In an optional embodiment, the clustering analysis is performed on each of the target feature vectors using a clustering algorithm to determine the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters, including:

[0011] Based on the agglomerative hierarchical clustering algorithm and the full-link criterion, cluster analysis is performed on each of the target feature vectors to merge the target feature vectors that meet the similarity conditions into the same data cluster until the merging stop condition is reached to obtain the corresponding initial classification topology structure;

[0012] A food classification topology structure is determined based on the initial classification topology structure.

[0013] In an optional implementation, determining the food classification topology structure based on the initial classification topology structure includes:

[0014] The number of data clusters in the initial classification topology structure is reduced based on the distance differences between the data clusters, so as to determine the food classification topology structure based on the reduced data clusters and the hierarchical relationships between the data clusters.

[0015] In an optional embodiment, the method further comprises:

[0016] Based on each data cluster corresponding to the food classification topological structure, a corresponding classification model parameter is determined, where the classification model parameter indicates the number of classification models required for classification and / or indicates the model complexity of the classification model required for classification.

[0017] In an optional implementation manner, the area ratio of the image area occupied by the food in the target food image is not less than a preset area ratio threshold; and the feature extraction of the target food images of the corresponding food under each category is performed to obtain the feature vector of the corresponding food, including:

[0018] Inputting the target food image of the corresponding food under each category into a preprocessing model, wherein the preprocessing model includes a patch embedding and an encoding layer;

[0019] Based on Patch Embedding, the target food image is divided into a plurality of image blocks, wherein each of the image blocks is added with an image region of an edge portion of an adjacent image block in a length direction and a width direction of the target food image;

[0020] Each of the image blocks is encoded into a vector representation and input into a coding layer to obtain a feature vector of the corresponding food.

[0021] In an optional embodiment, the preprocessing model further includes a global average pooling layer;

[0022] The processing of the feature vector by a preset processing method to obtain a corresponding target feature vector includes:

[0023] Inputting each of the feature vectors corresponding to the target food image into the global average pooling layer, summing up each of the feature vectors and evenly dividing them into each of the feature vectors, to obtain a feature average vector representing the distribution of the inherent features of the corresponding food in the target food image;

[0024] A corresponding target feature vector is determined based on the feature average vector.

[0025] In an optional implementation, determining the corresponding target feature vector based on the feature average vector includes:

[0026] The feature average vector is normalized and dimensionally reduced to obtain a corresponding target feature vector.

[0027] In a second aspect, the present invention provides a device for determining a food classification topology structure, comprising:

[0028] An information acquisition module is used to extract features of target food images of corresponding food in each category to obtain feature vectors of corresponding food;

[0029] The information processing module is used to process the feature vectors by a preset processing method to obtain corresponding target feature vectors; a clustering algorithm is used to perform clustering analysis on each of the target feature vectors to determine the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters.

[0030] In a third aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for determining a food classification topology structure as described in any one of the aforementioned embodiments is implemented.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program, and when the computer program is executed, the electronic device where the computer-readable storage medium is located is controlled to execute the method for determining the food classification topology structure described in any one of the aforementioned embodiments.

[0032] The beneficial effects of the embodiments of the present invention include, for example: providing a method, device, equipment and medium for determining the food classification topology structure, by extracting features of the target food images of the corresponding food in each category obtained by classification based on prior knowledge, and obtaining the food classification topology structure that conforms to the machine vision's cognitive logic for food based on image feature processing and cluster analysis, thereby providing a basis for food classification based on machine vision, thereby improving the reliability of food classification based on machine vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A schematic flow chart of a method for determining a food classification topology structure provided by an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a sub-step flow chart of S110 provided in an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of a sub-step flow chart of S120 provided in an embodiment of the present invention;

[0037] Figure 4 A schematic diagram of the logical architecture of a preprocessing model provided by an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of a sub-step flow chart of S130 provided in an embodiment of the present invention;

[0039] Figure 6 A schematic diagram of a sub-step flow chart of S131 provided in an embodiment of the present invention;

[0040] Figure 7 A schematic diagram of food classification provided by an embodiment of the present invention;

[0041] Figure 8 A schematic diagram of the overall process of a method for determining a food classification topology structure provided by an embodiment of the present invention;

[0042] Fig. 9 A schematic diagram of a food material hierarchical classification result provided by an embodiment of the present invention;

[0043] Fig.10 A schematic diagram of model selection corresponding to a food hierarchical classification result provided by an embodiment of the present invention;

[0044] Fig.11 A schematic diagram of food classification based on image features provided by an embodiment of the present invention;

[0045] Fig.12 A schematic diagram of the architecture of a device for determining a food classification topology structure provided by an embodiment of the present invention;

[0046] Fig.13 A structural block diagram of an electronic device provided by an embodiment of the present invention.

[0047] Icons: 11-electronic device; 111-controller; 112-ROM; 113-RAM; 114-bus; 115-I / O interface; 116-input unit; 117-output unit; 118-storage unit; 119-communication unit; 140-device for determining food classification topology structure; 141-information acquisition module; 142-information processing module. DETAILED DESCRIPTION

[0048] Nowadays, with the development of society, smart kitchen appliances such as smart cooking utensils, smart refrigerators, smart cleaning equipment, etc. are appearing more and more in people's work and life to realize smart cooking, storage, and cleaning of ingredients.

[0049] Given that different ingredients have different cooking methods, processing methods, storage methods, and cleaning methods, reliable classification and identification of ingredients during the processing process is a key factor in ensuring the compatibility of corresponding smart kitchen appliances.

[0050] At present, common food classification methods include code scanning, radio frequency identification, image recognition, etc. Among them, code scanning and radio frequency identification are limited in their application scenarios and cannot classify food without codes or labels. Image recognition is widely used in food classification scenarios because of its wide application scenarios.

[0051] Research has found that with the development of the imaging field and artificial intelligence disciplines, image recognition technology has been applied to various fields. A more important application in the smart kitchen appliance industry is to obtain the image data of the food and apply image recognition technology to obtain the corresponding food category.

[0052] Food culture has a long history and a wide variety of ingredients. For example, according to the "People's Republic of China Health Industry Standard WS / T 464-2015-Food Composition Data Expression Specification", food is divided into 20 categories according to its raw material properties, and each category is divided into several subcategories and subcategories.

[0053] Since the principle of image recognition technology is to use features extracted from images to identify food ingredients, if the images of different categories of food ingredients classified based on prior knowledge are similar, it is easy to misjudge the category when classifying food ingredients based on machine vision. In addition, with the development of the food industry, some semi-finished products such as original chicken nuggets and crispy pork have been born one after another. For semi-finished products, it is easy to have both in the classification of food ingredients based on machine vision. Therefore, in the existing technology, the reliability of food ingredient classification based on image recognition needs to be improved.

[0054] Based on the above research, an embodiment of the present invention provides a method, device, equipment and medium for determining the topological structure of food classification. Based on the characteristics of image data, image feature extraction, processing and cluster analysis are performed on different categories of food obtained by prior knowledge classification to obtain a food classification topological structure that conforms to the machine vision's cognitive logic for food, providing a theoretical basis for food classification based on machine vision, so as to improve the efficiency and reliability of food classification based on image recognition technology.

[0055] The defects existing in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present invention for the above problems below should all be the contributions made by the inventors in the invention process.

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0059] In the description of the present invention, it should be noted that the terms “first”, “second”, etc., if used, are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0060] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0061] See also Figure 1 , is a flow chart of a method for determining a food classification topology structure provided in an embodiment of the present invention, the method for determining a food classification topology structure includes S110, S120 and S130.

[0062] S110, performing feature extraction on the target food image of the corresponding food in each category to obtain a feature vector of the corresponding food.

[0063] S120, processing the feature vector by a preset processing method to obtain a corresponding target feature vector.

[0064] S130, performing cluster analysis on each of the target feature vectors using a clustering algorithm to determine a food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationships between the data clusters.

[0065] For different categories of ingredients obtained through prior knowledge classification, based on image feature extraction, processing, and cluster analysis, an ingredient classification topological structure that conforms to the machine vision's cognitive logic for ingredients is obtained. Compared with the ingredient categories obtained through prior knowledge classification, the classification standards in the ingredient classification topological structure are more compatible with the visual understanding of devices such as computers, so that when the device classifies ingredients based on image recognition, the similarity of each category of ingredients can be derived based on the ingredient classification topological structure, thereby providing a theoretical basis for ingredient classification based on machine vision, and thus improving the accuracy and reliability of ingredient classification based on machine vision.

[0066] Each category in S110 refers to an initial ingredient category, which is an ingredient classification based on human prior knowledge, and the initial ingredient category can be flexibly set. For example, based on the specific requirements of different projects, the ingredient categories that need to be image recognized and are known based on prior knowledge can be determined, wherein the corresponding ingredients under each category can be configured with corresponding recognition requirements and application scenarios for easy application and recognition.

[0067] The target food images of the corresponding food in each category can be obtained flexibly. For example, the images of the corresponding food in each category can be collected. In order to ensure a comprehensive and reliable representation of the food, the sources of the images can be multiple. For example, it can be shot on site, screened from existing data sets, or collated from user-generated content (UGC). For another example, it can include multiple images of food in different scenes and under different processing conditions.

[0068] Based on the specific requirements of different projects, the number of images collected for each ingredient can be flexibly selected. For example, for projects with high recognition accuracy requirements, the number of images collected for each ingredient can be large. For projects with average recognition accuracy requirements, the number of images collected for each ingredient can be moderate. For projects with low recognition accuracy requirements, the number of images collected for each ingredient can be small.

[0069] For example, in a certain project, the number of images collected for each ingredient may be more than 100, and these images cover the different processing states that the ingredient may encounter in the project, so as to facilitate subsequent reliable identification and processing.

[0070] In one implementation, all images collected for each food material constitute an image data set of the food material, and each image in the image data set can be directly used as a target food material image of the food material.

[0071] In another implementation, in order to improve the reliability of subsequent image feature extraction, processing, and analysis, each image in the image data set may be preprocessed, and the preprocessed image may be used as the target food image.

[0072] The preprocessing of images can be set flexibly. For example, you can filter each image in the image data set and remove images that do not meet the requirements of clarity, angle, etc. to reduce interference factors in subsequent processing. For another example, you can resize each image so that each image is in the same size range. For another example, you can crop each image so that the corresponding food is roughly located in the center of the image and meets the area ratio requirements to reduce interference and improve the convenience of image processing.

[0073] Exemplarily, the present embodiment provides an optional implementation of preprocessing each image in an image data set separately to obtain a target food image, including: for each image in each image data set, determining whether the area ratio of the image area occupied by the corresponding food in the image is less than a preset area ratio threshold; if so, cropping the image so that the area ratio of the image area occupied by the food in the image is not less than the preset area ratio threshold.

[0074] This embodiment provides another optional implementation of preprocessing each image in the image data set to obtain a target food image, including: for each image in each image data set, determining whether the area ratio of the corresponding food in the image reaches the ratio threshold, and whether the area where the corresponding food is located is located in the target area in the image. If the area ratio of the corresponding food is lower than the ratio threshold, or the area where the corresponding food is located is outside the target area in the image, the image is cropped with the area where the corresponding food is located as the center, and the cropped image is adjusted to a set size, so that in the image of the set size, the area ratio of the corresponding food reaches the ratio threshold, and the area where the corresponding food is located is located in the target area in the image.

[0075] Among them, the preset area ratio threshold and proportion threshold can be set flexibly, for example, it can be above 60%. Accordingly, preprocessing each image in the image data set can ensure that the area proportion of the corresponding food in the preprocessed image is higher than 60%.

[0076] In order to reduce the interference of background or other irrelevant elements, the target area can be centered. Accordingly, each image in the image data set is preprocessed so that the area where the corresponding food is located in the preprocessed image is located in the middle of the image.

[0077] In order to improve the convenience of subsequent image processing, the images can also be adjusted to a set size through preprocessing, so that after preprocessing, the target food images obtained are all of the set size, and in the images of the set size, the area ratio of the food reaches a ratio threshold (area ratio threshold), and the area where the food is located is located in the target area in the image.

[0078] There are many ways to extract features from target food images to obtain feature vectors of the corresponding food. For example, traditional image processing methods can be used, such as extracting texture features of images based on gray-level co-occurrence matrix (GLCM). For another example, deep learning models can be used to extract image features. Among them, deep learning models can include generative adversarial networks (GANs) such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), and autoencoders to extract image features based on the model. For another example, models based on transformer architecture such as Vision Transformer (ViT) can be used for image feature extraction.

[0079] Please refer to Figure 2 , provides a method for extracting feature vectors of target food images based on a preprocessing model for image feature extraction, wherein the preprocessing model includes Patch Embedding and a coding layer, including S111, S112 and S113.

[0080] S111, inputting the target food image of the corresponding food in each category into the preprocessing model.

[0081] S112, dividing the target food image into multiple image blocks based on Patch Embedding.

[0082] Each of the image blocks is added with an image area of ​​an edge portion of an adjacent image block in both the length direction and the width direction of the target food image.

[0083] S113, encoding each of the image blocks into a vector representation and inputting the vector into a coding layer to obtain a feature vector of the corresponding food.

[0084] After the characteristic vector of the food is obtained based on S110, in S120, the characteristic vector is processed by a preset processing method to obtain the corresponding target characteristic vector. The implementation method can be flexibly selected. For example, each characteristic vector can be summarized and analyzed, such as summarizing the characteristic vectors corresponding to each food, and analyzing each characteristic vector obtained by summarizing each food to obtain the target characteristic vector corresponding to each food, thereby reducing the processing complexity of each characteristic vector. This processing function can be implemented independently of the preprocessing model, or it can be integrated into the preprocessing model.

[0085] Taking the target feature vector obtained based on the preprocessing model as an example, in order to avoid the high processing complexity due to the existence of a large number of feature vectors, the preprocessing model can also include a global average pooling layer. After the encoding layer extracts the feature vector of each image block, the feature vector is input into the added global average pooling layer to reduce the complexity of processing.

[0086] Accordingly, please refer to Figure 3 In S120, the method of processing the feature vector by a preset processing method to obtain a corresponding target feature vector may include S121 and S122.

[0087] S121, input each of the feature vectors corresponding to the target food image into the global average pooling layer, sum up each of the feature vectors and evenly divide them into each of the feature vectors, and obtain a feature average vector representing the distribution of inherent features of the corresponding food in the target food image.

[0088] S122: Determine a corresponding target feature vector based on the feature average vector.

[0089] In this embodiment, the preprocessing model of ViT is used as an example. Figure 4 , the preprocessing model can include patch embedding, such as Figure 4 The image embedding layer includes the linear projection of flattened patches (Linear Projection of Flattened Patches), the transformer encoding layer (Transformer Encoder, also known as the Transformer encoding layer), the global average pooling layer (Adaptive Avg pool1d), etc. Among them, the Transformer encoding layer includes the multi-head attention mechanism and the neural network.

[0090] In the preprocessing model Figure 4In the case of the architecture shown, the feature average vector can be obtained in the following way: Each target food image is divided into multiple image blocks based on Patch Embedding. Among them, each image block is added with the image area of ​​the edge part of the image block adjacent to it in the length direction and width direction of the target food image to form an overlapping area, wherein the size of the overlapping area can be a set pixel. Each image block is encoded as a vector representation and input into the Transformer encoding layer, and feature extraction is performed based on the self-attention mechanism and neural network to obtain the feature vector of the corresponding food. For each target food image, the corresponding feature vectors are input into the global average pooling layer, and the feature vectors are summed and averaged to obtain the feature average vector that characterizes the distribution of the inherent features of the corresponding food in the target food image.

[0091] Among them, ViT is a model that applies the Transformer architecture to image recognition tasks. Unlike traditional CNNs, ViT processes pixel relationships in images through a self-attention mechanism and can process global information, which is in contrast to the local receptive field of CNNs. In addition, the model parameters pre-trained by ViT on large-scale image datasets such as ImageNet-21K make the obtained features more robust.

[0092] By modifying the Patch Embedding algorithm in the preprocessing model, compared with the existing Patch Embedding algorithm that divides the image into a fixed number of grid images such as 4*4, in this embodiment, considering that the area where the corresponding ingredients in the target ingredient image are located is located in the middle area of ​​the image, and the area ratio of the image area occupied by the ingredients is not less than the preset area ratio threshold, an overlapping area of ​​set pixels such as 20 pixels is set in the length and width directions when performing image segmentation based on the Patch Embedding algorithm. Based on the overlapping area, similar image content can be obtained from multiple angles, thereby increasing the spatial redundancy of the features and helping the preprocessing model to learn richer feature representations. In addition, by using overlapping areas, the influence of boundary pixels will be extended to multiple patches (image blocks), thereby reducing the boundary effect and making the features smoother.

[0093] The feature vectors obtained from each image block based on the Transformer coding layer include the spatial position of each image block. The spatial position may include the spatial arrangement and distribution of pixels in the image. The position of each pixel in the image together with the pixels around it constitute the spatial position (also called spatial structure) of the image. The spatial position contains many important features of the image, such as edges, textures, shapes, and spatial relationships of object parts. After the Transformer coding layer extracts features from the image block, a large number of feature vectors, such as more than 100,000, will be generated.

[0094] A large number of feature vectors will increase the risk of overfitting of the subsequent preprocessing model. Therefore, after the Transformer encoding layer extracts the feature vector, the feature vector is input into the additional global average pooling layer to reduce the complexity of the preprocessing model.

[0095] Based on the global average pooling layer, the feature vectors corresponding to each target food image are averaged, thereby effectively integrating multiple feature vectors of the entire target food image into a feature average vector, making the output feature average vector more global.

[0096] Considering that in the preprocessing model, each target food image is divided into multiple image blocks, and a feature vector is extracted for each image block, the amount of data is large. By averaging the feature vectors corresponding to each image block as the feature average vector of the entire target food image, the amount of feature data is effectively reduced, thereby reducing the complexity of subsequent processing of the preprocessing model.

[0097] Based on the preprocessing model, each target food image is processed according to the above process, and the feature average vector reflecting the global situation of the target food image can be extracted.

[0098] like Figure 4 As shown, the preprocessing model may also include commonly used position embedding (Position Embedding), normalization (Norm), fast Fourier transform (Fast Fourier Transform, FFT), classification tag (Cat cls_tokens), etc., and the corresponding content will not be described here.

[0099] When there is one target food image for each food, the inherent feature distribution of the food may be reflected based on the feature average vector corresponding to the target food image.

[0100] When each food corresponds to an image data set, and each image data set includes multiple target food images, given that each food image data set includes a large number of target food images, for each food, feature extraction is performed on each target food image in the image data set based on the preprocessing model, and more feature average vectors are extracted. In order to further reduce the complexity of processing, the feature average vectors corresponding to each target food image in each image data set can be further summed up, and then the sum can be evenly divided into each feature average vector, so as to obtain a feature average vector that can reflect the inherent feature distribution of each food.

[0101] After obtaining the feature average vector reflecting the inherent feature distribution of each food ingredient, the method for determining the target feature vector based on the feature average vector can be flexibly selected.

[0102] For example, given that the feature average vector contains many important features such as the edge, texture, shape, and spatial relationship of the object parts of the image, the data dimensions are large and the processing volume is large, in order to reduce the data complexity, the feature average vector can be subjected to data dimensionality reduction to further reduce the processing complexity. For another example, in order to eliminate the magnitude differences between the features of multiple ingredients, the feature average vector of each ingredient can be standardized. For another example, the feature average vector can be subjected to standardization and dimensionality reduction to obtain the corresponding target feature vector.

[0103] Among them, data dimensionality reduction can be implemented flexibly. For example, the feature average vector can be reduced in dimension based on the principal component analysis algorithm (PCA) to reduce the complexity of the feature average vector while retaining the main direction of data change, ensuring the structural integrity of the data after dimensionality reduction, so as to improve the efficiency and accuracy of subsequent clustering analysis, and make the determination of the food classification topology structure more efficient and practical.

[0104] Based on data dimensionality reduction, the feature average vector can be reduced to the target dimension, and the target dimension can be set flexibly. For example, the performance of the hardware device that executes the food classification topology structure determination solution and the accuracy requirements in each scenario can be adaptively selected. For example, if the hardware device performance is strong and the accuracy requirements are high, the target dimension can be set to be relatively high. If the hardware device performance is average and the accuracy requirements are average, the target dimension can be set to be relatively moderate. If the hardware device performance is weak and the accuracy requirements are low, the target dimension can be set to be relatively low.

[0105] The standardization process can be flexibly selected. For example, the average value of each feature data in each dimension can be subtracted from each feature average vector, and then divided by the standard deviation of the feature average vector to ensure that all features after standardization have zero mean and unit variance, that is, the mean of each feature is 0 and the variance is 1.

[0106] After the feature average vector is processed by data standardization, each indicator is in the same order of magnitude, so that a comprehensive comparative evaluation can be performed. The principle of standardization includes: subtracting the mean from the value and then dividing it by the standard deviation.

[0107] The principle of zero-meaning (also called centering) involves subtracting the mean from the value.

[0108] Unit variance is a measure of the dispersion of values ​​in a data set. When the variance of a data set is 1, it means that the standard deviation of the values ​​in the data set is also 1, indicating that the values ​​in the data set are centered on the mean and are evenly dispersed within a unit standard deviation. In order to make the variance of a data set 1, each value is usually divided by the original standard deviation of the data set.

[0109] Through centering and standardization, data with a mean of 0 and a standard deviation of 1 that obeys the standard normal distribution are obtained. Through data centering and standardization, errors caused by different dimensions, self-variation or large numerical differences are eliminated in regression analysis. Based on this processing, the target feature vector is obtained, which can improve the reliability of subsequent food classification.

[0110] After obtaining each target feature vector based on S120, in S130, a clustering algorithm is used to perform cluster analysis on each target feature vector, and a method for determining the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters can be flexibly selected, as long as a classification topology structure that characterizes the similarity of each type of food can be obtained. For example, a hierarchical clustering algorithm, a K-earest Neighbor (KNN) classification algorithm, etc. can be selected for cluster analysis.

[0111] In this embodiment, the clustering analysis principle is explained by taking the selection of an agglomerative hierarchical clustering algorithm to perform clustering analysis on each target feature vector as an example.

[0112] like Figure 5 As shown, a method for implementing S130 is provided, including S131 and S132.

[0113] S131, performing cluster analysis on each of the target feature vectors based on an agglomerative hierarchical clustering algorithm and a full-link criterion, so as to merge the target feature vectors that meet the similarity conditions into the same data cluster, until a merging stop condition is reached, so as to obtain a corresponding initial classification topology structure.

[0114] S132: Determine the food classification topology structure based on the initial classification topology structure.

[0115] Please refer to Figure 6 , a method for clustering analysis of each target feature vector based on an agglomerative hierarchical clustering algorithm and a full-link criterion in S131 is provided, including S1311, S1312 and S1313.

[0116] S1311, taking each target feature vector as a cluster, starting with each cluster, and calculating the distance between the clusters.

[0117] S1312: Determine the combination similarity between the clusters based on the distance.

[0118] S1313, using the full-link criterion, clusters whose similarity reaches a preset threshold are merged into the same data cluster until a merging stop condition is reached, thereby obtaining an initial classification topology structure.

[0119] In terms of similarity measurement, the measurement method can be flexibly selected. For example, distance, such as Euclidean distance, can be selected as the similarity measurement standard to comprehensively consider the similarity between different target feature vectors. The full link criterion is adopted to ensure the connectivity of the clusters during the merging process.

[0120] The merging stop condition can be flexibly set, for example, it can be set when all data points are classified into the same category, or when a preset number of data clusters is reached.

[0121] Taking the merging stop condition that all data points are classified into the same category as an example, after the initial classification topology structure is obtained through cluster analysis, the number of data clusters can be adjusted to obtain the desired food classification topology structure.

[0122] For example, in order to reduce the complexity of the topological structure, the number of data clusters in the initial classification topological structure may be reduced based on the distance difference between the data clusters, so as to determine the food classification topological structure based on the reduced data clusters and the hierarchical relationship between the data clusters.

[0123] For another example, a quantity range can be preset for each scenario. After obtaining the initial classification topology structure, determine whether the number of data clusters in the initial classification topology structure is within the preset quantity range. If so, directly determine the initial classification topology structure as the food classification topology structure. If not, adjust the preset threshold in S1313, and re-merge clusters based on the adjusted preset threshold until the number of data clusters in the latest initial classification topology structure is within the preset quantity range, and determine the latest initial classification topology structure as the food classification topology structure.

[0124] After obtaining the food classification topology, the similarity of each type of food classified based on prior knowledge when classified based on machine vision can be known based on the distance between each data cluster in the food classification topology, so as to determine the corresponding classification model parameters when classifying food based on machine vision based on the similarity. For example, the number of classification models required for classification can be indicated, and / or the model complexity of the classification model required for classification can be indicated. Exemplarily, the higher the similarity indicated in the food classification topology, the higher the model complexity of the required classification model.

[0125] In order to more clearly explain the implementation principle of this embodiment, the following scenario is taken as an example to illustrate the solution for determining the food classification topology structure.

[0126] See also Figure 7, provides an ingredient classification diagram including multiple known categories of ingredients obtained based on prior knowledge classification. These multiple ingredients, which have been classified based on prior knowledge, have different corresponding cooking methods, processing methods, storage methods, cleaning methods, etc. Therefore, in order to ensure the adaptability of smart kitchen appliances when processing different types of ingredients, in the process of intelligent processing of various ingredients, it is necessary to determine the appropriate classification model configuration strategy to reliably identify the types of each ingredient.

[0127] Given that the difficulty of identifying different types of food ingredients through machine vision varies, for a variety of food ingredients, it is key to reasonably determine the number of classification models to be selected and the accuracy requirements of each classification model to ensure the reliability of food type identification during the processing of a variety of food ingredients.

[0128] In order to provide a theoretical basis for the selection of classification models, please refer to Figure 8 , this embodiment provides a schematic diagram of the overall process of a method for determining a food classification topology structure.

[0129] like Figure 8 As shown in the figure, the types of food that need to be classified by machine vision are determined as follows: Figure 7 As an example, in the process of food classification, the food types collected are Figure 7 There are more than 100 images of each food in different scenes and processing states.

[0130] The collected images are preprocessed separately by cropping and resizing so that the food area accounts for more than 60% of each preprocessed image (target food image), and the food area is kept centered as much as possible.

[0131] Based on the preprocessing model using ViT, image features are extracted from each target food image obtained after preprocessing. When performing image segmentation based on the preprocessing model, an overlapping area of ​​20 pixels is set in the length and width directions. After the Transformer encoding layer of the preprocessing model extracts feature vectors for each segmented image block, based on the global average pooling layer, for each target food image, the feature vectors corresponding to each image block of the target food image are summed up and averaged to obtain the feature average vector corresponding to each target food image.

[0132] For each food ingredient, the feature average vectors of more than 100 target food ingredient images corresponding to it are summed up and averaged to obtain the feature average vector reflecting the inherent feature distribution of each food ingredient. Based on the feature average vectors corresponding to each food ingredient, each feature average vector is standardized and the PCA algorithm is used to reduce the dimension of the data to 10, thereby obtaining the target feature vector corresponding to each food ingredient.

[0133] The agglomerative hierarchical clustering algorithm is used to perform cluster analysis on each target feature vector to obtain the hierarchical classification result (initial classification topology structure).

[0134] like Fig. 9 As shown in FIG. 1 , it is a schematic diagram of a classification topology structure, wherein the horizontal axis is various ingredients, such as the corresponding labels of various ingredients, and the corresponding labels can be flexibly selected, such as A1 to An, n is a positive integer greater than 1, and can also be the name of the ingredient, such as pinyin, etc. The vertical axis is distance.

[0135] based on Fig. 9 It can be seen that each ingredient is divided into several subcategories and subcategories of subcategories. The hierarchical clustering algorithm regards the points of ingredient features as "trees" and iteratively merges the two "trees" with the closest distance to form a new "tree" until the merging termination condition is reached, thereby constructing a hierarchical tree. In the formed hierarchical tree, the height of the small trees at different levels represents the distance between the two points (categories). Obviously, the closer the distance between two points, the higher their similarity, which provides a basis for selecting a classification model in the application process. In short, by comparing the distance differences between different trees in the same level, the performance requirements of the classification model can be evaluated. The greater the distance difference, the lower the requirements for the performance of the classification model. Conversely, the smaller the distance difference, the higher the requirements for the performance of the classification model.

[0136] Since each node of the hierarchical tree can represent a classification model, and the more levels there are, the more classification models are required. Fig. 9 Based on this, the number of data clusters can be adjusted to achieve the desired classification effect. In this embodiment, the final number of classification models is determined by setting the minimum distance (preset threshold). For example, when the minimum distance is set to 30, Fig. 9 As shown in the dashed line, the trees below the dashed line will be classified into one category and will not be further classified. After similar processing, the final classification diagram can be obtained.

[0137] like Fig.10 and Fig.11 As shown in the figure, it is a schematic diagram of the adjusted classification topology and the corresponding food classification diagram. Figure 7For ingredients of known categories based on prior knowledge, the method for determining the ingredient classification topology structure in this embodiment is adopted. When classifying ingredients based on machine vision, only 8 classification models are needed to perform category division, which is consistent with the corresponding number of models.

[0138] Based on the solution in this embodiment, the food classification topology structure is obtained, so that in the selection of the classification model, by comparing the distances between different trees in the classification topology structure, the required model accuracy, equipment performance, etc. can be determined. Fig.10 For example, the distances between models 1, 3, and 6 are large, so a classification model with lower performance requirements can be selected to perform the classification task. Model 2 has relatively high requirements, while models 4, 5, 7, and 8 have even higher performance requirements. By providing a theoretical basis for the selection of classification models, the accuracy and reliability of food classification based on image recognition are improved.

[0139] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a device for determining a food component topology structure is provided below. Fig.12 , Fig.12 A functional module diagram of a device 140 for determining a food classification topology structure provided in an embodiment of the present invention. The device 140 for determining a food classification topology structure can be applied to Fig.13 It should be noted that the basic principle and technical effects of the device 140 for determining the food classification topology structure provided in this embodiment are similar to those of the above-mentioned Figure 1 The method embodiments shown are the same. For the sake of brief description, the parts not mentioned in this embodiment can be referred to the above Figure 1 The device 140 for determining the food classification topology structure includes an information acquisition module 141 and an information processing module 142 .

[0140] The information acquisition module 141 is used to extract features from the target food images of the corresponding food in each category to obtain feature vectors of the corresponding food.

[0141] The information processing module 142 is used to process the feature vectors by a preset processing method to obtain corresponding target feature vectors; and to perform cluster analysis on each target feature vector using a clustering algorithm to determine the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters.

[0142] The method, device, equipment and medium for determining the food classification topology structure provided by the embodiment of the present invention provide a method for determining the food classification topology structure, based on the initial food category (obtained by prior knowledge classification), multiple food images of corresponding food in each category are obtained, and the multiple food images are preprocessed to obtain multiple target food images of corresponding food in each category, and the area ratio of the food in the target food image to the image area is not less than the preset area ratio threshold. Feature extraction is performed on the target food image to obtain multiple feature vectors of the corresponding food in each category, and the multiple feature vectors of the corresponding food are summed up and averaged to obtain the feature average vector corresponding to each food. In order to eliminate the magnitude difference between features of different categories and reduce the complexity of the data, while retaining the main change direction of the data, each feature average vector is standardized and dimensionally reduced to obtain the target feature vector. Cluster analysis is performed on each target feature vector based on the agglomerative hierarchical clustering algorithm and the full link criterion to merge the target feature vectors that meet the similarity conditions into the same data cluster until no new data clusters are generated to obtain the corresponding initial classification topology. The number of data clusters in the initial classification topology structure is reduced based on the distance difference between data clusters, and the final food classification topology structure is determined based on the reduced data clusters and the hierarchical relationship between these data clusters. The food classification topology structure conforms to the cognitive logic of machine vision for food, and provides a theoretical basis for the selection of classification models in the process of food classification based on machine vision, thereby improving the accuracy and reliability of food classification.

[0143] Based on the scheme in this embodiment, the problem of blurred classification boundaries of food ingredients and their derivatives in the process of classification based on image recognition is improved, making the classification clearer, and providing a theoretical basis for the design of classification groups of food ingredients such as soybeans, soy milk, milk, tofu, etc. It provides a theoretical basis for the selection of classification models. For example, the number of classification models can be determined based on the classification topology, and a standard is provided for determining the selection and design of classification models. For example, a classification model is selected based on the distance between different food ingredients in the same data cluster. When the distance between the food ingredients is small, the selection requirements for the classification model will be higher to ensure the reliability of food classification.

[0144] See also Fig.13 , is a structural block diagram of an electronic device 11 for executing a method for determining a food classification topology structure provided in an embodiment of the present invention. The components, their connections and relationships, and their functions shown in this embodiment are merely examples and are not intended to limit the implementation of the embodiments of the present invention described and / or required in this embodiment.

[0145] like Fig.13As shown, the electronic device 11 includes a controller 111 and a memory. There is at least one controller 111, and the memory is such as a read-only memory (ROM112), a random access memory (RAM113), etc. Among them, the memory stores computer executable instructions that can be executed by at least one controller 111, and the controller 111 can perform various appropriate actions and processes according to the computer executable instructions stored in the read-only memory (ROM112) or the computer executable instructions loaded from the storage unit to the random access memory (RAM113). In RAM113, various programs and data required for the operation of the electronic device 11 can also be stored. The controller 111, ROM112 and RAM113 are connected to each other through a bus 114. The input / output interface (I / O interface 115) is also connected to the bus 114.

[0146] Multiple components in the electronic device 11 are connected to the I / O interface 115, including: an input unit 116, such as a keyboard, a mouse, a touch screen, etc.; an output unit 117, such as various types of displays, speaker components, etc.; a storage unit 118, such as a disk, an optical disk, etc.; and a communication unit 119, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 119 allows the electronic device 11 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The controller 111 can be various general and / or special processing components with processing and computing capabilities. Some examples of the controller 111 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The controller 111 performs the various methods and processes described above, such as a method for determining a food classification topology.

[0147] In some embodiments, the method for determining the food classification topology structure may be implemented as a computer executable instruction, which is tangibly contained in a computer readable storage medium, such as a storage unit. In some embodiments, part or all of the computer executable instructions may be loaded and / or installed on the electronic device 11 via the ROM 112 and / or the communication unit 119. When the computer executable instructions are loaded into the RAM 113 and executed by the controller 111, one or more steps of the method for determining the food classification topology structure described above may be performed. Alternatively, in other embodiments, the controller 111 may be configured to execute the method for determining the food classification topology structure in any other appropriate manner (e.g., by means of firmware).

[0148] Various implementations of the systems and techniques described above can be realized in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof.

[0149] These various embodiments may include being implemented in one or more computer programs that are executable and / or interpretable on a programmable system that includes at least one programmable processor. The programmable processor may be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] The computer executable instructions for implementing the method of the embodiment of the present invention can be written in any combination of one or more programming languages. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer executable instructions, when executed by the processor, implement the functions / operations specified in the flow chart and / or block diagram. The computer executable instructions can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0151] In the context of embodiments of the present invention, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device.

[0152] Computer readable storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of computer readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0153] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the embodiments of the present invention can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions of the embodiments of the present invention can be achieved, and this document does not limit this.

[0154] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for determining a food classification topology structure, characterized in that: include: Feature extraction is performed on target food images of corresponding food in each category to obtain feature vectors of corresponding food; Processing the feature vector by a preset processing method to obtain a corresponding target feature vector; Using a clustering algorithm to perform cluster analysis on each of the target feature vectors, so as to determine the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters; Based on each data cluster corresponding to the food classification topological structure, a corresponding classification model parameter is determined, where the classification model parameter indicates the number of classification models required for classification and / or indicates the model complexity of the classification model required for classification.

2. The method for determining the food classification topology structure according to claim 1, characterized in that: The clustering algorithm is used to perform cluster analysis on each of the target feature vectors to determine the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters, including: Based on the agglomerative hierarchical clustering algorithm and the full-link criterion, cluster analysis is performed on each of the target feature vectors to merge the target feature vectors that meet the similarity conditions into the same data cluster until the merging stop condition is reached to obtain the corresponding initial classification topology structure; A food classification topology structure is determined based on the initial classification topology structure.

3. The method for determining the food classification topology structure according to claim 2, characterized in that: The determining of the food classification topology structure based on the initial classification topology structure includes: The number of data clusters in the initial classification topology structure is reduced based on the distance differences between the data clusters, so as to determine the food classification topology structure based on the reduced data clusters and the hierarchical relationships between the data clusters.

4. The method for determining the food classification topology structure according to claim 1, characterized in that: The area ratio of the food in the target food image to the image area is not less than a preset area ratio threshold; the feature extraction of the target food images of the corresponding food in each category is performed to obtain the feature vector of the corresponding food, including: Inputting the target food image of the corresponding food in each category into a preprocessing model, wherein the preprocessing model includes a PatchEmbedding and an encoding layer; Based on Patch Embedding, the target food image is divided into a plurality of image blocks, wherein each of the image blocks is added with an image region of an edge portion of an adjacent image block in a length direction and a width direction of the target food image; Each of the image blocks is encoded into a vector representation and input into a coding layer to obtain a feature vector of the corresponding food.

5. The method for determining the food classification topology structure according to claim 4, characterized in that: The preprocessing model also includes a global average pooling layer; The processing of the feature vector by a preset processing method to obtain a corresponding target feature vector includes: Inputting each of the feature vectors corresponding to the target food image into the global average pooling layer, summing up each of the feature vectors and averaging them to each of the feature vectors, to obtain a feature average vector representing the distribution of the inherent features of the corresponding food in the target food image; A corresponding target feature vector is determined based on the feature average vector.

6. The method for determining the food classification topology structure according to claim 5, characterized in that: The determining the corresponding target feature vector based on the feature average vector includes: The feature average vector is normalized and dimensionally reduced to obtain a corresponding target feature vector.

7. A device for determining a food classification topology structure, characterized in that: include: An information acquisition module is used to extract features from target food images of corresponding food in each category to obtain feature vectors of the corresponding food; The information processing module is used to process the feature vectors by a preset processing method to obtain corresponding target feature vectors; a clustering algorithm is used to perform clustering analysis on each of the target feature vectors to determine the food classification topology structure based on the data clusters obtained by clustering and the hierarchical relationship between the data clusters.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for determining the food classification topology structure according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program, and when the computer program is executed, the electronic device where the computer-readable storage medium is located is controlled to execute the method for determining the food classification topology structure according to any one of claims 1 to 6.