A method for food image hierarchy construction and classification based on inter-class confusion

Abstract

The present invention relates to a hierarchical structure construction and hierarchical classification method based on inter-class confusion, comprising the following steps: S1: obtaining a food image dataset Food-M and training M-class flat classifiers, M=M (1) ; S2: According to M (k) The class-flat classifier obtains the degree of difficulty of classification confusion between food categories, k∈[1,K] and k∈Z; M (k) The food categories are divided into two categories, the number of categories in the two categories are respectively and the best merging method is obtained by traversal exploration, so that M (k) The class-flat classifier has the highest accuracy in distinguishing large categories. The present invention breaks through the limitations of existing hierarchical classification methods in large-scale classification scenarios where the hierarchical structure is not obvious, uses flat classification for large-category discrimination, and traverses to explore the best category merging method.

Description

Technical Field

[0001] The present invention relates to the technical field of food image analysis, and in particular to a food image hierarchical construction and classification method based on inter-class confusion. Background Art

[0002] Food recognition is fundamental to dietary monitoring and management. Many subsequent tasks, including calorie calculation, nutrient analysis, and dietary health management, require accurate food identification. The sheer number of food types required for identification is enormous, and this increased diversity leads to lower recognition accuracy, making it difficult to meet the demands of real-world applications. By building a hierarchical structure, replacing flat classification with hierarchical classification, we first identify broad categories, then make decisions within them. This eliminates constraints between broad categories and alleviates the problem of excessive variety.

[0003] Hierarchical classification methods based on inter-class confusion do not require inter-class hierarchical relationships, breaking through the narrow application scenarios of existing hierarchical classification methods. The unclear inter-class hierarchical relationships present the following challenges for hierarchical classification: it is difficult to construct a suitable hierarchical structure; the accuracy of large-class discrimination is too low; and the final classification effect is difficult to improve compared to flat classification. To address these issues, the present invention proposes a hierarchical structure construction and hierarchical classification method based on inter-class confusion. Summary of the Invention

[0004] To address the shortcomings of the prior art, the first objective of the present invention is to provide a hierarchical structure construction and hierarchical classification method based on inter-class confusion. This invention overcomes the limitations of existing hierarchical classification methods in large-scale classification scenarios where the hierarchical structure is not obvious. It uses flat classification to distinguish large categories and traverses to explore the optimal way to merge categories.

[0005] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0006] A hierarchical structure construction and hierarchical classification method based on inter-class confusion, characterized by comprising the following steps:

[0007] S1: Get the food image dataset Food-M and train M flat classifiers, M = M (1) ;

[0008] S2: According to M (k) The class-flat classifier obtains the degree of difficulty of classification confusion between food categories, k∈[1,K] and k∈Z; M (k) The food categories are divided into two categories, and the number of categories in the two categories is in and Traverse and explore to get the best merging method so that M (k) The class-flat classifier has the highest large-class discrimination accuracy;

[0009] S3: and Flat classifiers were trained for each of the two major categories;

[0010] S4: Order Repeat steps S2 to S3 until the

[0011] S5: For the food image dataset Food-M, a total of K large-category discriminators are set up layer by layer, and finally K A class flat classifier and a Class flat classifier, also trained to get K large class discriminators, respectively The class-flat classifier serves as , where k∈[1,K] and k∈Z.

[0012] Furthermore, in step S1, the dataset Food-M used can be any large-scale dataset. Unlike traditional hierarchical classification methods, it does not require a clear hierarchical structure between classes and is also applicable when M is thousands of classes. The dataset Food-M is divided into training set, validation set, and test set according to a ratio of 6:1:3, without considering the hierarchical relationship between classes, and a flat classifier of M classes is trained.

[0013] Using a flat classifier to make a large category judgment is to convert the small category judgment made by the flat classifier into the large category judgment to which the small category belongs after assuming a certain inter-class merging method.

[0014] Further: in step S2,

[0015] S2-1: Use M (k) The class flat classifier makes inferences on the validation set and compares the true label of each image in the validation set with the M (k) The predicted category given by the class flat classifier is saved to the text file predval.txt; and the output M (k) The inter-class confusion predicted by the class flat classifier is output in descending order according to the degree of confusion and saved in log.txt;

[0016] S2-2: Select several confused category pairs ranked top in the log.txt obtained in step S2-1 until a larger category is generated by merging the initial categories. The remaining All categories are classified into smaller categories to obtain the initial category merging method; the above-mentioned random initialization category merging method greatly shortens the traversal exploration of steps S2-3. Time for optimal category merging methods;

[0017] S2-3: Taking the initial category merging method obtained in step S2-2 as the starting state, iteratively explore the best category merging method; the iterative method is divided into two steps, first exchange traversal, then Plus one (i.e. minus one), the two-step iteration is performed alternately; without changing the number of categories of the larger category, the larger category is exchanged with the smaller category, and all categories are traversed. The merging method after the exchange is combined with the predval.txt file saved in step S2-1 to calculate the accuracy of the major classification under each merging method and save the best merging method; based on the best merging method obtained after the exchange traversal, Choose any category and put it in In; exchange traversal and Add one alternately until equal And the last step is the exchange traversal in the iterative method.

[0018] S2-4;M (k) The flat classifier makes a hierarchical judgment and M (k) Classes are divided into two major categories and and There is great flexibility in the selection of ; to make the accuracy of large-category discrimination high, The larger the setting, the The smaller the value, the smaller the value. To improve the efficiency of reasoning, The smaller the setting, the The bigger; and The selection of is a trade-off between the accuracy of large-scale discrimination and the efficiency of reasoning; We adhere to two principles: acc>acc threshold ; is the upper limit value of the number of categories contained in the artificially set larger category; acc is the accuracy rate of large category discrimination corresponding to the current hierarchical structure; acc threshold It is the lower limit value of the expected accuracy of large-category discrimination set artificially;

[0019] Set according to the number of loadable levels; in step S2-3, when swapping and traversing to measure the accuracy of large-category discrimination, if acc>acc threshold , then at this time That is If until Neither satisfies acc>acc threshold ,but

[0020] Further: in step S3,

[0021] The best merging method obtained in step S2-3 is used to train the smaller class. Flat classifier, used to make final category judgment; trained for larger categories Flat classifier; for Class flat classifier, if it satisfies Then the The class flat classifier is used to make the final category judgment. If it does not meet Then the The class flat classifier is used as a large class discriminator, and the The class is split into two major categories until it meets That is, for larger classes, always use Class hierarchy instead Class flat classification until satisfied The resulting hierarchical structure can be found in Figure 2 .

[0022] After obtaining the best merging method, the M class can be divided into two major classes at the minimum cost. and Because the large category judgment is completed by the flat classifier, this large category discrimination method will not add any classification errors on the basis of the flat classifier, that is, the samples correctly classified by the M-type flat classifier can also be classified into the correct large category by the large category discriminator, so that the corresponding or Flat classifier is used to judge. And the M-class flat classifier makes a wrong classification of small class judgment samples. (or ) The judgment errors between the small categories within the class are correct, and the corresponding large category judgment is correct, thus using a more targeted classifier than the M-class flat classifier. (or ) class flat classifier to judge, it is possible to correctly classify the sample that was misclassified by the M class flat classifier. and The samples that are confused with each other have already made mistakes when making the big category judgment, and then entered the wrong (or ) classifiers are bound to make incorrect category judgments. This error is the price to pay for dividing all categories into two large ones, and exploring the best way to merge categories has minimized this cost.

[0023] Further: In step S4, let Repeat steps S2 to S3, and always use Class hierarchy instead Class flat classification until satisfied

[0024] Further: In step S5, the final training is K A class flat classifier and a Class flat classifier, in addition there are K large class discriminators, namely The class flat classifier acts as a class with k∈[1,K] and k∈Z; combined with the hierarchical structure obtained by layer-by-layer traversal exploration, the final category judgment can be made.

[0025] A second object of the present invention is to provide an electronic device, characterized in that it includes:

[0026] one or more processors;

[0027] a storage device for storing one or more programs,

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above methods.

[0029] The third object of the present invention is to provide a computer-readable medium having a computer program stored thereon, wherein the computer program is characterized in that when the program is executed by a processor, any of the methods described above is implemented.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] The present invention combines a flat classifier for hierarchical judgment, hierarchical structure initialization, hierarchical structure traversal exploration, and loss function sharpening, wherein the hierarchical structure traversal exploration makes the hierarchical structure optimal and the cost of constructing the hierarchical structure is minimized. The present invention proposes solutions to challenges such as unclear hierarchical relationships between categories, too low accuracy of large categories, and inapplicability of hierarchical classification, and realizes a universal hierarchical construction and hierarchical classification method. Specifically, a flat classifier is used to act as a large category discriminator, and the best category merging method is traversed and explored. While improving the accuracy of large category discrimination, it does not increase any classification errors compared to a flat classifier, and the subsequent flat classification within the large category can eliminate the influence of categories within other large categories, and the flat classification can improve the classification effect. In addition, the classification within the large category is also a flat classification, which can also be replaced by the above-mentioned hierarchical classification method to continuously improve the classification effect. The present invention has no requirements for the hierarchical structure between categories and is universally applicable in all flat classification scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall framework diagram of the method of the present invention;

[0033] Figure 2 It is a hierarchical structure diagram of the method of the present invention;

[0034] Figure 3 This is a flow chart of model reasoning in the method of the present invention;

[0035] Figure 4 The method of the present invention is the M-type flat classifier on the Food-M dataset and Example comparison of Grad-CAM heatmaps for flat classifiers;

[0036] Figure 5 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0037] To help those skilled in the art better understand the technical solutions of the present invention, preferred embodiments of the present invention are described below in conjunction with specific examples. However, it should be understood that the drawings are for illustrative purposes only and are not to be construed as limiting the present invention. To better illustrate the present embodiments, certain components in the drawings may be omitted, enlarged, or reduced, and do not represent the dimensions of actual products. It is understandable that certain well-known structures and their descriptions may be omitted from the drawings. The positional relationships depicted in the drawings are for illustrative purposes only and are not to be construed as limiting the present invention.

[0038] The present invention will be further described below with reference to the accompanying drawings and examples, but they are not intended to limit the present invention.

[0039] like Figures 1 to 5 As shown in , a hierarchical structure construction and hierarchical classification method based on inter-class confusion is implemented based on a flat classifier acting as a large-category discriminator. The method includes flat classifier training, flat classifier acting as a large-category discriminator, traversal exploration of the best category merging method, and training of flat classifiers for each large category. Figure 1 , is a simple and general hierarchical structure construction and hierarchical classification model.

[0040] The following steps are involved:

[0041] S1: Get the food image dataset Food-M and train M flat classifiers, M = M (1) ;

[0042] The Food-M food image dataset contains M food categories and is partitioned into training, test, and validation sets in a 6:3:1 ratio. The validation set is used to select the optimal model to prevent overfitting on the training set. The selection of category merging methods based on broad category accuracy is performed only on the validation set, and the confusion on the test set is not considered a priori knowledge. The pre-trained ResNet101 model on ImageNet is used as the initialization model for model training. The resulting M-category flat classifier is used as the broad category discriminator to traverse and explore the optimal merging method. Furthermore, this M-category flat classifier serves as the benchmark for comparison with the final hierarchical classification method.

[0043] S2: According to M (k) The class-flat classifier obtains the degree of difficulty of classification confusion between food categories, k∈[1,K] and k∈Z; M (k) The food categories are divided into two categories, and the number of categories in the two categories is in and Traverse and explore to get the best merging method so that M (k) The class-flat classifier has the highest large-class discrimination accuracy, that is, it makes the large-class discrimination error of the M-class flat classifier the lowest, and makes the cost of dividing the M class into two large classes the minimum.

[0044] In step S2,

[0045] S2-1: Use M (k) The class flat classifier makes inferences on the validation set and compares the true label of each image in the validation set with the M (k) The predicted category given by the class flat classifier is saved to the text file predval.txt; and the output M (k) The inter-class confusion predicted by the class flat classifier is output in descending order according to the degree of confusion and saved in log.txt;

[0046] S2-2: Select several confused category pairs ranked top in the log.txt obtained in step S2-1 until a larger category is generated by merging the initial categories. The remaining All categories are classified into smaller categories to obtain the initial category merging method;

[0047] S2-3: Taking the initial category merging method obtained in step S2-2 as the starting state, iteratively explore the best category merging method; the iterative method is divided into two steps, first exchange traversal, then Plus one (i.e. minus one), the two-step iteration is performed alternately; without changing the number of categories of the larger category, the larger category is exchanged with the smaller category, and all categories are traversed. The merging method after the exchange is combined with the predval.txt file saved in step S2-1 to calculate the accuracy of the major classification under each merging method and save the best merging method; based on the best merging method obtained after the exchange traversal, Choose any category and put it in In; exchange traversal and Add one alternately until equal And the last step is the exchange traversal in the iterative method.

[0048] S2-4;M (k) The flat classifier makes a hierarchical judgment and M (k) Classes are divided into two major categories and and There is great flexibility in the selection of ; to make the accuracy of large-category discrimination high, The larger the setting, the The smaller the value, the smaller the value. To improve the efficiency of reasoning, The smaller the setting, the The bigger; and The selection of is a trade-off between the accuracy of large-scale discrimination and the efficiency of reasoning; We adhere to two principles: acc>acc threshold ; is the upper limit value of the number of categories contained in the artificially set larger category; acc is the accuracy rate of large category discrimination corresponding to the current hierarchical structure; acc threshold It is the lower limit value of the expected accuracy of large-category discrimination set artificially;

[0049] Set according to the number of loadable levels; in step S2-3, when swapping and traversing to measure the accuracy of large-category discrimination, if acc>acc threshold , then at this time That is If until Neither satisfies acc>acc threshold ,but

[0050] The above steps S2-1, S2-2, S2-3, and S2-4 are a single category-level exploration, which will be performed K times, M times (k) Take separately Where k∈[1,K] and k∈Z. K can be set larger to achieve a deeper hierarchy and more classifiers in exchange for a greater improvement in accuracy. The setting is very flexible and is also applicable to classification scenarios with an initial number M of thousands of categories, showing strong adaptability. It traverses and explores the hierarchical structure without requiring inherent relationships between categories, making it more universally applicable than traditional hierarchical classification methods.

[0051] S3: and The two categories are trained with flat classifiers respectively; the training is the same as step S1 for training the M category flat classifier. Flat classifier and All flat classifiers load the ResNet101 pre-trained model on ImageNet as the initialization model.

[0052] S4: Order Repeat steps S2 to S3 until the The resulting inter-class hierarchy is as follows Figure 2 As shown;

[0053] S5: For the food image dataset Food-M, a total of K large-category discriminators are set up layer by layer, and finally K A class flat classifier and a Class flat classifier, also trained to get K large class discriminators, respectively The class-flat classifier acts as , where k∈[1,K] and k∈Z. In the inference phase, as Figure 3 As shown, the input image passes through layers of large-scale discriminators (M (k) Flat classifier), make a major category judgment, and then enter the corresponding category according to the corresponding major category judgment Class or Class discriminator. Only Flat classifier and the last level Only the flat classifier can give the final category judgment. Other flat classifiers can only serve as large-category discriminators to decide which classifier in the next level the image should be sent to.

[0054] Using a flat classifier as a large-category discriminator is different from traditional large-category discrimination. It no longer needs to learn large-category features and does not require a clear hierarchical structure between classes. Traditional hierarchical classification methods require a good hierarchical relationship between classes, the ability to learn good large-category features, and a sufficiently high large-category discrimination accuracy. The error rate of large-category discrimination cannot be too large. Only in this way can the advantage of the low-level classifier in terms of increased accuracy due to the reduction in the number of categories be truly brought into play. Using a flat classifier as a large-category discriminator will not increase any classification errors on the basis of the flat classifier, that is, it is (k) The samples correctly classified by the flat classifier can also be classified into the correct large category by the large category discriminator, so as to further use the corresponding Class or The flat classifier is used to judge. (k) The class flat classifier made a misclassified small class judgment sample, if it is (or ) The judgment error between the small categories within the class is correct, and then the M (k) Flat classifiers are more targeted (or ) class flat classifier to judge, it is possible to put M (k) Class flat classifier misclassifies the sample correctly. For a single level k, M (k) Class flat classifier, Flat classifier and The flat classifier is better; overall, compared with the M-class flat classifier, K A class flat classifier and a The final category judgment made by the class-flat classifier is more accurate.

[0055] Test model effectiveness: Figure 4 The following are representative examples of the test in the embodiment. The first row is three examples of original images, namely steamed hairy crabs, Figure 4 -(1), celery fried shrimp, such as Figure 4 -(2), gimbap, such as Figure 4 -(3); Figure 4 -(4), 4-(5), and 4-(6) correspond to the three original images one by one. Grad-Cam heatmaps on the M-class flat classifier; 4-(7), 4-(8), and 4-(9) are Grad-Cam heatmaps on the M-class flat classifier corresponding to the three original images. These three images all have classification errors on the original M-class flat classifier, but they can make correct major category judgments layer by layer according to the major category merging method until they are The flat classifier makes the correct final category decision; the Grad-Cam heat map visualizes the image area that the model focuses on when making the corresponding prediction category. The red part is the image area that is focused on, and the blue part is the opposite. It can be seen from the Grad-Cam heat map that when making the corresponding category decision, The flat classifier focused on the hairy crab, shrimp, and gimbap areas, and thus made the correct category prediction; on the contrary, the M-class flat classifier focused on the surrounding areas and made the wrong category prediction, predicting steamed hairy crab as steamed pork with rice flour, celery and shrimp as corn salad, and gimbap as rice. Obviously, Compared with the M-class flat classifier, the class flat classifier needs to classify fewer categories, is more targeted, can learn better features, and has a higher classification accuracy.

[0056] The present invention was experimented on the VireoFood-172 public dataset, and the invented hierarchical classification method was proven to be superior to flat classification in large-scale classification tasks without obvious hierarchical structure between classes. This breaks through the inapplicability of traditional hierarchical structure construction and hierarchical classification methods in classification tasks without clear hierarchical structure between classes.

[0057] Through the description of the above embodiments, it will be clear to those skilled in the art that the facilities of the present invention can be implemented by means of software plus the necessary general hardware platform. The embodiments of the present invention can be implemented using existing processors, or by a dedicated processor used for this or other purposes for an appropriate system, or by a hard-wired system. The embodiments of the present invention also include non-transitory computer-readable storage media, which include machine-readable media for carrying or having machine-executable instructions or data structures stored thereon; such machine-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer or other machine with a processor. For example, such machine-readable media can include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of machine-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computer or other machine with a processor. When information is transmitted or provided to a machine via a network or other communication connection (hard-wired, wireless, or a combination of hard-wired or wireless), the connection is also considered a machine-readable medium.

[0058] Based on the description and drawings of the present invention, those skilled in the art can easily manufacture or use a biological sequence processing and model training method of the present invention, and can produce the positive effects described in the present invention.

[0059] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A hierarchical structure construction and hierarchical classification method based on inter-class confusion, characterized by: The steps include: S1: Get the food image dataset Food-M and train M flat classifiers, M = M (1) ; S2: According to M (k) The class-flat classifier obtains the degree of difficulty of classification confusion between food categories, k∈[1,K] and k∈Z; M (k) The food categories are divided into two categories, and the number of categories in the two categories is in and Traverse and explore to get the best merging method so that M (k) The class-flat classifier has the highest large-class discrimination accuracy; S3: and Flat classifiers were trained for each of the two major categories; S4: Order Repeat steps S2 to S3 until the S5: For the food image dataset Food-M, a total of K large-category discriminators are set up layer by layer, and finally K A class flat classifier and a Class flat classifier, also trained to get K large class discriminators, respectively The class-flat classifier acts as , where k∈[1,K] and k∈Z; In step S1, the dataset Food-M is divided into training set, validation set and test set according to the ratio of 6:1:3, without considering the hierarchical relationship between classes, and an M-class flat classifier is trained; In step S2, S2-1: Use M (k) The class flat classifier makes inferences on the validation set and compares the true label of each image in the validation set with the M (k) The predicted category given by the class flat classifier is saved to the text file predval.txt; and the output M (k) The inter-class confusion predicted by the class flat classifier is output in descending order according to the degree of confusion and saved in log.txt; S2-2: Select several confused category pairs ranked top in the log.txt obtained in step S2-1 until a larger category is generated by merging the initial categories. The remaining All categories are classified into smaller categories to obtain the initial category merging method; S2-3: Taking the initial category merging method obtained in step S2-2 as the starting state, iteratively explore the best category merging method; the iterative method is divided into two steps, first exchange traversal, then Add one, and the two steps are iterated alternately; without changing the number of categories of the larger category, the larger category is exchanged with the smaller category, and all categories are traversed. The merging method after the exchange is combined with the predval.txt file saved in step S2-1 to calculate the accuracy of the major classification under each merging method and save the best merging method; based on the best merging method obtained after the exchange traversal, Choose any category and put it in In; exchange traversal and Add one alternately until equal And the last step is the exchange traversal in the iterative method; S2-4;M (k) The flat classifier makes a hierarchical judgment and M (k) The categories are divided into two categories, the number of which are and and There is great flexibility in the selection; To make the classification accuracy of large categories high, The larger the setting, the The smaller the value, the smaller the value. To improve the efficiency of reasoning, The smaller the setting, the The bigger; and The selection of is a trade-off between the accuracy of large-scale discrimination and the efficiency of reasoning; Adhere to two principles: acc>acc threshold ; is the upper limit value of the number of categories contained in the artificially set larger category; acc is the accuracy rate of large category discrimination corresponding to the current hierarchical structure; acc threshold It is the lower limit value of the expected accuracy of large-category discrimination set artificially; Set according to the number of loadable levels; in step S2-3, when swapping and traversing to measure the accuracy of large-category discrimination, if acc>acc threshold , then at this time That is If until Neither satisfies acc>acc threshold ,but 2. A hierarchical structure construction and hierarchical classification method based on inter-class confusion according to claim 1, characterized in that: In step S3, The best merging method obtained in step S2-3 is used to train the smaller class. Flat classifier, used to make final category judgment; trained for larger categories Flat classifier; for Class flat classifier, if it satisfies Then the The class flat classifier is used to make the final category judgment. If it does not meet Then the The class flat classifier is used as a large class discriminator, and the The class is split into two major categories until it meets 3. The hierarchical structure construction and hierarchical classification method based on inter-class confusion according to claim 1, characterized in that: In step S4, let Repeat steps S2 to S3, and always use Class hierarchy instead Class flat classification until satisfied 4. The hierarchical structure construction and hierarchical classification method based on inter-class confusion according to claim 1, characterized in that: In step S5, the final training results are K A class flat classifier and a Class flat classifier, in addition there are K large class discriminators, namely The class flat classifier acts as a class with k∈[1,K] and k∈Z; combined with the hierarchical structure obtained by layer-by-layer traversal exploration, the final category judgment can be made.

5. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

6. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images