Long-Tail Data Classification Method, System and Medium Based on Class Activation Heatmap

Abstract

The present invention discloses a long-tail data classification method, system and medium based on class activation heatmaps. The method includes: obtaining a long-tail data set; constructing a long-tail data classification model; pre-training a feature extraction module and a classification module on the long-tail data set, and freezing the parameters of the feature extraction module after completion; iteratively training the long-tail data classification model on the long-tail data set, including: obtaining a data set of few-shot classes, inputting it into the class activation map module to generate learnable prototypes and standard class activation heatmaps; converting the standard class activation heatmaps into single-channel maps, and re-extracting the feature vectors of the few-shot class data set to obtain classification results; iteratively training the class activation map module and the classification module, updating the parameters until convergence to obtain a trained model; obtaining the classification results of the long-tail data set to be predicted. The present invention effectively solves the problem of data imbalance and accurately classifies few-shot classes without reducing the classification performance of multi-sample classes.

Description

Technical Field

[0001] The present invention belongs to the technical field of long-tail data processing, and in particular relates to a long-tail data classification method, system and medium based on a class activation heat map. Background Art

[0002] In recent years, artificial intelligence and related industries have been growing rapidly and have become the focus of academia, industry and governments around the world. The State Council has issued the "New Generation Artificial Intelligence Development Plan", highlighting the national strategic position of artificial intelligence research and industry. In the field of computer vision, deep neural networks have made many breakthroughs; one of the important reasons is the large number of available data sets (such as ImageNet). However, the data in these data sets is often unbalanced, and the number of data in different categories varies greatly. The model learned from unbalanced data has a low accuracy rate for the judgment of few-sample categories. Many early studies solved the problem of data imbalance by artificially balancing data, such as sample sampling strategies, class-related loss functions, etc.; although the problem of data imbalance was solved to a certain extent, it also reduced the performance on multi-sample categories and there was a risk of overfitting. Recently, studies have shown that unbalanced data distribution has a great impact on the classifier in the neural network. The size of the classification weight corresponding to each category is positively correlated with the number of data in the learning model, which causes the network to always tend to produce judgments on multi-sample categories. Therefore, how to train a model with accurate judgment of few-sample categories on long-tail data is a difficult problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a long-tail data classification method, system and medium based on class activation heat map. The present invention introduces a class activation map module into the constructed long-tail data classification model, pre-trains the parameters of the feature extraction module and freezes them, and then retrains the long-tail data classification model through iterative training, thereby effectively solving the data imbalance problem and accurately classifying the few-sample categories without reducing the classification performance of the multi-sample categories.

[0004] In order to achieve the above object, the present invention provides a long-tail data classification method based on class activation heat map, comprising the following steps:

[0005] Obtain a long-tail data set, wherein the long-tail data set contains multiple categories, and each sample is labeled with its category, and the number of samples in different categories has a large difference;

[0006] Constructing a long-tail data classification model, the long-tail data classification model includes a feature extraction module, a class activation map module and a classification module; the class activation map module is located between the feature extraction module and the classification module;

[0007] Pre-train the feature extraction module and the classification module on the long-tail dataset using the cross-entropy loss function. After pre-training, freeze the parameters of the feature extraction module to obtain the categories of the long-tail dataset.

[0008] Iteratively train the long-tail data classification model on the long-tail dataset, including:

[0009] Define the categories in the long-tail dataset with the data volume less than τ as the few-shot categories, and obtain the dataset of the few-shot categories.

[0010] Input the dataset of the few-shot categories into the class activation map module to generate learnable prototypes and standard class activation map heatmaps.

[0011] Convert the standard class activation heatmap into a single-channel map, and input it into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot category dataset, and input them into the classification model to obtain the classification results.

[0012] Use the cross-entropy loss function to iteratively train the class activation map module and the classification module, update the parameters of the class activation map module and the classification module until convergence, and obtain the trained long-tail data classification model.

[0013] Input the long-tail dataset to be predicted into the trained long-tail data classification model to obtain the predicted classification results.

[0014] As a preferred technical solution, the pre-training of the feature extraction module and the classification module on the long-tail dataset using the cross-entropy loss function is specifically as follows:

[0015] Input the long-tail dataset into the feature extraction module to extract the feature vector F of each sample in the long-tail dataset.

[0016] For each category c in the long-tail dataset, perform global average pooling operation on the feature vector F of each sample and the category c to obtain the global representation x c ;

[0017] Through the global representation x c and the normalized parameters of the classification module obtain the category score s c , where w c represents the parameters of the classification module for the category c;

[0018] After obtaining the category scores of each category in the long-tail dataset, determine the classification results through the Softmax function.

[0019] Use the cross-entropy loss function to update the parameters of the feature extraction module.

[0020] As a preferred technical solution, the category score is expressed as:

[0021]

[0022] Among them, <,> represents the dot product operation, g is a hyperparameter, x is the global representation of the feature vector F after global average pooling, and w c is the parameter of the classification module for class c;

[0023] The Softmax function is expressed as:

[0024]

[0025] where Softmax(s c ) is the predicted probability of the Softmax function for each class score s c , s j represents the class score of the j-th class;

[0026] The classification result is the class with the highest predicted probability;

[0027] The cross-entropy loss function is expressed as:

[0028]

[0029] where p = [p0, ···, p C-1 represents the predicted probabilities of all classes, p i represents the predicted probability of the i-th class, y = [y0, ···, y C-1 is the one-hot representation of the sample class. When the sample belongs to the i-th class, y i = 1, otherwise y i = 0; C is the number of classes.

[0030] As a preferred technical solution, generating the learnable prototype and the standard class activation map heatmap is specifically as follows:

[0031] For each few-shot class c′, use the dataset of this few-shot class to generate a learnable prototype Φ c′ ;

[0032] Use the feature vectors obtained from K samples in the few-shot class c′ through the feature extraction module, and take the average of each column of the i-th few-shot feature vector to convert it into a row Finally, merge them as the initial value of the prototype Φ c′ :

[0033]

[0034] Use the learnable prototype Φ c′ to perform a convolution operation on the feature vectors of the few-shot class dataset to obtain the standard class activation heatmap The formula is:

[0035]

[0036] Where, Φ c′ represents the learnable prototype of the few-shot class c′; F c′ is the feature vector of the few-shot class.

[0037] As a preferred technical solution, the re-extraction of the feature vector of the few-shot class dataset is specifically:

[0038] Perform a convolution operation on the standard class activation heatmap to convert it into a single-channel map

[0039] Input the single-channel map into the feature extraction module with frozen parameters to re-extract the feature vector of the few-shot class dataset which is expressed as:

[0040]

[0041] Where, σ(.) is the Sigmoid function, represents the Hadamard product, F c′ is the feature vector of the few-shot class c′ dataset.

[0042] As a preferred technical solution, the classification result obtained by inputting into the classification model is specifically:

[0043] Perform a global average pooling operation on the re-extracted feature vector of the few-shot class dataset to obtain the global representation x c′ ;

[0044] Generate the class score s c′ of the few-shot class c′ through the global representation x c′ and w c′ after normalization, and the formula is:

[0045]

[0046] Where, 〈,〉 represents the dot product operation, g is a hyperparameter, and w c′ is the parameter of the classification module for class c′;

[0047] After obtaining the class score of the few-shot class c′, determine the classification result through the Softmax function.

[0048] As a preferred technical solution, the cross-entropy loss function is expressed as:

[0049]

[0050] Among them, p = [p0, ···, p C′-1 represents the predicted probabilities of all few-shot classes, and p i represents the predicted probability of the i-th few-shot class, and y = [y0, ···, y C′-1 is the one-hot representation of the few-shot classes. When the few-shot sample belongs to the i-th few-shot class, y i = 1, otherwise y i = 0; C′ is the number of few-shot classes.

[0051] As a preferred technical solution, the feature extraction module of the long-tail data classification model adopts a ResNeXt-50 network;

[0052] During the iterative training process, the long-tail data classification model uses an SGD optimizer with a momentum of 0.9 to optimize the parameters, the batch size is 256, and it is trained for 200 epochs during the pre-training process and 10 epochs during the iterative training of the class activation map module and the classification module; the training uses a cosine learning rate that gradually decreases from 0.1 to 0, and the hyperparameter g is set to 0.5 and 16 respectively during the pre-training process and the iterative training of the class activation map module and the classification module.

[0053] On the other hand, the present invention also provides a long-tail data classification system based on a class activation heat map, including a data acquisition module, a model construction module, a pre-training module, an iterative training module, and a result output module;

[0054] The data acquisition module is used to acquire a long-tail data set; the long-tail data set contains multiple classes, and each sample is labeled with the class it belongs to, and the number of samples in different classes has a large difference;

[0055] The model construction module is used to construct a long-tail data classification model; the long-tail data classification model includes a feature extraction module, a class activation map module, and a classification module; the class activation map module is located between the feature extraction module and the classification module;

[0056] The pre-training module is used to pre-train the feature extraction module and the classification module on the long-tail data set using the cross-entropy loss function. After the pre-training is completed, the parameters of the feature extraction module are frozen to obtain the classes of the long-tail data set;

[0057] The iterative training module is used to iteratively train the long-tail data classification model on the long-tail data set, including:

[0058] Define the classes in the long-tail data set with a data volume less than τ as few-shot classes to obtain a data set of few-shot classes;

[0059] Input the dataset of few-shot classes into the class activation map module to generate learnable prototypes and standard class activation map heatmaps;

[0060] Convert the standard class activation heatmap into a single-channel map, input it into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot class dataset, and input it into the classification model to obtain the classification result;

[0061] Use the cross-entropy loss function to iteratively train the class activation map module and the classification module, update the parameters of the class activation map module and the classification module until convergence, and obtain the trained long-tail data classification model;

[0062] The result output module is used to input the long-tail dataset to be predicted into the trained long-tail data classification model to obtain the predicted classification result.

[0063] In another aspect, the present invention provides a computer-readable storage medium storing a program, characterized in that when the program is executed by a processor, the above-mentioned long-tail data classification method based on class activation heatmaps is implemented.

[0064] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0065] Aiming at the problem that the existing long-tail data classification model has a low classification accuracy for few-shot classes in the long-tail dataset, the present invention proposes a long-tail data classification method based on class activation heatmaps. By normalizing the classification module parameters and sample global representations in long-tail recognition, using the normalized classification module for representation learning, and simply setting a normalization factor for the normalized classification module, the representation learning and classification learning are effectively improved; at the same time, a class activation map module (CAMC module) is introduced to improve the learning and prediction of the classification module, enabling the model to make predictions based on important regions in the image; through empirical research, it shows that the model can effectively improve the classification accuracy of the tail classes without sacrificing the accuracy of the head classes; and experiments are carried out on different datasets (ImageNet-LT, Places-LT, iNaturalist 2018, CIFAR10-LT, CIFAR100-LT), achieving good performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0067] Figure 1It is a flowchart of the long-tail data classification method based on class activation heatmap in an embodiment of the present invention;

[0068] Figure 2 It is a schematic diagram of pre-training of the feature extraction module and the classification module in an embodiment of the present invention;

[0069] Figure 3 It is a flowchart of pre-training of the feature extraction module and the classification module in an embodiment of the present invention;

[0070] Figure 4 It is a schematic diagram of iterative training of the long-tail data classification model in an embodiment of the present invention;

[0071] Figure 5 It is a flowchart of iterative training of the long-tail data classification model in an embodiment of the present invention;

[0072] Figure 6 It is a schematic diagram of the class activation map module in an embodiment of the present invention;

[0073] Figure 7 It is a schematic structural diagram of the long-tail data classification system based on class activation heatmap in an embodiment of the present invention;

[0074] Figure 8 It is a schematic structural diagram of the computer-readable storage medium in an embodiment of the present invention. Detailed implementation manners

[0075] In order to enable those skilled in the art of the present technology to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0076] Referring to "embodiment" in the present application means that the specific features, structures or characteristics described in combination with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in the present application may be combined with other embodiments.

[0077] In reality, data usually exhibits a long-tailed distribution, where some categories account for the majority of the dataset, while the samples of most rare categories are limited. For example, in animal data, the vast majority of samples collected in nature belong to common head categories (such as cats and dogs), while only a small number of samples can be collected for the vast majority of tail categories (such as snow leopards and pandas). This causes the problem of data imbalance and also brings troubles to data classification. Based on this, as Figure 1 shown, this embodiment proposes a long-tailed data classification method based on class activation heatmaps, including the following steps:

[0078] S1. Obtain a long-tailed dataset; the obtained long-tailed dataset is required to contain multiple categories, label each sample with its belonging category, and there are significant differences in the number of samples in different categories. For example, the ImageNet-LT dataset involves 1000 categories, with a single category containing at most 1280 images and at least only 5 images.

[0079] S2. Build a long-tailed data classification model, including a feature extraction module, a class activation map module, and a classification module; the class activation map module is located between the feature extraction module and the classification module.

[0080] In this embodiment, ResNeXt-50 is used as the feature extraction module. After obtaining the sample feature vector, when calculating the few-shot class scores, the class activation map module (CAMC module) is used to adjust the feature vector and then perform global average pooling, and finally the prediction probability of each category is obtained through the Softmax function, and the category with the highest prediction probability is selected as the prediction result. It should be noted that networks with the same function as the ResNeXt-50 network are applicable to the method of the present invention, and will not be elaborated one by one in this embodiment.

[0081] S3. As Figure 2 shown, regardless of the differences in the data volume of each category in the long-tailed dataset, use the cross-entropy loss function to pre-train the feature extraction module and the classification module on the long-tailed dataset. After the pre-training is completed, freeze the parameters of the feature extraction module to obtain the categories of the long-tailed dataset. The specific pre-training steps are as Figure 3 shown:

[0082] S31. Input the long-tailed dataset into the feature extraction module to extract the feature vector F of each sample in the long-tailed dataset.

[0083] S32. For each category c in the long-tailed dataset, perform global average pooling operation on the feature vector F of each sample and the category c to obtain the global representation x c ;

[0084] S33. Through the global representation x c and the normalized classification module parameters Obtain the class score s c , and the formula is:

[0085]

[0086] where <,> represents the dot product operation, g is a hyperparameter, x is the global representation after global average pooling of the feature vector F, and w c is the parameter of the classification module for class c;

[0087] S34. After obtaining the class scores of each class in the long-tail dataset, determine the classification result through the Softmax function; the Softmax function is expressed as:

[0088]

[0089] where Softmax(s c ) is the predicted probability of the Softmax function for each class score s c , s j represents the class score of the j-th class, and the class with the highest predicted probability is selected as the classification result.

[0090] S35. Use the cross-entropy loss function (Cross Entropy Loss) to update the parameters of the feature extraction module until convergence. The cross-entropy loss function of the feature extraction module is expressed as:

[0091]

[0092] where p = [p0, ···, p C-1 represents the predicted probabilities of all classes, p i represents the predicted probability of the i-th class, y = [y0, ···, y C-1 is the one-hot representation of the sample class. When the sample belongs to the i-th class, y i = 1, otherwise y i = 0; C is the number of classes.

[0093] S4. As Figure 4 shown, the present invention adds a class activation map module (CAMC module) after the feature extraction module and before the classification module to increase the weight of the key representation, and then iteratively trains the long-tail data classification model on the long-tail dataset. The specific process steps are as Figure 5 shown, including:

[0094] S41. On the same long-tail dataset, considering the differences in the data volume of each class, define the classes with a data volume less than τ in the long-tail dataset as few-shot classes, and obtain the dataset of few-shot classes;

[0095] S42. AsFigure 6 As shown, the dataset of the few-shot class is input into the class activation map module to generate a learnable prototype and a standard class activation map heatmap. Specifically:

[0096] For each few-shot class \(c'\), a learnable prototype \(\varPhi\) is generated using the dataset of this few-shot class c′ ;

[0097] Using the feature vectors obtained from the \(K\) samples in the few-shot class \(c'\) through the feature extraction module, and taking the average of each column of the \(i\)-th sample feature vector to convert it into a row Finally, they are merged as the initial value of \(\varPhi\): c′

[0098]

[0099] Using the learnable prototype \(\varPhi\) c′ Perform a convolution operation on the feature vectors of the few-shot class dataset to obtain the standard class activation heatmap The formula is:

[0100]

[0101] where \(\varPhi\) c′ represents the learnable prototype of the few-shot class \(c\) ′ ; \(F\) c′ is the feature vector of the few-shot class.

[0102] S43. Convert the standard class activation heatmap into a single-channel map, and input it into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot class dataset, and input them into the classification model to obtain the classification result. Specifically:

[0103] Perform a convolution operation on the standard class activation heatmap to convert it into a single-channel map

[0104] Input the single-channel map into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot class dataset Extraction, expressed as:

[0105]

[0106] where \(\sigma(.)\) is the Sigmoid function, represents the Hadamard product, and \(F\) c′ is the feature vector of the few-shot class \(c'\) dataset.

[0107] Then, perform global average pooling on the re-extracted feature vectors of the few-shot class dataset to obtain the global representation \(x\)​c′ ;

[0108] Generate the class score c of the few-shot class c′ through the global representations x c′ and w c′ after normalization, and the formula is: c′ , the formula is:

[0109]

[0110] where <,> represents the dot product operation, g is a hyperparameter, and w c′ is the parameter of the classification module for the class c′;

[0111] After obtaining the class score of the few-shot class c′, determine the classification result through the Softmax function.

[0112] S34. Use the cross-entropy loss function to iteratively train the class activation map module and the classification module, update the parameters of the class activation map module and the classification module until convergence, and obtain the trained long-tail data classification model;

[0113] The cross-entropy loss function is expressed as:

[0114]

[0115] where p = [p0, ···, p C′-1 represents the predicted probabilities of all few-shot classes, and p i represents the predicted probability of the i-th few-shot class, y = [y0, ···, y C′-1 is the one-hot representation of the few-shot classes, where y i = 1 when the few-shot sample belongs to the i-th few-shot class, and otherwise y i = 0; C′ is the number of few-shot classes.

[0116] In this embodiment, during the iterative training process, the long-tail data classification model uses the SGD optimizer with momentum 0.9 to optimize the parameters, the batch size is 256, and it is trained for 200 epochs during the pre-training process and 10 epochs during the iterative training of the class activation map module and the classification module; the training uses the cosine learning rate to gradually decrease from 0.1 to 0, and the hyperparameter g is set to 0.5 and 16 respectively during the pre-training process and the iterative training of the class activation map module and the classification module.

[0117] S5. Input the long-tail dataset to be predicted into the trained long-tail data classification model to obtain the predicted classification result.

[0118] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously.

[0119] Based on the same idea as the long-tail data classification method based on class activation heatmap in the above embodiments, the present invention also provides a long-tail data classification system based on class activation heatmap, which can be used to execute the above long-tail data classification method based on class activation heatmap. For the sake of convenience of description, in the structural schematic diagram of the long-tail data classification system embodiment based on class activation heatmap, only the parts related to the embodiments of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than those illustrated, or combine certain components, or arrange different components.

[0120] As Figure 7 shown, another embodiment of the present invention provides a long-tail data classification system based on class activation heatmap, including a data acquisition module, a model construction module, a pre-training module, an iterative training module, and a result output module;

[0121] Among them, the data acquisition module is used to acquire the long-tail data set;

[0122] The model construction module is used to construct a long-tail data classification model; the model includes a feature extraction module, a class activation map module, and a classification module; among them, the class activation map module is located between the feature extraction module and the classification module;

[0123] The pre-training module is used to pre-train the feature extraction module and the classification module on the long-tail data set using the cross-entropy loss function. After the pre-training is completed, the parameters of the feature extraction module are frozen to obtain the categories of the long-tail data set;

[0124] The iterative training module is used to iteratively train the long-tail data classification model on the long-tail data set, including:

[0125] Define the categories in the long-tail data set with the data volume less than τ as few-shot categories, and obtain the data set of few-shot categories;

[0126] Input the data set of few-shot categories into the class activation map module to generate a learnable prototype and a standard class activation map heatmap;

[0127] Convert the standard class activation heatmap into a single-channel map, and input it into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot category data set, and input them into the classification model to obtain the classification result;

[0128] Iteratively train the class activation map module and the classification module using the cross-entropy loss function, update the parameters of the class activation map module and the classification module until convergence, and obtain a trained long-tail data classification model;

[0129] The result output module is used to input the long-tail data set to be predicted into the trained long-tail data classification model to obtain the predicted classification result.

[0130] It should be noted that the long-tail data classification system based on the class activation heat map of the present invention corresponds one-to-one with the long-tail data classification method based on the class activation heat map of the present invention. The technical features and their beneficial effects described in the embodiments of the above-mentioned long-tail data classification method based on the class activation heat map are applicable to the embodiments of the long-tail data classification system based on the class activation heat map. For specific content, please refer to the description in the method embodiments of the present invention, which will not be repeated here. This is hereby declared.

[0131] In addition, in the implementation manner of the long-tail data classification system based on the class activation heat map in the above embodiments, the logical division of each program module is only an example. In actual applications, according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the long-tail data classification system based on the class activation heat map is divided into different program modules to complete all or part of the functions described above.

[0132] As Figure 8 shown, in one embodiment, a computer-readable storage medium is provided, which stores a program in a memory. When the program is executed by a processor, the long-tail data classification method based on the class activation heat map is implemented, specifically:

[0133] Obtain a long-tail data set; the long-tail data set contains multiple categories, each sample is labeled with its belonging category, and the number of samples in different categories has a large difference;

[0134] Construct a long-tail data classification model; the model includes a feature extraction module, a class activation map module, and a classification module; the class activation map module is located between the feature extraction module and the classification module;

[0135] Pre-train the feature extraction module and the classification module on the long-tail data set using the cross-entropy loss function. After the pre-training is completed, freeze the parameters of the feature extraction module to obtain the categories of the long-tail data set;

[0136] Iteratively train the long-tail data classification model on the long-tail data set, including:

[0137] Define the categories in the long-tail data set with the data volume less than τ as the few-shot categories to obtain the data set of the few-shot categories;

[0138] Input the dataset of few-shot classes into the class activation map module to generate learnable prototypes and the heatmap of the standard class activation map;

[0139] Convert the heatmap of the standard class activation into a single-channel map, input it into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot class dataset, and input it into the classification model to obtain the classification result;

[0140] Use the cross-entropy loss function to iteratively train the class activation map module and the classification module, update the parameters of the class activation map module and the classification module until convergence, and obtain the trained long-tail data classification model;

[0141] Input the long-tail dataset to be predicted into the trained long-tail data classification model to obtain the predicted classification result.

[0142] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0143] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0144] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A long-tail data classification method based on class activation heatmaps, characterized in that, It includes the following steps: Obtain a long-tail image dataset, which contains multiple categories, label each image sample with its belonging category, and there are significant differences in the number of image samples of different categories; Construct a long-tail data classification model, which includes a feature extraction module, a class activation map module, and a classification module; the class activation map module is located between the feature extraction module and the classification module; Pre-train the feature extraction module and the classification module on the long-tail image dataset using the cross-entropy loss function. After the pre-training is completed, freeze the parameters of the feature extraction module to obtain the categories of the long-tail image dataset; Iteratively train the long-tail data classification model on the long-tail image dataset, including: Define the categories with the data volume less than τ in the categories of the long-tail image dataset as few-shot categories to obtain a dataset of few-shot categories; Input the images in the dataset of few-shot categories into the class activation map module to generate learnable prototypes and standard class activation heatmaps; Convert the standard class activation heatmap into a single-channel map, input it into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot category dataset, and input them into the classification model to obtain the classification results; Use the cross-entropy loss function to iteratively train the class activation map module and the classification module, update the parameters of the class activation map module and the classification module until convergence to obtain a trained long-tail data classification model; Input the images in the long-tail data set to be predicted into the trained long-tail data classification model to obtain the predicted classification results.

2. The long-tail data classification method based on class activation heatmap according to claim 1, characterized in that, The pre-training of the feature extraction module and the classification module on the long-tail image dataset using the cross-entropy loss function is specifically: Input the long-tail image dataset into the feature extraction module to extract the feature vector F of each image sample in the long-tail image dataset; For each category c in the long-tail image dataset, a global average pooling operation is performed on the feature vector F of each image sample and the category c to obtain a global representation x c ; Through the global representation x c and the normalized classification module parameters obtain the class score s c , where w c represents the parameters of the classification module for class c; After obtaining the category scores of each category in the long-tail image dataset, determine the classification results through the Softmax function; Use the cross-entropy loss function to update the parameters of the feature extraction module.

3. The long-tail data classification method based on class activation heatmap according to claim 2, wherein The category scores are expressed as: Among them, <,> represents the dot product operation, g is a hyperparameter, x is the global representation after the global average pooling of the feature vector F, and w c is the parameter of the classification module for class c; The Softmax function is expressed as: Among them, Softmax(s c ) is the predicted probability of the Softmax function for each class score s c , where s j represents the class score of the j-th class; The classification result is the category with the highest predicted probability; The cross-entropy loss function is expressed as: Among them, p = [p0, ···, p C-1 represents the predicted probabilities of all categories, and p i represents the predicted probability of the i-th category. y = [y0, ···, y C-1 is the one-hot representation of the sample category. When the sample belongs to the i-th category, y i = 1, otherwise y i = 0; C is the number of categories.

4. The long-tail data classification method based on class activation heatmap according to claim 2, characterized in that, The generation of learnable prototypes and standard class activation heatmaps is specifically: For each few-shot class c′, generate a learnable prototype Φ using the dataset of that few-shot class c′ ; Use the feature vectors obtained from K samples in the few-shot class c′ through the feature extraction module, and average each column of the i-th few-shot feature vector to convert it into a row Finally, merge them as the prototype Φ c′ Initial value of: Using the learnable prototype Φ c′ Perform a convolution operation on the feature vectors of the few-shot class dataset to obtain a standard class activation heatmap The formula is: Among them, Φ c′ represents the learnable prototype of the few-shot class c′; F c′ represents the feature vector of the few-shot class.

5. The long-tail data classification method based on class activation heatmap according to claim 4, wherein The re-extraction of the feature vectors of the few-shot category dataset is specifically: Standard class activation heatmap Perform a convolution operation to convert it into a single-channel map Input the single-channel image into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot category dataset, which is expressed as: Extraction where, σ(.) is the Sigmoid function, represents the Hadamard product, F c′ is the feature vector of the few-shot class c ′ data set.

6. The long-tail data classification method based on class activation heatmap according to claim 5, wherein The input into the classification model to obtain the classification results is specifically: Feature vectors of the re-extracted few-shot class dataset Perform global average pooling operation to obtain the global representation x c′ ; Through the global representation x c′ and w c′ After normalization, generate the few-shot class c ′ The class score s of c′ , the formula is: where <,> represents the dot product operation, g is a hyperparameter, and w c′ is the parameter of the classification module for class c ′ ; Obtain the class score of the few-shot class c ′ After that, the classification result is determined by the Softmax function.

7. The method for classifying long-tail data based on class activation heatmap according to claim 6, wherein The cross-entropy loss function is expressed as: Among them, p = [p0, ···, p C′-1 represents the predicted probabilities of all few-shot classes, and p i represents the predicted probability of the i-th few-shot class. y = [y0, ···, y C′-1 is the one-hot representation of the few-shot classes. When the few-shot sample belongs to the i-th few-shot class, y i = 1; otherwise, y i = 0; C′ is the number of few-shot classes.

8. The method for classifying long-tail data based on class activation heatmaps according to claim 7, characterized in that The feature extraction module of the long-tail data classification model uses a ResNet-50 network; During the iterative training process, the long-tail data classification model uses an SGD optimizer with a momentum of 0.9 to optimize the parameters, the batch size is 256, it is trained for 200 epochs during the pre-training process, and it is trained for 10 epochs during the iterative training of the class activation map module and the classification module; the training uses a cosine learning rate that gradually decreases from 0.1 to 0, and the hyperparameter g is set to 0.5 and 16 respectively during the pre-training process and the iterative training of the class activation map module and the classification module.

9. A long-tail data classification system based on class activation heatmaps, characterized in that, Applied to the long-tail data classification method based on class activation heatmap described in any one of claims 1-8, it includes a data acquisition module, a model construction module, a pre-training module, an iterative training module, and a result output module; The data acquisition module is used to acquire a long-tail image dataset; the long-tail image dataset contains multiple categories, each image sample is labeled with its belonging category, and the number of image samples in different categories has a large difference; The model construction module is used to construct a long-tail data classification model; the long-tail data classification model includes a feature extraction module, a class activation map module, and a classification module; the class activation map module is located between the feature extraction module and the classification module; The pre-training module is used to pre-train the feature extraction module and the classification module on the long-tail image dataset using the cross-entropy loss function. After the pre-training is completed, the parameters of the feature extraction module are frozen to obtain the categories of the long-tail image dataset; The iterative training module is used to iteratively train the long-tail data classification model on the long-tail image dataset, including: Defining the categories in the long-tail image dataset with the data volume less than τ as few-shot categories to obtain a dataset of few-shot categories; Inputting the images in the dataset of few-shot categories into the class activation map module to generate learnable prototypes and standard class activation heatmaps; Converting the standard class activation heatmap into a single-channel map, inputting it into the feature extraction module with frozen parameters to re-extract the feature vectors of the few-shot category dataset, and inputting them into the classification model to obtain classification results; Using the cross-entropy loss function to iteratively train the class activation map module and the classification module, updating the parameters of the class activation map module and the classification module until convergence to obtain a trained long-tail data classification model; The result output module is used to input the images in the long-tail data set to be predicted into the trained long-tail data classification model to obtain the predicted classification results.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by a processor, it implements the long-tail data classification method based on class activation heatmap described in any one of claims 1-8.

Images