An image multi-label classification method based on frequent item set

Abstract

The application discloses a kind of frequent item set-based image multi-label classification method, comprising the following steps: (1) in image dataset, using frequent item set mining algorithm to mine the frequent item set of dataset;(2) utilize the frequent item set mined to construct tree structure's image multi-label classification network;(3) the feature extraction network and image multi-label classification network are combined to obtain total classification network, wherein the feature vector output by feature extraction network is as the input of image multi-label classification network;(4) training total classification network model until model converges;(5) the image to be classified is input into trained total classification network model, and the classification result of multi-label is obtained.The frequent item set generated by frequent item set mining algorithm of the application can automatically generate more reasonable image multi-label classification network, and higher image multi-label classification precision can be provided.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image classification, and particularly relates to a frequent item set-based image multi-label classification method. BACKGROUND

[0002] The image classification task is one of the most basic and important tasks in the field of computer vision. The image classification task is ubiquitous in real life. For example, in the medical field, we need to determine whether there is a risk of suffering from certain diseases according to CT pictures. For a human image, we can classify it as a man or a woman. In many cases, the classification of a sample is diverse, for example, a human picture can be both a man and a teacher.

[0003] Most of the current multi-label classification tasks use deep learning models, for example, Jiawei Zhao et al. proposed a Transformer-based Dual Relation Graph for Multi-label Image Recognition in the International Conference on Computer Vision in 2021. The work optimizes the Transformer in the deep learning model, thereby performing multi-label classification of pictures.

[0004] The Chinese patent document with the publication number CN114004992A discloses a multi-label classification model training method and an image multi-label classification method. The multi-label classification model training method performs multi-granularity segmentation on the sampled pictures, combines and splices the segmented pictures of each granularity to obtain a multi-granularity spliced picture sequence, and then inputs the multi-granularity spliced picture sequence into a Transformer model to complete the training of the model.

[0005] The Chinese patent document with the publication number CN115546530A discloses an image multi-label classification method, which includes training an embedding model for image multi-label classification through federated learning. During the training process, the server uses a divergence regular function with label relationship to optimize the category embedding matrix, which is a parameter of the embedding model. The trained embedding model is used for image multi-label classification.

[0006] However, different data sets have different characteristics, and most of the network structures in the current deep learning model for actual multi-label classification tasks are manually confirmed, and cannot automatically generate network structures according to the co-occurrence information between the multi-labels of the data set, so the co-occurrence information between the labels of the data set is actually not fully utilized. At the same time, the existing model is still in the form of multi-task for multi-label classification task, that is, the labels always share most of the neural network layers, which will lead to the decline of classification accuracy. SUMMARY

[0007] The application provides a frequent item set-based image multi-label classification method, which generates a more reasonable image multi-label classification network through co-occurrence information generated by a frequent item set mining algorithm, and can provide higher image multi-label classification accuracy.

[0008] A frequent item set-based image multi-label classification method, comprising the following steps:

[0009] (1) In the image data set, a frequent item set mining algorithm is used to mine the frequent item set of the data set;

[0010] (2) The mined frequent item set is used to construct a tree-shaped image multi-label classification network;

[0011] (3) The feature extraction network and the image multi-label classification network are combined to obtain a total classification network, wherein the feature vector output by the feature extraction network is used as the input of the image multi-label classification network;

[0012] (4) The total classification network model is trained until the model converges;

[0013] (5) The image to be classified is input into the trained total classification network model to obtain the multi-label classification result.

[0014] Further, in step (1), the frequent item set mining algorithm used is Apriori.

[0015] The specific process of step (2) is as follows:

[0016] (2-1) For the frequent item set set L, sort according to the frequency sup of each item set in the data set from large to small, and the item set I with higher sup is arranged in front after sorting;

[0017] (2-2) Define N as the total number of classifications, and N is also equal to the total number of item sets with size 1;

[0018] First, generate N nodes in an empty graph, each node maintains an item set with size 1, which has only one item corresponding to different classification labels, and the parent nodes of the N nodes are all empty at this time; meanwhile, let k be equal to 1, and L kthe kth largest frequent item set in the frequent item set set L;

[0019] (2-3) extracting all items in L k from the root node of the tree where the corresponding point in the graph is located, and putting them into set R, and setting the union set of all item sets in all nodes in set R as S; if all items in the union set S composed of all item sets maintained in all nodes in set R have appeared in the frequent items at this time, that is, S∈L k , a new node is generated, and the item set information of the node is defined as L k , the parent node of all nodes in R is set to the newly generated node, and the parent node of the newly generated node is set to null;

[0020] (2-4) adding 1 to k, and repeating step (2-3) if L k exists at this time;

[0021] (2-5) after the traversal is completed, if all nodes do not form a tree, a new node is generated, the item set of the new node is the set of all items, and the root node set of all trees in the graph is set as R, and the parent node of all nodes in R is set to the new node, finally forming an image multi-label classification network in a tree structure;

[0022] (2-6) filling neural network layers in the image multi-label classification network to ensure that each leaf node in the tree structure can generate a probability that a sample belongs to the item of the leaf node.

[0023] In step (2-6), filling neural network layers in the image multi-label classification network is specifically:

[0024] using a 5-layer Block structure in a non-leaf node, and the FC of each layer of the Block structure is a fully connected neural network layer with an input length of V and an output length of V; using a fully connected neural network layer with an input length of V and an output length of 1 and a Sigmod activation function in a leaf node to output the probability of belonging to the label;

[0025] for all leaf nodes with a probability greater than 0.5, obtaining the label corresponding to the item, and all labels are used as the classification of the sample.

[0026] In step (3), the picture data set has a picture length of H, a width of W, and a channel number of C; using a 1-layer fully connected neural network layer with an input length of H*W*C and an output length of V and a 5-layer Block structure as a feature extraction network, wherein all FCs in the 5-layer Block structure are fully connected neural network layers with an input length of V and an output length of V.

[0027] In step (4), the picture with the dimension (C, H, W) is flattened into a one-dimensional vector with the length H*W*C, and is input into the network training, and the loss function is defined as the sum of the cross-entropy and the regularization Loss during the training process.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] 1. The image multi-label classification network is constructed according to the sup of the frequent item set from large to small, and the finally generated network is a tree structure. For the whole tree structure, the larger the sup is, the earlier the new node is created for connection, and the deeper the depth in the tree structure is. The deeper the depth is, the more parameters are used for classification, and the higher the classification accuracy is.

[0030] 2. The present application fully utilizes the co-occurrence information between the labels in the data set to dynamically construct a reasonable image multi-label classification network. For different data sets, the image multi-label classification network can be automatically generated, which fills the gap of the previous technology that the multi-label classification network is relatively fixed and mostly manually determined, and the co-occurrence information between the multi-labels in the data set is not fully utilized. At the same time, since the constructed image multi-label classification network is a tree structure, the paths of the different labels on the tree are different, which means that the neural network layers passed by different labels are different. This is completely different from the multi-task classification method used in most works. At the same time, the structure of the image multi-label classification network conforms to the characteristic that the more frequent the label appears, the deeper the depth in the tree is, in other words, the more neural network parameters are used, which can more effectively predict. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The flowchart of the image multi-label classification method based on the frequent item set of the present application;

[0032] Figure 2 The example using the Apriori algorithm in the embodiment of the present application;

[0033] Figure 3 The image multi-label classification network structure diagram generated by using the frequent item set in Figure 2

[0034] The block structure diagram in the embodiment of the present application. Figure 4 DETAILED DESCRIPTION

[0035] The present application will be further described in detail below in combination with the drawings and embodiments. It should be pointed out that the following embodiments are intended to facilitate the understanding of the present application, and do not limit the present application in any way.

[0036] As Figure 1 ​As shown, a frequent item set-based image multi-label classification method includes the following steps:

[0037] S01, in the image data set, using data mining algorithm Apriori mining frequent item set L.

[0038] First define I = {I1, I2, … I m} is an item set, where I1, I2, … l m are all items in the item set, and each item can correspond to a label. Then define D as the existing image data set, which contains n image data, and the label set of the i-th data is T i Define the count of item set I as:

[0039]

[0040] That is, count(I) is the number of times the item set I appears in the image data set, and in order to obtain the frequency of item set I in the data set, the formula is defined as:

[0041]

[0042] From the above formula, the frequency of each item set in the data set can be obtained, and a threshold sup_threshold is defined. The item set whose sup is greater than sup_threshold is called a frequent item set. For item set I, if there is no sup of I's superset and I is equal, then such a frequent item set can be defined as a closed frequent item set.

[0043] The most naive way to get frequent item sets is to enumerate all possible item sets. For each enumerated item set I, traverse the entire data set to obtain sup, and finally judge whether sup is greater than threshold sup_threshold. However, the time complexity of this method is exponential, which is not suitable for hundreds of classification numbers in practical applications.

[0044] In order to obtain frequent item sets more efficiently, the famous algorithm Apriori in the field of frequent item set mining is used. Apriori mainly uses a simple and obvious property: for any item set I, if it is not a frequent item set, then for any superset of I, it cannot be a frequent item set. In other words, for a frequent item set, any non-empty subset of it is also a frequent item set. The basic idea of the Apriori algorithm is to use this property to mine frequent item sets.

[0045] The item set with sup greater than sup_threshold is defined as a frequent item set, and after using the Apriori algorithm, we can obtain the set L of all frequent item sets and the sup corresponding to each frequent item set I in L. S02, using the mined frequent item sets to construct a tree-shaped image multi-label classification network.

[0046] (2-1) For the set L of frequent item sets, we sort them in descending order according to the sup of each item set. The higher the sup of an item set I after sorting, the higher it is placed in front.

[0047] (2-2) Define N as the total number of classifications, which is also equal to the total number of item sets with size 1. We first generate N nodes in an empty graph, each of which maintains an item set with size 1 containing only one item corresponding to different classifications, and the parent nodes of these N nodes are all empty. At the same time, we set k equal to 1. k is the kth largest frequent item set in the set L of frequent item sets.

[0048] (2-3) Take out all the items in L k from the root nodes of the corresponding trees in the graph and put them into the set R. The union of all item sets in the set R is S, and if all items in the union S of the item sets maintained by all nodes in the set R have appeared in the current traversal of the frequent item set, that is, S∈L k , a new node is generated, and the item set information of this node is defined as L k . The parent nodes of all nodes in R are set to the newly generated node, and the parent node of the newly generated node is set to empty.

[0049] (2-4) Increment k, and if L k exists at this time, repeat step (2-3).

[0050] (2-5) After the traversal is completed, if all nodes do not form a tree, a new node is generated, and the item set of this new node is the set of all items. Set the set of root nodes of all trees in the graph as R, and set the parent nodes of all nodes in R to the new node. At this time, since we are constructing from bottom to top, we will eventually form a tree-shaped network. And since we prefer to use the most frequent frequent item to connect nodes, the frequent item is more frequent, and the depth of the tree-shaped network is also deeper, so we can use more parameters for prediction.

[0051] (2-6) Fill the tree structure with any neural network layer. Take the simplest example, if the length of the feature vector output by the feature extraction network is V, then we can use a fully connected neural network layer with input length V and output length V in the non-leaf node, and a fully connected neural network layer with input length V and output length 1 in the leaf node, and then use the Sigmod activation function to obtain the probability of the item set belonging to the leaf node.

[0052] The neural network layer filling the tree structure has diversity, and can also be filled with convolutional neural network or Transformer or other neural network structure, as long as each leaf node can generate a probability of the sample belonging to the item of the leaf node.

[0053] (2-7) The size of the item set of the leaf node is 1, and the leaf node will output the probability of the sample belonging to the item. After inputting the sample and passing through the neural network, all items in the item set corresponding to the leaf node with a probability greater than 0.5 are taken out, and the multi-label classification of the image sample can be obtained.

[0054] As shown in Figure 2 , it is an example of using an Apriori algorithm according to an embodiment of the present application, Figure 3 is an image multi-label classification network structure diagram generated by using the frequent item set in Figure 1 .

[0055] S03, combine the feature extraction network and the image multi-label classification network to obtain the total classification network, wherein the feature vector output by the feature extraction network is used as the input of the image multi-label classification network.

[0056] Definition Figure 4The network structure in the Block is first using a Relu function, then passing through a first fully connected layer FC and using a Relu function again, then passing through a fully connected layer FC again, and the result obtained by adding the input is the output of the Block. The application uses 1 layer of fully connected neural network layer plus 5 layers of Block as a feature extraction network. Since the picture used in the embodiment process is 32 long, 32 wide, and 3 channels, and is flattened into a single-dimensional vector of 3072 long, the first layer of fully connected neural network layer has an input length of 3072 and an output length of 128. The 5 layers of Block have the same structure, and all the FCs are fully connected neural network layers with an input length of 128 and an output length of 128. After the picture of 32 long, 32 wide, and 3 channels is flattened into a vector of 3072 long and passed through the first layer of fully connected neural network layer, a single-dimensional feature vector of 128 long is generated, and the output of the 5 layers of Block structure is always a single-dimensional vector of 128 long. Finally, a single-dimensional feature vector of 128 long is generated for the multi-label classification network.

[0057] In the image multi-label classification network of the tree structure in the embodiment of the application, the Block structure in the non-leaf node is also used, and the FC of each layer is a fully connected neural network layer with an input length of 128 and an output length of 128. Figure 4 In the leaf node, a fully connected neural network layer with an input length of 128 and an output length of 1 and a Sigmod activation function are used to output the probability of belonging to the label.

[0058] For all leaf nodes with a probability greater than 0.5, the label corresponding to the item is obtained, and all labels are classified as the sample.

[0059] S04, flattening the picture matrix of 32 long, 32 wide, and 3 channels into a one-dimensional vector of 3072 long to send into the network for training, training the total classification network model until the model converges, and defining the loss function as the sum of the cross entropy and the regularization loss.

[0060] S05, flattening the image to be classified and inputting it into the trained total classification network model to obtain the multi-label classification result.

[0061] For different data sets, the application automatically generates a multi-label classification network using the co-occurrence information between labels. The network can configure more neural network layer parameters for labels that appear more frequently in the data set, which will help improve the accuracy of the image multi-label classification task. Meanwhile, the automatically generated neural network can use different neural network layers for different classification labels, which can improve the accuracy and avoid the defects caused by manually determining the neural network.

[0062] In order to obtain the co-occurrence information of the labels in the data set, the application uses the algorithm in the field of data mining for mining. By using the frequent item set mined from the data, a technology of automatically generating an image multi-label classification network by using the frequent item set is innovatively proposed. The network generated by using the technology will meet the requirement that the more frequent the item set is, the more neural network parameters are used, so as to provide higher multi-label classification accuracy.

[0063] The above-described embodiments have described the technical solutions and beneficial effects of the application in detail. It should be understood that the above-described is only a specific embodiment of the application and is not used to limit the application. Any modification, supplement and equivalent replacement made within the principle range of the application shall be included in the protection range of the application.

Claims

1. A multi-label image classification method based on frequent itemsets, characterized in that, Includes the following steps: (1) In the image dataset, the frequent itemset mining algorithm is used to mine the frequent itemsets of the dataset; (2) Construct a tree-structured image multi-label classification network using the mined frequent itemsets; the specific process is as follows: (2-1) For a set of frequent itemsets L, sort them from largest to smallest according to the frequency sup of each itemset in the dataset. After sorting, the itemset I with the higher sup will be ranked earlier. (2-2) Define N as the total number of categories, and N is equal to the total number of itemsets with an itemset size of 1; First, generate N nodes in an empty graph. Each node maintains an itemset of size 1, which contains only one item corresponding to a different category label. At this point, the parent nodes of these N nodes are all empty. Also, let k equal 1, L... k Let sup be the k-th largest frequent itemset in the set of frequent itemsets L; (2-3) L k Take all items from the root node of the tree corresponding to the point in the graph and put them into set R. Let S be the union of the item sets of all nodes in set R. If all items in the union S of the item sets maintained by all nodes in set R have appeared in the frequent items traversed at this time, that is, S∈L k When a new node is generated, the itemset information of that node is defined as L. k Set the parent node of all nodes in R to the newly generated node, and set the parent node of the newly generated node to null. (2-4) Increment k by 1. If L exists at this time... k Repeat steps (2-3); (2-5) After the traversal is completed, if all nodes do not form a tree, a new node is generated. The itemset of the new node is the set of all items. Let R be the set of root nodes of all trees in the graph. Set the parent node of all nodes in R as the new node, and finally form a tree-structured image multi-label classification network. (2-6) Fill the neural network layer in the image multi-label classification network to ensure that each leaf node in the tree structure can generate a probability that a sample belongs to the item of that leaf node; (3) The feature extraction network and the image multi-label classification network are combined to obtain the overall classification network, wherein the feature vector output by the feature extraction network is used as the input of the image multi-label classification network; (4) Train the overall classification network model until the model converges; (5) Input the image to be classified into the trained total classification network model to obtain the multi-label classification result.

2. The image multi-label classification method based on frequent itemsets according to claim 1, characterized in that, In step (1), the frequent itemset mining algorithm used is Apriori.

3. The image multi-label classification method based on frequent itemsets according to claim 1, characterized in that, In steps (2-6), filling the neural network layers in the image multi-label classification network specifically involves: In non-leaf nodes, a 5-layer Block structure is used, where each Block structure's fully connected layer is a fully connected neural network layer with an input length of V and an output length of V. In leaf nodes, a fully connected neural network layer with an input length of V and an output length of 1 and a Sigmoid activation function are used to output the probability of belonging to that label. For all leaf nodes with a probability greater than 0.5, obtain the label corresponding to their item, and use all labels as the classification of the image sample.

4. The image multi-label classification method based on frequent itemsets according to claim 1, characterized in that, In step (3), let the length of the image in the image dataset be H, the width be W, and the number of channels be C; use a fully connected neural network layer with an input length of H*W*C and an output length of V, plus a 5-layer Block structure as the feature extraction network; wherein, all FCs in the 5-layer Block structure are fully connected neural network layers with an input length of V and an output length of V.

5. The image multi-label classification method based on frequent itemsets according to claim 1, characterized in that, In step (4), the image with dimension (C,H,W) is flattened into a one-dimensional vector with length H*W*C and fed into the network for training. During the training process, the loss function is defined as the sum of cross-entropy and regularization loss.

Images