Image classification method based on semantic relationship graph

Abstract

The image classification method based on semantic relationship graph disclosed in the present invention is specifically as follows: 1) prepare an image training sample set, and construct a semantic relationship graph according to the image training sample set; 2) build a network model of the image classification method based on the semantic relationship graph according to the image training sample set and the semantic relationship graph; 3) construct a semantic relevance objective function; 4) construct a local geometric structure consistency objective function; 5) construct a total objective function according to the semantic relevance objective function and the local geometric structure consistency objective function; 6) train the network model built in step 2) on the training sample set according to the total objective function constructed in step 5); 7) input the image to be classified into the trained network model in step 6), and obtain the predicted category of the image at the classification layer. The method can learn image feature vectors with more semantic relevance, and can significantly improve the image classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision image classification, and in particular relates to an image classification method based on a semantic relationship graph. Background Art

[0002] Image classification refers to the process of classifying an input image into appropriate categories based on the content contained in the image. Traditional image classification methods include two stages: image feature extraction and classifier discrimination. Deep convolutional neural networks can learn feature vectors and classifiers suitable for classification tasks from the original image end-to-end. Currently, the best image classification methods are based on deep convolutional neural networks. Although deep convolutional neural networks have achieved the best classification accuracy on almost all image classification datasets, in specific practical applications, the classification accuracy has not met people's expectations.

[0003] At present, most methods to improve image classification accuracy are achieved by increasing the complexity of the network model or increasing the number of labeled training samples. Increasing the complexity of the network model will consume more computing resources, affecting the use of these methods in lightweight devices and mobile terminals; manually labeling a large number of training samples requires a lot of manpower, material and financial resources, and often requires relevant professional knowledge. These image classification methods driven by large models and big data limit their practical application scope.

[0004] In conclusion, how to improve the image classification accuracy without increasing the complexity of the network model and the number of labeled training samples is of great research significance. Summary of the invention

[0005] The purpose of the present invention is to provide an image classification method based on a semantic relationship graph, which does not rely on a deep convolutional neural network structure, can learn a vector representation of the latent semantic information of a category label, can learn an image feature vector with more semantic relevance, and can significantly improve the image classification accuracy.

[0006] The technical solution adopted by the present invention is an image classification method based on a semantic relationship graph, comprising the following steps:

[0007] 1) Prepare an image training sample set and construct a semantic relationship graph based on the image training sample set;

[0008] 2) According to the image training sample set and the semantic relationship graph, a network model of the image classification method based on the semantic relationship graph is built;

[0009] 3) Constructing semantic relevance objective function;

[0010] 4) Construct local geometric structure consistency objective function;

[0011] 5) constructing the overall objective function based on the semantic relevance objective function and the local geometric structure consistency objective function;

[0012] 6) training the network model constructed in step 2) on the training sample set according to the total objective function constructed in step 5);

[0013] 7) Input the image to be classified into the network model trained in step 6) and obtain the predicted category of the image at the classification layer.

[0014] The present invention is also characterized in that

[0015] The specific implementation method of step 1) is as follows:

[0016] Step 1.1: Assumptions is the image training sample set, X i represents the i-th training sample image, c i ∈{l1,l2,…,l C} represents the training sample image X i The category labels, {l1,l2,…,l C} represents the category label set of the image training sample set, l j represents the j-th category label of the training sample set, j = 1, 2, ..., C, C represents the number of category labels in the training sample set, and N represents the total number of training samples;

[0017] Step 1.2: Build a semantic relationship graph: Build an undirected graph in is a vertex set, ε={(l i ,l j )|i,j=1,2,…,C} is an edge set, A=(a ij ) C×C is the adjacency matrix, a ij Represents the category label l i With l j The similarity between Among them, w i and w j Represents the category label l i and l j The word embedding vector of , exp(·) represents the exponential function, ||·|| represents the modulus of the vector, and the parameter η is greater than zero and is an adjustable parameter.

[0018] The specific implementation method of building a network model of the image classification method based on the semantic relationship graph in step 2) is as follows:

[0019] A deep convolutional neural network is selected, the last layer of the deep convolutional neural network is a classification layer, the number of neurons in the classification layer is set to C, and the activation function of the classification layer is a softmax activation function; the penultimate layer of the deep convolutional neural network is used as a feature layer, and the feature layer is preceded by a plurality of convolutional layers; the value of a node in a semantic relationship graph is set to a word embedding vector of its corresponding category label, and then a vector representation of the latent semantic information of the corresponding category label is learned on the semantic relationship graph through a two-layer graph convolutional network; the semantic relevance objective function and the local geometric structure consistency objective function constructed by the present invention act on the vector representation of the latent semantic information of the feature layer and the category label at the same time.

[0020] The specific implementation method of step 3) is as follows:

[0021] Define the training sample image X i The eigenvector of i , that is, the training sample image X i Input to the network model built in step 2) and the output at the feature layer is x i , assuming that the dimension of the feature vector is D;

[0022] In the network model built in step 2), the input of the first layer of the graph convolution network is a semantic relationship graph, and the value of each node in the semantic relationship graph is the word embedding vector of its corresponding category label; the output of the first layer of graph convolution is the input of the second layer of graph convolution. After the second layer of graph convolution operation, the output of the second layer is still a graph, and the value of each node in the output graph is a D-dimensional vector, which represents the vector representation of the latent semantic information of the corresponding category label;

[0023] Assume that the class label l i The vector representation of the latent semantic information is v li , the semantic relevance objective function L1 adopted is:

[0024]

[0025] in, Represents the category label c i The vector representation of the latent semantic information.

[0026] During the network training process, minimizing the semantic relevance objective function can enable the network model to learn image feature vectors with more semantic relevance.

[0027] The specific implementation method of step 4) is as follows:

[0028] The objective function used to construct the consistency of local geometric structure is:

[0029]

[0030] in, The parameter σ is greater than zero and is an adjustable parameter.

[0031] During the network training process, minimizing the local geometric structure consistency objective function can make the local geometric structures of the feature space and the vector representation space of the latent semantic information consistent. That is, if the vector representations of the latent semantic information of two images are close to each other, then their corresponding feature vectors should also be close to each other.

[0032] The overall objective function constructed in step 5) for:

[0033]

[0034] in represents the softmax classification loss function of the training sample set, represents the layer parameter set of the network model built in step 2), L1 and L2 are the semantic relevance objective function and the local geometric structure consistency objective function respectively, and the parameters λ1 and λ2 are both greater than zero, which are used to adjust the balance between the three terms on the right side of the above formula.

[0035] The beneficial effects of the present invention are:

[0036] (1) The method of the present invention proposes to learn the vector representation of the implicit semantic information of the category label through a graph convolutional network.

[0037] (2) The semantic relevance objective function proposed by the method of the present invention enables the network model to learn image feature vectors with more semantic relevance.

[0038] (3) The local geometric structure consistency objective function proposed by the method of the present invention can make the local geometric structures of the feature space and the vector representation space of the latent semantic information consistent.

[0039] (4) The method of the present invention can significantly improve the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of a network model of an image classification method based on a semantic relationship graph of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The present invention provides an image classification method based on a semantic relationship graph. Figure 1 As shown, the following steps are included:

[0043] 1) Prepare an image training sample set and construct a semantic relationship graph based on the image training sample set;

[0044] The specific implementation method of step 1) is as follows:

[0045] Step 1.1: Assumptions is the image training sample set, X i represents the i-th training sample image, c i ∈{l1,l2,…,l C} represents the training sample image X i The category labels, {l1,l2,…,l C} represents the category label set of the image training sample set, l j represents the j-th category label of the training sample set, j = 1, 2, ..., C, C represents the number of category labels in the training sample set, and N represents the total number of training samples;

[0046] Step 1.2: Build a semantic relationship graph: Build an undirected graph in is a vertex set, ε={(l i ,l j )|i,j=1,2,…,C} is an edge set, A=(a ij ) C×C is the adjacency matrix, a ij Represents the category label l i With l j The similarity between Among them, w i and w j Represents the category label l i and l j The word embedding vector of , exp(·) represents the exponential function, ||·|| represents the modulus of the vector, and the parameter η is greater than zero and is an adjustable parameter.

[0047] 2) According to the image training sample set and the semantic relationship graph, a network model of the image classification method based on the semantic relationship graph is built;

[0048] The specific implementation method of building a network model of the image classification method based on the semantic relationship graph in step 2) is as follows:

[0049] See also Figure 1As shown in the invention: a deep convolutional neural network is selected, the last layer of the deep convolutional neural network is a classification layer, the number of neurons in the classification layer is set to C, and the activation function of the classification layer is a softmax activation function; the penultimate layer of the deep convolutional neural network is used as a feature layer, and the feature layer is preceded by several convolutional layers; the value of the node in the semantic relationship graph is set to the word embedding vector of its corresponding category label, and then the vector representation of the latent semantic information of the corresponding category label is learned on the semantic relationship graph through a two-layer graph convolutional network; the semantic relevance objective function and the local geometric structure consistency objective function constructed by the present invention act on the vector representation of the latent semantic information of the feature layer and the category label at the same time.

[0050] 3) Constructing semantic relevance objective function;

[0051] The specific implementation method of step 3) is as follows:

[0052] Define the training sample image X i The eigenvector of i , that is, the training sample image X i Input to the network model built in step 2) and the output at the feature layer is x i , assuming that the dimension of the feature vector is D;

[0053] In the network model built in step 2), the input of the first layer of the graph convolution network is a semantic relationship graph, and the value of each node in the semantic relationship graph is the word embedding vector of its corresponding category label; the output of the first layer of graph convolution is the input of the second layer of graph convolution. After the second layer of graph convolution operation, the output of the second layer is still a graph, and the value of each node in the output graph is a D-dimensional vector, which represents the vector representation of the latent semantic information of the corresponding category label;

[0054] Assume that the class label l i The vector representation of the latent semantic information is The semantic relevance objective function L1 adopted is:

[0055]

[0056] in, Represents the category label c i The vector representation of the latent semantic information.

[0057] During the network training process, minimizing the semantic relevance objective function can enable the network model to learn image feature vectors with more semantic relevance.

[0058] 4) Construct local geometric structure consistency objective function;

[0059] The specific implementation method of step 4) is as follows:

[0060] The objective function used to construct the consistency of local geometric structure is:

[0061]

[0062] in, The parameter σ is greater than zero and is an adjustable parameter.

[0063] During the network training process, minimizing the local geometric structure consistency objective function can make the local geometric structures of the feature space and the vector representation space of the latent semantic information consistent. That is, if the vector representations of the latent semantic information of two images are close to each other, then their corresponding feature vectors should also be close to each other.

[0064] 5) constructing the overall objective function based on the semantic relevance objective function and the local geometric structure consistency objective function;

[0065] The overall objective function constructed in step 5) for:

[0066]

[0067] in represents the softmax classification loss function of the training sample set, represents the layer parameter set of the network model built in step 2), L1 and L2 are the semantic relevance objective function and the local geometric structure consistency objective function respectively, and the parameters λ1 and λ2 are both greater than zero, which are used to adjust the balance between the three terms on the right side of the above formula.

[0068] 6) training the network model constructed in step 2) on the training sample set according to the total objective function constructed in step 5);

[0069] 7) Input the image to be classified into the network model trained in step 6) and obtain the predicted category of the image at the classification layer.

[0070] Example

[0071] The proposed method is experimentally verified on two commonly used deep convolutional neural networks (Quick-model and NIN) and two image classification datasets (CIFAR10 and CIFAR100). Each image in the CIFAR10 and CIFAR100 datasets is a 32×32 RGB three-channel color image.

[0072] The CIFAR-10 dataset consists of 60,000 images in 10 categories, of which there are 50,000 training sample images (5,000 for each category) and 10,000 test sample images (1,000 for each category). The CIFAR100 dataset consists of 60,000 images in 100 categories, of which there are 50,000 training sample images (500 for each category) and 10,000 test sample images (100 for each category).

[0073] Table 1 shows the classification accuracy (%) of different methods based on Quick-model on CIFAR10 and CIFAR100 datasets. In Table 1, Quick-model+softmax and Quick-model+method of the present invention respectively represent the classification results of Quick-model trained using softmax classification loss function and method of the present invention. Table 2 shows the classification accuracy (%) of different methods based on NIN on CIFAR10 and CIFAR100 datasets. In Table 2, NIN+softmax and NIN+method of the present invention respectively represent the classification results of NIN trained using softmax classification loss function and method of the present invention.

[0074] It can be seen from Tables 1 and 2 that the methods proposed in the present invention are significantly better than the classification accuracy of the corresponding models trained directly using the softmax classification loss function, which fully proves the effectiveness of the methods of the present invention.

[0075] Table 1

[0076] method CIFAR10 CIFAR100 Quick-model+softmax 76.53 44.13 Quick-model+method of the present invention 82.54 49.77

[0077] Table 2

[0078] method CIFAR10 CIFAR100 NIN+softmax 89.59 64.32 NIN+method of the present invention 91.76 67.50

Claims

1. An image classification method based on a semantic relationship graph, characterized in that: The following steps are involved: 1) Prepare an image training sample set and construct a semantic relationship graph based on the image training sample set; The specific implementation method of step 1) is as follows: Step 1.1: Assumptions is the image training sample set, X i represents the i-th training sample image, c i ∈{l1,l2,…,l C } represents the training sample image X i The category labels, {l1,l2,…,l C } represents the category label set of the image training sample set, l j represents the j-th category label of the training sample set, j = 1, 2, ..., C, C represents the number of category labels in the training sample set, and N represents the total number of training samples; Step 1.2: Build a semantic relationship graph: Build an undirected graph in is a vertex set, ε={(l i ,l j )|i,j=1,2,…,C} is an edge set, A=(a ij ) C×C is the adjacency matrix, a ij Represents the category label l i With l j The similarity between Among them, w i and w j Respectively represent the category label l i and l j The word embedding vector, exp(·) represents the exponential function, ||·|| represents the modulus of the vector, and the parameter η is greater than zero; 2) According to the image training sample set and the semantic relationship graph, a network model of the image classification method based on the semantic relationship graph is built; 3) Constructing semantic relevance objective function; The specific implementation method of step 3) is as follows: Define the training sample image X i The eigenvector of i , that is, the training sample image X i Input to the network model built in step 2) and the output at the feature layer is x i , assuming that the dimension of the feature vector is D; In the network model built in step 2), the input of the first layer of the graph convolution network is a semantic relationship graph, and the value of each node in the semantic relationship graph is the word embedding vector of its corresponding category label; the output of the first layer of graph convolution is the input of the second layer of graph convolution. After the second layer of graph convolution operation, the output of the second layer is still a graph, and the value of each node in the output graph is a D-dimensional vector, which represents the vector representation of the latent semantic information of the corresponding category label; Assume that the class label l i The vector representation of the latent semantic information is v li , the semantic relevance objective function L1 adopted is: Among them, v ci Represents the category label c i The vector representation of the latent semantic information; 4) Construct local geometric structure consistency objective function; 5) constructing the overall objective function based on the semantic relevance objective function and the local geometric structure consistency objective function; 6) Training the network model constructed in step 2) on the training sample set according to the total objective function constructed in step 5); 7) Input the image to be classified into the network model trained in step 6) and obtain the predicted category of the image at the classification layer.

2. The image classification method based on semantic relationship graph according to claim 1, characterized in that: The specific implementation method of building a network model of the image classification method based on the semantic relationship graph in step 2) is as follows: A deep convolutional neural network is selected, the last layer of the deep convolutional neural network is the classification layer, the number of neurons in the classification layer is set to C, and the activation function of the classification layer is the softmax activation function; the penultimate layer of the deep convolutional neural network is used as the feature layer, and the feature layer is preceded by several convolutional layers; the value of the node in the semantic relationship graph is set to the word embedding vector of its corresponding category label, and then a two-layer graph convolutional network is used on the semantic relationship graph to learn the vector representation of the latent semantic information of the corresponding category label.

3. The image classification method based on semantic relationship graph according to claim 1, characterized in that: The specific implementation method of step 4) is as follows: The objective function used to construct the consistency of local geometric structure is: in, The parameter σ is greater than zero.

4. The image classification method based on semantic relationship graph according to claim 3 is characterized in that: The overall objective function constructed in step 5) for: in represents the softmax classification loss function of the training sample set, represents the layer parameter set of the network model built in step 2), L1 and L2 are the semantic relevance objective function and the local geometric structure consistency objective function respectively, and the parameters λ1 and λ2 are both greater than zero, which are used to adjust the balance between the three terms on the right side of the above formula.

Images