Class classification method, class classification device, class classification program, learning method, learning device, and learning program
The deep learning-based method generates a continuously changing weight vector for each class to address the challenge of wide-ranging feature distributions, achieving accurate class classification and improved performance in multimodal scenarios.
Patent Information
- Application Number
- PCT/JP2024/017081
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-04-01
- Filing Date
- 2024-05-08
- Publication Date
- 2024-12-05
AI Technical Summary
Existing class classification methods face challenges in accurately classifying inputs when feature values within a class are distributed over a wide range, as they struggle to express the multimodal distribution of features using a single weight vector.
A deep learning-based method that generates a weight vector continuously changing according to the feature vector for each class, using a trained generator to calculate scores for each class and classify the input data, allowing for accurate classification even with wide-ranging feature values.
This approach enables accurate class classification by effectively expressing multiple variations of features within a class, improving classification performance and interpretability, especially in scenarios with diverse and multimodal data distributions.
Smart Images

Figure JP2024017081_05122024_PF_FP_ABST
Abstract
Description
Classification method, classification device, classification program, learning method, learning device, and learning program
[0001] The present disclosure relates to a technique for performing class classification using a deep learning model, and a technique for training a deep learning model used to perform class classification.
[0002] In conventional classification tasks, a linear classifier is used, which assigns one weight vector to each class. Therefore, when features that follow a multimodal distribution are input to the classifier, it is difficult to classify the input image based on the features.
[0003] To address this issue, for example, Non-Patent Document 1 proposes a method of representing multiple classes using multiple weight vectors instead of one weight vector.
[0004] However, with the above-described conventional techniques, it is difficult to accurately classify classes when the feature values within a class are distributed over a wide range, and further improvements are needed.
[0005] Wenguan Wang, Cheng Han, Tianfei Zhou, Dongfang Liu, “Visual Recognition with Deep Nearest Centroids”, ICLR2023, February 2, 2023
[0006] The present disclosure has been made to solve the above problems, and aims to provide a technology that can accurately perform class classification even when feature values within a class are distributed over a wide range.
[0007] A class classification method according to the present disclosure is a computer-executed class classification method, including: obtaining a feature vector extracted from input data; using a trained generator to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes depending on the value of the feature vector; using a trained calculator to calculate a score for each of the plurality of classes based on the feature vector and the plurality of weight vectors generated for each of the plurality of classes; classifying the classification target of the input data into one of the plurality of classes based on the plurality of scores calculated for each of the plurality of classes; and outputting the classification result.
[0008] According to the present disclosure, accurate class classification can be performed even when feature values within a class are distributed over a wide range.
[0009] 1 is a block diagram showing the configuration of a classification device according to the present embodiment. FIG. 2 is a block diagram showing the configuration of a first weight generation unit shown in FIG. 1. FIG. 3 is a block diagram showing the configuration of a learning device according to the present embodiment. FIG. 4 is a flowchart for explaining the operation of a classification device according to the present embodiment. FIG. 5 is a flowchart for explaining the operation of a learning device according to the present embodiment. FIG. 6 is a diagram showing the distribution of features on a feature space in classification by conventional DNC. FIG. 7 is a diagram showing the distribution of features on a feature space in classification by the classification device according to the present embodiment. FIG. 8 is a diagram showing evaluation results of two conventional classification methods and the classification method of the present embodiment for two data sets. FIG. 9 is a diagram showing evaluation results of two conventional classification methods and the classification method of the present embodiment for two types of labels in CIFAR-20. FIG. 10 is a diagram showing evaluation results of two conventional classification methods and the classification method of the present embodiment for two types of labels in ImageNet.
[0010] (Findings underlying the present disclosure) Classification tasks have traditionally been used in a variety of tasks, such as image identification, object detection, and semantic segmentation, and improving classification accuracy is an important challenge. A network model for classifying images generally consists of a backbone that extracts features from images and a classifier that classifies each class based on the extracted features. The classifier identifies classes by comparing the magnitude of scores (similarity) calculated by the inner product of a single weight vector and the feature for each class. In recent years, in order to improve classification performance, the number of input images has increased, and the types of input images have also increased, resulting in a more diverse distribution of input images. As the distribution of input images becomes more diverse, it becomes impossible to represent the data sets of each class in feature space using a single set, and the distribution of the data sets becomes multimodal. In this case, it becomes difficult to represent the distribution within a class using a single weight vector.
[0011] To address this issue, Non-Patent Document 1 proposes a method for representing multiple classes using multiple weight vectors instead of a single weight vector. This method calculates multiple centroid vectors (clustering centroids) within each class using online clustering to obtain complex feature representations within each class. In the inference process, a centroid vector closest to a feature extracted from an input image is searched for, and the class to which the found centroid vector belongs is output as the inference result. In other words, in this method, one selected centroid vector is used for class identification, and the other centroid vectors that were not selected are not used in learning. For example, even if an input image has features common to two centroid vectors, the learning is performed by forcibly assigning the image to one of the two centroid vectors, and the parameters are updated using the error for the assigned centroid vector. Therefore, when the feature values within a class are distributed over a wide range, it becomes difficult for the backbone to accurately learn the properties of the feature values, making accurate class classification difficult.
[0012] In order to solve the above problems, the following techniques are disclosed.
[0013] (1) A class classification method according to one aspect of the present disclosure is a class classification method executed by a computer, including: obtaining a feature vector extracted from input data; using a trained generator to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes depending on the value of the feature vector; using a trained calculator to calculate a score for each of the plurality of classes based on the feature vector and the plurality of weight vectors generated for each of the plurality of classes; classifying the classification target of the input data into one of the plurality of classes based on the plurality of scores calculated for each of the plurality of classes; and outputting the classification result.
[0014] According to this configuration, a weight vector that continuously changes depending on the value of the feature vector is generated for each of a plurality of classes to be classified. Then, scores for each of the plurality of classes are calculated based on the feature vector and the plurality of weight vectors generated for each of the plurality of classes. Then, the classification target of the input data is classified into one of the plurality of classes based on the plurality of scores calculated for each of the plurality of classes.
[0015] Therefore, a weight vector that changes continuously depending on the value of the feature vector is generated for each of the multiple classes to be classified, making it possible to represent multiple variations in the feature within a class, and enabling accurate class classification even when the feature within a class is distributed over a wide range.
[0016] (2) In the class classification method described in (1) above, the generator may have, as a parameter, a matrix composed of a plurality of row vectors for each of the plurality of classes, and generating the weight vector may include generating the weight vector for each of the classes using the value of the feature vector and the matrix.
[0017] According to this configuration, a weight vector for each class can be generated using the value of the feature vector and the matrix.
[0018] (3) In the class classification method described in (2) above, generating the weight vector may include calculating, for each of the plurality of classes, a weight of a linear combination of each of the plurality of row vectors from the value of the feature vector and a plurality of row vectors constituting a matrix assigned to each of the plurality of classes, and linearly combining the plurality of row vectors using the weight to generate a weight vector for each of the classes.
[0019] According to this configuration, a feature vector can be assigned to each element of each matrix using continuous values, and multiple variations of features within a class can be represented.
[0020] (4) In the classification method described in (2), the matrix may be an orthonormal matrix. With this configuration, the multimodality of the data can be expressed by an orthogonal basis, thereby improving classification performance.
[0021] (5) In the classification method described in (3), each element of the weight of the linear combination may be positive. This configuration improves interpretability for multiple variations in features within a class.
[0022] (6) In the classification method according to any one of (1) to (5), the input data may be image data. With this configuration, the classification target of the image data can be classified into one of a plurality of classes.
[0023] (7) In the classification method described in (6) above, the classification target of the input data may be the image data. With this configuration, class classification can be performed for each image data, as in image recognition.
[0024] (8) In the classification method described in (6) above, the classification target of the input data may be each of a plurality of pixels constituting the image data. With this configuration, class classification can be performed for each of a plurality of pixels constituting the image data, like semantic segmentation.
[0025] (9) In the classification method described in (6) above, the classification target of the input data may be a bounding box surrounding an object included in the image data. With this configuration, class classification can be performed for each bounding box surrounding an object included in the image data, as in object detection.
[0026] (10) In the classification method according to any one of (1) to (5), the input data may be time-series data. With this configuration, the classification target of the time-series data can be classified into one of a plurality of classes.
[0027] (11) A learning method according to another aspect of the present disclosure is a computer-executed learning method, including: obtaining a feature vector extracted from input data; using an untrained generator to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes depending on the value of the feature vector; using an untrained calculator to calculate a score for each of the classes based on the feature vector and a plurality of weight vectors generated for each of the classes; calculating an error between the plurality of scores calculated for each of the classes and a correct label associated with a classification target of the input data; and updating parameters of at least one of the generator and the calculator based on the error.
[0028] According to this configuration, a weight vector that continuously changes depending on the value of the feature vector is generated for each of a plurality of classes to be classified. Then, scores for each of the plurality of classes are calculated based on the feature vector and the plurality of weight vectors generated for each of the plurality of classes. Then, parameters of at least one of the generator and the calculator are updated based on the error between the plurality of scores calculated for each of the plurality of classes and the correct label associated with the classification target of the input data.
[0029] Therefore, a weight vector that changes continuously depending on the value of the feature vector is generated for each of the multiple classes to be classified, and the generator and calculator trained using the generated weight vector can represent multiple variations of the features within a class, allowing accurate class classification even when the features within a class are distributed over a wide range.
[0030] (12) In the learning method described in (11) above, the generator may have, as a parameter, a matrix composed of a plurality of row vectors for each of the plurality of classes, and generating the weight vector may include generating the weight vector for each of the classes using the value of the feature vector and the matrix.
[0031] According to this configuration, a weight vector for each class can be generated using the value of the feature vector and the matrix.
[0032] (13) In the learning method described in (12) above, generating the weight vector may include calculating, for each of the plurality of classes, a weight of a linear combination of each of the plurality of row vectors from the value of the feature vector and a plurality of row vectors constituting a matrix assigned to each of the plurality of classes, and linearly combining the plurality of row vectors using the weight to generate a weight vector for each of the classes.
[0033] According to this configuration, a feature vector can be assigned to each element of each matrix using continuous values, and multiple variations of features within a class can be represented.
[0034] (14) In the learning method described in (12), the matrix may be an orthonormal matrix. With this configuration, multimodality of data can be expressed by an orthogonal basis, thereby improving classification performance.
[0035] (15) In the learning method described in (13), each element of the weight of the linear combination may be positive. This configuration improves interpretability for multiple variations of features within a class.
[0036] (16) The learning method according to any one of (11) to (15) above may further include extracting the feature vector from the input data using an untrained extractor, and updating the parameters may include simultaneously updating the parameters of the extractor, the generator, and the calculator based on the error.
[0037] In particular, in Non-Patent Document 1, the parameters of the feature extractor and the weight generator are updated alternately by online clustering, making it difficult to simultaneously update the parameters of the feature extractor and the weight generator. In contrast, the above configuration makes it possible to simultaneously update the parameters of the extractor, generator, and calculator, for example, by backpropagation or gradient descent.
[0038] The present disclosure can be realized not only as a classification method that performs the above-described characteristic processing, but also as a classification device having a characteristic configuration corresponding to the characteristic processing performed by the classification method. Furthermore, the present disclosure can also be realized as a computer program that causes a computer to execute the characteristic processing included in such a classification method. Therefore, the following other aspects can also achieve the same effects as the above-described classification method.
[0039] (17) A class classification device according to another aspect of the present disclosure includes an acquisition unit that acquires a feature vector extracted from input data; a generation unit that uses a trained generator to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes depending on the value of the feature vector; a score calculation unit that uses a trained calculator to calculate a score for each of the plurality of classes based on the feature vector and the plurality of weight vectors generated for each of the plurality of classes; a classification unit that classifies a classification target of the input data into one of the plurality of classes based on the plurality of scores calculated for each of the plurality of classes; and an output unit that outputs the classification result.
[0040] (18) A class classification program according to another aspect of the present disclosure causes a computer to function in the following manner: acquire a feature vector extracted from input data; use a trained generator to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes depending on the value of the feature vector; use a trained calculator to calculate a score for each of the plurality of classes based on the feature vector and the plurality of weight vectors generated for each of the plurality of classes; classify the classification target of the input data into one of the plurality of classes based on the plurality of scores calculated for each of the plurality of classes; and output the classification result.
[0041] (19) A non-transitory computer-readable recording medium according to another aspect of the present disclosure records the class classification program.
[0042] Furthermore, the present disclosure can be realized not only as a learning method that executes the characteristic processes described above, but also as a learning device or the like that has a characteristic configuration corresponding to the characteristic processes executed by the learning method. Furthermore, the present disclosure can also be realized as a computer program that causes a computer to execute the characteristic processes included in such a learning method. Therefore, the same effects as those of the above learning method can also be achieved in the following other aspects.
[0043] (20) A learning device according to another aspect of the present disclosure includes an acquisition unit that acquires a feature vector extracted from input data; a generation unit that uses an untrained generator to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes depending on the value of the feature vector; a score calculation unit that uses an untrained calculator to calculate a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; an error calculation unit that calculates an error between the plurality of scores calculated for each of the plurality of classes and a correct label associated with a classification target of the input data; and an update unit that updates parameters of at least one of the generator and the calculator based on the error.
[0044] (21) A learning program according to another aspect of the present disclosure causes a computer to function in the following manner: acquire a feature vector extracted from input data; use an untrained generator to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes depending on the value of the feature vector; use an untrained calculator to calculate a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; calculate an error between the plurality of scores calculated for each of the plurality of classes and a correct label associated with the classification target of the input data; and update parameters of at least one of the generator and the calculator based on the error.
[0045] (22) A non-transitory computer-readable recording medium according to another aspect of the present disclosure records the learning program.
[0046] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.
[0047] FIG. 1 is a block diagram showing the configuration of a classification device 1 according to this embodiment.
[0048] The classification device 1 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. Note that the classification device 1 may be realized, for example, by hardware implementation using the processing circuit, or by execution of a software program stored in the memory by the processing circuit or distributed from an external server, or by a combination of these hardware and software implementations.
[0049] The classification device 1 shown in FIG. 1 includes a data acquisition unit 11, a feature extraction unit 12, a weight generation unit 13, a class score calculation unit 14, a classification unit 15, and an output unit 16.
[0050] The data acquisition unit 11 acquires image data including a classification target. The image data is an example of input data. The data acquisition unit 11 may receive image data from an external device via a communication unit (not shown), or may read image data stored in a memory (not shown). Examples of the external device include a personal computer, a server, or a camera. The data acquisition unit 11 may acquire image data in accordance with an acquisition instruction from a user. In this specification, image data may also be simply referred to as an image.
[0051] The feature extraction unit 12 uses a trained extraction model (extractor) to extract a feature vector from the image data acquired by the data acquisition unit 11. The feature extraction unit 12 extracts a d-dimensional feature vector according to the number of classes to be classified. The feature extraction unit 12 inputs the image data acquired by the data acquisition unit 11 to the extraction model and acquires the feature vector output from the extraction model. The extraction model is trained by deep learning, which is a type of machine learning.
[0052] The weight generation unit 13 acquires feature vectors extracted from image data. The weight generation unit 13 acquires the feature vectors extracted by the feature extraction unit 12. The weight generation unit 13 uses a trained generative model (generator) to generate, for each of a plurality of classes to be classified, a weight vector that continuously changes according to the value of the feature vector. The generative model has, for each of the plurality of classes, a matrix composed of a plurality of row vectors as a parameter. The weight generation unit 13 generates a weight vector for each class using the value of the feature vector and the matrix. For each of the plurality of classes, the weight generation unit 13 calculates a linear combination weight for each of the plurality of row vectors from the value of the feature vector and the plurality of row vectors that make up the matrix assigned to each class, and linearly combines the plurality of row vectors using the weights to generate a weight vector for each class. The matrix is an orthonormal matrix. Each element of the weight of the linear combination is positive.
[0053] The weight generation unit 13 includes a first weight generation unit 131 to an N-th weight generation unit 13N corresponding to the number of classes to be classified. For example, if the number of classes to be classified is three, the weight generation unit 13 includes a first weight generation unit 131 to a third weight generation unit 133. The first weight generation unit 131 to the N-th weight generation unit 13N generate weight vectors corresponding to the first to N-th classes to be classified, respectively.
[0054] Fig. 2 is a block diagram showing the configuration of the first weight generation unit 131 shown in Fig. 1. Note that the configurations of the second weight generation unit 132 to the N-th weight generation unit 13N are the same as the configuration of the first weight generation unit 131.
[0055] The first weight generation unit 131 includes a weight vector generation unit 31 and a parameter storage unit 32. The weight vector generation unit 31 calculates a linear combination weight for each of a plurality of row vectors from the value of the feature vector and a plurality of row vectors constituting a matrix assigned to the first class, and generates a weight vector for the first class by linearly combining the plurality of row vectors using the weights. The weight vector generation unit 31 inputs the feature vector extracted by the feature extraction unit 12 to a generative model and obtains a weight vector corresponding to the first class output from the generative model. The generative model is trained by deep learning, which is a type of machine learning. The parameter storage unit 32 stores in advance parameters to be used in the trained generative model. The parameters are matrices assigned to the first class.
[0056] The class score calculation unit 14 uses a trained calculation model (calculator) to calculate scores for each of the multiple classes based on the feature vector extracted by the feature extraction unit 12 and the multiple weight vectors generated for each of the multiple classes by the weight generation unit 13. The class score calculation unit 14 inputs the feature vector and the multiple weight vectors to the calculation model and obtains the scores output from the calculation model. The calculation model is trained by deep learning, which is one type of machine learning.
[0057] The class classification unit 15 classifies the classification target of the image data into one of a plurality of classes based on the plurality of scores calculated for each of the plurality of classes by the class score calculation unit 14. The class classification unit 15 classifies the classification target of the image data into the class corresponding to the largest score among the plurality of scores calculated by the class score calculation unit 14.
[0058] The output unit 16 outputs the classification result obtained by the classifying unit 15. For example, the output unit 16 may output the classification result to a display unit connected to the classifying device 1. The display unit may display the classification result. The output unit 16 may output the acquired image data and the classification result to the display unit.
[0059] In the DNC (Deep Nearest Centroids) method of Non-Patent Document 1, when a feature z belonging to a class c is given, the feature z is assigned to a certain centroid vector w c i In other words, there is a problem that the similarity sim(w c i , z) is the similarity sim(w c j , z), the center of gravity vector w c i and the center of gravity vector w c j For example, sim(w c i , z) > sim(w c j , z), the feature z is the centroid vector w c i The center of gravity vector w c j The property of being similar to is ignored.
[0060] Therefore, the classification device 1 of this embodiment uses a matrix W consisting of n (<d) weight vectors. c ∈R n*d By using the above, it becomes possible to represent the multi-modal distribution of the feature z in the feature space without using online clustering.
[0061] Specifically, the feature z and the matrix W c The similarity to cT a c , z). In this case, online clustering is not used, and the matrix W is calculated by the gradient descent method. c To be able to update the assignment vector, the assignment vector is a continuous value (a c ∈R n + )
[0062] When the L2 distance with temperature is used as the similarity measure, the logit function l c (z;κ) is expressed by the following formula (1).
[0063]
[0064] In the above formula (1), κ is a hyperparameter greater than 0, z is a feature vector, and W cT is the matrix W c is the transpose matrix of a c is the assignment vector.
[0065] Here, the matrix W c is an orthonormal matrix. That is, the matrix W c Is W c W cT =I n The constraint is satisfied. cT is the matrix W c is the transpose matrix of I n is an n-dimensional identity matrix. c is always full rank. Also, the matrix W c Due to the orthonormality of c =ReLU(W c z) is set. Therefore, the matrix W c is an orthogonal matrix, the logit function of the above equation (1) is expressed by the following equation (2).
[0066]
[0067] In the above formula (2), κ is a hyperparameter greater than 0, z is a feature vector, and W c is the matrix of class c (c=1, 2, ..., C), and W cT is the matrix W c , and ReLU is an activation function, also called the rectified linear unit.
[0068] Furthermore, similarly to DNC, when the feature amount is L2 normalized and the constant term is removed, the logit function of the above equation (2) is expressed by the following equation (3).
[0069]
[0070] W in Equation (2) cT ReLU (Wc z) represents the weight vector for each class generated by the weight generation unit 13. In addition, the logit function l in Equation (2) and Equation (3) c (z;κ) represents the score of each class calculated by the class score calculation unit 14.
[0071] In this embodiment, in order to improve interpretability within the prototype learning framework, we propose a model that has high interpretability for variations within a class by converting the weight vector in class classification into a matrix and imposing orthogonal constraints on the weight matrix.
[0072] Next, the configuration of the learning device 2 according to this embodiment will be described.
[0073] FIG. 3 is a block diagram showing the configuration of the learning device 2 according to this embodiment.
[0074] The learning device 2 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. Note that the learning device 2 may be realized, for example, by hardware implementation using the processing circuit, or by execution of a software program stored in the memory by the processing circuit or distributed from an external server, or by a combination of these hardware and software implementations.
[0075] The learning device 2 shown in Fig. 3 includes a data acquisition unit 17, a feature extraction unit 12, a weight generation unit 13, a class score calculation unit 14, an error calculation unit 18, and a parameter update unit 19. In the learning device 2 shown in Fig. 3, the same components as those in the classification device 1 shown in Fig. 1 are denoted by the same reference numerals, and their description will be omitted.
[0076] The data acquisition unit 17 acquires image data including a classification target and a correct label associated with the classification target of the image data. The image data is an example of input data. The image data and the correct label are training data used in machine learning. The data acquisition unit 17 may receive the image data and the correct label from an external device via a communication unit (not shown), or may read the image data and the correct label stored in a memory (not shown). The external device may be, for example, a personal computer or a server. The data acquisition unit 17 may acquire the image data and the correct label in accordance with an acquisition instruction from a user. In this specification, image data may also be simply referred to as an image.
[0077] The feature extraction unit 12 uses an untrained extraction model (extractor) to extract feature vectors from image data acquired by the data acquisition unit 17. The weight generation unit 13 uses an untrained generative model (generator) to generate, for each of a plurality of classes, a weight vector that continuously changes depending on the value of the feature vector. The class score calculation unit 14 uses an untrained calculation model (calculator) to calculate scores for each of a plurality of classes based on the feature vector extracted by the feature extraction unit 12 and the plurality of weight vectors generated for each of the classes by the weight generation unit 13.
[0078] The error calculation unit 18 calculates the error between the multiple scores calculated for each of the multiple classes by the class score calculation unit 14 and the correct label associated with the classification target of the image data. The error calculation unit 18 calculates the cross entropy error for each of the multiple classes by substituting the multiple scores and the correct label into a loss function. The loss function is a logit function l c It is expressed by the following equation (4) using (z;κ).
[0079]
[0080] In the above formula (4), x i represents the image data, and y i represents the correct label of the image data, and f θ (x i ) is the image data xi represents the feature vector of
[0081] The parameter update unit 19 updates the parameters of the extraction model, the generative model, and the calculation model based on the error calculated by the error calculation unit 18. The parameter update unit 19 simultaneously updates the parameters of the extraction model, the generative model, and the calculation model by using the error backpropagation method so as to minimize the error calculated by the error calculation unit 18. Specifically, the parameters of the generative model are a plurality of matrices for a plurality of classes, and the parameter update unit 19 updates the plurality of matrices.
[0082] In this embodiment, the parameter update unit 19 updates the parameters of the extraction model, the generation model, and the calculation model in one learning session, but the present disclosure is not particularly limited to this, and at least one parameter of the extraction model, the generation model, and the calculation model may be updated in one learning session.
[0083] Note that the logit function l c The softmax cross entropy error minimization problem using (z; k) is expressed by the following equation (5).
[0084]
[0085] This problem is an optimization problem with orthogonal constraints. For such an optimization problem with orthogonal constraints, for example, conventionally, Lagrange multipliers have been used and orthogonal constraints have been added to the regularization term of the objective function. This method is effective in terms of calculation time when there are many orthogonal constraints, but requires adjustment of the regularization parameter. Therefore, in this embodiment, the orthogonal constraints are calculated using the weight matrix W c Therefore, Riemannian optimization is adopted for optimization.
[0086] The learning device 2 of this embodiment learns using values obtained by projecting feature vectors onto an orthogonal matrix. By using the feature vectors projected onto an orthogonal matrix for classification, the multimodality of the data can be expressed by an orthogonal basis, thereby improving classification performance.
[0087] Next, the operation of the classification device 1 according to this embodiment will be described.
[0088] FIG. 4 is a flowchart for explaining the operation of the classification device 1 according to this embodiment.
[0089] First, in step S11, the data acquisition unit 11 acquires image data including a classification target.
[0090] Next, in step S12, the feature extraction unit 12 extracts a feature vector from the image data acquired by the data acquisition unit 11 using the trained extraction model.
[0091] Next, in step S13, the weight generation unit 13 uses the trained generation model to generate a weight vector that changes continuously depending on the value of the feature vector for each of the multiple classes to be classified.
[0092] Next, in step S14, the class score calculation unit 14 uses the trained calculation model to calculate scores for each of the multiple classes based on the feature vector extracted by the feature extraction unit 12 and the multiple weight vectors generated for each of the multiple classes by the weight generation unit 13.
[0093] Next, in step S15, the classification unit 15 classifies the classification target of the image data into one of a plurality of classes based on the plurality of scores calculated for each of the plurality of classes by the class score calculation unit 14.
[0094] Next, in step S16 , the output unit 16 outputs the classification result obtained by the classification unit 15 .
[0095] In this way, a weight vector that continuously changes depending on the value of the feature vector is generated for each of the multiple classes to be classified. Then, scores for each of the multiple classes are calculated based on the feature vector and the multiple weight vectors generated for each of the multiple classes. Then, the classification target of the input data is classified into one of the multiple classes based on the multiple scores calculated for each of the multiple classes.
[0096] Therefore, a weight vector that changes continuously depending on the value of the feature vector is generated for each of the multiple classes to be classified, making it possible to represent multiple variations in the feature within a class, and enabling accurate class classification even when the feature within a class is distributed over a wide range.
[0097] Next, the operation of the learning device 2 according to this embodiment will be described.
[0098] FIG. 5 is a flowchart for explaining the operation of the learning device 2 according to this embodiment.
[0099] First, in step S21, the data acquisition unit 17 acquires image data including a classification target and a correct label associated with the classification target of the image data.
[0100] Next, in step S22, the feature extraction unit 12 extracts a feature vector from the image data acquired by the data acquisition unit 17 using the untrained extraction model.
[0101] Next, in step S23, the weight generation unit 13 uses the untrained generative model to generate a weight vector that changes continuously depending on the value of the feature vector for each of the multiple classes to be classified.
[0102] Next, in step S24, the class score calculation unit 14 uses the unlearned calculation model to calculate scores for each of the multiple classes based on the feature vector extracted by the feature extraction unit 12 and the multiple weight vectors generated for each of the multiple classes by the weight generation unit 13.
[0103] Next, in step S25, the error calculation unit 18 calculates the error between the scores calculated for each of the classes by the class score calculation unit 14 and the correct label associated with the classification target of the image data.
[0104] Next, in step S26, the parameter update unit 19 updates the parameters of the extraction model, the generation model, and the calculation model based on the errors calculated by the error calculation unit 18.
[0105] The extraction model, the generative model, and the calculation model are trained by repeatedly performing the processes of steps S21 to S26 using a plurality of image data and a plurality of correct labels. The trained extraction model, the generative model, and the calculation model are used in the classification device 1.
[0106] In this way, a weight vector that continuously changes according to the value of the feature vector is generated for each of the multiple classes to be classified. Then, scores for each of the multiple classes are calculated based on the feature vector and the multiple weight vectors generated for each of the multiple classes. Then, parameters of at least one of the generative model and the calculation model are updated based on the error between the multiple scores calculated for each of the multiple classes and the correct label associated with the classification target of the input data.
[0107] Therefore, a weight vector that changes continuously depending on the value of the feature vector is generated for each of the multiple classes to be classified, and the generative model and calculation model trained using the generated weight vector can represent multiple variations of the features within the class, allowing accurate class classification even when the features within the class are distributed over a wide range.
[0108] Next, a comparison result between the classification by the DNC in Non-Patent Document 1 and the classification by the classification device 1 according to the present embodiment will be described.
[0109] FIG. 6 is a diagram showing the distribution of features in a feature space in class classification by conventional DNC, and FIG. 7 is a diagram showing the distribution of features in a feature space in class classification by the class classification device 1 according to this embodiment.
[0110] In Figures 6 and 7, CIFAR-10 is used as the dataset, and multiple image data are classified into classes representing airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Points in Figures 6 and 7 indicate the distribution of features in feature space 301, and multiple sets of points indicate features classified into multiple classes. A zoomed distribution diagram 302 shows the feature distribution of the bird class. Stars 303 in Figure 6 represent the center of gravity vectors of the DNC, and star-shaped marks 304 in Figure 7 represent the weight vectors (matrix) of this embodiment.
[0111] In conventional DNC, the input feature z belonging to class c is calculated by one of the centroid vectors w c i The center of gravity vector w c i is updated by online clustering so as to approach the average vector within each batch. Therefore, as shown in Figure 6, the centroid vector represents the center of each class in the feature space 301.
[0112] On the other hand, in this embodiment, the input feature z belonging to class c is calculated by dividing the input feature z by the orthogonal weight vector {W c 1 , W c 2 , ..., W c n} a linear combination Σa c i W c i At this time, the assignment vector a c =ReLU(W c z) is 0≦a c i ≦1 and ||a c || 2 ≦1, as shown in FIG. 7, each element W c i represents the edge of each class on the feature space 301. This allows the present embodiment to obtain a detailed representation that could not be expressed by DNC.
[0113] Next, we will explain the results of verification of the effectiveness of this embodiment in image classification problems. In this verification, we compared three classification methods: a conventional classification method (ResNet), the classification method of Non-Patent Document 1 (DNC), and the classification method of this embodiment.
[0114] The classification problem was evaluated using CIFAR-100 and ImageNet as datasets. ResNet-50 and ResNet-101 were used as feature extraction methods. The hyperparameters for the classification method of this embodiment were n = 10 and κ = 50.0. The experimental results show the average and standard deviation of three experiments conducted with different random numbers.
[0115] 8 is a diagram showing evaluation results of two conventional classification methods and the classification method of this embodiment for two data sets. The two data sets are CIFAR-100 and ImageNet, the two conventional classification methods are ResNet and DNC, and the evaluation indices are Top-1 Accuracy and Top-5 Accuracy. Top-1 Accuracy is an evaluation indices that determine a correct answer when the most probable class in the classification results matches the class of the correct label. Top-5 Accuracy is an evaluation indices that determine a correct answer when the class of the correct label is included in the top five classes with the highest probabilities in the classification results.
[0116] In CIFAR-100 and ImageNet, the classification method of this embodiment outperforms the conventional ResNet and DNC. In particular, in the Top-5 Accuracy of CIFAR-100, the performance of DNC deteriorates, while the classification method of this embodiment shows a performance improvement of 0.8% to 0.9%, indicating that the distance relationship between features can be more accurately learned.
[0117] Next, the results of another experiment on the effectiveness of this embodiment in an image classification problem will be described.
[0118] As classes become coarser, the intra-class distribution becomes more complex and the feature distribution becomes multimodal. To evaluate performance with respect to multimodality, an experiment was conducted in which a model was trained using coarse-grained labels and evaluated using fine-grained labels. Evaluation with fine-grained labels was performed using top-1 nearest neighbor accuracy. The network architecture and training details used in the experiment are the same as those in Figure 8.
[0119] The datasets used were CIFAR-100 and ImageNet. CIFAR-100 contains 20 coarse-grained labels (CIFAR-20) and 100 fine-grained labels. For ImageNet, 127 coarse-grained labels (ImageNet-127) were obtained from the 1,000 fine-grained labels by performing top-down clustering with a fixed distance from the root node of the WordNet tree.
[0120] Fig. 9 shows the evaluation results of two conventional classification methods and the classification method of this embodiment for two types of CIFAR-20 labels. Fig. 10 shows the evaluation results of two conventional classification methods and the classification method of this embodiment for two types of ImageNet labels. The two types of labels are coarse-grained labels and fine-grained labels.
[0121] In the CIFAR-20 data shown in Figure 9, it can be seen that the classification method of this embodiment is superior to conventional classification methods in terms of the accuracy of fine-grained labels. Also, in the ImageNet-127 data shown in Figure 10, it can be seen that the classification method of this embodiment is superior to conventional classification methods in terms of the accuracy of fine-grained labels. This shows that the classification method of this embodiment can more accurately represent the multimodality within a class than DNC, which performs online clustering.
[0122] In the present embodiment, the input data is image data, but the present disclosure is not particularly limited to this, and the input data may be time-series data or audio data.
[0123] In the present embodiment, the classification target of the input data is image data, but the present disclosure is not particularly limited to this. The classification target of the input data may be each of the multiple pixels that make up the image data. The classification target of the input data may also be a bounding box that surrounds an object included in the image data.
[0124] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Furthermore, the program may be executed by another independent computer system by recording the program on a recording medium and transferring it, or by transferring the program via a network.
[0125] Some or all of the functions of the device according to the embodiment of the present disclosure are typically realized as an LSI (Large Scale Integration), which is an integrated circuit. These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Furthermore, the integrated circuit is not limited to an LSI, and may be realized using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor, which can reconfigure the connections and settings of circuit cells within an LSI.
[0126] Furthermore, some or all of the functions of the device according to the embodiment of the present disclosure may be realized by a processor such as a CPU executing a program.
[0127] Furthermore, all the numbers used above are merely examples to specifically explain the present disclosure, and the present disclosure is not limited to the numbers used as examples.
[0128] The order in which the steps are executed in the above flowchart is merely an example for specifically explaining the present disclosure, and other orders may be used as long as similar effects are obtained. Also, some of the steps may be executed simultaneously (in parallel) with other steps.
[0129] The technology disclosed herein can accurately perform class classification even when features within a class are distributed over a wide range, and is therefore useful as a technology for performing class classification using a deep learning model and a technology for learning a deep learning model used to perform class classification.
Claims
1. A classification method executed by a computer, comprising: obtaining a feature vector extracted from input data; generating, for each of a plurality of classes to be classified, a weight vector that changes continuously according to the value of the feature vector, using a trained generator; calculating, using a trained calculator, a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; classifying the classification target of the input data into one of the plurality of classes based on the multiple scores calculated for each of the plurality of classes; and outputting the classification result.
2. The method of claim 1, wherein the generator has as a parameter a matrix composed of a plurality of row vectors for each of the plurality of classes, and generating the weight vector includes generating the weight vector for each of the classes using the value of the feature vector and the matrix.
3. The method of claim 2, wherein generating the weight vector includes calculating, for each of the plurality of classes, a weight of a linear combination of each of the plurality of row vectors from the value of the feature vector and a plurality of row vectors constituting a matrix assigned to each of the plurality of classes, and linearly combining the plurality of row vectors using the weights to generate a weight vector for each of the plurality of classes.
4. The method of claim 2, wherein the matrix is an orthogonal matrix.
5. The method according to claim 3, wherein each element of the weights in the linear combination is positive.
6. The classification method according to any one of claims 1 to 5, wherein the input data is image data.
7. The class classification method according to claim 6, wherein the classification target of the input data is the image data.
8. The class classification method according to claim 6, wherein the classification targets of the input data are each of a plurality of pixels constituting the image data.
9. The method according to claim 6, wherein the classification target of the input data is a bounding box surrounding an object included in the image data.
10. The classification method according to any one of claims 1 to 5, wherein the input data is time-series data.
11. A learning method executed by a computer, comprising: obtaining a feature vector extracted from input data; using an untrained generator to generate a weight vector for each of a plurality of classes to be classified, the weight vector changing continuously according to the value of the feature vector; using an untrained calculator to calculate a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; calculating an error between the plurality of scores calculated for each of the plurality of classes and a correct label corresponding to the classification target of the input data; and updating parameters of at least one of the generator and the calculator based on the error.
12. The learning method according to claim 11, wherein the generator has as a parameter a matrix composed of a plurality of row vectors for each of the plurality of classes, and generating the weight vector includes generating the weight vector for each of the classes using the value of the feature vector and the matrix.
13. The learning method according to claim 12, wherein generating the weight vector includes calculating, for each of the plurality of classes, a weight of a linear combination of each of the plurality of row vectors from the value of the feature vector and a plurality of row vectors constituting a matrix assigned to each of the plurality of classes, and linearly combining the plurality of row vectors using the weights to generate a weight vector for each of the plurality of classes.
14. The method according to claim 12, wherein the matrix is an orthogonal matrix.
15. The learning method according to claim 13, wherein each element of the weights of the linear combination is positive.
16. The learning method according to any one of claims 11 to 15, further comprising extracting the feature vector from the input data using an untrained extractor, and updating the parameters comprises simultaneously updating the parameters of the extractor, the generator, and the calculator based on the error.
17. A class classification device comprising: an acquisition unit that acquires a feature vector extracted from input data; a generation unit that uses a trained generator to generate, for each of a plurality of classes to be classified, a weight vector that changes continuously depending on the value of the feature vector; a score calculation unit that uses a trained calculator to calculate a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; a classification unit that classifies the classification target of the input data into one of the plurality of classes based on the multiple scores calculated for each of the plurality of classes; and an output unit that outputs the classification result.
18. A class classification program which causes a computer to function as follows: obtaining a feature vector extracted from input data; using a trained generator, generating a weight vector for each of a plurality of classes to be classified, the weight vector changing continuously according to the value of the feature vector; using a trained calculator, calculating a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; classifying the classification target of the input data into one of the plurality of classes based on the plurality of scores calculated for each of the plurality of classes; and outputting the classification result.
19. A learning device comprising: an acquisition unit that acquires a feature vector extracted from input data; a generation unit that uses an untrained generator to generate a weight vector for each of a plurality of classes to be classified, the weight vector changing continuously depending on the value of the feature vector; a score calculation unit that uses an untrained calculator to calculate a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; an error calculation unit that calculates an error between the multiple scores calculated for each of the plurality of classes and a correct label corresponding to the classification target of the input data; and an update unit that updates parameters of at least one of the generator and the calculator based on the error.
20. A learning program that causes a computer to function as follows: obtaining a feature vector extracted from input data; using an untrained generator, generating a weight vector for each of a plurality of classes to be classified, the weight vector changing continuously according to the value of the feature vector; using an untrained calculator, calculating a score for each of the plurality of classes based on the feature vector and a plurality of weight vectors generated for each of the plurality of classes; calculating an error between the plurality of scores calculated for each of the plurality of classes and a correct label corresponding to the classification target of the input data; and updating parameters of at least one of the generator and the calculator based on the error.
Citation Information
Patent Citations
Device for classifying defects and method for adjusting classification
WO2011010557A1
Object detection device, learning method, and recording medium
WO2021090473A1
Cited By
Lightweight traffic identification method based on domain name class frequency perception and PON (Passive Optical Network) equipment
CN121333676A
