Clustering device and clustering method

Abstract

Provided is a clustering device comprising a control unit that clusters, by K-Multiple-Means, n data points within a data space that is a phase space including one or a plurality of data points, the clustering device being such that, in a process for obtaining a solution to an optimization problem obtained by fixing a matrix S representing the degree of similarity between each of the data points and each of m prototypes within the data space, said process being executed in alternating optimization in the K-Multiple-Means, the control unit obtains, as the solution, an eigenvector of a matrix M representing the degree of similarity between the prototypes.

Description

Clustering device and clustering method

[0001] The present invention relates to a clustering device and a clustering method.

[0002] Clustering, which calculates clusters from data with high similarity, is an important process in data analysis such as segmentation, data analysis for determining the startup order of a group of servers, and data analysis for sound source separation. K-means is one of the most well-known clustering methods (see Non-Patent Document 1). K-means calculates clusters by minimizing the sum of squared errors between data points and their assigned representative points, called prototypes.

[0003] However, because K-means calculates clusters based on the distance to the prototype, it has the drawback of being unable to effectively calculate clusters as shown in Figure 12 when given non-spherically distributed data such as that shown in Figure 11. In this figure, the data points are color-coded by cluster. Also, the points represented by squares in this figure are prototypes.

[0004] To overcome this drawback, K-Multiple-Means has been proposed (see Non-Patent Document 2). This clustering method uses more than c subclusters to calculate c clusters. Specifically, this method first divides data into m subclusters (m is greater than c) as shown in Figure 13, and then groups the subclusters into c clusters as shown in Figure 14. K-Multiple-Means uses one or more prototypes to represent each cluster, so it can effectively represent non-spherical clusters.

[0005] Theoretically, K-Multiple-Means treats clustering as a graph partitioning problem for a bipartite graph. As shown in Figure 15, in this problem, we consider a data point x ｉ (i is an integer between 1 and n) and prototype a ｊ(j is an integer between 1 and m), and the similarities between nodes in this graph are updated so that the graph has c connected components, as shown in Figure 16. As a result, n data points with m prototypes are divided into c connected components, and the connected components correspond to clusters.

[0006] Furthermore, if a bipartite graph has c connected components, the rank of the Laplacian matrix for the bipartite graph is n+m-c (see Non-Patent Document 3). Therefore, K-Multiple-Means solves the clustering of n data points with m prototypes as a partitioning problem of a bipartite graph with a constraint on the rank of the Laplacian matrix. Note that the Laplacian matrix here is a matrix that can be calculated from the similarity between the data points and the prototypes (see Non-Patent Document 4).

[0007] JB MacQueen, Some Methods for Classification and Analysis of Multi-Variate Observations, Berkeley Symposium on Mathematical Statistics and Probability, 1967. Feiping Nie et al., K-Multiple-Means: A Multiple-Means Clustering Method with Specified KClusters, KDD, 2019. Ulrike von Luxburg, A Tutorial on Spectral Clustering, Stat. Comput., 2007.Feiping Nie et al., Learning A Structured Optimal Bipartite Graph for Co-Clustering, NIPS, 2017.

[0008] K-Multiple-Means performs alternating optimization on the n x m similarity matrix of a bipartite graph to solve the partitioning problem. Specifically, it updates the prototype of the subcluster by repeatedly updating the similarity between nodes so as to satisfy the rank constraint using c singular vectors corresponding to the c largest singular values ​​of the similarity matrix. However, it takes nm to calculate the singular value decomposition of the n x m similarity matrix to find the singular vectors. ２ This requires a computational cost of the order of 1. Therefore, the processing time of K-Multiple-Means may become long.

[0009] In view of the above circumstances, an object of the present invention is to provide a technique for reducing the time required to execute K-Multiple-Means.

[0010] One aspect of the present invention is a clustering device that includes a control unit that clusters n data points in a data space, which is a topological space containing one or more data points, using K-Multiple-Means, and the control unit obtains, as the solution, an eigenvector of a matrix M that represents the similarity between the prototypes in a process executed in alternating optimization in K-Multiple-Means, where the process obtains a solution to an optimization problem obtained by fixing a matrix S that represents the similarity between each of the data points in the data space and each of m prototypes.

[0011] One aspect of the present invention is a clustering method executed by a clustering device, comprising: a control unit that clusters n data points in a data space, which is a topological space containing one or more data points, using K-Multiple-Means; the control unit obtains, as the solution, an eigenvector of a matrix M that represents the similarity between the prototypes in a process executed in alternating optimization in K-Multiple-Means, the process being a process of obtaining a solution to an optimization problem obtained by fixing a matrix S that represents the similarity between each of the data points in the data space and each of m prototypes; and a clustering step in which the control unit executes the K-Multiple-Means.

[0012] The present invention makes it possible to provide a technique for reducing the time required to execute K-Multiple-Means.

[0013] FIG. 1 is an explanatory diagram illustrating an overview of a clustering device according to an embodiment. FIG. 2 is a diagram illustrating an algorithm 1 according to an embodiment. FIG. 3 is a diagram illustrating an example of the hardware configuration of a clustering device according to an embodiment. FIG. 4 is a flowchart illustrating an example of the flow of processing executed by a clustering device according to an embodiment. FIG. 5 is a diagram illustrating an algorithm 6 according to a modified example. FIG. 6 is a diagram illustrating an algorithm 7 according to a modified example. FIG. 1 is a diagram illustrating a conventional technique. FIG. 2 is a diagram illustrating a conventional technique. FIG. 3 is a diagram illustrating a conventional technique. FIG. 4 is a diagram illustrating a conventional technique. FIG. 5 is a diagram illustrating a conventional technique. FIG. 6 is a diagram illustrating a conventional technique.

[0014] 1 is an explanatory diagram illustrating an overview of a clustering device 1 according to an embodiment. The clustering device 1 includes a control unit 11 including a processor 91, such as a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU), and a memory 92, which are connected via a bus.

[0015] The control unit 11 classifies data points in a topological space (hereinafter referred to as a "data space") containing one or more data points using K-Multiple-Means.

[0016] K-Multiple-Means is a clustering technique that uses alternating optimization and has a matrix S as one of the variables in the alternating optimization. The matrix S is a matrix that represents the similarity between each data point and each prototype in the data space. In other words, similarity is closeness in the data space. Therefore, the matrix S can also be said to be a matrix that represents the closeness between each data point and each prototype in the data space.

[0017] As is well known, alternating optimization is a technique in which some of multiple variables are fixed and an optimal solution is obtained while exchanging the fixed variables, so in alternating optimization performed by K-Multiple-Means, there is a stage (or step) in which optimization is performed with a fixed matrix S. In other words, in alternating optimization performed by K-Multiple-Means, processing is executed to obtain a solution to an optimization problem obtained by fixing the matrix S.

[0018] For simplicity, the process executed in the alternating optimization in K-Multiple-Means to obtain a solution to an optimization problem with a fixed matrix S will be referred to as an optimal step solution acquisition process.

[0019] A well-known technique for the optimal step solution acquisition process is, for example, tilde S eigenvector acquisition process. Although more details will be described later in <Outline of K-Multiple-Means for Executing Optimal Step Solution Acquisition Process>, the tilde S eigenvector acquisition process is a process for obtaining a matrix made up of left singular vectors and a matrix made up of right singular vectors corresponding to the maximum eigenvalue in the tilde S matrix, which is a matrix derived from matrix S and will be described later in detail. The obtained matrix made up of left singular vectors and the obtained matrix made up of right singular vectors are respectively solutions to the above-mentioned optimization problem obtained by fixing matrix S.

[0020] The Fujiwara process is a type of optimal step solution acquisition process, and is different from the S eigenvector acquisition process. Specifically, the Fujiwara process is a process for obtaining eigenvectors of a matrix M that represents the similarity between prototypes. The obtained eigenvectors are the solution to the optimization problem obtained by fixing the matrix S. More specifically, the matrix M is the product of the S matrix and the transpose of the S matrix.

[0021] As will be described later, K-Multiple-Means, which performs the Fujiwara process, has a lower computational cost than K-Multiple-Means, which performs the tilde S eigenvector acquisition process.

[0022] Therefore, the control unit 11 executes, for example, Fujihara-type K-Multiple-Means, which is a process executed in alternating optimization in K-Multiple-Means, and which obtains the eigenvectors of matrix M as a solution in a process of obtaining a solution to an optimization problem obtained by fixing matrix S.

[0023] <Technical Significance of Fujiwara Processing> The technical significance of Fujiwara Processing will be explained. To explain the technical significance of Fujiwara Processing, we will first provide an overview of K-Multiple-Means, which executes the process of obtaining the optimal step solution. After providing an overview of K-Multiple-Means, which executes the process of obtaining the optimal step solution, we will explain the technical significance of Fujiwara Processing.

[0024] <<Outline of K-Multiple-Means that executes the optimal step solution acquisition process>> If d is the number of dimensions of the data, the i-th data point is the vector x ｉ = [x ｉ [1], ..., x ｉ The j-th prototype is represented as vector a ｊ = [a ｊ [1], ..., a ｊ Using this, the m × d matrix A = [a １ , ..., a ｍ ] Ｔ The matrix A is called the prototype matrix.

[0025] the i-th data point x ｉ is connected to the j-th prototype by similarity s[i][j]. The smaller the quantity of the following formula (1), the larger the value of s[i][j].

[0026]

[0027] The expression in the following formula (2) is L ２ Represents the norm.

[0028]

[0029] Here, the n×m matrix whose (i, j) element is s[i][j] is the aforementioned matrix S. It is clear from formula (1) that matrix S indicates the degree of match between each classification object and each prototype in the data space. Note that matrix S represents a bipartite graph consisting of data points and prototypes. More specifically, each element of matrix S represents the weight of each edge in the bipartite graph.

[0030] To divide n data points with m prototypes into c clusters, K-Multiple-Means calculates the Laplacian matrix L of the following equation (3).

[0031]

[0032] In equation (3), matrix D is a (n+m) × (n+m) diagonal matrix defined by equation (4) below, and its i-th diagonal element d[i] is given by equation (5) below. Note that w[i][j] means the element in row i and column j of matrix W, as expressed by equation (6) below.

[0033]

[0034]

[0035]

[0036] If the rank of the Laplacian matrix L is n+m-c, then the bipartite graph of data points and prototypes has c connected components, so n data points with m prototypes are partitioned into c clusters.

[0037] Therefore, in K-Multiple-Means, clusters are obtained by solving the optimization problem expressed by the following equation (7).

[0038]

[0039] The expression in the following equation (8) represents the Frobenius norm.

[0040]

[0041] In equation (7), α and β are both real numbers greater than or equal to zero and are hyperparameters in the optimization problem. β is set to a sufficiently large value during optimization. ｉ represents the i-th smallest eigenvalue of the Laplacian matrix L. ｍ is a column vector of length m with all elements being 1. ｍ = [1, ..., 1] Ｔ is.

[0042] In this optimization problem, the first and second terms in equation (7) are equivalent to assigning n data points to m prototypes based on weighted squared error, where the hyperparameter α influences how sparsely the connections between data points and prototypes are. The third term is equivalent to a rank constraint on the Laplacian matrix L, which groups the m prototypes into c clusters. In other words, adding the third term to the constraint problem results in c clusters of prototypes in the solution to the constraint problem.

[0043] As described above, the hyperparameter β is a sufficiently large value, so that the quantity in the following equation (9) becomes small in the optimal solution, and the rank constraint is satisfied.

[0044]

[0045] Furthermore, the optimization problem can be rewritten as the following equation (10): where F is a matrix consisting of (n+m)×c singular vectors.

[0046]

[0047] where tr(·) represents the trace of the matrix.

[0048] K-Multiple-Means solves the problem of Equation (10) using alternating optimization. Specifically, K-Multiple-Means solves the problem of Equation (10) by alternately updating F, matrix S, and prototype matrix A. More specifically, first, matrix S and prototype matrix A are fixed and F is updated, and then F and prototype matrix A are fixed and matrix S is updated. These updates are repeated until the rank constraint of the Laplacian matrix L is satisfied.

[0049] Finally, the prototype is updated by fixing F and matrix S and updating the prototype matrix A. K-Multiple-Means repeats these processes until the prototype converges. After convergence, c clusters are obtained from c connected components. In this way, in the alternating optimization of K-Multiple-Means, matrix S, F, and prototype matrix A are variables.

[0050] First, consider the step in which matrix S and prototype matrix A are fixed in the alternating optimization. When matrix S and prototype matrix A are fixed, L = I - D －１／２ W.D. －１／２ Therefore, the optimization problem of equation (10) is expressed by the following equation (11).

[0051]

[0052] In K-Multiple-Means, this problem is solved to obtain F. The process of obtaining F is the optimal step solution acquisition process. K-Multiple-Means is a technique that does not care how F is obtained, but F can be obtained, for example, by the tilde S eigenvector acquisition process.

[0053] <<<<Explanation Using Mathematical Formulas for the tilde S Eigenvector Acquisition Process>>> When F is acquired by the tilde S eigenvector acquisition process, the problem of equation (11) can be solved by calculating the singular vectors of the matrix expressed by the following equation (12) (see Non-Patent Document 4). Hereinafter, the matrix on the left side of equation (12) is referred to as the tilde S matrix.

[0054]

[0055] Here, the elements are all real numbers in the n × n matrix D ｎ and an m × m matrix D whose elements are all real numbers. ｍ are diagonal matrices that satisfy the relationship expressed by the following equation (13).

[0056]

[0057] n × c matrix U whose elements are all real numbers ｃ and an m × c matrix V whose elements are all real numbers.ｃ Let U and V be matrices consisting of left and right singular vectors corresponding to the c largest eigenvalues ​​in the tilde S matrix, respectively. Then, the matrix of the following equation (14) (hereinafter referred to as the “optimal U matrix”) and the matrix of the following equation (15) (hereinafter referred to as the “optimal V matrix”) are optimal solutions to the problem expressed by equation (11).

[0058]

[0059]

[0060] Therefore, F is expressed by the following equation (16).

[0061]

[0062] This is the tilde S eigenvector acquisition process. The optimal U matrix and the optimal V matrix are solutions to the optimization problem expressed by equation (11) or equation (12), respectively. In the tilde S eigenvector acquisition process, as is clear from the transformations of the equations up to this point, it is necessary to calculate the singular value decomposition of the tilde S matrix in order to obtain the optimal U matrix and the optimal V matrix. Therefore, performing the tilde S eigenvector acquisition process requires a high computational cost.

[0063] Next, consider the step in alternating optimization where F and the prototype matrix A are fixed. When F and the prototype matrix A are fixed, the problem of equation (10) is reduced to the problem expressed by the following equation (17).

[0064]

[0065] This problem has a closed-form solution (see reference 1).

[0066] Reference 1: Feiping Nie et al., Clustering and Projected Clustering with Adaptive Neighbors, KDD, 2014.

[0067] Specifically, the elements of the matrix S are expressed by the following equation (18).

[0068]

[0069] Here, l (lowercase l) is a hyperparameter that gives the number of prototypes close to each data point. The symbol in the following equation (19) represents the set of l neighboring prototypes for the i-th data point. D[i,j] is the set of x ｉ and a ｉ It represents the distance between ｉ,ｌ＋１ Ha x ｉ represents the distance of the (l+1)th closest prototype to

[0070]

[0071] In equation (18), D[i,j] is given by the following equation (20).

[0072]

[0073] where f ｉ represents the i-th row vector in F. To find the closest prototype, the distance given by equation (13) must be calculated for all pairs of data points and prototypes. Note that the matrix S is a sparse matrix with l non-zero elements in each row, and its elements are normalized to satisfy the condition shown in equation (21) below.

[0074]

[0075] The hyperparameter α is expressed by the following equation (22): Therefore, α can be set by adjusting the number l (lowercase L) of adjacent prototypes.

[0076]

[0077] When F and the matrix S are fixed, the problem expressed by equation (10) can be expressed by the following equation (23).

[0078]

[0079] Therefore, the prototype can be calculated by the following equation (24).

[0080]

[0081] K-Multiple-Means is such a technique.

[0082] Incidentally, K-Multiple-Means, which obtains F by executing the tilde S eigenvector acquisition process explained so far, can calculate clusters effectively, but the tilde S eigenvector acquisition process results in a long processing time. This problem will be explained in order to explain the technical significance of the Fujiwara process.

[0083] <<K-Multiple-Means Problem of Obtaining F by Executing .tilde.S Eigenvector Acquisition Processing>> Since the size of the .tilde.S matrix expressed in equation (12) is n × m, in order to calculate the singular vectors from the singular value decomposition of the .tilde.S matrix and update F in equation (16), ２ Since calculating the distance between a data point and a prototype from equation (20) requires a computational cost of nm(c+d), calculating the nearby prototypes from equation (18) and calculating the matrix S requires a computational cost of nm(c+d+log m)).

[0084] Therefore, the number of iterations required to obtain c connected components is t ｃ Let t ａ If the number of iterations until the prototype converges is nm(d+c+m)t, the calculation cost of K-Multiple-Means is ｃ +nmdt ａ ) The memory cost of K-Multiple-Means is (n(m+d)). This means that an amount of memory on the order of n-th is required to store the data points. The sizes of matrix F, matrix S, and prototype matrix A are c(n+m), nm, and md, respectively, where c<m<n.

[0085] When the tilde S eigenvector acquisition process is used, the calculation cost is thus high. On the other hand, when the Fujiwara process is used, the calculation cost is reduced compared to the tilde S eigenvector acquisition process. This will be explained.

[0086] <<Explanation Using Formulas for Fujiwara Processing>> In Fujiwara processing, instead of the eigenvectors of the tilde S matrix, the eigenvectors of matrix M, which is the product of the tilde S matrix and the transpose of the tilde S matrix, are obtained as a solution to the optimization problem expressed by equation (11) or (12). Matrix M can be expressed mathematically as the following equation (25).

[0087]

[0088] From equation (12), the n×m tilde S matrix can also be said to represent the similarity between data points and prototypes, similar to the matrix S. Therefore, the m×m matrix M can be said to represent the similarity between prototypes.

[0089] The matrix M is the product of a tilde S matrix and the transpose of the tilde S matrix, and as can be seen from equation (25), it is a symmetric matrix, a type of square matrix. Therefore, the singular vectors of the tilde S matrix are the eigenvectors of the matrix M. Note that, as can be seen from equation (25), the tilde S matrix must be obtained to calculate the matrix M. Therefore, the tilde S matrix may be obtained quickly by, for example, pruning unnecessary calculations and using a lower limit value for the distance between the data points and the prototype.

[0090] <Effects of Obtaining Eigenvectors of Matrix M> For an n × m tilde S matrix, m is smaller than n (i.e., m<n). Therefore, the tilde S matrix is ​​a general matrix with a rank of at most m. On the other hand, matrix M is a symmetric matrix. In general, obtaining the eigenvectors of a symmetric matrix requires less computational cost than obtaining the singular vectors of a general matrix. This is because obtaining the singular vectors requires repeated singular value decomposition, which is computationally expensive.

[0091] Therefore, the eigenvectors can be obtained with lower calculation cost by obtaining the eigenvectors of the matrix M. Note that the power method is used to obtain the eigenvectors of a square matrix.

[0092] Also, this is not a general theory, but a reason unique to K-Multiple-Means. In K-Multiple-Means, the number of prototypes is much smaller than the number of data points (i.e., m<<n). Therefore, the process of obtaining the eigenvectors of the m x m matrix M can be performed with low computational cost due to the small size of matrix M. This is one of the reasons why K-Multiple-Means is unique.

[0093] Furthermore, when the eigenvalues ​​of matrix M are obtained by the power method, the m×m matrix M is divided into small block matrices (hereinafter referred to as "M block matrices") corresponding to connected components during the repeated calculation process, and therefore the eigenvectors are obtained from the M block matrices that are smaller in size than matrix M. In this respect, too, use of matrix M makes it possible to obtain eigenvectors with lower calculation cost.

[0094] In this way, Fujiwara-type K-Multiple-Means, which is a K-Multiple-Means that performs Fujiwara processing, has a lower calculation cost than K-Multiple-Means that performs tilde S eigenvector acquisition processing. Therefore, Fujiwara-type K-Multiple-Means can reduce the time required to execute K-Multiple-Means.

[0095] <<Regarding the Correspondence Between the Eigenvectors of Matrix M and the Singular Vectors of the tilde S Matrix>> Here, we will explain the correspondence between the eigenvectors of matrix M and the singular vectors of the tilde S matrix, and further explain the effect of Fujiwara-type K-Multiple-Means. As described above, the Fujiwara processing avoids performing singular value decomposition of the tilde S matrix and uses matrix M given by equation (25). In fact, the following Fujiwara's 1st Theorem holds for matrix M.

[0096] <<<<Fujiwara's 1st Theorem>>> λ ｉ Let be the i-th largest eigenvalue of the symmetric matrix M, and q ｉ λ ｉ Let u be the eigenvector corresponding to ｉ and v ｉand are respectively the left singular vector and the right singular vector corresponding to the i-th largest singular value of the tilde S matrix. In this case, the matrix M is ｉ = σ ｉ ２ The i-th left singular vector and right singular vector of the tilde S matrix are expressed by the following equation (26).

[0097]

[0098] Next, we prove Fujiwara's first theorem.

[0099] <<<<Proof of Fujiwara's First Theorem>>> Σ=diag(σ １ ,・・・,σ ｍ ) is a diagonal matrix of singular values, and U = [u1,...,u ｍ ] and V=[v1,...,v ｍ ] is a matrix consisting of singular vectors, the matrix M is M = VΣU Ｔ UΣV Ｔ = VΣ ２ V Ｔ Furthermore, Λ = diag(λ １ ,・・・,λ ｍ ) is a diagonal matrix consisting of the eigenvalues ​​of matrix M, and Q = [q １ ,・・・,q ｍ ] is a matrix consisting of eigenvectors of matrix M, then M = QΛQ Ｔ As a result, VV Ｔ =V Ｔ V = I and QQ Ｔ =Q Ｔ Since Q = I, V = Q and Σ ２ =Λ. Therefore, v ｉ =q ｉ and λ ｉ = σ ｉ ２ Furthermore, since the relationship of the following equation (27) holds, the relationship of the following equation (28) holds. As a result, the relationship of the following equation (29) holds.

[0100] Q.E.D. The proof of Fujiwara's first theorem is complete.

[0101]

[0102]

[0103]

[0104] Fujiwara's First Theorem states that the left and right singular vectors of the tilde S matrix can be calculated from the eigenvalue decomposition of the m x m matrix M. Therefore, it is not necessary to calculate the singular value decomposition of the tilde S matrix. Furthermore, since the similarity between the data points and the prototypes is updated to satisfy the rank constraint, the bipartite graph of the data points and the prototypes is divided into multiple connected components during the repeated calculation process.

[0105] As a result, the matrix M is divided into multiple block matrices (i.e., M block matrices) during the iterative calculation. Specifically, let M' be a matrix obtained by rearranging the prototypes in the matrix M, and let m ｉ is the number of prototypes contained in the i-th connected component, and the matrix M ｉ ´をm ｉ ×m ｉ is the i-th M block matrix and b is the number of M block matrices corresponding to the connected components, the following equation (30) holds.

[0106]

[0107] Note that P is a matrix for rearranging prototypes. The matrix P is an m x m orthogonal matrix, with only one element of 1 in each row and example, and the other elements are 0. If p[i][j] = 1, the i-th row is rearranged to the j-th row.

[0108] Next, we consider the M-block matrix M ｉ An example of an algorithm for obtaining ' is shown as Algorithm 1.

[0109] FIG. 2 shows the M-block matrix M using the matrix P in the embodiment. ｉ 1 is a diagram showing Algorithm 1, which is an algorithm for obtaining 0′. ｎ= [0, ..., 0] is a row vector of length n, the symbol in the following equation (31) represents the i-th column vector of the tilde S matrix, r represents a column vector of length n, and the symbol in the following equation (32) is a matrix representing the set of prototypes contained in the i-th connected component. Hereinafter, the symbol in the following equation (31) will be referred to as the i-th tilde c column vector.

[0110]

[0111]

[0112] In Algorithm 1, r is first set based on the non-zero elements of the tilde S matrix (lines 2 to 4 of Algorithm 1). Next, in Algorithm 1, element m[i][j] of matrix M is obtained based on the inner product of r and the i-th tilde c column vector according to equation (25) (lines 5 to 8 of Algorithm 1). Next, in Algorithm 1, the connected components of matrix M and the permutation matrix P are obtained (lines 9 to 15 of Algorithm 1). Next, in Algorithm 1, matrix M' is obtained according to equation (30).

[0113] In fact, such an algorithm 1 is an M-block matrix M ｉ ' can be calculated efficiently. This is shown by Fujiwara's 2nd Theorem.

[0114] <<<Fujiwara's 2nd Theorem>>> Algorithm 1 is based on the tilde S matrix and calculates the M-block matrix M with computational cost on the order of nml. １ ´ , ..., M ｂ ´ can be obtained.

[0115] Prove Fujiwara's second theorem.

[0116] <<<<Proof of Fujiwara's Second Theorem>>> To obtain a vector r of length m, ２This is because, as described in lines 1 and 2 of Algorithm 1, a process of making a vector of length m into a zero vector is performed m times. Since the number of non-zero elements in the i-th column vector is (nl) / m and matrix M is an m x m matrix, a calculation cost of the order of nml is required to obtain matrix M. Also, the number of connected components of matrix M is m. ２ Since matrix M' is a matrix obtained by rearranging the prototypes of matrix M, matrix M' is obtained by ２ Therefore, the computational cost of Algorithm 1 is on the order of nml.

[0117] Q.E.D. The proof of Fujiwara's second theorem is complete.

[0118] Since the matrix M′ in equation (30) is a block diagonal matrix, the eigenvectors of the matrix M are M block matrices M ｉ Specifically, λ ｉ,ｊ M block matrix M ｉ Let q be the j-th largest eigenvalue of q ｉ，ｊ λ ｉ,ｊ M-block matrix M corresponding to ｉ ´, and λ ｊ´ λ ｉ,ｊ Let q be the eigenvectors of M corresponding to ｊ´ λ ｊ´ is the eigenvector of the matrix M corresponding to, then the following equation (33) holds.

[0119]

[0120] In this way, the M-block matrix M ｉ Based on this, the eigenvalues ​​and eigenvectors of the matrix M can be obtained at high speed. In this sense, Fujihara-type K-Multiple-Means can reduce the time required to execute K-Multiple-Means.

[0121] 3 is a diagram illustrating an example of a hardware configuration of the clustering device 1 according to an embodiment. The clustering device 1 includes a control unit 11 that is a control unit including a processor 91, such as a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU), and a memory 92, all connected via a bus, and executes a program. By executing the program, the clustering device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0122] More specifically, the processor 91 reads out a program stored in the storage unit 13 and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the clustering device 1 functions as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0123] The control unit 11 controls the operation of each functional unit included in the clustering device 1. The control unit 11 executes, for example, Fujihara-type K-Multiple-Means. By executing Fujihara-type K-Multiple-Means, the control unit 11 classifies (i.e., clusters) one or more data points to be clustered.

[0124] The control unit 11 acquires, for example, information stored in the storage unit 13. Specifically, the process of acquiring information stored in the storage unit 13 is a read process.

[0125] The interface unit 12 includes a communication interface for connecting the clustering device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits information indicating each clustering target (hereinafter referred to as "clustering target information"). The interface unit 12 acquires the clustering target information by communicating with the device that transmits the clustering target information.

[0126] The interface unit 12 may be configured to include input devices such as a mouse, a keyboard, a touch panel, etc. The interface unit 12 may be configured as an interface that connects these input devices to the clustering device 1. In this way, the input devices of the interface unit 12 accept input of various information to the clustering device 1 via wired or wireless connections. Note that information does not necessarily have to be input to the communication interface of the interface unit 12, but may also be input to the input devices of the interface unit 12.

[0127] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes, for example, a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, and a speaker. The interface unit 12 may be configured as an interface that connects these display devices or speakers to the clustering device 1. Therefore, the interface unit 12 outputs, for example, information input to an input device of the interface unit 12 as an image or sound.

[0128] The storage unit 13 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 13 stores various information related to the clustering device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may exist on a cloud, for example.

[0129] 4 is a flowchart showing an example of the flow of processing executed by the clustering device 1 in this embodiment. The control unit 11 acquires clustering target information (step S101). Next, the control unit 11 executes Fujihara-type K-Multiple-Means on the clustering targets indicated by the clustering target information (step S102). By executing Fujihara-type K-Multiple-Means, each clustering target indicated by the clustering target information obtained in step S101 is classified.

[0130] The clustering device 1 of this embodiment configured as described above executes Fujiwara-type K-Multiple-Means. Therefore, for the reasons explained in <Effects of Obtaining Eigenvectors of Matrix M>, which includes explanations of Fujiwara's First Theorem and Fujiwara's Second Theorem, the clustering device 1 can reduce the time required to execute K-Multiple-Means.

[0131] (Modification) <First Technique for Further Accelerating Acquisition of Eigenvalues ​​and Eigenvectors of Matrix M> Incidentally, it is possible to further speed up acquisition of the largest eigenvalue and eigenvector of matrix M. This will be explained.

[0132] As explained earlier, the M-block matrix M ｉ Based on the c eigenvalues ​​and eigenvectors of ', the c left singular vectors and the c right singular vectors of the 'S' matrix are obtained. According to Fujiwara's first theorem, the eigenvalues ​​of matrix M and the singular values ​​of the 'S' matrix are non-negative. Therefore, the c singular vectors are obtained for each M-block matrix M. ｉ ' is obtained by applying the power method to find c eigenvalues ​​and eigenvectors. However, this process is performed for b M-block matrices M ｉ Since this is a process of calculating c eigenvectors for ', the power method must be calculated bc times. However, the number of times the power method is executed can be reduced by performing the process based on Fujiwara's 3rd Theorem below (hereinafter referred to as "Fujiwara's first sub-process." A more precise definition will be given later.)

[0133] <<Fujiwara's 3rd Theorem>> Let b be a column vector of length m defined by the following equation (34), and P ｉ Let P = [P １ ,・・・,P ｂ ] the i-th m × m ｉ Let b be a submatrix of ｉ ´ to b ｉ ´=P ｉ Ｔ The length of b is m ｉ is a column vector of the i-th M-block matrix Mｉ The largest eigenvalue of ´ is λ ｉ,１ = 1, and its eigenvector is q ｉ,１ = b ｉ ´.

[0134]

[0135] Prove Fujiwara's third theorem.

[0136] <<Proof of Fujiwara's Third Theorem>> B ｉ is expressed by the matrix PS Ｔ SP Ｔ the i-th m ｉ ×m ｉ (hereinafter referred to as "B block matrix").

[0137]

[0138] where B is the block matrix B ｉ represents the similarity between prototypes in the i-th connected component. ｉ ' is the B block matrix B ｉ As the Laplacian matrix of ｉ ' is expressed by the following equation (36).

[0139]

[0140] B block matrix B ｉ is one connected component, so L ｉ ' has one eigenvalue that is 0, and its eigenvector is expressed by the following equation (37) (see Reference 2).

[0141] Reference 2: Ulrike von Luxburg, A Tutorial on Spectral Clustering, Stat. Comput., 2007.

[0142]

[0143] Furthermore, S Ｔ The (i, j) element of S is expressed by the following formula (38), and the following formula (39) holds from formula (21). Therefore, S Ｔ The sum of the i-th row elements of S is expressed by the following equation (40).

[0144]

[0145]

[0146]

[0147] Furthermore, the following equation (41) holds true from equations (3) to (6).

[0148]

[0149] Therefore, S Ｔ The sum of the i-th row elements of S is expressed by the following equation (42).

[0150]

[0151] From equation (35), the matrix PS Ｔ SP Ｔ is S Ｔ It is a matrix obtained by rearranging the rows and columns of S. Therefore, if the matrix S Ｔ The i-th row of S is the B block matrix B ｉ If the i'th row of the equation (36) is rearranged, ｉ The following equation (43) holds for the i'th diagonal element in

[0152]

[0153] As a result, the following equation (44) holds true, so the vector b ｉ ´ is L ｉ ' is the eigenvector corresponding to the eigenvalue 0.

[0154]

[0155] Furthermore, from the formulas (3) to (6) and (13), D ｎ =I, the following equation (45) holds true from equations (12), (25), and (30).

[0156]

[0157] Therefore, from equations (30) and (35), each M block matrix M ｉ ' or each B block matrix B ｉ The following equation (46) holds true for

[0158]

[0159] On the other hand, if λ ｉ,ｊ ´ to L ｉ ' is the jth largest eigenvalue of L ｉ ', the property of the following equation (47) holds (see Reference 3).

[0160]

[0161] Reference 3: Fan RK Chung, Spectral Graph Theory, American Mathematical Society, 1996.

[0162] Therefore, from equation (36), L ｉ ´=I-M ｉ ', and from Fujiwara's first theorem and equation (33), we can obtain the M-block matrix M ｉ Since ' has non-negative eigenvalues, the M-block matrix M ｉ For each eigenvalue of ´, the following holds:

[0163]

[0164] Furthermore, L ｉ ´=I-M ｉ ' is the eigenvector b corresponding to the eigenvalue 0 ｉ ', M block matrix M ｉ ´ is λ ｉ，１ = 1 corresponding eigenvector q ｉ，１ = b ｉ ´.

[0165] Q.E.D. The proof of Fujiwara's third theorem is complete.

[0166] Here, the eigenvectors obtained from Fujiwara's third theorem are not normalized. Therefore, based on Fujiwara's third theorem, each M-block matrix M ｉ In the process of obtaining the largest eigenvalue and eigenvector of ' (i.e., Fujiwara's first sub-process), the eigenvector q ｉ，１ Instead of (49), the eigenvector on the left side of the equation (49) is obtained as follows: As can be seen from the equation (49), the eigenvector on the left side of the equation (49) is the eigenvector q ｉ，１Therefore, in other words, the eigenvector on the left side of equation (49) is the normalized eigenvector q ｉ，１ is.

[0167]

[0168] In addition, the eigenvector q ｉ，１ is obtained from the elements of the diagonal matrix D.

[0169] In view of the above, if we define Fujiwara's first sub-processing more precisely, the first sub-processing of Fujiwara is defined as the M-block matrix M ｉ The eigenvector corresponding to the largest eigenvalue of ' is the eigenvector corresponding to the eigenvalue with a value of 1. According to Fujiwara's third theorem, the eigenvalue with a value of 1 is the M-block matrix M ｉ ' is the maximum eigenvalue, the eigenvector obtained by Fujiwara's first sub-processing is the eigenvector corresponding to the maximum eigenvalue.

[0170] <<Effects of Executing Fujiwara's First Sub-Processing>> In this way, Fujiwara's third theorem shows that it is possible to obtain the maximum eigenvalue without calculating the eigenvalue. Therefore, by Fujiwara's first sub-processing, which is a process for obtaining the maximum eigenvalue and its corresponding eigenvector based on Fujiwara's third theorem, each of the M block matrices M ｉ The maximum eigenvalue and eigenvector of the ' can be obtained more quickly. Therefore, the control unit 11 may execute the first Fujiwara sub-processing in the Fujiwara-type K-Multiple-Means. This allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0171] <Second Technique for Further Accelerating Acquisition of Eigenvalues ​​and Eigenvectors of Matrix M> It is possible to further speed up the process of acquiring the eigenvalues ​​and eigenvectors of matrix M. According to Fujiwara's Third Theorem, the largest eigenvalue and its eigenvector of a block matrix can be efficiently acquired as described above without using the power method. However, it is necessary to obtain the c largest eigenvalues ​​and their eigenvectors of the block matrix.

[0172] Therefore, if the number b of block matrices is smaller than the number c of clusters, it is necessary to obtain c-b eigenvalues ​​and eigenvectors from the block matrices using the power method. To quickly calculate these eigenvalues ​​and eigenvectors, for example, the upper limit of the eigenvalues ​​may be calculated using the following Fujiwara's second sub-process. Unnecessary calculations are pruned by executing Fujiwara's second sub-process.

[0173] <<Fujiwara's second sub-process>> M block matrix M ｉ ', λ represented by the symbol on the left side of the following equation (50) when j>1. ｉ，ｊ The upper limit of is obtained according to the following equation (50).

[0174]

[0175] λ ｉ，ｊ For the upper limit of , the following Fujiwara's 4th Theorem holds:

[0176] <<Fujiwara's 4th Theorem>> M-block matrix M ｉ ´, λ ｉ，ｊ The upper limit of λ ｉ，ｊ That is, the M block matrix M ｉ For ', the following equation (51) holds:

[0177]

[0178] Prove Fujiwara's fourth theorem.

[0179] <<Proof of Fujiwara's Fourth Theorem>> When j=2, the following equation (52) holds; otherwise, the following equation (53) holds. Therefore, the following equation (54) holds. Furthermore, the following equation (55) holds (see Reference 4), so that, as per Fujiwara's First Theorem, the M-block matrix M ｉ The eigenvalues ​​of λ′ are non-negative. Therefore, the following equation (56) holds. Furthermore, the following equation (57) holds. Therefore, λ ｉ，ｊ Regarding the upper limit value of , the following equation (58) holds.

[0180]

[0181]

[0182]

[0183]

[0184]

[0185]

[0186]

[0187] Reference 4: Gene H. Golub et al., Matrix Computations (3rd ed.), Johns Hopkins University Press, 1996.

[0188] Q.E.D. The proof of Fujiwara's fourth theorem is complete.

[0189] <<Effects of Executing Fujiwara's Second Sub-Processing>> In this way, by executing Fujiwara's second sub-processing, the above-mentioned c-b eigenvalues ​​and eigenvectors can be obtained more quickly. Therefore, the control unit 11 may execute Fujiwara's second sub-processing in Fujiwara-type K-Multiple-Means. This allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0190] <Third technique for further speeding up the acquisition of eigenvalues ​​and eigenvectors of matrix M> <<Algorithm 2>> An example of a speed-up algorithm using Fujiwara's third theorem (or execution of Fujiwara's first sub-processing) and Fujiwara's fourth theorem (or execution of Fujiwara's second sub-processing) explained so far will be explained as Algorithm 2.

[0191] 5 shows Algorithm 2, which is an algorithm for obtaining eigenvectors corresponding to the c largest eigenvalues ​​of matrix M in the modified example. Algorithm 2 uses Fujiwara's third theorem and Fujiwara's fourth theorem to obtain the M-block matrix M corresponding to the c largest eigenvalues ​​of matrix M. ｉIn Algorithm 2, the symbols in the following formula (59) represent a set of c largest eigenvalues, and the symbols in the following formula (60) represent b M-block matrices M ｉ represents the set of ´.

[0192]

[0193]

[0194] In Algorithm 2, first, c dummy eigenvalues ​​whose eigenvalue is 0 are added to the set represented by the symbol in Equation (59) for initialization (lines 1 and 2 of Algorithm 2). Next, in Algorithm 2, according to Fujiwara's third theorem (i.e., by executing Fujiwara's first subprocess), the M-block matrix M ｉ The largest eigenvalue of ' and its eigenvector are obtained (lines 3-10 of Algorithm 2).

[0195] Specifically, the M-block matrix M ｉ To specify the largest eigenvalue for ', the pair (i,j) is set to (i,1) (line 4 of Algorithm 2). Then, the largest eigenvalue and its eigenvector are obtained (lines 5-6 of Algorithm 2). Then, to obtain the second largest eigenvalue, the M-block matrix M ｉ ' to the matrix q corresponding to the largest eigenvalue ｉ,ｊ λ ｉ,ｊ q ｉ,ｊ Ｔ is subtracted (line 7 of Algorithm 2). Then λ ｉ,ｊ The set represented by the symbols in Equation (59) is updated using (lines 8-9 of Algorithm 2). In addition, to obtain the second largest eigenvalue, the pair (i, j) is updated to (i, j+1) (line 10 of Algorithm 2).

[0196] Next, if b<c, c-b eigenvalues ​​and their eigenvectors are obtained according to Fujiwara's fourth theorem (i.e., by executing Fujiwara's second subprocess) (lines 11 to 21 of Algorithm 2).

[0197] Specifically, the M-block matrix M ｉ' is estimated, and if its maximum value is smaller than the obtained eigenvalue, the M-block matrix M ｉ The eigenvalues ​​and eigenvectors of ' are obtained by the power method (lines 13-15 of Algorithm 2). Then, the M-block matrix M ｉ ' and the set represented by the symbol in Equation (59) and (i, j) are updated (lines 16 to 20 of Algorithm 2). These processes are repeated until the maximum upper limit value of the eigenvalue becomes smaller than the previously obtained eigenvalue (line 21 of Algorithm 2).

[0198] Next, c eigenvalues ​​and eigenvectors of the matrix M are obtained according to equation (33) (lines 22-23 of Algorithm 2).

[0199] In this way, by executing Algorithm 2, eigenvectors corresponding to the c largest eigenvalues ​​of matrix M can be quickly obtained. Therefore, the control unit 11 may execute Algorithm 2 in Fujihara-type K-Multiple-Means. This allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0200] Incidentally, for Algorithm 2, the following Fujiwara's 5th Theorem holds.

[0201] <<Fujiwara's 5th Theorem>> t ｐ Let m be the number of iterations in the power method. Algorithm 2 calculates the c largest eigenvalues ​​of matrix M and their eigenvectors as (m / b) ２ t ｐ + (mc / b) order of computational cost.

[0202] Prove Fujiwara's 5th Theorem.

[0203] <<Proof of Fujiwara's Fifth Theorem>> In Algorithm 2, the maximum eigenvalue and its eigenvector of each block matrix can be obtained with a computational cost on the order of m. According to Equation (50), the upper limit of the eigenvalue can be obtained with a computational cost on the order of m. Since the size of each block matrix is ​​(m / b) × (m / b), the computational cost of the power method is (m / b)２ t ｐ The vector q ｉ，ｊ Since the length of is m / b, the eigenvectors of the matrix M can be obtained in the order of (mc / b) according to equation (33). Therefore, the computational cost of Algorithm 2 is (m / b) ２ t ｐ It is on the order of +(mc / b).

[0204] Q.E.D. Fujiwara's fifth theorem is proven.

[0205] <<Effects of Executing Algorithm 2>> K-Multiple-Means, which executes the tilde S eigenvector acquisition process, needs to perform singular value decomposition of an n × m tilde S matrix to obtain c singular vectors. On the other hand, Algorithm 2 obtains eigenvalues ​​and their eigenvectors from a small block matrix. Singular vectors are obtained from the eigenvalues ​​and their eigenvectors. Therefore, Fujihara-type K-Multiple-Means, which executes Algorithm 2, can obtain singular vectors more quickly than K-Multiple-Means, which executes the tilde S eigenvector acquisition process. Therefore, the control unit 11 may execute, for example, Fujihara-type K-Multiple-Means, which executes Algorithm 2. This allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0206] <Fourth technique for further increasing the speed of obtaining eigenvalues ​​and eigenvectors of matrix M> As shown in equation (25), in Fujihara-type K-Multiple-Means, the tilde S matrix is ​​used to obtain a block matrix. The tilde S matrix is ​​obtained using the matrix S as shown in equation (12). As shown in equation (18), the matrix S is obtained by calculating a prototype that is close to each data point. Therefore, to obtain the tilde S matrix, it is necessary to calculate a nearby prototype based on the distance "D[i,j]" shown in the following equation (61).

[0207]

[0208]

[0209]

[0210] where β is a hyperparameter. Ｅ [i,j] is a vector x of length d ｉ and a ｊ The distance between the Ｆ [i,j] is a vector f of length c ｉ and f ｎ＋ｊ Therefore, if the number of dimensions d and the number of clusters c are large, the calculation cost for calculating the distance D[i,j] becomes large. As a result, the time required for the process to obtain the tilde S matrix becomes long. Therefore, a technique for shortening the time required to obtain the tilde S matrix will be described. More specifically, the distance D Ｆ A technique for obtaining [i,j] more quickly will be described.

[0211] Well then, D Ｆ We will now discuss a technique for quickly obtaining [i,j] by pruning unnecessary calculations. For simplicity, we will consider the case where b<c. This is the case where the number of block matrices is smaller than the number of clusters. Since the block matrices represent connected components, if b<c, the number of connected components will be smaller than the number of clusters.

[0212] f ｉ,ｂ f ｉ = [f ｉ,ｂ , f ｉ,c-ｂ ] such that ｉ Let f be a subvector of length b ｎ＋ｊ , ｂ f ｎ＋ｊ = [f ｎ＋ｊ,ｂ , f ｎ＋ｊ,c-ｂ ] such that ｎ＋ｊ Let f be a subvector of length b. More specifically, ｉ，ｂ and f ｎ＋ｊ，ｂ and respectively ｉ and f ｎ＋ｊ Let f be a vector representing the first b elements of f and b<c. ｉ,ｂ and f ｎ＋ｊ , ｂ For and , the following Fujiwara's 6th Theorem holds:

[0213] <<Fujiwara's 6th Theorem>> The symbol in the following equation (64) is x ｉ represents a connected component containing ｊ If we represent a connected component containing n, then the following equation (66) holds: ｉ and ｊ and x respectively. ｉ and a connected component containing ｊ represents the number of data points contained in the connected component containing

[0214]

[0215]

[0216]

[0217] Prove Fujiwara's 6th Theorem.

[0218] <<Proof of Fujiwara's Sixth Theorem>> As shown in equation (26), the eigenvectors for the b largest eigenvalues ​​of the block matrix correspond to the singular vectors of the tilde S matrix. Therefore, from equation (16), the subvector f ｉ，ｂ corresponds to the eigenvectors for the b largest eigenvalues. ｋ A matrix U that satisfies the condition that k is less than or equal to b ｃ is the k-th column vector in, then, as shown by Fujiwara's first theorem, the following equation (67) and V = Q hold. Therefore, from Fujiwara's third theorem, the following equation (68) holds.

[0219]

[0220]

[0221] x ｉ If is included in the k-th connected component, then x ｉ The connected components containing M ｋ′ ', the following equation (69) is established from equations (33) and (49).

[0222]

[0223] From equations (3), (5), and (13), D ｎ=I, the following equation (70) holds from equation (12). ｉ ' is an M-block matrix M ｋ′ ' corresponding matrix SD ｍ －１／２ P Ｔ The length of the i-th row vector of ｋ′ is a sub-vector of, the following equation (71) holds from equation (16).

[0224]

[0225]

[0226] matrix SD ｍ －１／２ P Ｔ The i-th row vector s ｉ ' is the block matrix M ｋ′ ', the result of equation (71) is used to obtain the relationship of equation (72) below.

[0227]

[0228] Furthermore, as shown in equation (21), each row of the matrix S is normalized. Therefore, the following equation (73) holds true.

[0229]

[0230] Therefore, the following equation (74) holds from equations (3), (5), and (21).

[0231]

[0232] Therefore, x ｉ is included in the k-th connected component, the following equation (75) holds from equation (74).

[0233]

[0234] On the other hand, x ｉ If is not included in the k-th connected component, the following equation (76) holds from equation (33).

[0235]

[0236] Furthermore, since the eigenvectors are normalized as shown in equation (49), aｊ is included in the k-th connected component and a ｊ is the matrix M ｋ When the j'th row of ' is represented by the equations (16), (26), and (49), the following equation (77) is obtained.

[0237]

[0238] Furthermore, from Fujiwara's third theorem, the following equation (78) holds.

[0239]

[0240] Therefore, from equation (74), a ｊ is included in the k-th connected component, the following equation (79) holds.

[0241]

[0242] On the other hand, a ｊ If is not included in the k-th connected component, the following equation (80) holds from equation (33).

[0243]

[0244] Therefore, from equations (75), (76), (79), and (80), x ｉ and a ｊ If and are in the same connected component, the following equation (81) holds.

[0245]

[0246] Therefore, the following equation (82) holds true.

[0247]

[0248] Also x ｉ and a ｊ If and are in different connected components, the following equation (83) holds.

[0249]

[0250] Q.E.D. Fujiwara's 6th Theorem is now proven.

[0251] <<According to equation (66), f ｉIn this way, Fujiwara's 6th theorem states that the first b elements are obtained by dividing the first b elements by the distance D Ｆ [i, j] can be obtained with a computational cost of the order of 1, not the order of b. Therefore, according to equation (66), ｉ By performing a process to obtain the first b elements of the distance D Ｆ [i, j] can be obtained more quickly. Therefore, the control unit 11 may execute, for example, Fujiwara's K-Multiple-Means, which executes Fujiwara's third sub-process. This allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0252] <Fifth technique for further increasing the speed of obtaining eigenvalues ​​and eigenvectors of matrix M> ｉ Fujiwara's third subprocess was explained as a technique for obtaining the first b elements of f more quickly. ｉ The process of obtaining the remaining c−b elements of can also be speeded up, so it will be explained below.

[0253] Fujiwara's 7th Theorem follows from Fujiwara's 6th Theorem.

[0254] <<Fujiwara's 7th Theorem>> When the following equations (84) and (85) are satisfied, if b<c, then the distance D Ｆ [i, j] is obtained by the following equation (86).

[0255]

[0256]

[0257]

[0258] Prove Fujiwara's 7th Theorem.

[0259] <<Proof of Fujiwara's Seventh Theorem>> Since the following equation (87) is satisfied, the following equation (88) is satisfied.

[0260]

[0261]

[0262] When the following equation (89) is satisfied, d[i]=1, and therefore, the following equation (90) holds true from Fujiwara's sixth theorem.

[0263]

[0264]

[0265] When the following equation (91) is satisfied, the following equation (92) holds true from Fujiwara's sixth theorem.

[0266]

[0267]

[0268] Since the relationship of the above formula (91) is satisfied, x ｉ and a ｉ are included in different connected components. Therefore, if <•> represents the inner product of vectors, the following equation (93) is obtained from equation (33).

[0269]

[0270] Therefore, the following equation (94) is obtained.

[0271]

[0272] Therefore, when the relationship of the above formula (91) is satisfied, the following formula (95) is satisfied.

[0273]

[0274] Q.E.D. Fujiwara's 7th Theorem is now proven.

[0275] <<Effect of performing the calculation of equation (86)>> Fujiwara's 7th theorem states that x ｉ and a ｊ If and are in the same connected component, then f ｉ，ｃ－ｂ Since is a vector of length c−b, by carrying out equation (86), f ｉ Skip the first b elements of Ｆ Fujiwara's seventh theorem shows that [i, j] can be obtained with a computational cost of the order of c - b. ｉ and aｊ When the distances D Ｆ This shows that [i, j] can be calculated to the order of 1. Therefore, Fujiwara's seventh theorem states that if b<c, the distance D can be calculated by performing the calculation of equation (86) (hereinafter referred to as "Fujiwara's fourth sub-process"). Ｆ It is shown that [i, j] can be obtained quickly.

[0276] Therefore, when b<c, the control unit 11 may execute, for example, Fujiwara's fourth sub-process, Fujiwara's K-Multiple-Means, which allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0277] Note that if b is equal to or greater than c, the M-block matrix M ｉ The eigenvector of the maximum eigenvalue of ' is vector f ｉ Therefore, it is clear from Fujiwara's 7th theorem that the distance D Ｆ [i, j] is obtained by the following equation (96).

[0278]

[0279] <<Effect of Calculating Equation (96)>> Equation (96) states that if b is equal to or greater than c, then the distance D Ｆ It shows that [i, j] can be calculated to the order of 1. Equation (96) further shows that if b or c is greater than or equal to Ｆ [i, j] is f ｉ and f ｎ＋ｊ Therefore, if b is greater than or equal to c, the distance D can be obtained without calculating F. Ｆ [i, j] is obtained.

[0280] From the above, if b is equal to or greater than c, the control unit 11 may, for example, perform the calculation of equation (96) (hereinafter referred to as "Fujiwara's fifth sub-processing"). That is, if b is equal to or greater than c, the control unit 11 may, for example, perform Fujiwara-type K-Multiple-Means, which executes Fujiwara's fifth sub-processing. This allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0281] <Sixth technique for further increasing the speed of obtaining eigenvalues ​​and eigenvectors of matrix M> Up to this point, the distance D in equation (63) Ｆ An example of a technique for speeding up the process of obtaining [i, j] has been explained. Next, the distance D in equation (62) E An example of a technique for speeding up the process of obtaining [i, j] will be described.

[0282] As mentioned earlier, the distance D Ｆ [i, j] can be calculated quickly. However, the distance D E Since the computational cost of [i,j] is on the order of d, the processing time required to find the nearest prototype increases when the dimensionality of the data points is large. This processing time can be reduced by approximating the data points and prototypes using singular value decomposition, as described below.

[0283] A data point x with dimensionality d ｉ = [x ｉ [1], ..., x ｉ [d] and prototype a ｊ = [a ｊ [1], ..., a ｊ To approximate [d], a singular value decomposition of rank d' is used on the sampled data points.

[0284] Specifically, the singular value decomposition of rank d' for X' is U ｄ′ Σ ｄ′ V Ｔ ｄ′ In addition, x defined in equation (97) is calculated as follows: ｉ Approximation of (hereinafter referred to as "tilde x ｉ ") and a defined by equation (98) ｉ Approximation of (hereinafter referred to as "tilde a ｉ ") and are calculated using the following equations (99) and (100), respectively. ｉ and tilde a ｉ Both are d-dimensional vectors. １ ´, ..., x ｍ ´] Ｔ is an m×d matrix obtained by randomly sampling m data points.

[0285]

[0286]

[0287]

[0288]

[0289] <<Fujiwara Distance>> Nearby prototypes can be obtained quickly by using the Fujiwara distance defined by the following equation (101). ｉ and a ｉ is the lower limit of the distance between

[0290]

[0291]

[0292]

[0293] Here, we will explain Fujiwara's 8th Theorem, which is a theorem regarding the properties of the Fujiwara distance.

[0294] <<Fujiwara's 8th Theorem>> The Fujiwara distance satisfies the relationship of the following formula (104).

[0295]

[0296] Prove Fujiwara's 8th Theorem.

[0297] <<Proof of Fujiwara's Eighth Theorem>> The following equation (105) is obtained from equation (61) and equation (62).

[0298]

[0299] Since the singular value decomposition is an orthonormal transformation, the following equation (106) is obtained.

[0300]

[0301] Therefore, if the Cauchy-Schwarz inequality is applied, the following equation (107) holds.

[0302]

[0303] Therefore, the following equation (108) holds true.

[0304]

[0305] Q.E.D. Fujiwara's 8th Theorem is now proven.

[0306] <<The effect of using Fujiwara distance>> Tilde x ｉ and tilde a ｉ Since is a vector of length d', the Fujiwara distance can be obtained faster than D[i,j]. Therefore, calculating the Fujiwara distance allows for fast finding of nearby prototypes.

[0307] Based on the above, the control unit 11 may, for example, calculate the Fujiwara distance (hereinafter referred to as "Fujiwara's sixth sub-processing"). That is, the control unit 11 may, for example, execute Fujiwara-type K-Multiple-Means, which executes Fujiwara's sixth sub-processing. This allows the clustering device 1 to further reduce the time required to execute K-Multiple-Means.

[0308] <Seventh Technique for Further Accelerating Acquisition of Eigenvalues ​​and Eigenvectors of Matrix M> <<Algorithm 3>> Algorithm 3 will now be described as one of algorithms for calculating the tilde S matrix at higher speed.

[0309] 6 shows Algorithm 3, which is one of the algorithms for calculating the tilde S matrix in the modified example. The symbols in the following equation (109) are used for the data points x ｉ represents the set of l+1 nearest prototypes to

[0310]

[0311] If β is not zero, then Algorithm 3 calculates the distance D Ｆ [i, j] is calculated (lines 1 to 7 of Algorithm 3). Specifically, if b<c, the distance D Ｆ Calculate [i, j] (lines 4-5 of Algorithm 3), otherwise calculate the distance D Ｆ[i, j] is calculated (lines 6-7 of Algorithm 3).

[0312] Next, Algorithm 3 calculates the nearest prototype for each data point (lines 8 to 17 of Algorithm 3). ｉ The set represented by the symbol in equation (109) is initialized by adding l+1 dummy prototypes whose distance to is infinite (lines 9-10 of Algorithm 3). As is well known, computers have an upper limit to the values ​​they can handle, so they cannot handle infinite distances. An infinite distance means a distance that is defined in advance as a distance that represents infinity.

[0313] Next, Algorithm 3 calculates the Fujiwara distance to the prototype (lines 11-12 of Algorithm 3). If a prototype can be a neighboring prototype, Algorithm 3 calculates the exact distance to the prototype (lines 13-14 of Algorithm 3) and updates the neighboring prototypes if necessary (lines 15-17 of Algorithm 3).

[0314] Then, in Algorithm 3, each element of matrix S is calculated from the obtained prototype (lines 18-19 of Algorithm 3). Finally, the tilde S matrix is ​​calculated according to the relationship in equation (12) (line 20 of Algorithm 3).

[0315] The computational cost of Algorithm 3 is the cost shown by Fujiwara's 9th Theorem below.

[0316] <<Fujiwara's 9th Theorem>> Algorithm 3 obtains the tilde S matrix with a computational cost on the order of n(md'+ld)).

[0317] Prove Fujiwara's 9th Theorem.

[0318] <<Proof of Fujiwara's 9th Theorem>> If b<c, then x ｉ and a ｊ If and are in the same connected component, then according to equation (86), the distance D ＦThe computational cost of computing [i, j] is on the order of c-b. ｉ and a ｊ If the distances D Ｆ The computational cost of computing [i, j] is of the order of 1. Since there are m / c prototypes in the same connected component as xi, the distance D Ｆ The computational cost of computing [i, j] is on the order of nm.

[0319] On the other hand, if b is equal to or greater than c, the distance D Ｆ The computational cost of computing [i, j] is of the order of 1. Therefore, the distance D Ｆ The computational cost of calculating [i, j] is on the order of nm. Furthermore, the computational cost of calculating the Fujiwara distance according to equation (101) is nmd', and the computational cost of calculating the accurate distance from equation (61) is on the order of nld. Note that accurate here means that the Fujiwara distance and the distance of equation (61) calculated by K-Multiple-Means, which executes the tilde S eigenvector acquisition process, are the same value.

[0320] Since the elements of matrix S are obtained from the distances of adjacent prototypes, matrix S can be obtained with a computational cost on the order of nl according to equation (18). Since the number of non-zero elements in matrix S is nl, obtaining the tilde S matrix from matrix S using equation (12) requires a computational cost on the order of nl. Therefore, the elements of the tilde S matrix can be obtained with a computational cost on the order of n(md' + ld).

[0321] Q.E.D. Fujiwara's 9th Theorem is now proven.

[0322] 7 shows Algorithm 4, which is an example of an algorithm executed by the control unit 11 in the modified example. In Algorithm 4, the singular value decomposition of the sampled data points is calculated (lines 2 and 3 of Algorithm 4). Then, in Algorithm 4, an approximation of the singular value decomposition is calculated (lines 4 and 5 of Algorithm 4).

[0323] In Algorithm 4, the sampled data points are set as prototypes in the same way as in K-Multiple-Means, which performs the process of obtaining the tilde S eigenvectors (lines 7 to 9 of Algorithm 4). Then, in Algorithm 4, the tilde S matrix is ​​calculated and the hyperparameter β = α is set (lines 11 to 12 of Algorithm 4).

[0324] Then, in Algorithm 4, iterative calculations are performed (lines 13 to 35 of Algorithm 4). Specifically, the tilde S matrix is ​​obtained using Algorithm 3 (line 15 of Algorithm 4), and the block matrix is ​​calculated using Algorithm 1 (line 18 of Algorithm 4).

[0325] If b<c, β is doubled, and after executing Algorithm 2 to obtain the eigenvalues ​​and their eigenvectors, F is obtained (lines 21-25 of Algorithm 4). If b>c, β is halved. Note that F is not calculated here.

[0326] Then, by executing Algorithm 3, the tilde S matrix is ​​obtained (Algorithm 4, line 29). If b = c, the iterative calculation stops, and the prototypes and their approximations are updated (Algorithm 4, lines 32-34). In Algorithm 4, these processes are repeated until the prototypes converge (Algorithm 4, line 35). In Algorithm 4, after the iterative calculation, c clusters are obtained from the connected components of the bipartite graph (Algorithm 4, line 37).

[0327] Algorithm 4 satisfies Fujiwara's 10th Theorem below.

[0328] <<Fujiwara's 10th Theorem>> t ｆ If F is the number of iterations in updating F, the computational cost required to execute Algorithm 4 is ndd' + md ２ + ((m / b) ２ t ｐ +cn)t ｆ +n(md'+ld+ml)tｃ +(nd+mdd')t ａ is.

[0329] <<Proof of Fujiwara's 10th Theorem>> The singular value decomposition of m sampled data points is md ２ The computational cost for approximating the data points and the prototype is (n+m)dd'. According to Fujiwara's fifth theorem, the computational cost for calculating the eigenvalues ​​and their eigenvectors is ((m / b) ２ t ｐ Since the computational cost of obtaining F is on the order of cn, when b<c, the update cost of F is ((m / b) ２ t ｐ +cn)t ｆ is.

[0330] When b>c, the computational cost required to update β is t ｃ -t ｆ According to Fujiwara's 9th theorem, the computational cost required to obtain the tilde S matrix is ​​on the order of n(md'+ld). And according to Fujiwara's 2nd theorem, the computational cost required to obtain the M block matrix is ​​on the order of nml. Therefore, in the repeated calculations, the tilde S matrix and the M block matrix are n(md'+ld+ml)t ｃ This can be obtained with a computational cost of the order of .

[0331] The prototype and its approximation are (nd+mdd')t ａ Therefore, the computational cost required to execute Algorithm 4 is ndd' + md ２ + ((m / b) ２ t ｐ +cn)t ｆ +n(md'+ld+ml)t ｃ +(nd+mdd')t ａ is.

[0332] Q.E.D. Fujiwara's 10th Theorem is now proven.

[0333] Algorithm 4 satisfies Fujiwara's 11th Theorem below.

[0334] <<Fujiwara's 11th Theorem>> The memory cost of Algorithm 4 is on the order of n(n+d)+dd'.

[0335] Prove Fujiwara's 11th Theorem.

[0336] <<Proof of Fujiwara's 11th Theorem>> Compared to K-Multiple-Means, which executes the S-eigenvector acquisition process, Algorithm 4 requires matrices M, Q, P, Σ, and V. ｄ′ Furthermore, in Algorithm 4, the upper bounds of the eigenvalues, the approximate data points, and the approximate prototypes must be retained. The matrices M, Q, and P have sizes m×m.

[0337] Also, Σ and V ｄ′ To store the eigenvalues, memory amounts on the order of m and dd' are required, respectively. Furthermore, to store the upper bounds of the eigenvalues, the approximate data points, and the approximate prototypes, memory amounts on the order of m, nd', and md' are required, respectively. Therefore, the additional memory amount required in Algorithm 4 is nm + (n + d)d'.

[0338] On the other hand, the memory size required to execute K-Multiple-Means, which performs the process of acquiring the tilde S eigenvectors, is n(m+d). Therefore, the memory size required to execute Algorithm 4 is n(m+d)+dd'.

[0339] Q.E.D. Fujiwara's 11th Theorem is now proven.

[0340] Algorithm 4 satisfies Fujiwara's 12th Theorem below.

[0341] <<Fujiwara's 12th Theorem>> Algorithm 4 returns the same clustering result as K-Multiple-Means, which performs the tilde S eigenvector acquisition process.

[0342] Prove Fujiwara's 12th Theorem.

[0343] <<Proof of Fujiwara's 12th Theorem>> Algorithm 4 uses Algorithm 2 to calculate c eigenvalues ​​and their eigenvectors of an M block matrix. ｉ According to Fujiwara's third theorem, λ ｉ，ｊ = 1, and q ｉ，ｊ = b ｉ Therefore, the maximum eigenvalue and its eigenvector are calculated exactly in Algorithm 4. The meaning of "exactly" is that the maximum eigenvalue and its eigenvector calculated by Fujiwara's third theorem are the same as the maximum eigenvalue and its eigenvector calculated using the power method.

[0344] Furthermore, if the c largest eigenvalues ​​of the M block matrix cannot be obtained in Algorithm 2, the following equation (110) holds due to the properties of the upper limit of the eigenvalue in Fujiwara's Fourth Theorem.

[0345]

[0346] Therefore, the calculations for the c largest eigenvalues ​​and their eigenvectors are not pruned by executing Algorithm 2. As a result, Algorithm 2 accurately obtains the c largest eigenvalues ​​and their eigenvectors using Equation (33). Therefore, Algorithm 4 can accurately calculate F. Note that the term "accurate" here means that the eigenvalues ​​and their eigenvectors calculated by Algorithm 2 are the same as those calculated by the power method.

[0347] Furthermore, in Algorithm 4, matrix S is obtained by calculating the nearest prototype for each data point from Algorithm 3. According to Fujiwara's eighth theorem, the following equation (111) holds. Therefore, the following equation (112) holds for the nearest prototype in Algorithm 3.

[0348]

[0349]

[0350] As a result, when calculating matrix S, close prototypes are not pruned by executing algorithm 3. Therefore, matrix S is obtained accurately. Note that "exactly" here means that the close prototypes obtained by pruning with algorithm 3 are the same as the close prototypes obtained without pruning. As a result, F and matrix S are obtained accurately by algorithm 4, and therefore the clustering results obtained by executing algorithm 4 are the same as those of K-Multiple-Means, which executes the process of acquiring eigenvectors of tilde S.

[0351] Q.E.D. Fujiwara's 12th Theorem is now proven.

[0352] Fujiwara's 10th and 11th theorems indicate that executing Algorithm 4 on large-scale data requires a computational cost on the order of the following equation (113) and a memory cost on the order of the following equation (114).

[0353]

[0354]

[0355] This means that for large data sets, m<<n, d'<d<<n, and t ｆ <t ｃ In Algorithm 4, d' is set to log d. On the other hand, for large-scale data, d<<m and c<<m. Therefore, in order to execute K-Multiple-Means, which executes the S eigenvector acquisition process, nm ２ t ｃ +nmdt ａ This requires a computational cost on the order of n(m+d) and a memory amount on the order of n(m+d).

[0356] Therefore, when Algorithm 4 is executed in clustering of large-scale data, clustering is performed with a smaller computational cost than when K-Multiple-Means, which executes the S-eigenvector acquisition process, is executed. Furthermore, when Algorithm 4 is executed in clustering of large-scale data, the memory overhead is negligibly small.

[0357] <Technique for More Efficient Initialization of Prototype> <<First Example of Algorithm for More Efficient Initialization of Prototype>> As mentioned above, the clustering of Algorithm 4 obtains the same clustering results as K-Multiple-Means, which performs the process of obtaining tilde S eigenvectors. Here, an example of a technique for improving the accuracy of clustering will be described.

[0358] In Fujiwara-type K-Multiple-Means such as Algorithm 4, prototypes are randomly initialized, similar to K-Multiple-Means that executes the S-eigenvector acquisition process. To improve the accuracy of clustering, prototypes may be initialized using k-means++ (see Reference 5).

[0359] Reference 5: David Arthur et al., k-means++: the advantages of careful seeding, SODA, 2007.

[0360] k-means++ is a technique that repeatedly samples data points as prototypes based on the distance to the nearest prototype. Therefore, k-means++ can effectively set prototypes based on the distribution of data, thereby improving the accuracy of clustering.

[0361] However, k-means++ calculates the closest prototype for each data point, which results in a computational cost on the order of nmd.

[0362] Therefore, we will explain Algorithm 5 as a prototype initialization algorithm that solves this problem. Note that Algorithm 5 is an algorithm that can be executed when the number of prototypes m is fixed.

[0363] FIG. 8 shows Algorithm 5, which is one of the prototype initialization algorithms in the modified example. ｃ [i] is the data point x ｉrepresents the distance of the nearest prototype from D. In Algorithm 5, first, one data point is sampled and used as a prototype (lines 1 and 2 of Algorithm 5). Then, in Algorithm 5, for each data point, D ｃ [i] is initialized (lines 3-4 of Algorithm 5).

[0364] Next, in Algorithm 5, similar to k-means++, ｃ [i] is used to repeatedly set the prototype (lines 6-7 of Algorithm 5). ｃ Singular value decomposition is used to quickly update [i].

[0365] As seen in the proof of Fujiwara's 8th theorem, x ｊ and a ｉ＋１ For and , the following equation (115) holds.

[0366]

[0367] Therefore, only when the following equation (116) holds, D ｃ [i] is updated (lines 9-14 of Algorithm 5).

[0368]

[0369] The calculation cost and memory cost required to execute Algorithm 5 are the quantities shown in Fujiwara's 13th Theorem below.

[0370] <<<<Fujiwara's 13th Theorem>>>> Execution of Algorithm 5 requires a calculation cost on the order of the following equation (117) and a memory cost on the order of n.

[0371]

[0372] Prove Fujiwara's 13th Theorem.

[0373] <<<<Proof of Fujiwara's 13th Theorem>>> The computational costs of calculating an approximate prototype and an approximate distance are on the order of mdd' and nmd', respectively. Even if the prototypes are sampled randomly, a computational cost on the order of nd' log m is required to accurately calculate the distance. Note that accurate here means obtaining a distance D[i, j] that is not an approximate distance.

[0374] Also D ｃ Sampling data points using [i] requires computational cost on the order of nm. ｃ Storing [i] requires a memory cost on the order of n. Therefore, the computational cost and memory cost required to execute Algorithm 5 are on the order of the cost of the above equation (117) and the cost on the order of n, respectively.

[0375] Q.E.D. Fujiwara's 13th Theorem is now proven.

[0376] Therefore, when executing Algorithm 4, the control unit 11 may execute Algorithm 5 instead of lines 7 to 9 of Algorithm 4. In such a case, the accuracy of clustering is further improved. Furthermore, the above-mentioned problem of the calculation cost becoming high on the order of nmd is also solved because the number of calculations of the distance between the data point and the prototype is reduced by calculating the approximate distance. "Solved" means that the calculation cost becomes an order of magnitude smaller than nmd.

[0377] <<Second Example of an Algorithm for More Efficient Initialization of Prototypes>> Incidentally, the number of prototypes, m, as shown in Algorithm 4, is a hyperparameter in Fujihara-type K-Multiple-Means. In Fujihara-type K-Multiple-Means, clusters are obtained by grouping prototypes. Therefore, the number of prototypes, m, affects the accuracy of clustering. A simple technique for determining the number of prototypes is to perform cross-validation while changing the number of prototypes. However, because the number of prototypes takes values ​​from c to n, there is an issue that cross-validation requires a long processing time for large-scale data. Another issue is that cross-validation cannot be performed if the data points are not labeled. Therefore, a technique for determining the number of prototypes based on the distribution of the data is desired.

[0378] Therefore, we will explain Algorithm 6 as a prototype initialization algorithm that solves these problems. When executing Algorithm 4, the control unit 11 may execute Algorithm 6 instead of lines 7 to 9 of Algorithm 4. In this case, the above problem is solved for reasons that will be described later.

[0379] Fig. 9 shows Algorithm 6 in a modified example. Algorithm 6 is a technique for determining the number m of prototypes according to the distribution of data. Algorithm 6 will be explained in more detail. The symbols in the following equation (118) in Fig. 9 represent a ｉ represents the set of data points whose nearest prototype is

[0380]

[0381] In Algorithm 6, prototypes are added one by one until the following equation (119) holds (lines 3 to 6 of Algorithm 6).

[0382]

[0383] Equation (119) indicates that the added prototype is the closest prototype of one data point. When equation (119) holds, the entire data point is well represented by other prototypes. Therefore, when equation (119) holds, there is no need to add prototypes. As a result, the number of prototypes m is determined by the distribution of the data.

[0384] Note that in Algorithm 6, each prototype is sampled from a data point, so that each prototype is the closest prototype for at least one data point (i.e., the size of the set represented by the symbols in Equation (118) is 1 or greater).

[0385] Algorithm 6 may be used to determine the prototype instead of Algorithm 5. Since Algorithm 6 uses Algorithm 5, its computational cost and memory cost are the same as Algorithm 5.

[0386] As described above, the control unit 11 may execute Algorithm 6 instead of lines 7 to 9 of Algorithm 4 when executing Algorithm 4, which solves the above-mentioned problem. This is because, in Algorithm 6, the user does not need to specify the number m of prototypes.

[0387] <<Third Example of a More Efficient Prototype Initialization Algorithm>> As described above, Fujihara-type K-Multiple-Means obtains the same results as K-Multiple-Means, which performs the S-eigenvector acquisition process. Therefore, Fujihara-type K-Multiple-Means can effectively cluster non-spherically distributed data.

[0388] DBSCAN (see Reference 6) is a well-known technique for clustering non-spherically distributed data.

[0389] Reference 6: Martin Ester et al., A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise, KDD, 1996.

[0390] DBSCAN is a technology that can calculate not only clusters but also outliers of data. An outlier is a data point that exists alone in an area where there is not much data. Here, we will explain Algorithm 7 as an algorithm that effectively calculates not only prototypes but also outliers.

[0391] Because outliers exist in areas where there is not much data, the distance between the outlier and its neighboring prototype is large. Therefore, when calculating the distance between a data point and a prototype, the distance between the outlier and its neighboring prototype is larger than that between data points that are not outliers. Algorithm 7 is an algorithm devised to detect outliers based on this knowledge.

[0392] Fig. 10 shows a modified version of Algorithm 7. In Fig. 10, θ represents the average distance between a data point and its neighboring prototype. The symbols in the following equation (120) represent a set of outliers.

[0393]

[0394] The symbols in equation (121) below represent the set of data points used to calculate the clusters.

[0395]

[0396] In Algorithm 7, first, a prototype is obtained by executing Algorithm 6 (line 1 of Algorithm 7). Then, in Algorithm 7, a prototype close to each data point is obtained by executing Algorithm 3, and their average distance θ is obtained (lines 2 and 3 of Algorithm 7).

[0397] Next, Algorithm 7 detects outliers using the distance of the prototypes that are close to the data point. Specifically, if a data point is far from the nearby prototype, the data is added as an outlier to the set represented by the symbol in Equation (120) (lines 10-11 of Algorithm 7). On the other hand, if not, the data is added to the set represented by the symbol in Equation (121) (lines 12-13 of Algorithm 7).

[0398] Finally, the ID (Identification) of the data point for which the cluster is calculated is set using the set represented by the symbol in Equation (121) (lines 16 to 19 of Algorithm 7). The computational cost and memory cost required to execute Algorithm 7 are the quantities shown in Fujiwara's 14th Theorem below.

[0399] <<<<Fujiwara's 14th Theorem>>>> Execution of Algorithm 7 requires a calculation cost on the order of the following equation (122) and a memory cost on the order of nl.

[0400]

[0401] Prove Fujiwara's 14th Theorem.

[0402] <<<<Proof of Fujiwara's 14th Theorem>>>> Calculating a prototype by executing Algorithm 5 requires a computational cost on the order of the following equation (123).

[0403]

[0404] Execution of Algorithm 3 to find the nearest prototype for each data point requires a computational cost of the order of n(md'+ld). Computing the set represented by the symbols in Equation (120) and the set represented by the symbols in Equation (121) requires a computational cost of the order of nl.

[0405] Furthermore, storing the set represented by the symbols in equation (120) and the set represented by the symbols in equation (121) requires a memory cost on the order of n. Storing the set represented by the symbols in equation (19) for each data point requires a memory cost on the order of n. Therefore, executing Algorithm 7 requires a computational cost on the order of equation (123) and a memory cost on the order of n.

[0406] Q.E.D. Fujiwara's 14th Theorem is now proven.

[0407] The reason why Algorithm 7 can effectively calculate not only prototypes but also outliers is that it can effectively obtain outliers that exist alone in areas where there is not much data, from the distance between the data point and the prototype. The definition of "effective" means that the execution result of Algorithm 7 does not result in an execution result where no outliers are found.

[0408] <Others> Fujihara-type K-Multiple-Means may be used in any technology that requires clustering. Therefore, Fujihara-type K-Multiple-Means may be used, for example, for segmentation, for determining the startup order of a group of servers, or for sound source separation. Therefore, clustering device 1 has industrial applicability.

[0409] For example, when Fujiwara-type K-Multiple-Means is used for segmentation, the clustering targets are specifically each pixel in the image that is the target of segmentation. For example, when Fujiwara-type K-Multiple-Means is used to determine the startup order of a group of servers, the clustering targets are specifically nodes connected to the servers whose startup order is to be determined. For example, when Fujiwara-type K-Multiple-Means is used for sound source separation, the clustering targets are specifically sensors (i.e., microphones) that pick up sound emitted from the sound source that is the target of separation.

[0410] The clustering device 1 may be implemented using a plurality of information processing devices communicably connected via a network, in which case the respective functional units of the clustering device 1 may be distributed and implemented among the plurality of information processing devices.

[0411] Note that all or part of the functions of the clustering device 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0412] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0413] REFERENCE SIGNS LIST 1... clustering device, 11... control unit, 12... interface unit, 13... storage unit, 91... processor, 92... memory

Claims

1. A control unit that clusters n data points in a data space, which is a phase space including one or more data points, by K-Multiple-Means, 2. The clustering device according to claim 1, wherein the matrix M is a product of the tilde S matrix described on the left side of the following formula (C1) defined by the following formula (C1) and the transpose of the tilde S matrix. wherein the control unit obtains, as the solution of an optimization problem obtained by fixing a matrix S representing the similarity between each data point in the data space and each of m prototypes, an eigenvector of a matrix M representing the similarity between the prototypes, in a process executed in the alternating optimization in K-Multiple-Means. A clustering device.

3. The control unit executes a first sub-process of Fujiwara, which is a process of obtaining an eigenvector corresponding to an eigenvalue with a value of 1 as an eigenvector corresponding to the maximum eigenvalue of a block matrix in the matrix M. The clustering device according to claim 1.

4. A clustering method executed by a clustering device, the clustering device including a control unit that clusters n data points in a data space, which is a phase space including one or more data points, by K-Multiple-Means, wherein the control unit obtains, as the solution of an optimization problem obtained by fixing a matrix S representing the similarity between each data point in the data space and each of m prototypes, an eigenvector of a matrix M representing the similarity between the prototypes, in a process executed in the alternating optimization in K-Multiple-Means, the clustering method including a clustering step in which the control unit executes the K-Multiple-Means.

Images