Information processing system, information processing method and program

Abstract

To enable mitigation of calculation load.SOLUTION: An information processing system comprises: data acquisition means that acquires vector data of a plurality of dimensions; clustering means that groups pieces of data having a small relative distance on the basis of the relative distance of vector data acquired by the data acquisition means, and performs hierarchical clustering in which grouping is repeated; and density calculation means that calculates a density of each of clusters of the vector data. When performing hierarchical clustering, the clustering means divides or integrates each cluster on the basis of the density of the cluster calculated by the density calculation means.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] There is known an information processing system that includes a clustering means that converts input data into multi-dimensional vector data and groups data that are close in relative distance based on the relative distance of the converted vector data (see, for example, patent documents). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6562984 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to avoid a huge combinatorial explosion when calculating the relative distance between vector data, the above system uses the K-nearest neighbor method to group vectors that are close to each other. However, when grouping is performed by specifying the number of clusters in advance, as in the K-nearest neighbor method, repeated iterative calculations are required, which can result in an excessively large computational load.

[0005] The present disclosure has been made to solve such problems, and a main object of the present disclosure is to provide an information processing system, an information processing method, and a program that can reduce the calculation load. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present disclosure is to a data acquisition means for acquiring multi-dimensional vector data; a clustering unit that performs hierarchical clustering by grouping data that are close in relative distance based on the relative distance of the vector data acquired by the data acquisition unit and repeating the grouping, a density calculation means for calculating the density of each cluster of the vector data; When performing the hierarchical clustering, the clustering means divides or merges each cluster based on the density of each cluster calculated by the density calculation means. Information Processing Systems is. In this aspect, The clustering means may extract at least two vectors by comparing the density of each vector component of the multidimensional vector data, extract concentrated points in each of the extracted vectors where the vector values are concentrated, extract a region of interest in which to concentrate clustering based on the extracted concentrated points, and perform the hierarchical clustering on the extracted region of interest and a surrounding region surrounding the region of interest. In this aspect, the clustering means divides the cluster when it is determined that the density of the cluster calculated by the density calculation means is equal to or greater than a threshold; If it is determined that the density of each cluster is not equal to or greater than a threshold, it may be determined whether the current number of clusters is equal to or greater than a target value, and if it is determined that the current number of clusters is equal to or greater than the target value, the hierarchical clustering may be stopped. In this aspect, When the clustering means determines that the density of each cluster has reached a predetermined value during the hierarchical clustering, the clustering means may temporarily stop the hierarchical clustering and output the current number of clusters and the vectors included in each cluster. In this aspect, The density may be a Gini coefficient. In order to achieve the above object, one aspect of the present disclosure is to obtaining multi-dimensional vector data; a step of performing hierarchical clustering by grouping data having close relative distances based on the relative distances of the acquired vector data and repeating the grouping; calculating the density of each cluster of the vector data; When performing the hierarchical clustering, dividing or merging each cluster based on the calculated density of each cluster; information processing method, is. In order to achieve the above object, one aspect of the present disclosure is to A process for acquiring multi-dimensional vector data; a process of performing hierarchical clustering by grouping data having close relative distances based on the relative distances of the acquired vector data and repeating the grouping; A process of calculating the density of each cluster of the vector data; a process of dividing or merging each cluster based on the calculated density of each cluster when performing the hierarchical clustering; A program that causes a computer to execute is. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide an information processing system, an information processing method, and a program that can reduce the calculation load. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic system configuration of a processing system according to an embodiment of the present invention. [Figure 2] 10 is a flowchart illustrating an example of a processing flow by the processing system according to the present embodiment. [Figure 3] FIG. 10 is a diagram comparing the advantages and disadvantages of the top-down method, the bottom-up method, and the method according to the present embodiment. [Figure 4]1 is a block diagram illustrating an example of a schematic system configuration of an information processing system according to an embodiment of the present invention. [Figure 5] 10 is a flowchart showing an example of the flow of an information processing method by the information processing system according to the present embodiment. [Figure 6] FIG. 10 is a diagram showing an example of the Gini coefficient of each component of a high-dimensional vector after each morpheme is projected onto the high-dimensional vector. [Figure 7] 1 is an example of a graph showing a joint distribution consisting of vectors α and β. [Figure 8] FIG. 10 is a diagram illustrating an example of a location where vector values are concentrated. [Figure 9] FIG. 2 is a diagram illustrating an example of a region of interest and a surrounding region. [Figure 10] FIG. 10 is a diagram showing cluster dividing lines. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. For example, when creating a trained model for a document search system or a large-scale language model used to generate documents, the input data is converted into multi-dimensional vector data, and data with close relative distances is grouped based on the relative distances of the converted vector data, a process known as clustering.

[0010] In order to avoid an explosive increase in the number of combinations when calculating the relative distance between vector data, it is envisaged to group vectors that are close to each other using the K-nearest neighbor method.

[0011] However, when using unsupervised clustering, such as the K-nearest neighbor method, where the number of clusters (number of groups) is specified in advance, repeated iterative calculations are required, which can result in a huge amount of calculations and an excessively large computational load.

[0012] The information processing system according to this embodiment executes a processing method that solves the problems that may arise in the K-means method described above.

[0013] For example, in unsupervised clustering, which involves clustering large-scale, high-dimensional data without determining the center of the cluster in advance, the number of dimensions that must be considered becomes enormous. This can lead to a situation known as the curse of dimensionality, making it impossible to perform clustering. There are two ways to solve the curse of dimensionality in this type of clustering:

[0014] (1) Stop the calculation when the required number of clusters is reached. (2) Adopt a method that does not depend on initial values and avoid iterative calculations.

[0015] The information processing system according to this embodiment considers the above methods (1) and (2) and performs the following improved hierarchical clustering as a method for avoiding the curse of dimensionality. In this embodiment, when stopping the clustering when the required number of clusters is reached, attention is paid to the density of the data in order to consider "which clusters should be divided with priority and which clusters should be agglomerated as desired."

[0016] The information processing system according to this embodiment calculates the Gini coefficient, which indicates the density of a vector, for each high-dimensional vector (for example, 200 dimensions for word2vec and 768 dimensions for BERT).

[0017] If the function of vector values is L(x), the Gini coefficient is 1- 2∫ 1 It is expressed as 0L(x)dx. The more uniform the data, the closer to 0 the Gini coefficient will be, and conversely, the more densely the data is, the closer to 1 the coefficient will be.

[0018] For example, the Gini coefficient can be calculated by counting the number of words in group 1, the number of words in group 2, the number of words in group 3, ..., the number of words in group N within a certain range, calculating the ratio of each to the total number of words, multiplying these ratios (= sum of products), and subtracting them from 1.

[0019] While one possible method for calculating data density is to use the center of gravity, the method using the Gini coefficient as in this embodiment has the advantage of being easier to calculate. This method using the Gini coefficient only counts the number of data items included in a group or the number of types of groups, and requires only simple arithmetic operations (counting the number, calculating the ratio, multiplying and accumulating the ratio, and multiplying and accumulating the ratio by 1), making the calculations easier.

[0020] The information processing system according to this embodiment selects multiple vectors with high Gini coefficients and performs hierarchical clustering based on the values of those vectors. Hierarchical clustering is a method of grouping data with high similarity between vectors that are close in distance, and then repeatedly grouping the data into larger groups. This method makes it possible to visually understand the structure of the data and hidden relationships.

[0021] The information processing system according to this embodiment can suppress an increase in the computational load, and can significantly reduce the time required to create the same trained model. Such a time-saving effect contributes to solving the problem of reducing the time required to create a trained model for a large-scale language model.

[0022] The essential issue with this embodiment is how to obtain desired features from data with hundreds of millions of dimensions, known as big data, without exploding the amount of calculation required. At the same time, it is necessary to appropriately compress high-dimensional data and retain its features.

[0023] The information processing system according to this embodiment uses the K-means method and centroids to perform "clustering" to find commonalities in data, and reduces the computational complexity exponentially (o(N 2)) and stopping the calculation midway reduces it linearly (=o(1 / N)). This allows you to obtain the necessary features from big data.

[0024] In particular, with regard to language data, the minimum corpus can contain 100,000 articles, 71,000 words, and the vector dimensions of each word can be as large as several hundred. For this reason, obtaining features while minimizing the amount of calculations can be an essential technology for document retrieval and summarization.

[0025] On the other hand, if big data exists in a dense manner throughout the entire data set, it becomes difficult to reduce the amount of calculation required to observe that trend. However, if there is a bias in the distribution of data within the entire data set, such as when it becomes sparse or dense, it can be said to be efficient to focus more on calculations in the dense areas and complete calculations more easily in the sparse areas.

[0026] Therefore, as described above, the information processing system according to this embodiment focuses on data density, calculates the Gini coefficient, which indicates the density of vectors, selects multiple vectors with high Gini coefficients, and performs hierarchical clustering based on the values of those vectors.

[0027] Next, an example of the hardware configuration of a processing system according to this embodiment will be described. Fig. 1 is a block diagram showing a schematic system configuration of a processing system according to this embodiment. The processing system 1 according to this embodiment includes a morphological analysis unit 2, a relative distance calculation unit 3, and the information processing system 4 described above.

[0028] The information processing system 4 has a hardware configuration of a typical computer, including, for example, a processor 4a such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), an internal memory 4b such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a storage device 4c such as an HDD (Hard Disk Drive) or an SDD (Solid State Drive), an input / output I / F 4d for connecting peripheral devices such as a display, and a communication I / F 4e for communicating with devices outside the device.

[0029] 2 is a flowchart showing an example of a processing flow by the processing system according to this embodiment. The processing system 1 according to this embodiment finds, for example, common factors that substitute for the meaning of text from text, and is applicable to other natural language processing services (search, document summarization, translation, etc.).

[0030] To find common factors in texts, it is necessary to calculate the similarity of the texts, and for this purpose, the words contained in the texts are quantified. To quantify the similarity, the processing system 1 according to this embodiment executes the following steps S101 to S103.

[0031] First, for example, text data is input to the morphological analysis unit 2 (step S101). The morphological analysis unit 2 performs morphological analysis to output units of words or morphemes from the input text data (step S102).

[0032] Based on the text decomposed into morpheme units by the morphological analysis unit 2, the relative distance calculation unit 3 maps and quantifies words and morphemes to relative positions based on the usage of each language. That is, the relative distance calculation unit 3 generates high-dimensional vector data by projecting the words and morphemes onto high-dimensional (e.g., several hundred dimensions) vectors (step S103). The relative distance calculation unit 3 outputs this high-dimensional vector data to the information processing system 4.

[0033] The information processing system 4 performs improved hierarchical clustering, which will be described later, based on the high-dimensional vector output from the relative distance calculation unit 3 (step S104).

[0034] Here, the method of searching for common factors by clustering is well known, so a detailed explanation will be omitted. For example, clustering methods include a top-down method (K-center method) that divides the entire data, and a bottom-up method (hierarchical clustering) that groups the two data that are closest to each other from the data and repeats this grouping.

[0035] Figure 3 compares the advantages and disadvantages of the top-down method, the bottom-up method, and the improved hierarchical clustering method.

[0036] The information processing system 4 according to this embodiment takes into consideration the above advantages and disadvantages and performs improved hierarchical clustering that eliminates the drawbacks of the top-down method.

[0037] 4 is a block diagram showing an example of a schematic system configuration of an information processing system according to this embodiment. The information processing system 4 according to this embodiment includes a data acquisition unit 41 that acquires high-dimensional vector data, a density calculation unit 42 that calculates the density of each cluster of the high-dimensional vector acquired by the data acquisition unit 41, and a clustering unit 43 that performs hierarchical clustering based on the density of each cluster calculated by the density calculation unit 42.

[0038] FIG. 5 is a flowchart showing an example of the flow of an information processing method by the information processing system according to this embodiment.

[0039] The data acquisition unit 41 acquires high-dimensional vector data from, for example, the relative distance calculation unit 3 (step S201).

[0040] The density calculation unit 42 calculates, for example, the Gini coefficient of each cluster as the density of each cluster of the high-dimensional vector acquired by the data acquisition unit 41 (step S202).

[0041] The clustering unit 43 determines whether the Gini coefficient of each cluster calculated by the density calculation unit 42 is equal to or greater than a threshold (step S203). The threshold for the Gini coefficient may be set in the clustering unit 43 in advance.

[0042] If the clustering unit 43 determines that the Gini coefficient of the cluster calculated by the density calculation unit 42 is equal to or greater than the threshold value (YES in step S203), it divides the cluster (step S204) and returns to the above (step S202).

[0043] On the other hand, if the clustering unit 43 determines that the Gini coefficient of each cluster calculated by the density calculation unit 42 is not equal to or greater than the threshold value (NO in step S203), the clustering unit 43 determines whether the current number of clusters is equal to or greater than a target value (step S205). Note that the target value for the number of clusters (number of common factors) may be set in the clustering unit 43 in advance.

[0044] If the clustering unit 43 determines that the current number of clusters is equal to or greater than the target value (YES in step S205), it stops the hierarchical clustering (step S206). On the other hand, if the clustering unit 43 determines that the current number of clusters is not equal to or greater than the target value (NO in step S205), it returns to the above (step S204).

[0045] As described above, the information processing system 4 according to this embodiment calculates the density of high-dimensional vectors for each cluster, and repeats dividing clusters with high density and dense data until the target number of clusters is reached.

[0046] The improved hierarchical clustering according to this embodiment makes it possible to set a target number of clusters in advance and perform calculations while avoiding the drawbacks of dependence on initial values and increased computational complexity. This improved hierarchical clustering allows, for example, clustering words or morphemes until a desired number of clusters is reached, and then summarizing them as common factors.

[0047] Next, we will explain in more detail the improved hierarchical clustering performed by the clustering unit 43. The clustering unit 43 according to this embodiment is characterized in that it can significantly reduce the calculation load for large-scale data without reducing accuracy.

[0048] As described above, if the distance between high-dimensional vectors is calculated without any computational ingenuity, it is necessary to compare the dimension of the vector times the number of words. However, it is not necessarily the case that the "distance between words is close" to create a cluster in all dimensions, i.e., data is densely packed. Therefore, the clustering unit 43 according to this embodiment calculates the Gini coefficient, which indicates the degree of data density, for each dimension (each vector component) of the high-dimensional vector.

[0049] Vector components with a high Gini coefficient have densely packed data. This means that further clustering is required to separate common factors. On the other hand, vector components with a low Gini coefficient have uniformly spread data. This means that common factors can be separated without further clustering.

[0050] For the above reasons, the clustering unit 43 according to this embodiment concentrates the clustering by the improved hierarchical clustering described above on areas where data is particularly concentrated. Next, a method for identifying concentrated areas of the improved hierarchical clustering described above will be described in detail.

[0051] The clustering unit 43 first calculates the Gini coefficient of each vector component based on the high-dimensional vector acquired by the data acquiring unit 41. Fig. 6 is a diagram showing an example of the Gini coefficient of each component of the high-dimensional vector after each morpheme is projected onto the high-dimensional vector.

[0052] For example, the clustering unit 43 compares the Gini coefficients of the components of the calculated high-dimensional vectors as shown in Fig. 6. Then, the clustering unit 43 extracts two vectors, for example, a vector α consisting of vector components with the highest Gini coefficient and a vector β consisting of vector components with the second highest Gini coefficient.

[0053] Figure 7 is an example of a graph showing the joint distribution of vectors α and β. In the joint distribution shown in Figure 7, the darker parts of the graph are areas where the vector values are densely concentrated, and the lighter parts are areas where the vector values are sparsely distributed.

[0054] As shown in FIG. 7, the clustering unit 43 determines the areas where vector values are dense and the areas where vector values are sparse, and then determines the concentrated areas for improved hierarchical clustering based on the results.

[0055] Fig. 8 is a diagram showing an example of a location where vector values are densely packed. As shown in Fig. 8, the location where vector values are particularly densely packed is i for the component of vector α and ii for the component of vector β.

[0056] Here, the clustering unit 43 extracts i and ii, where the vector values are particularly dense, so that the common portion between these two is a predetermined value. Specifically, the common portion between i and ii is set as a region of interest, and the region surrounding the region of interest is set as a peripheral region. The clustering unit 43 extracts i and ii so that the peripheral region accounts for a predetermined ratio (e.g., 45%) of the entire region. The predetermined ratio may be set experimentally, taking into account, for example, the amount of calculation. Fig. 9 is a diagram showing an example of the region of interest and the peripheral region calculated as described above.

[0057] In the above description, the clustering unit 43 extracts two vectors, but it may also extract three or more vectors and extract the region of interest and the surrounding region based on the extracted vectors.

[0058] The clustering unit 43 performs the improved hierarchical clustering shown in Fig. 5 on the region of interest and the surrounding regions extracted as described above. The reason for performing hierarchical clustering on not only the region of interest but also the surrounding regions is that it is difficult to perform appropriate clustering on the boundary portion of the region of interest. For this reason, the clustering unit 43 according to this embodiment performs the improved hierarchical clustering on not only the region of interest but also the surrounding regions of the region of interest.

[0059] The clustering unit 43 may temporarily stop the improved hierarchical clustering if it determines that the Gini coefficient of each cluster has reached a predetermined value (for example, if it determines that the number of data items has decreased by about 20%). For example, when the clustering unit 43 temporarily stops the hierarchical clustering, it randomly selects one representative vector from among the vectors included in each cluster.

[0060] The clustering unit 43 assigns a name indicating the cluster to each vector included in the cluster. This allows the clustering unit 43 to output the number of clusters and the vectors included in the cluster (for example, their names, morpheme A, etc.). The clustering unit 43 performs the output using, for example, a display device or a printer. The user can adjust the threshold and target value of the Gini coefficient (described later) according to the output result from the clustering unit 43.

[0061] This allows for the cluster division lines to be adjusted according to the data density, as shown in FIG. 10, minimizing the effects of density fluctuations and enabling division of densely packed areas with greater emphasis.

[0062] After the temporary suspension, the clustering unit 43 resumes the improved hierarchical clustering, and when it determines that the number of clusters has reached the target value, it finally stops the improved hierarchical clustering.

[0063] The target value for the number of clusters may be determined, for example, based on the number of different meanings of words. This value represents the number of different meanings of words in a pseudo-like manner. For this reason, the target number of clusters may be increased when finely classifying meanings, and may be decreased when roughly classifying meanings and broadening the range of synonyms. In this embodiment, the target number of clusters is set to, for example, 100,000 different meanings, and the target value is set to 100,000.

[0064] As described above, the information processing system 4 according to this embodiment includes a data acquisition unit 41 that acquires multidimensional vector data, a clustering unit 43 that performs hierarchical clustering by grouping data that are close in relative distance based on the relative distance of the vector data acquired by the data acquisition unit 41 and repeating the grouping, and a density calculation unit 42 that calculates the density of each cluster of vector data. When performing hierarchical clustering, the clustering unit 43 divides clusters whose density is equal to or greater than a threshold, based on the density of each cluster calculated by the density calculation unit 42.

[0065] According to the information processing system 4 of this embodiment, the density of each cluster of a high-dimensional vector is calculated, and clusters with high density and dense data are divided, and this process is repeated until the target number of clusters is reached. This makes it possible to set the target number of clusters in advance and perform calculations while avoiding the disadvantages of dependence on initial values and increased calculation volume, and reducing the calculation load.

[0066] Although several embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0067] In the above embodiment, when performing hierarchical clustering, the clustering unit 43 divides clusters whose density is high, equal to or higher than a threshold, and whose vector data is dense, based on the density of each cluster calculated by the density calculation unit 42. However, this is not limited to this. For example, when performing hierarchical clustering, the clustering unit 43 may combine clusters whose density is low, below a threshold, and whose vector data is sparse, based on the density of each cluster calculated by the density calculation unit 42.

[0068] As described above, the density of each cluster of a high-dimensional vector is calculated, and clusters with low density and sparse data are merged, repeating this process until the target number of clusters is reached. As with splitting clusters, this avoids the disadvantages of dependence on initial values and increased calculation volume, reducing the calculation load, while enabling calculations to be performed by setting the target number of clusters in advance.

[0069] The present disclosure can also be implemented by causing a processor to execute a computer program to perform the processing shown in FIG. 5, for example.

[0070] The program can be stored and supplied to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)).

[0071] The program may be provided to the computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to the computer via a wired communication path such as an electrical wire or optical fiber, or via a wireless communication path.

[0072] Each component constituting the information processing system 4 according to the above-described embodiment can be realized not only by a program, but also in part or in whole by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). [Explanation of symbols]

[0073] 1 Processing system, 2 Morphological analysis unit, 3 Relative distance calculation unit, 4 Information processing system, 41 Data acquisition unit, 42 Density calculation unit, 43 Clustering unit

Claims

1. a data acquisition means for acquiring multi-dimensional vector data; a clustering unit that performs hierarchical clustering by grouping data that are close in relative distance based on the relative distance of the vector data acquired by the data acquisition unit and repeating the grouping, a density calculation means for calculating the density of each cluster of the vector data; When performing the hierarchical clustering, the clustering means divides or merges each cluster based on the density of each cluster calculated by the density calculation means. Information processing system.

2. 2. The information processing system according to claim 1, the clustering means extracts at least two vectors by comparing the densities of each vector component of the multidimensional vector data, extracts densely-spaced points where vector values are densely-spaced in each of the extracted vectors, extracts a region of interest in which clustering is to be concentrated based on the extracted densely-spaced points, and performs the hierarchical clustering on the extracted region of interest and a peripheral region surrounding the region of interest. Information processing system.

3. 2. The information processing system according to claim 1, the clustering means divides the cluster when it is determined that the density of the cluster calculated by the density calculation means is equal to or greater than a threshold; If it is determined that the density of each cluster is not equal to or greater than the threshold, it is determined whether the current number of clusters is equal to or greater than a target value, and if it is determined that the current number of clusters is equal to or greater than the target value, the hierarchical clustering is stopped. Information processing system.

4. 2. The information processing system according to claim 1, When the clustering means determines that the density of each cluster has reached a predetermined value during the hierarchical clustering, the clustering means temporarily stops the hierarchical clustering and outputs the current number of clusters and the vectors included in each cluster. Information processing system.

5. 2. The information processing system according to claim 1, The density is the Gini coefficient. Information processing system.

6. obtaining multi-dimensional vector data; a step of performing hierarchical clustering by grouping data having close relative distances based on the relative distances of the acquired vector data and repeating the grouping; calculating the density of each cluster of the vector data; When performing the hierarchical clustering, dividing or merging each cluster based on the calculated density of each cluster; An information processing method, including:

7. A process for acquiring multi-dimensional vector data; a process of performing hierarchical clustering by grouping data having close relative distances based on the relative distances of the acquired vector data and repeating the grouping; A process of calculating the density of each cluster of the vector data; a process of dividing or merging each cluster based on the calculated density of each cluster when performing the hierarchical clustering; A program that causes a computer to execute the following.

Images