Evaluation apparatus, evaluation method, and program

Abstract

To provide an evaluation apparatus which narrows down causes for unsatisfactory performance of a language processing model.SOLUTION: An evaluation apparatus has: an acquisition unit for acquiring embedding generated by using an embedding layer included in a language processing model for a natural language sentence included in each of a plurality of training data pieces; a clustering unit for clustering the plurality of embedding; a calculation unit for calculating an evaluation index obtained by evaluating the embedding result, referring to the label included in each of the plurality of training data pieces; and an evaluation unit for evaluating a quality of the embedding layer. Thus, it is possible to optimize the performance of a machine-leaned language processing model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to techniques for evaluating the performance of a language processing model. [Background technology]

[0002] In recent years, it has been known to generate a language processing model that performs a desired language processing task by fine-tuning a general-purpose natural language processing model as a pre-learning model. The performance of such a language processing model is affected by, for example, the training data used to generate the language processing model, the pre-learning model, the learning algorithm, the hyperparameters adopted in the language processing model, and the like. For the purpose of improving the performance of such a language processing model, it is known to use a technique such as grid search to adjust the hyperparameters. In addition, for example, Patent Document 1 describes a technique for improving the quality of training data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-19341 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the performance of a language processing model is poor, it is difficult to narrow down which of the above-mentioned training data, pre-learning model, learning algorithm, hyperparameters, etc. is the main cause. The technology described in Patent Document 1 is effective when it is known that the main cause is the quality of the training data, but when this is not the case, there is a possibility that the performance of the language processing model cannot be improved even if the quality of the training data is improved. Therefore, it is important to narrow down the cause when the performance of the language processing model is poor.

[0005] The present disclosure has been made in consideration of the above problems, and an exemplary objective thereof is to provide a technique for narrowing down causes of poor performance of a language processing model. [Means for solving the problem]

[0006] An evaluation device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring embeddings generated for natural language sentences included in each of a plurality of training data fragments using an embedding layer included in a language processing model, a clustering means for clustering the embeddings, a calculation means for calculating an evaluation index that evaluates the result of the clustering by referring to labels included in each of the plurality of training data fragments, and an evaluation means for evaluating the quality of the embedding layer based on the evaluation index.

[0007] An evaluation method according to an exemplary aspect of the present disclosure includes an acquisition process in which at least one processor acquires embeddings generated for natural language sentences included in each of a plurality of training data pieces using an embedding layer included in a language processing model; a clustering process in which the at least one processor performs clustering of the embeddings; a calculation process in which the at least one processor calculates an evaluation index for evaluating a result of the clustering by referring to labels included in each of the plurality of training data pieces; and an evaluation process in which the at least one processor evaluates quality of the embedding layer based on the evaluation index.

[0008] A program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an evaluation device, and causes the computer to function as an acquisition means that acquires embeddings generated for natural language sentences included in each of a plurality of training data pieces using an embedding layer included in a language processing model, a clustering means that performs clustering of the embeddings, a calculation means that calculates an evaluation index that evaluates the result of the clustering by referring to labels included in each of the plurality of training data pieces, and an evaluation means that evaluates the quality of the embedding layer based on the evaluation index. Effect of the Invention

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is provided in that a technique for narrowing down causes of poor performance of a language processing model can be provided. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of an evaluation device according to the present disclosure. [Diagram 2] FIG. 1 is a flow diagram showing the flow of an evaluation method according to the present disclosure. [Diagram 3] FIG. 1 is a block diagram showing a configuration of an evaluation system according to the present disclosure. [Figure 4] FIG. 1 is a flow diagram illustrating the flow of an evaluation method according to the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram illustrating an example of an embedding set according to the present disclosure. [Figure 6] FIG. 13 is a scatter plot illustrating an example of an embedding set according to the present disclosure. [Figure 7] FIG. 13 is a schematic diagram showing an example of a result of multiple clustering operations according to the present disclosure. [Figure 8] FIG. 11 is a flow diagram showing a detailed example of a calculation process according to the present disclosure. [Figure 9] FIG. 13 is a diagram illustrating an example of a series of evaluation indexes according to the present disclosure. [Figure 10]FIG. 2 is a diagram illustrating a first example of a judgment criterion according to the present disclosure. [Figure 11] FIG. 13 is a diagram illustrating a second example of a determination criterion according to the present disclosure. [Figure 12] FIG. 11 is a flow diagram showing a detailed example of a calculation process according to the present disclosure. [Figure 13] FIG. 11 is a flow diagram showing a detailed example of a calculation process according to the present disclosure. [Figure 14] FIG. 1 is a scatter plot illustrating embeddings according to the present disclosure. [Figure 15] FIG. 13 is a diagram illustrating an example of an evaluation result according to the present disclosure. [Figure 16] FIG. 13 is a diagram illustrating an example of an evaluation result according to the present disclosure. [Figure 17] FIG. 2 is a block diagram showing an example configuration of a computer that functions as each device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present inventors have focused on the fact that the causes of poor performance of a language processing model can be narrowed down according to the quality of the embedding layer included in the language processing model, and have invented an evaluation device for evaluating the quality of an embedding layer. If the performance of a language processing model is poor despite the quality of the embedding layer being good, the cause is likely to be in the language processing task layer (e.g., hyperparameters). On the other hand, if the quality of the embedding layer is poor, there is a high possibility that there is a problem with the training data, pre-learning model, or learning algorithm involved in the generation process of the embedding layer. In this way, by using the evaluation device according to the present invention, the causes of poor performance of a language processing model can be narrowed down according to the evaluation result of the quality of the embedding layer.

[0012] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, an embodiment obtained by appropriately combining the technical means employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, an embodiment obtained by appropriately omitting a part of the technical means employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, an embodiment that does not exhibit the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0013] First Exemplary Embodiment A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. The scope of application of each technical means adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means adopted in this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical obstacle occurs. In addition, each technical means shown in the drawings referred to for explaining this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure to the extent that no particular technical obstacle occurs.

[0014] (Configuration of Evaluation Device 1) The configuration of the evaluation device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the evaluation device 1. As shown in FIG. 1, the evaluation device 1 includes an acquisition unit 11, a clustering unit 12, a calculation unit 13, and an evaluation unit 14. The acquisition unit 11 is an example of a configuration for realizing an acquisition means. The clustering unit 12 is an example of a configuration for realizing a clustering means. The calculation unit 13 is an example of a configuration for realizing a calculation means. The evaluation unit 14 is an example of a configuration for realizing an evaluation means. The acquisition unit 11 acquires embeddings generated using an embedding layer included in a language processing model for natural language sentences included in each of a plurality of training data pieces. The clustering unit 12 performs clustering of the plurality of embeddings. The calculation unit 13 calculates an evaluation index that evaluates the result of the clustering by referring to the labels included in each of the plurality of training data pieces. The evaluation unit 14 evaluates the quality of the embedding layer based on the evaluation index.

[0015] (Effects of evaluation device 1) As described above, the evaluation device 1 is configured to include the above-mentioned acquisition unit 11, clustering unit 12, calculation unit 13, and evaluation unit 14. Therefore, the evaluation device 1 has the effect of providing a technique for narrowing down the cause of poor performance of a language processing model according to the evaluation result of the embedding layer.

[0016] (Example of program implementation) When the evaluation device 1 is configured by a computer including at least one processor and a memory, the memory stores the following program according to this exemplary embodiment. The program causes the computer to function as the evaluation device 1, and causes the computer to function as the acquisition unit 11, the clustering unit 12, the calculation unit 13, and the evaluation unit 14 described above.

[0017] (Evaluation method S1 flow) The flow of the evaluation method S1 will be described with reference to FIG. 2. FIG. 2 is a flow diagram showing the flow of the evaluation method S1. As shown in FIG. 2, the evaluation method S1 includes an acquisition process S11, a clustering process S12, a calculation process S13, and an evaluation process S14. In the acquisition process S11, at least one processor acquires embeddings generated using an embedding layer included in a language processing model for natural language sentences included in each of a plurality of training data pieces. In the clustering process S12, at least one processor performs clustering of the plurality of embeddings. In the calculation process S13, at least one processor calculates an evaluation index that evaluates the clustering result by referring to a label included in each of a plurality of training data pieces. In the evaluation process S14, at least one processor evaluates the quality of the embedding layer based on the evaluation index.

[0018] (Effect of evaluation method S1) As described above, the evaluation method S1 employs a configuration including the above-mentioned acquisition process S11, clustering process S12, calculation process S13, and evaluation process S14. Therefore, the evaluation method S1 has the effect of providing a technique for narrowing down the cause of poor performance of a language processing model according to the evaluation result of the embedding layer.

[0019] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be given the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means adopted in this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure, as long as no particular technical hindrance occurs. In addition, each technical means shown in each drawing referred to for explaining this exemplary embodiment can be adopted in other exemplary embodiments included in this disclosure, as long as no particular technical hindrance occurs.

[0020] (Configuration of Evaluation Device 1A) Fig. 3 is a block diagram showing a configuration of an evaluation system 10 including an evaluation device 1A. As shown in Fig. 3, the evaluation system 10 includes the evaluation device 1A, a learning device 2, and a display device 3. The evaluation device 1A and the learning device 2 may be connected to each other so as to be able to communicate with each other, but they do not have to be connected to each other so as to be able to communicate with each other. The display device 3 is connected to the evaluation device 1A and displays information under the control of the evaluation device 1A.

[0021] The evaluation device 1A is a device that evaluates the quality of an embedding layer included in a language processing model M2. In this exemplary embodiment, a deep learning model that executes a classification task of a natural language sentence is applied as the language processing model M2. The language processing model M2 is trained to output a classification of a natural language sentence when the natural language sentence is input.

[0022] The learning device 2 is a device that generates a language processing model M2 by performing fine tuning on a pre-learning model M1 using a training dataset DS1. The training dataset DS1 and the language processing model M2 are stored in a storage device (not shown) accessible from the evaluation device 1A. In this exemplary embodiment, an example will be described in which the evaluation device 1A refers to the training dataset DS1 used in generating the language processing model M2. However, a part or all of the training dataset DS1 referred to by the evaluation device 1A does not necessarily have to be the training dataset used in generating the language processing model M2, and may be a training dataset for evaluation.

[0023] The pre-learning model M1 takes a natural language sentence as input and outputs its embedding. The embedding represents the natural language sentence as a vector in a high-dimensional feature space. Each element of the embedding is a numerical value, that is, the embedding represents the natural language sentence as a numerical value. Examples of the pre-learning model M1 include BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and T5 (Text-to-Text Transfer Transformer). For example, the embedding output by Japanese BERT (bert-base-japanese-whole-word-masking) is a 768-dimensional vector. Also, the embedding output by Japanese GPT2 (Japanese-gpt2-medium) is a 1024-dimensional vector. Also, the embedding output by Japanese T5 (t5-base-japanese) is a 768-dimensional vector. However, the pre-learning model M1 is not limited to the above examples.

[0024] The training data set DS1 includes a plurality of training data fragments. The training data fragments include information (hereinafter also referred to as a pair) that associates a natural language sentence with a label indicating a classification of the natural language sentence. As an example, the label "TRUE" indicates a classification that the natural language sentence associated with the label is correct information, and the label "FALSE" indicates a classification that the natural language sentence associated with the label is incorrect information. In this case, examples of the training data fragments include a pair of the label "TRUE" and the natural language sentence "The capital of Japan is Tokyo", a pair of the label "FALSE" and the natural language sentence "Osaka is the prefecture with the largest population in Japan", and the like. As another example, the training data set DS1 may be composed of training data fragments that include articles included in a news corpus as natural language sentences and categories of the articles as labels. Note that the fields of the natural language sentences and the types of labels included in the training data fragments are not limited to the above examples. Also, it is desirable that the number of types of labels included in the training data set DS1 is two or more.

[0025] The language processing model M2 is a model generated by fine-tuning the pre-learning model M1 by the learning device 2 using the training data set DS1. The language processing model M2 includes an embedding layer L1 and a classification layer L2. The embedding layer L1 is a layer that receives a natural language sentence as input and outputs an embedding. For the same natural language sentence as input, the embedding output from the embedding layer L1 and the embedding output from the pre-learning model M1 may not be the same and may be different. This is because the embedding layer L1 is fine-tuned based on the pre-learning model M1. The classification layer L2 is a layer that receives an embedding as input and outputs a classification. The language processing model M2 is trained so that when a natural language sentence of a training data piece is input to the embedding layer L1, a label associated with the natural language sentence is output from the classification layer L2.

[0026] The evaluation device 1A includes an acquisition unit 11A, a clustering unit 12A, a calculation unit 13A, an evaluation unit 14A, and an output unit 15A. The acquisition unit 11A acquires embeddings generated for natural language sentences included in each training data piece of a training dataset DS1 using an embedding layer L1 included in a language processing model M2.

[0027] The clustering unit 12A performs clustering of a plurality of embeddings a plurality of times with different numbers of clusters. The number of times of performing clustering with different numbers of clusters may be two or more, and is not particularly limited. For example, the clustering unit 12A may apply, as the number of clusters, a number M=N×m obtained by multiplying the number of types of labels N included in the training data set DS1 by m (m=1, 2, 3, ...). In other words, the number of clusters M may be a multiple of the number of types of labels N. In the following description, m=1 is omitted because it is unlikely that the labels and the clustering will completely match, and the description will continue with m=2, 3, .... As an example, when a label is assigned to a piece of training data using a deep neural network (DNN), the classification by the DNN becomes nonlinear. For this reason, it may be considered that the classification (label) by the DNN and the linear clustering almost never completely match. However, m is not limited to starting from 2. The plurality of embeddings acquired by the acquisition unit 11A are clustered into M clusters. It should be noted that any known technique can be applied to the clustering technique. For example, one example of an applicable clustering technique is the k-means method, which is an example of a non-hierarchical technique, but is not limited to this.

[0028] The calculation unit 13A calculates an evaluation index by referring to a cluster that satisfies an occupancy condition for considering the cluster to be occupied by multiple embeddings generated from multiple training data pieces containing the same label, among the multiple clusters indicated by the clustering result. Here, "embedding generated from a training data piece containing a certain label" represents a natural language sentence associated with the label in the training data piece. In the following, for the sake of simplicity, "embedding generated from a training data piece containing a certain label" is also described as "embedding associated with a certain label". In other words, a cluster satisfies the occupancy condition, which indicates that the cluster is considered to be occupied by an embedding associated with a certain label. A cluster that satisfies the occupancy condition is highly likely to correspond appropriately to the label. Therefore, the more clusters that satisfy the occupancy condition, the more likely the clustering result is appropriate. Therefore, an evaluation index can be calculated by referring to a cluster that satisfies the occupancy condition.

[0029] The evaluation unit 14 evaluates the quality of the embedding layer L1 based on the evaluation indexes obtained by evaluating the results of each of the clustering operations performed multiple times. Hereinafter, the "evaluation indexes obtained by evaluating the results of each of the clustering operations performed multiple times" will also be referred to as a "series of evaluation indexes." For example, the evaluation unit 14 may determine whether the embedding layer L1 is appropriate based on whether the series of evaluation indexes satisfies a judgment criterion. The judgment criterion will be described in detail later.

[0030] Moreover, when neither the first evaluation result nor the second evaluation result obtained using the same training data set DS1 meets the judgment criterion, the evaluation unit 14A evaluates that the quality of the training data set DS1 does not meet the criterion. Here, the first evaluation result is an evaluation result of the quality of the embedding layer obtained by applying the language processing model M2 as the language processing model and operating the acquisition unit 11A, the clustering unit 12A, the calculation unit 13A, and the evaluation unit 14A. Moreover, the second evaluation result is an evaluation result of the quality of the embedding layer obtained by applying the language processing model M2-1 as the language processing model and operating the acquisition unit 11A, the clustering unit 12A, the calculation unit 13A, and the evaluation unit 14A. The language processing model M2-1 is a model different from the language processing model M2. For example, the language processing model M2-1 may be a model generated by fine-tuning a pre-learning model M1-1 different from the pre-learning model M1 using the same training data set DS1.

[0031] The output unit 15A outputs the evaluation result of the quality of the embedding layer L1, the clustering result, and some or all of the evaluation indexes. The output unit 15A is an example of a configuration that realizes an output means.

[0032] (Evaluation method S1A flow) The evaluation device 1A configured as above executes the evaluation method S1A. The flow of the evaluation method S1A will be described with reference to Fig. 4. Fig. 4 is a flow diagram illustrating the flow of the evaluation method S1A. As shown in Fig. 4, the evaluation method S1A includes steps S11A to S15A.

[0033] In step S11A, the acquisition unit 11A acquires embeddings of natural language sentences included in each training data piece of the training data set DS1, using the embedding layer L1 included in the language processing model M2.

[0034] FIG. 5 is a schematic diagram showing an example of embedding acquired in step S11A. The training data set DS1 shown in FIG. 5 includes training data fragments D1 to D8. For example, the training data fragment D1 includes a pair of natural language sentence labels "Label 1" and "Text 1". The label "Label 1" indicates the classification of the natural language sentence "Text 1". The embedding E1 is an embedding obtained by inputting the natural language sentence "Text 1" into the embedding layer L1 of the language processing model M2, and each element is represented by a numerical value. The training data fragments D2 to D8 and the embeddings E2 to E8 are also described in the same manner as the training data fragment D1 and the embedding E1. As described above, for example, the "embedding E1" acquired from "Text 1" associated with "Label 1" is also described as "embedding E1 associated with Label 1", etc. In the training data set DS1, the label is one of "Label 1", "Label 2", and "Label 3", and the number of types of labels N is three. In this way, an embedding set ES1 is generated from the training dataset DS1.

[0035] FIG. 6 is a scatter plot showing the embedding set ES1 shown in FIG. 5. In the scatter plot, high-dimensional embeddings (e.g., the number of dimensions of the pre-training model M1) are compressed to two dimensions and plotted. In the example of FIG. 6, the training dataset DS contains 63 training data pieces, so figures showing 63 embeddings are plotted. Embeddings associated with "label 1" are represented by black circles. Embeddings associated with "label 2" are represented by black triangles. Embeddings associated with "label 3" are represented by black rectangles. The number of embeddings associated with each label is 21.

[0036] In step S12A shown in FIG. 4, the clustering unit 12A performs clustering of a plurality of embeddings a plurality of times with different numbers of clusters. FIG. 7 is a schematic diagram showing the results of clustering performed a plurality of times with different numbers of clusters on the embedding set ES1 shown in FIG. 6. Lines surrounding a plurality of embeddings in the clustering results G1 to G4 represent clusters. For example, in the clustering result G1, the number of clusters M is 6, which is obtained by multiplying the number of label types N=3 by m=2. In addition, in the clustering result G2, the number of clusters M is 9, which is obtained by multiplying N=3 by m=3. In addition, in the clustering result G3, the number of clusters M is 12, which is obtained by multiplying N=3 by m=4. In addition, in the clustering result G4, the number of clusters M is 15, which is obtained by multiplying N=3 by m=5. That is, in this example, the number of clusters is a multiple of the number of label types, 3. 7 shows the results of four rounds of clustering, with the number of clusters set to 6, 9, 12, and 15. The number of clustering rounds is expected to be about 3 to 10. However, the number of clustering rounds is not limited to this range, and may be set according to the evaluation accuracy or calculation cost.

[0037] In step S13A shown in Fig. 4, the calculation unit 13A calculates an evaluation index by referring to clusters that satisfy the occupancy condition among the multiple clusters indicated by the clustering result. As an example, the calculation unit 13A calculates, as the evaluation index, a ratio of the total number of embeddings included in one or multiple clusters that satisfy the occupancy condition to the total number of multiple embeddings acquired from the training dataset DS1. A detailed example of the step S13A will be described with reference to Fig. 8.

[0038] FIG. 8 is a flow diagram showing a detailed example of the calculation process in step S13A. In the following description, each of M clusters indicated by the clustering result is described as the i-th cluster (i=0, 1, ..., M-1) based on the processing order. The processing order may be arbitrary, and for example, the i-th cluster may be randomly selected from clusters that have not yet been processed. As shown in FIG. 8, the calculation process S13A includes steps S101 to S111. Steps S101 to S111 indicate a series of processes performed for one clustering result, and are repeated the number of times that clustering has been performed. In the following, an example in which steps S101 to S111 are performed for the clustering result G1 will be mainly described, but steps S101 to S111 are similarly performed for each of the clustering results G2 to G4.

[0039] In step S101, the calculation unit 13A counts the total number of clustered embeddings D. In the example of Fig. 6, since the training data set DS1 includes 63 training data pieces D, D = 63.

[0040] In step S102, the calculation unit 13A counts the number of label types N. In the example of Fig. 6, as described above, the training data set DS1 includes three types of labels, label 1, label 2, and label 3, so the number of label types N = 3.

[0041] In step S103, the calculation unit 13A counts the number of clusters M. For example, in the case of the clustering result G1 in FIG.

[0042] In step S104, the calculation unit 13A initializes an iterator i for sequentially processing a plurality of clusters included in the corresponding clustering result to 0. The calculation unit 13A also initializes a variable e for counting the total number of embeddings included in a cluster that satisfies the occupancy condition to 0.

[0043] In step S105, the calculation unit 13A judges whether i is less than M (i.e., whether there is a cluster that has not yet been processed). If it is judged as Yes in step S105, a series of processes from the next step S106 to S109 is executed for the i-th cluster. The case where it is judged as No in step S105 will be described later.

[0044] In step S106, the calculation unit 13A counts the number di of embeddings included in the i-th cluster. For example, assume that the cluster G11 is processed 0th in the clustering result G1 in Fig. 7. In this case, the number d0 of embeddings included in the cluster G11 is 11.

[0045] In step S107, the calculation unit 13A counts the number dk of embeddings associated with each k-th (k=1, 2, ..., N) label among the N types of labels in the i-th cluster. For example, assume that cluster G11 is processed 0th in the clustering result G1 in FIG. 7. In this case, the number d0_1 of embeddings associated with the first label "Label 1" is 0. The number d0_2 of embeddings associated with the second label "Label 2" is 1. The number d0_3 of embeddings associated with the third label "Label 3" is 10.

[0046] In step S108, the calculation unit 13A judges whether the i-th cluster satisfies the occupancy condition. As described above, the occupancy condition is a condition for considering a certain cluster to be occupied by embeddings associated with the same label. The occupancy condition may be, for example, a condition that the ratio of embeddings associated with the same label in a certain cluster is equal to or greater than a threshold T. However, the occupancy condition is not limited to the above example. The ratio of embeddings associated with the same label in a cluster is calculated by the formula "di_k / di". The threshold T may be a predetermined ratio (e.g., 70%) or may be calculated based on a formula for calculating the threshold T. An example of such a formula is shown in the following formula (1).

[0047] T=1-1 / N …(1) In formula (1), N is the number of label types. For example, in the example of the training data set DS1 in FIG. 6, the number of label types is N=3, so T=1−1 / 3≈0.67.

[0048] For example, assume that cluster G11 is processed 0th in the clustering result G1 of FIG. 7. The ratio of embeddings associated with label 3 in cluster G11 is calculated as 10 / 11 ≈ 0.91 based on the number of embeddings d0_3 = 10 and the number of embeddings included in cluster G11 d0 = 11. The ratio 0.91 is greater than threshold T = 0.67. Therefore, cluster G11 satisfies the occupancy condition. In other words, cluster G11 is occupied by embeddings associated with label 3.

[0049] Also, for example, assume that cluster G12 is processed first in the clustering result G1 in FIG. 7. In cluster G12, the ratio of embeddings associated with label 1 is calculated to be d1_1 / d1 = 2 / 7 ≒ 0.29. The ratio of embeddings associated with label 2 is calculated to be d1_2 / d1 = 1 / 7 ≒ 0.14. The ratio of embeddings associated with label 3 is calculated to be d1_3 / d1 = 4 / 7 ≒ 0.57. Since the ratio of embeddings associated with any label is not greater than or equal to threshold T, cluster G12 does not satisfy the occupancy condition.

[0050] If the result of step S108 is Yes, the process proceeds to step S109. In step S109, the calculation unit 13A adds the number di of embeddings in the i-th cluster to the variable e. This is because the i-th cluster satisfies the occupancy condition, and therefore all of the di embeddings included in the cluster are considered to be associated with occupying labels.

[0051] In step S110, the calculation unit 13A increments the iterator i, and repeats the process from step S105.

[0052] If the answer is No in step S108, the process skips step S109 and proceeds to step S110. In other words, if the i-th cluster does not satisfy the occupancy condition, the iterator i is incremented without adding the variable e, and the process from step S105 is repeated.

[0053] Next, a case where the answer is determined to be No in step S105 will be described. In this case, the processes of steps S106 to S109 have been completed for all clusters included in the clustering result. In other words, the variable e indicates the total number of embeddings included in clusters that satisfy the occupancy condition among the multiple clusters indicated by the clustering result. Therefore, in this case, the process of the next step S111 is executed.

[0054] In step S111, the calculation unit 13A calculates the ratio e / D of the total number e of embeddings included in one or more clusters that satisfy the occupancy condition to the total number D of embeddings subjected to clustering as an evaluation index for evaluating the clustering result.

[0055] The calculation unit 13A executes steps S101 to S111 for each result of clustering performed multiple times for the same multiple embeddings, thereby acquiring a series of evaluation indices. FIG. 9 is a diagram showing an example of a series of evaluation indices obtained from clustering results G1 to G4 for the embedding set ES1. In FIG. 9, the series of evaluation indices vs1 is a string in which evaluation indices v1, v2, v3, and v4 are arranged in ascending order of the number of clusters M. The evaluation index v1 is an evaluation index obtained from the clustering result G1 shown in FIG. 7, and corresponds to the number of clusters M=6. The evaluation index v2 is an evaluation index obtained from the clustering result G2, and corresponds to the number of clusters M=9. The evaluation index v3 is an evaluation index obtained from the clustering result G3, and corresponds to the number of clusters M=12. The evaluation index v4 is an evaluation index obtained from the clustering result G4, and corresponds to the number of clusters M=15.

[0056] This concludes the detailed example of step S13A. Next, in step S14A shown in FIG. 4, the evaluation unit 14A refers to the series of evaluation indexes and evaluates the quality of the embedding layer L1 based on the judgment criteria. For example, the evaluation result by the evaluation unit 14A may indicate whether the quality of the embedding layer L1 is good or bad depending on whether the judgment criteria are satisfied. Two examples of the judgment criteria will be described below.

[0057] A first example of the criterion is that the relationship between the number of clusters and the evaluation index satisfies a predetermined condition. As an example, the criterion may include a condition for one or both of the slope and intercept of the approximation line obtained from the series of evaluation indexes. Such an approximation line is obtained by performing linear approximation with the evaluation index as the objective variable and the number of clusters as the explanatory variable.

[0058] FIG. 10 is a diagram for explaining a first example of the judgment criterion. In FIG. 10, the horizontal axis x indicates the number of clusters M, the vertical axis y indicates the evaluation index, and the series vs1 of the evaluation index shown in FIG. 9 is plotted. A linear function F1 represents a straight line obtained by linearly approximating the series vs1 of the evaluation index, with a slope a=0.0762 and an intercept b=0.4524. For example, if the judgment criterion is "a>0 and b>0.4", the series vs1 of the evaluation index is judged to satisfy the judgment criterion. For example, if the judgment criterion is "a>0 and b>0.8", the series vs1 of the evaluation index is judged to not satisfy the judgment criterion.

[0059] A second example of the criterion is that the series of evaluation indexes is included in a predetermined area in a graph in which the series of evaluation indexes are plotted. The predetermined area may be rectangular or of another shape. FIG. 11 is a diagram for explaining the second example of the criterion. In FIG. 11, the horizontal axis x indicates the number of clusters M, and the vertical axis y indicates the evaluation indexes. In FIG. 11, the series of evaluation indexes vs2, vs3, and vs4 obtained by a specific example described later are plotted. The series of evaluation indexes vs2 is composed of evaluation indexes in the results of clustering performed nine times with different numbers of clusters M for multiple embeddings obtained from a training data set DS2 (not shown). The training data set DS2 includes three types of labels (N=3), namely, label 1, label 2, and label 3, and in the nine clustering operations, a multiple of the number of label types N=3 is adopted as the number of clusters M. The series of evaluation indexes vs3 and vs4 are also explained in the same manner as the series vs1, but the training data sets used are different. The series of evaluation indices vs3 is obtained from the result of clustering for the ending set obtained from the training data set DS3 (not shown). The series of evaluation indices vs4 is obtained from the result of clustering for the ending set obtained from the training data set DS4 (not shown). The training data sets DS3 and DS4 are described in the same manner as the training data set DS2, but some or all of the training data pieces included are different.

[0060] 11, a predetermined area A1 is set as a judgment criterion. The predetermined area A1 indicates a rectangular area where y≧0.85. That is, in this case, the judgment criterion is that the evaluation index is always 0.85 or more regardless of changes in the number of clusters M. Since the evaluation index series vs2 is included in the predetermined area A1, it is judged that the judgment criterion is satisfied. Since neither of the evaluation index series vs3 and vs4 is included in the predetermined area A1, it is judged that the judgment criterion is not satisfied. This concludes the explanation of the example of the judgment criterion in step S14A.

[0061] In step S15A shown in FIG. 4, the output unit 15A outputs to the display device 3 some or all of the evaluation results by the evaluation unit 14A, the clustering results by the clustering unit 12A, and the evaluation indexes calculated by the calculation unit 13A.

[0062] For example, the information output to the display device 3 may include information indicating whether the quality of the embedding layer is good or bad as an evaluation result by the evaluation unit 14A. An example of information indicating that the quality of the embedding layer is good is text information such as "The embedding layer of this language processing model has been determined to be appropriate." Also, an example of information indicating that the quality of the embedding layer is not good is text information such as "The embedding layer of this language processing model has been determined to be inappropriate. Please check the training dataset and consider changing the pre-learning model."

[0063] Furthermore, for example, the information output to the display device 3 may include an image (e.g., FIG. 7) in which a figure indicating a cluster is superimposed on a scatter plot of embeddings compressed into two dimensions as a result of clustering by the clustering unit 12A.

[0064] Furthermore, for example, the information output to the display device 3 may include an image (e.g., FIG. 10 or FIG. 11) in which a figure showing the judgment criteria is superimposed on a graph plotting a series of evaluation indexes calculated by the calculation unit 13A.

[0065] (Variation 1) The above-mentioned evaluation method S1A can be modified to include step S13B instead of step S13A. In step S13B, the calculation unit 13A may calculate an evaluation index by referring to one or more embeddings generated from a plurality of training data pieces including a label that satisfies a remaining condition for considering a cluster to be left without being removed from the cluster, for each of the plurality of clusters indicated by the clustering result.

[0066] The remaining condition may be that a first ratio indicating a ratio of one or more embeddings associated with a certain label in a certain cluster satisfies a predetermined condition with respect to a second ratio of training data fragments including the label in the entirety of the plurality of training data fragments. The predetermined condition may be that the first ratio is equal to or greater than the second ratio. However, the remaining condition is not limited to the above example. Here, in a certain cluster, embeddings associated with labels other than the label satisfying the remaining condition are considered to have been removed. As a result, it can be considered that one or more embeddings associated with labels satisfying the remaining condition remain in the i-th cluster. Hereinafter, "embeddings associated with labels satisfying the remaining condition in a certain cluster" will also be referred to as "embeddings remaining in a certain cluster". The greater the total number of embeddings remaining in each of the plurality of clusters indicated by the clustering result, the more likely the clustering result is appropriate. Therefore, an evaluation index can be calculated by referring to the remaining embeddings. A detailed example of the calculation process in step S13B will be described with reference to FIG.

[0067] Fig. 12 is a flow diagram showing a detailed example of the calculation process in step S13B. As shown in Fig. 12, the calculation process S13B includes steps S201 to S212. Steps S201 to S211 indicate a series of processes performed for one clustering result, and are repeated the number of times that clustering has been performed. The following mainly describes an example in which steps S201 to S212 are performed for the clustering result G1 shown in Fig. 7, but steps S201 to S212 are also similarly performed for the clustering results G2 to G4.

[0068] In step S201, the calculation unit 13A counts the total number of clustered embeddings D. In the example of Fig. 6, since the training data set DS1 includes 63 training data pieces D, D = 63.

[0069] In step S202, the calculation unit 13A counts the number of label types N. In the example of Fig. 6, as described above, the number of label types N is three, namely label 1, label 2, and label 3, so N = 3.

[0070] In step S203, the calculation unit 13A calculates a ratio Rk of one or more embeddings associated with the k-th label for each of the N types of labels. The ratio Rk is an example of the second ratio. In the example of Fig. 6, there are 21 embeddings associated with label 1, label 2, and label 3, respectively, so R1 = 33.33%, R2 = 33.33%, and R3 = 33.33% (here, for simplicity, numbers rounded to two decimal places are shown as the ratios, and the same applies to the following).

[0071] In step S204, the calculation unit 13A counts the number of clusters M. For example, in the case of the clustering result G1 in FIG.

[0072] In step S205, the calculation unit 13A initializes an iterator i for sequentially processing multiple clusters included in the corresponding clustering result to 0. The calculation unit 13A also initializes a variable s for counting the total number of one or multiple embeddings associated with a label that satisfies the remaining condition to 0.

[0073] In step S206, the calculation unit 13A judges whether i is less than M (i.e., whether there is a cluster that has not yet been processed). If the judgment in step S206 is Yes, a series of processes from the next steps S207 to S210 is executed for the i-th cluster. The case where the judgment in step S206 is No will be described later.

[0074] In step S207, the calculation unit 13A counts the number di of embeddings included in the i-th cluster. For example, assume that the cluster G11 is processed 0th in the clustering result G1 in Fig. 7. In this case, d0=11.

[0075] In step S208, the calculation unit 13A calculates a ratio rk of embeddings associated with the kth label for each of the N types of labels in the i-th cluster. The ratio rk is an example of the first ratio described above. The ratio rk is calculated by dividing the number of embeddings associated with the kth label in the i-th cluster by the number di described above. For example, a case where the cluster G11 is processed 0th in the clustering result G1 of FIG. 7 will be described. In this case, the number of embeddings associated with the label 1 is 0, so r1=0 / 11=0.00%. The number of embeddings associated with the label 2 is 1, so r2=1 / 19≒9.09%. The number of embeddings associated with the label 3 is 10, so r3=10 / 11≒90.91%.

[0076] In step S209, the calculation unit 13A determines that the K-th label satisfying rK≧RK satisfies the remaining condition. Note that there may be multiple types of labels that satisfy the remaining condition. In other words, multiple values may be specified for K. Further, the calculation unit 13A counts the number si of one or more embeddings remaining in the i-th cluster. As described above, the one or more remaining embeddings are one or more embeddings associated with the label K that satisfies the remaining condition. For example, in cluster G11, since r1<R1, label 1 does not satisfy the remaining condition. Since r2<R2, label 2 does not satisfy the remaining condition. Since r3>R3, label 3 satisfies the remaining condition. Therefore, the number s0 of embeddings associated with label 3 remaining in cluster G11 is 10.

[0077] In step S210, the calculation unit 13A adds the number si of embeddings remaining in the i-th cluster to the variable s.

[0078] In step S211, the calculation unit 13A increments the iterator i and repeats the process from step S206.

[0079] The case where it is determined as No in step S206 will be described. In this case, the processes of steps S207 to S210 have been completed for all the clusters included in the clustering result. That is, the variable s indicates the total number of embeddings remaining in each of the plurality of clusters indicated by the clustering result. Therefore, in this case, the process of the next step S212 is executed.

[0080] In step S212, as an evaluation index for evaluating the clustering result, the calculation unit 13A calculates the ratio s / D of the total number s of embeddings remaining in each cluster to the total number D of embeddings for which clustering is performed.

[0081] The calculation unit 13A executes steps S201 to S212 for each result of clustering performed multiple times for the same multiple embeddings, thereby acquiring a series of evaluation indexes.

[0082] (Variation 2) The above-mentioned evaluation method S1A can be modified to include step S13C instead of step S13A. In step S13C, the calculation unit 13A may calculate the evaluation index by referring to a comparison result between the number of training data fragments including the same label among the multiple training data fragments (hereinafter referred to as the first number) and the number of training data fragments included in one or more clusters including at least one embedding generated from the training data fragment including the label among the multiple clusters indicated by the clustering result (hereinafter referred to as the second number).

[0083] Here, the smaller the difference between the first number and the second number, the less embeddings associated with other labels are included in the cluster including the embedding associated with the label, and the more likely the clustering result is appropriate. Therefore, the evaluation index can be calculated by referring to the comparison result of the first number and the second number. A detailed example of the calculation process in step S13C will be described with reference to FIG.

[0084] Fig. 13 is a flow diagram showing a detailed example of the calculation process in step S12C. As shown in Fig. 13, the calculation process S13C includes steps S301 to S314. Steps S301 to S314 indicate a series of processes performed for one clustering result, and are repeated the number of times that clustering has been performed. The following mainly describes an example in which steps S301 to S314 are performed for the clustering result G1 shown in Fig. 7, but steps S301 to S314 are also similarly performed for the clustering results G2 to G4.

[0085] In step S301, the calculation unit 13A counts the number of label types N. In the example of Fig. 6, as described above, the number of label types N is three, namely label 1, label 2, and label 3, so N = 3.

[0086] In step S302, the calculation unit 13A initializes an iterator i for sequentially processing multiple clusters included in the corresponding clustering result to 0. The calculation unit 13A also initializes a variable t for adding a set product to 0. The set product is a value obtained by dividing the total number of embeddings associated with a certain label (the first number described above) by the total number of embeddings included in one or more clusters including at least one embedding associated with the label (the second number described above). The set product is an example of a number indicating a comparison result between the first number and the second number.

[0087] In step S303, the calculation unit 13A counts the number of clusters M. For example, in the case of the clustering result G1 in FIG.

[0088] In step S304, the calculation unit 13A judges whether k is smaller than N (i.e., whether there is a label that has not yet been processed). If it is judged as Yes in step S304, a series of processes from the next steps S305 to S312 are executed for the k-th label. The case where it is judged as No in step S304 will be described later.

[0089] In step S305, the calculation unit 13A counts the total number Dk of embeddings associated with the k-th label. The total number Dk indicates the above-mentioned first number. In the clustering result G1 of FIG. 7, D1=21, D2=21, and D3=21.

[0090] In step S306, the calculation unit 13A initializes an iterator i for sequentially processing multiple clusters included in the corresponding clustering result to 0. The calculation unit 13A also initializes a variable dk for counting the total number of embeddings (the above-mentioned second number) included in one or more clusters including at least one embedding associated with the kth label to 0.

[0091] In step S307, the calculation unit 13A judges whether i is less than M (i.e., whether there is a cluster that has not yet been processed). If it is judged as Yes in step S307, a series of processes from the next step S308 to S310 is executed for the i-th cluster. The case where it is judged as No in step S307 will be described later.

[0092] In step S308, the calculation unit 13A judges whether or not at least one embedding associated with the kth label is included in the i-th cluster. For example, in the case of the first label (label 1), cluster G11 does not have any embedding associated with label 1, so the result is determined as No. Cluster G12 has two embeddings associated with label 1, so the result is determined as Yes. If the result in step S308 is Yes, the process from the next step S309 is executed. If the result in step S308 is No, the processes from steps S309 to S310 are not executed, and the process from step S311 described below is executed.

[0093] In step S309, the calculation unit 13A counts the number p of embeddings included in the i-th cluster. For example, when the cluster G11 in the clustering result G1 in Fig. 7 is the target, p = 11.

[0094] In step S310, the calculation unit 13A adds the number p of embeddings in the i-th cluster to the variable dk.

[0095] In step S311, the calculation unit 13A increments the iterator i, and repeats the processes from step S307.

[0096] A case where the result of step S307 is No will be described. In this case, for the kth label, the processes of steps S308 to S310 have been completed for all clusters included in the clustering result. In other words, the variable dk indicates the total number (second number) of embeddings included in one or more clusters that include at least one of the kth labels. In this case, the process of the next step S312 is executed.

[0097] In step S312, the calculation unit 13A adds the set product Dk / dk for the k-th label to the variable t.

[0098] In step S313, the calculation unit 13A increments the iterator k, and repeats the processes from step S304.

[0099] A case where the result of step S304 is No will be described. In this case, the processes of steps S305 to S312 have been completed for all N types of labels. In other words, the variable t indicates the sum of the set products for each label. In this case, the process of the next step S314 is executed.

[0100] In step S314, the calculation unit 13A calculates an average value t / N of the set products for each label as an evaluation index for evaluating the clustering result.

[0101] The calculation unit 13A executes steps S301 to S314 for each result of clustering performed multiple times for the same multiple embeddings, thereby acquiring a series of evaluation indexes.

[0102] (Specific examples) A specific example of an experiment to evaluate the quality of the embedding layer L1 using the evaluation device 1A will be described. In this example, the quality of the embedding layer L1 was evaluated using different training data sets DS2, DS3, and DS4. In addition, three types of evaluation indexes calculated by calculation processes S13A, S13B, and S13C were used as evaluation indexes.

[0103] The training dataset DS2 is generated by extracting articles from an existing news corpus in three categories, A, B, and C, and contains 2286 training data fragments. Each training data fragment contains information that associates an article with one of the three categories.

[0104] The training dataset DS3 is generated by adding a training dataset DSα including 2289 training data fragments to the training dataset DS2, and includes a total of 4575 training data fragments. The added training dataset DSα was generated by extracting articles of three other categories different from any of categories A, B, and C from the news corpus, and disguising each of the extracted articles as any of categories A, B, and C. Disguising means associating any of categories A, B, and C with an article that is actually not any of categories A, B, and C. The training dataset DS3 is a training data fragment in which about half of the total of 4575 items include incorrect labels.

[0105] The training dataset DS4 is generated by adding a training dataset DSβ containing 4932 training data fragments to the training dataset DS2, and contains a total of 7218 training data fragments. The added training dataset DSβ was generated by extracting articles of six other categories different from categories A, B, and C from the news corpus, and disguising each of the extracted articles as either category A, B, or C. The training dataset DS4 contains training data fragments, of which about two-thirds contain incorrect labels, out of a total of 7218 items.

[0106] Fig. 14 is a scatter plot in which the embeddings (high dimension) acquired by the acquisition unit 11A in this specific example are compressed into two dimensions and plotted. In Fig. 14, an embedding set ES2 shows multiple embeddings acquired from the training data set DS2. An embedding set ES3 shows multiple embeddings acquired from the training data set DS3. An embedding set ES4 shows multiple embeddings acquired from the training data set DS4. Since the embedding set ES2 was acquired from the training data set DS2 that does not include erroneous training data pieces, it is distributed so that each category can be more clearly distinguished compared to the embedding sets ES3 and ES4 acquired from the training data sets DS3 and DS4 that include erroneous training data pieces.

[0107] The evaluation result of the embedding sets ES2, ES3, and ES4 using the evaluation index calculation process S13A is as described with reference to FIG. 11. That is, since the evaluation index series vs2 is included in the predetermined area A1, it is determined that the judgment criterion is satisfied. In this case, in step S15A, the quality of the embedding layer L1 is output as good. Since neither of the evaluation index series vs3 nor vs4 is included in the predetermined area A1, it is determined that the judgment criterion is not satisfied. That is, the quality of the embedding layer L1 is evaluated as not good. In this case, in step S15A, it is output as "It has been determined that the embedding layer of this language processing model is not appropriate. Please check the training data set and consider changing the pre-learning model." In this way, it can be seen that if the quality of the training data set DS3 (or DS4) is not good, an appropriate evaluation result of "Please check the training data set and consider changing the pre-learning model" is output.

[0108] FIG. 15 is a diagram for explaining an example of an evaluation result performed using the evaluation index calculation process S13B on the result of multiple clustering operations performed on the embedding set ES3. FIG. 15 is explained in the same manner as FIG. 11, except that evaluation index series vs2_2, vs3_2, and vs4_2 are plotted instead of evaluation index series vs2, vs3, and vs4. The evaluation index series vs2_2, vs3_2, and vs4_2 are evaluation index series calculated using the evaluation index calculation process S13B. Even when the evaluation index calculation process S13B is used, the evaluation index series vs2_2 is included in the predetermined area A1, so it is determined that the judgment criterion is satisfied. The evaluation index series vs3_2 and vs4_2 are not included in the predetermined area A1, so it is determined that the judgment criterion is not satisfied. Therefore, just as in the case of using the evaluation index calculation process S13A, when the evaluation index calculation process S13B is used, if the quality of the training dataset DS3 (or DS4) is poor, an appropriate evaluation result such as "Please check the training dataset and consider changing the pre-learning model" is output.

[0109] FIG. 16 is a diagram for explaining an example of an evaluation result performed using the evaluation index calculation process S13B on the result of multiple clusterings performed on the embedding set ES4. FIG. 16 is explained in the same manner as FIG. 11, except that evaluation index series vs2_3, vs3_3, and vs4_3 are plotted instead of evaluation index series vs2, vs3, and vs4. The evaluation index series vs2_3, vs3_3, and vs4_3 are evaluation index series calculated using the evaluation index calculation process S13C. Even when the evaluation index calculation process S13C is used, the evaluation index series vs2_3 is included in the predetermined area A1, so it is determined that the judgment criterion is met. The evaluation index series vs3_3 and vs4_3 are not included in the predetermined area A1, so it is determined that the judgment criterion is not met. Therefore, just as in the case of using the evaluation index calculation process S13A, when the evaluation index calculation process S13C is used, if the quality of the training dataset DS3 (or DS4) is poor, an appropriate evaluation result such as "Please check the training dataset and consider changing the pre-learning model" is output.

[0110] (Variation 3) The evaluation method S1A can be modified as follows. When the evaluation device 1A determines that the evaluation result of the embedding layer quality obtained by executing steps S11A to S14A for the first time (hereinafter referred to as the first evaluation result) does not satisfy the judgment criterion, the evaluation device 1A may execute steps S11A to S14A for the second time. In steps S11A to S14A for the second time, the same training dataset DS1 as in the first time and a language processing model M2-1 different from that in the first time are used. As described above, the language processing model M2-1 may be a model generated using the same training dataset DS1 based on a pre-learning model M1-1 different from the pre-learning model M1. By executing steps S11A to S14A for the second time, an evaluation result of the embedding layer quality (hereinafter referred to as the second evaluation result) is obtained.

[0111] In this modification, after executing step S14A for the second time, the evaluation unit 14A judges whether the second evaluation result satisfies the judgment criterion. If the second evaluation result does not satisfy the judgment criterion like the first evaluation result, the evaluation unit 14A evaluates that the quality of the training data set DS1 does not satisfy the criterion. In other words, the cause of poor performance of the language processing models M2 and M2-1 is narrowed down to the training data set DS1.

[0112] In addition, when the first evaluation result does not satisfy the judgment criterion, the evaluation unit 14A may execute steps S11A to S14A multiple times (for example, the second time, the third time, ...) without being limited to the second time. For example, when the first evaluation result does not satisfy the judgment criterion, the evaluation unit 14A may obtain two or more evaluation results (a second evaluation result, a third evaluation result, ...) using the same training data set DS1 and two or more different language processing models (M2-1, M2-2, ...). For example, the multiple language processing models (M2, M2-1, M2-2, ...) may be obtained by fine-tuning a plurality of pre-learning models (M1, M1-1, M1-2, ...) different from each other using the same training data set DS1. In this case, the evaluation unit 14A may evaluate the quality of the training data set DS1 based on the statistics of three or more evaluation results (a first evaluation result, a second evaluation result, a third evaluation result, ...). For example, when a predetermined ratio or more of the three or more evaluation results do not satisfy the standard, the evaluation unit 14A may evaluate that the quality of the training dataset DS1 does not satisfy the standard.

[0113] In this case, the output unit 15A may output information indicating that the quality of the training data set DS1 does not meet the standard in step S15A. An example of such information is text information such as "It has been determined that none of the embedding layers of multiple language processing models generated using the same training data set is appropriate. The cause may be the training data set. Please check."

[0114] Furthermore, when the first evaluation result satisfies the judgment criterion, the evaluation device 1A may execute step S15A in the same manner as described above without executing the second or subsequent times. Furthermore, the evaluation device 1A may execute the second or subsequent times regardless of the first evaluation result.

[0115] (Examples) The evaluation devices 1 and 1A according to the above-described exemplary embodiments can be applied, for example, in the field of health care or medical care. As an example, the evaluation devices 1 and 1A can be used to narrow down the cause of poor performance of a language processing model that has been machine-learned to classify electronic medical records recorded by doctors regarding patients.

[0116] (Effects of evaluation device 1A) As described above, the evaluation device 1A employs a configuration in which the clustering unit 12 performs clustering multiple times with different numbers of clusters, and the evaluation unit 14 evaluates the quality of the embedding layer based on an evaluation index that evaluates the results of each of the multiple clusterings. In addition to the effects of the evaluation device 1, the above configuration provides the effect of being able to evaluate the quality of the embedding layer more accurately than when it is based on an evaluation index that evaluates the results of a single clustering.

[0117] Moreover, in the evaluation device 1A, the calculation unit 13A calculates the evaluation index by referring to a cluster that satisfies an occupancy condition for considering the cluster to be occupied by multiple embeddings generated from multiple training data pieces containing the same label, among multiple clusters indicated by the clustering result. According to the above configuration, in addition to the effect of the evaluation device 1, the effect of being able to evaluate the quality of the embedding layer with higher accuracy can be obtained.

[0118] In addition, the evaluation device 1A employs a configuration in which the calculation unit 13A calculates an evaluation index by referring to one or more embeddings generated from a plurality of training data pieces including a label that satisfies a remaining condition for considering a cluster to be left in the cluster without being removed, in each of a plurality of clusters indicated by the clustering result. According to the above configuration, in addition to the effect of the evaluation device 1, the effect of being able to evaluate the quality of the embedding layer with higher accuracy can be obtained.

[0119] Moreover, in the evaluation device 1A, the calculation unit 13A calculates an evaluation index by referring to a comparison result between (1) the number of training data pieces including the same label among the multiple training data pieces, and (2) the number of embeddings included in one or multiple clusters including at least one embedding generated from the training data pieces including the label among the multiple clusters indicated by the clustering result. With the above configuration, in addition to the effect of the evaluation device 1, the effect of being able to evaluate the quality of the embedding layer with higher accuracy can be obtained.

[0120] The evaluation device 1A is further configured to include an output unit 15A that outputs the evaluation result of the embedding layer quality, the clustering result, and some or all of the evaluation indexes. With the above configuration, in addition to the effects of the evaluation device 1, the user can recognize the evaluation result of the embedding layer quality, the clustering result, and some or all of the evaluation indexes.

[0121] (Other variations) In the above-described exemplary embodiment 2, an example has been described in which the language processing model is a model that executes a classification task. However, the language processing model is not limited to a classification task, and may be a model that executes other language processing tasks.

[0122] Furthermore, in the above-described exemplary embodiment 2, the evaluation device 1A has been described as being included in the evaluation system 10, but the evaluation device 1A does not necessarily have to be included in the evaluation system 10. For example, the evaluation device 1A does not have to be able to access the training dataset DS1 itself and the embedding layer L1 itself, as long as the evaluation device 1A can access a storage device in which embeddings generated using the embedding layer L1 for each of the training data pieces included in the training dataset DS1 are stored.

[0123] [Software implementation example] Some or all of the functions of the evaluation devices 1, 1A (hereinafter also referred to as "the above-mentioned devices") may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0124] In the latter case, each of the above devices is realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in Fig. 17. Fig. 17 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0125] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0126] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0127] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0128] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0129] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0130] (Appendix A1) An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A clustering means for clustering a plurality of said embeddings; a calculation means for calculating an evaluation index by referring to a label included in each of the plurality of training data pieces and evaluating a result of the clustering; an evaluation means for evaluating the quality of the embedding layer based on the evaluation index; The evaluation device comprises:

[0131] (Appendix A2) the clustering means performs the clustering a plurality of times with different numbers of clusters; The evaluation means evaluates quality of the embedding layer based on the evaluation index obtained by evaluating each result of the clustering performed multiple times. 2. An evaluation device as described in Appendix A1.

[0132] (Appendix A3) the calculation means calculates the evaluation index by referring to a cluster that satisfies an occupancy condition for considering the cluster to be occupied by a plurality of embeddings generated from a plurality of training data pieces including the same label, among the plurality of clusters indicated by the result of the clustering; 2. An evaluation device according to claim 1 or 2.

[0133] (Appendix A4) The calculation means calculates the evaluation index by referring to one or more embeddings generated from a plurality of training data pieces including a label that satisfies a remaining condition for considering the cluster to be left without being removed from the cluster in each of the plurality of clusters indicated by the result of the clustering. 13. The evaluation device according to claim 12, wherein the evaluation device is a

[0134] (Appendix A5) The calculation means calculates the evaluation index by referring to a comparison result between (1) the number of training data pieces including the same label among the plurality of training data pieces, and (2) the number of embeddings included in one or more clusters including at least one embedding generated from the training data pieces including the label among the plurality of clusters indicated by the clustering result. 13. An evaluation device according to any one of appendices A1 to A4.

[0135] (Appendix A6) an output means for outputting the embedding layer quality evaluation result, the clustering result, and some or all of the evaluation indexes; The evaluation device according to any one of appendices A1 to A5, further comprising:

[0136] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0137] (Appendix B1) An acquisition process in which at least one processor acquires embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; a clustering process in which the at least one processor performs clustering of a plurality of the embeddings; A calculation process in which the at least one processor calculates an evaluation index that evaluates a result of the clustering by referring to a label included in each of the plurality of training data pieces; an evaluation process in which the at least one processor evaluates the quality of the embedding layer based on the evaluation index; An evaluation method, comprising:

[0138] (Appendix B2) In the clustering process, the at least one processor performs the clustering a plurality of times with different numbers of clusters; In the evaluation process, the at least one processor evaluates quality of the embedding layer based on the evaluation index obtained by evaluating each result of the clustering performed multiple times. Evaluation method described in Appendix B1.

[0139] (Appendix B3) In the calculation process, the at least one processor calculates the evaluation index by referring to a cluster that satisfies an occupancy condition for considering the cluster to be occupied by a plurality of embeddings generated from a plurality of training data pieces including the same label, among the plurality of clusters indicated by the result of the clustering. Evaluation method described in Appendix B1 or B2.

[0140] (Appendix B4) In the calculation process, the at least one processor calculates the evaluation index by referring to one or more embeddings generated from a plurality of training data pieces including a label that satisfies a remaining condition for considering the cluster to be left without being removed from the cluster in each of the plurality of clusters indicated by the clustering result. The evaluation method described in any one of Appendices B1 to B3.

[0141] (Appendix B5) In the calculation process, the at least one processor calculates the evaluation index by referring to a comparison result between (1) the number of training data pieces including the same label among the multiple training data pieces, and (2) the number of embeddings included in one or more clusters including at least one embedding generated from the training data pieces including the same label among the multiple clusters indicated by the clustering result. The evaluation method described in any one of Appendices B1 to B4.

[0142] (Appendix B6) an output process in which the at least one processor outputs the evaluation result of the quality of the embedding layer, the clustering result, and some or all of the evaluation indexes; The evaluation method of any one of Appendices B1 to B5, further comprising:

[0143] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0144] (Appendix C1) A program for causing a computer to function as an evaluation device, The computer, An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A clustering means for clustering a plurality of said embeddings; a calculation means for calculating an evaluation index by referring to a label included in each of the plurality of training data pieces and evaluating a result of the clustering; an evaluation means for evaluating the quality of the embedding layer based on the evaluation index; A program that functions as a

[0145] (Appendix C2) the clustering means performs the clustering a plurality of times with different numbers of clusters; The evaluation means evaluates quality of the embedding layer based on the evaluation index obtained by evaluating each result of the clustering performed multiple times. 2. The program described in Appendix C1.

[0146] (Appendix C3) the calculation means calculates the evaluation index by referring to a cluster that satisfies an occupancy condition for considering the cluster to be occupied by a plurality of embeddings generated from a plurality of training data pieces including the same label, among the plurality of clusters indicated by the result of the clustering; 2. A program as described in Appendix C1 or C2.

[0147] (Appendix C4) The calculation means calculates the evaluation index by referring to one or more embeddings generated from a plurality of training data pieces including a label that satisfies a remaining condition for considering the cluster to be left without being removed from the cluster in each of the plurality of clusters indicated by the result of the clustering. 13. A program according to any one of appendices C1 to C3.

[0148] (Appendix C5) The calculation means calculates the evaluation index by referring to a comparison result between (1) the number of training data pieces including the same label among the plurality of training data pieces, and (2) the number of embeddings included in one or more clusters including at least one embedding generated from the training data pieces including the label among the plurality of clusters indicated by the clustering result. 13. A program according to any one of appendices C1 to C4.

[0149] (Appendix C6) The computer, an output means for outputting the embedding layer quality evaluation result, the clustering result, and some or all of the evaluation indexes; Further functioning as 13. A program according to any one of appendices C1 to C5.

[0150] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0151] (Appendix D1) at least one processor, the at least one processor comprising: An acquisition process of acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A clustering process for clustering a plurality of said embeddings; A calculation process of calculating an evaluation index by referring to the labels included in each of the plurality of training data pieces and evaluating the result of the clustering; an evaluation process for evaluating the quality of the embedding layer based on the evaluation index; An evaluation device that performs the above.

[0152] The evaluation device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0153] (Appendix D2) In the clustering process, the at least one processor performs the clustering a plurality of times with different numbers of clusters; In the evaluation process, the at least one processor evaluates quality of the embedding layer based on the evaluation index obtained by evaluating each result of the clustering performed multiple times. 13. An evaluation device as described in Appendix D1.

[0154] (Appendix D3) In the calculation process, the at least one processor calculates the evaluation index by referring to a cluster that satisfies an occupancy condition for considering the cluster to be occupied by a plurality of embeddings generated from a plurality of training data pieces including the same label, among the plurality of clusters indicated by the result of the clustering. 13. The evaluation device according to claim 12, wherein the evaluation device is a

[0155] (Appendix D4) In the calculation process, the at least one processor calculates the evaluation index by referring to one or more embeddings generated from a plurality of training data pieces including a label that satisfies a remaining condition for considering the cluster to be left without being removed from the cluster in each of the plurality of clusters indicated by the clustering result. 13. The evaluation device according to claim 12, wherein the evaluation device is a

[0156] (Appendix D5) In the calculation process, the at least one processor calculates the evaluation index by referring to a comparison result between (1) the number of training data pieces including the same label among the multiple training data pieces, and (2) the number of embeddings included in one or more clusters including at least one embedding generated from the training data pieces including the same label among the multiple clusters indicated by the clustering result. 5. The evaluation device according to claim 1, wherein the evaluation device is a

[0157] (Appendix D6) the at least one processor: an output process for outputting the embedding layer quality evaluation result, the clustering result, and some or all of the evaluation indexes; Further execute 13. The evaluation device according to any one of appendices D1 to D5.

[0158] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0159] (Appendix E1) A program for causing a computer to function as an evaluation device, the program comprising: An acquisition process of acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A clustering process for clustering a plurality of said embeddings; A calculation process of calculating an evaluation index by referring to the labels included in each of the plurality of training data pieces and evaluating the result of the clustering; an evaluation process for evaluating the quality of the embedding layer based on the evaluation index; A non-transitory recording medium on which a program for executing a program is recorded. [Explanation of symbols]

[0160] 1. 1A Evaluation Device 2 Learning device 3 Display device 10 Rating System 11, 11A Acquisition Department 12, 12A Clustering Section 13, 13A calculation section 14, 14A Evaluation Section 15A output section C1 Processor C2 Memory

Claims

1. An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A clustering means for clustering a plurality of said embeddings; a calculation means for calculating an evaluation index by referring to a label included in each of the plurality of training data pieces and evaluating a result of the clustering; an evaluation means for evaluating the quality of the embedding layer based on the evaluation index; The evaluation device comprises:

2. the clustering means performs the clustering a plurality of times with different numbers of clusters; The evaluation means evaluates quality of the embedding layer based on the evaluation index obtained by evaluating each result of the clustering performed multiple times. The evaluation device according to claim 1 .

3. the calculation means calculates the evaluation index by referring to a cluster that satisfies an occupancy condition for considering the cluster to be occupied by a plurality of embeddings generated from a plurality of training data pieces including the same label, among the plurality of clusters indicated by the result of the clustering; The evaluation device according to claim 1 .

4. The calculation means calculates the evaluation index by referring to one or more embeddings generated from a plurality of training data pieces including labels that satisfy a remaining condition for considering the cluster to be left without being removed from the cluster in each of the plurality of clusters indicated by the result of the clustering. The evaluation device according to claim 1 .

5. The calculation means calculates the evaluation index by referring to a comparison result between (1) the number of training data pieces including the same label among the plurality of training data pieces, and (2) the number of embeddings included in one or more clusters including at least one embedding generated from the training data pieces including the same label among the plurality of clusters indicated by the clustering result. The evaluation device according to claim 1 .

6. an output means for outputting the embedding layer quality evaluation result, the clustering result, and some or all of the evaluation indexes; The evaluation device according to claim 1 , further comprising:

7. An acquisition process in which at least one processor acquires embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; a clustering process in which the at least one processor performs clustering of a plurality of the embeddings; A calculation process in which the at least one processor calculates an evaluation index for evaluating a result of the clustering by referring to a label included in each of the plurality of training data pieces; an evaluation process in which the at least one processor evaluates the quality of the embedding layer based on the evaluation index; An evaluation method, comprising:

8. A program for causing a computer to function as an evaluation device, The computer, An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A clustering means for clustering a plurality of said embeddings; a calculation means for calculating an evaluation index by referring to a label included in each of the plurality of training data pieces and evaluating a result of the clustering; an evaluation means for evaluating the quality of the embedding layer based on the evaluation index; A program that functions as a

Images