Evaluation apparatus, evaluation method, and program

The evaluation device and method analyze embedding layers to diagnose performance issues in language processing models, effectively identifying causes beyond training data quality, thereby facilitating targeted enhancements.

JP2024167768APending Publication Date: 2024-12-04NEC CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023084071
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-22
Publication Date
2024-12-04

AI Technical Summary

Technical Problem

Existing language processing models face challenges in identifying the specific cause of poor performance, as existing techniques are inadequate for pinpointing issues beyond training data quality, particularly when hyperparameters, pre-learning models, or learning algorithms are at fault.

Method used

An evaluation device and method that utilize an embedding layer to generate image data from embeddings, detect characteristics, and evaluate the quality of the embedding layer to identify the root cause of performance issues in language processing models.

Benefits of technology

Enables precise identification of the cause of poor performance in language processing models, allowing for targeted improvements by pinpointing issues in training data, pre-learning models, or hyperparameters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167768000001_ABST
    Figure 2024167768000001_ABST
Patent Text Reader

Abstract

To narrow down a cause of a case where performance of a language processing model is not satisfactory.SOLUTION: An evaluation apparatus (1) includes: an acquisition section (11) for acquiring embeddings of natural language sentences that are respectively included in a plurality of training data pieces with use of an embedding layer included in a language processing model; a generation section (12) for generating image data by converting each of elements of a matrix which includes each of the plurality of embeddings as rows or columns into pixel values; a detection section (13) for detecting a feature of the image data; and an evaluation section (14) for evaluating quality of the embedding layer based on the feature of the image data.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for evaluating the performance of a language processing model. [Background technology]

[0002] In recent years, it has been known to generate a language processing model that performs a desired language processing task by fine-tuning a general-purpose natural language processing model as a pre-learning model. The performance of such a language processing model is affected by, for example, the training data used to generate the language processing model, the pre-learning model, the learning algorithm, the hyperparameters adopted in the language processing model, and the like. For the purpose of improving the performance of such a language processing model, it is known to use a technique such as grid search to adjust the hyperparameters. In addition, for example, Patent Document 1 describes a technique for improving the quality of training data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-19341 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the performance of a language processing model is poor, it is difficult to narrow down which of the above-mentioned training data, pre-learning model, learning algorithm, hyperparameters, etc. is the main cause. The technology described in Patent Document 1 is effective when it is known that the main cause is the quality of the training data, but when this is not the case, there is a possibility that the performance of the language processing model cannot be improved even if the quality of the training data is improved. Therefore, it is important to narrow down the cause when the performance of the language processing model is poor.

[0005] One aspect of the present invention has been made in consideration of the above problems, and an example of a purpose of the present invention is to provide a technique for narrowing down the cause when the performance of a language processing model is poor. [Means for solving the problem]

[0006] An evaluation device according to one embodiment of the present invention includes an acquisition means for acquiring embeddings generated using an embedding layer included in a language processing model for natural language sentences included in each of a plurality of training data pieces, a generation means for generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value, a detection means for detecting characteristics of the image data, and an evaluation means for evaluating the quality of the embedding layer based on the characteristics of the image data. An evaluation method according to one embodiment of the present invention is an evaluation method executed by at least one processor, which includes the steps of: obtaining embeddings generated using an embedding layer included in a language processing model for natural language sentences respectively included in a plurality of training data pieces; generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value; detecting features of the image data; and evaluating the quality of the embedding layer based on the features of the image data.

[0007] A program according to one embodiment of the present invention causes a computer to function as: an acquisition means for acquiring embeddings generated using an embedding layer included in a language processing model for natural language sentences respectively included in a plurality of pieces of training data; a generation means for generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value; a detection means for detecting characteristics of the image data; and an evaluation means for evaluating the quality of the embedding layer based on the characteristics of the image data. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to narrow down the causes of poor performance of a language processing model. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a configuration of an evaluation device according to a first exemplary embodiment of the present invention. [Diagram 2] FIG. 1 is a flow chart showing the flow of an evaluation method according to a first exemplary embodiment of the present invention. [Diagram 3] FIG. 11 is a block diagram showing a configuration of an evaluation system including an evaluation device according to a second exemplary embodiment of the present invention. [Figure 4] FIG. 11 is a flow diagram illustrating the flow of an evaluation method according to an exemplary embodiment 2 of the present invention. [Diagram 5] FIG. 11 is a schematic diagram showing an example of embedding in exemplary embodiment 2 of the present invention. [Figure 6] FIG. 11 is a flow chart showing a detailed flow of a generation process in the exemplary embodiment 2 of the present invention. [Figure 7] FIG. 11 is a schematic diagram showing an example of a matrix in exemplary embodiment 2 of the present invention. [Figure 8] FIG. 11 is a schematic diagram showing a specific example of pixel value allocation in exemplary embodiment 2 of the present invention. [Figure 9] FIG. 11 is a schematic diagram showing an example of image data in the second exemplary embodiment of the present invention. [Figure 10] FIG. 11 is a flow chart showing a detailed flow of a detection process in the second exemplary embodiment of the present invention. [Figure 11] FIG. 11 is a schematic diagram showing an example of a first boundary in the second exemplary embodiment of the present invention. [Figure 12] FIG. 11 is a flow chart showing a detailed flow of an evaluation process in the second exemplary embodiment of the present invention. [Figure 13] FIG. 11 is a diagram showing image data in a specific example of exemplary embodiment 2 of the present invention. [Figure 14] A figure showing a first boundary and a second boundary in a specific example of exemplary embodiment 2 of the present invention. [Figure 15]FIG. 2 is a diagram illustrating an example of a hardware configuration of an evaluation device according to each exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present inventors have focused on the fact that the causes of poor performance of a language processing model can be narrowed down according to the quality of the embedding layer included in the language processing model, and have invented an evaluation device for evaluating the quality of an embedding layer. If the performance of a language processing model is poor despite the quality of the embedding layer being good, the cause is likely to be in the language processing task layer (e.g., hyperparameters). On the other hand, if the quality of the embedding layer is poor, there is a high possibility that there is a problem with the training data, pre-learning model, or learning algorithm involved in the generation process of the embedding layer. In this way, by using the evaluation device according to the present invention, the causes of poor performance of a language processing model can be narrowed down according to the evaluation result of the quality of the embedding layer.

[0011] Example Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0012] (Configuration of Evaluation Device 1) The configuration of an evaluation device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the evaluation device 1. As shown in Fig. 1, the evaluation device 1 includes an acquisition unit 11, a generation unit 12, a detection unit 13, and an evaluation unit 14.

[0013] The acquisition unit 11 acquires embeddings generated using an embedding layer included in a language processing model for natural language sentences included in each of a plurality of training data pieces. The generation unit 12 generates image data by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value. The detection unit 13 detects characteristics of the image data. The evaluation unit 14 evaluates the quality of the embedding layer based on the characteristics of the image data.

[0014] (Example of program implementation) When the evaluation device 1 is configured by a computer including at least one processor and a memory, the memory stores the following program according to this exemplary embodiment. The program causes the computer to function as an acquisition unit 11 that acquires embeddings generated by using an embedding layer included in a language processing model for natural language sentences included in each of a plurality of training data pieces, a generation unit 12 that generates image data by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value, a detection unit 13 that detects features of the image data, and an evaluation unit 14 that evaluates the quality of the embedding layer based on the features of the image data.

[0015] (Evaluation method S1 flow) The evaluation device 1 configured as above executes the evaluation method S1 according to this exemplary embodiment. The flow of the evaluation method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the evaluation method S1. As shown in Fig. 2, the evaluation method S1 includes steps S11 to S14.

[0016] In step S11, the at least one processor obtains embeddings generated using an embedding layer included in the language processing model for natural language sentences included in each of the multiple training data pieces. In step S12, the at least one processor generates image data by converting each element of a matrix including each of the multiple embeddings as a row or column into a pixel value. In step S13, the at least one processor detects features of the image data. In step S14, the at least one processor evaluates the quality of the embedding layer based on the features of the image data.

[0017] As described above, in this exemplary embodiment, for natural language sentences included in each of a plurality of training data pieces, an embedding generated using an embedding layer included in a language processing model is obtained, image data is generated by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value, features of the image data are detected, and the quality of the embedding layer is evaluated based on the features of the image data. Therefore, according to this exemplary embodiment, it is possible to obtain an effect that it is possible to narrow down the causes of poor quality of the language processing model according to the evaluation result of the embedding layer.

[0018] Exemplary embodiment 2 An evaluation device 1A according to a second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0019] (Configuration of Evaluation Device 1A) Fig. 3 is a block diagram showing a configuration of an evaluation system 10 including an evaluation device 1A. As shown in Fig. 3, the evaluation system 10 includes the evaluation device 1A, a learning device 2, and a display device 3. The evaluation device 1A and the learning device 2 may be connected to each other so as to be able to communicate with each other, but they do not have to be connected to each other so as to be able to communicate with each other. The display device 3 is connected to the evaluation device 1A and displays information under the control of the evaluation device 1A.

[0020] The evaluation device 1A is a device that evaluates the quality of an embedding layer included in a language processing model M2. In this exemplary embodiment, a deep learning model that executes a classification task of a natural language sentence is applied as the language processing model M2. The language processing model M2 is trained to output a classification of a natural language sentence when the natural language sentence is input.

[0021] The learning device 2 is a device that generates a language processing model M2 by performing fine tuning on a pre-learning model M1 using a training dataset DS1. The training dataset DS1 and the language processing model M2 are stored in a storage device (not shown) accessible from the evaluation device 1A. In this exemplary embodiment, an example will be described in which the evaluation device 1A refers to the training dataset DS1 used in generating the language processing model M2. However, a part or all of the training dataset DS1 referred to by the evaluation device 1A does not necessarily have to be the training dataset used in generating the language processing model M2, and may be a training dataset for evaluation.

[0022] The pre-training model M1 takes a natural language sentence as input and outputs its embedding. The embedding represents the natural language sentence as a vector in a high-dimensional feature space. Each element of the embedding is a numerical value, that is, the embedding represents the natural language sentence as a numerical value. Examples of the pre-training model M1 include BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and T5 (Text-to-Text Transfer Transformer). For example, the embedding output by Japanese BERT (bert-base-japanese-whole-word-masking) is a 768-dimensional vector. Also, the embedding output by Japanese GPT2 (Japanese-gpt2-medium) is a 1024-dimensional vector. Also, the embedding output by Japanese T5 (t5-base-japanese) is a 768-dimensional vector. However, the pre-training model M1 is not limited to the above-mentioned specific examples.

[0023] The training data set DS1 includes a plurality of training data fragments. The training data fragments include information (hereinafter also referred to as a pair) that associates a natural language sentence with a label indicating a classification of the natural language sentence. As an example, the label "TRUE" indicates a classification that the natural language sentence associated with the label is correct information, and the label "FALSE" indicates a classification that the natural language sentence associated with the label is incorrect information. In this case, examples of the training data fragments include a pair of the label "TRUE" and the natural language sentence "The capital of Japan is Tokyo", a pair of the label "FALSE" and the natural language sentence "Osaka is the prefecture with the largest population in Japan", etc. Note that the types and number of labels that can be included in the training data fragments are not limited to the above examples.

[0024] The language processing model M2 is a model generated by fine-tuning the pre-learning model M1 by the learning device 2 using the training data set DS1. The language processing model M2 includes an embedding layer L1 and a classification layer L2. The embedding layer L1 is a layer that receives a natural language sentence as input and outputs an embedding. For the same natural language sentence as input, the embedding output from the embedding layer L1 and the embedding output from the pre-learning model M1 may not be the same and may be different. This is because the embedding layer L1 is fine-tuned based on the pre-learning model M1. The classification layer L2 is a layer that receives an embedding as input and outputs a classification. The language processing model M2 is trained so that when a natural language sentence of a training data piece is input to the embedding layer L1, a label associated with the natural language sentence is output from the classification layer L2.

[0025] The evaluation device 1A includes an acquisition unit 11A, a generation unit 12A, a detection unit 13A, an evaluation unit 14A, and an output unit 15A. The acquisition unit 11A acquires embeddings generated for natural language sentences included in each training data piece of a training dataset DS1 using an embedding layer L1 included in a language processing model M2.

[0026] The generating unit 12A generates image data by converting each element of a matrix into a pixel value from a matrix in which each embedding is arranged as a row or column in an order based on the label. For example, the generating unit 12A may generate image data from a matrix in which a plurality of rows included in the matrix are rearranged based on a first representative value calculated from each element of each row. Also, for example, the generating unit 12A may generate image data from a matrix in which a plurality of columns included in the matrix are rearranged based on a second representative value calculated from each element of each column. Also, for example, the generating unit 12A may convert each element of the matrix into a pixel value by assigning pixel values ​​that represent a range between a plurality of colors in a predetermined number of gradations to elements of the matrix from the minimum value to the maximum value.

[0027] The detection unit 13A detects a first boundary based on a change in pixel value as a feature of the image data. For example, the detection unit 13A may detect a candidate parallel to a row or column direction corresponding to embedding in the image data as the first boundary among candidates for the first boundary detected based on a change in pixel value. Also, for example, the detection unit 13A may detect a parallel line as the first boundary when the total length of one or more parallel candidates included in a parallel line parallel to one side of the image data is equal to or greater than a predetermined ratio of the length of the side.

[0028] The evaluation unit 14A evaluates the quality of the embedding layer based on a comparison result between the first boundary and a second boundary indicating a boundary between rows or columns corresponding to embeddings having different labels in the image data. For example, the evaluation unit 14A may evaluate the quality of the embedding layer based on a recall indicating a ratio of the number of first boundaries determined to match the second boundaries to the number of second boundaries. Also, for example, the evaluation unit 14A may evaluate the quality of the embedding layer based on a detection magnification indicating a ratio of the number of first boundaries to the number of second boundaries. The output unit 15A outputs one or both of the evaluation result by the evaluation unit 14A and the image data generated by the generation unit 12A.

[0029] Moreover, when neither the first evaluation result nor the second evaluation result obtained using the same training data set DS1 meets the judgment criterion, the evaluation unit 14A evaluates that the quality of the training data set DS1 does not meet the criterion. Here, the first evaluation result is an evaluation result of the quality of the embedding layer obtained by applying the language processing model M2 as the language processing model and causing the acquisition unit 11A, the generation unit 12A, the detection unit 13A, and the evaluation unit 14A to function. Moreover, the second evaluation result is an evaluation result of the quality of the embedding layer obtained by applying the language processing model M2-1 as the language processing model and causing the acquisition unit 11A, the generation unit 12A, the detection unit 13A, and the evaluation unit 14A to function. The language processing model M2-1 is a model different from the language processing model M2. For example, the language processing model M2-1 may be a model generated by fine-tuning a pre-learning model M1-1 different from the pre-learning model M1 using the same training data set DS1.

[0030] (Evaluation method S1A flow) The evaluation device 1A configured as above executes the evaluation method S1A. The flow of the evaluation method S1A will be described with reference to Fig. 4. Fig. 4 is a flow diagram illustrating the flow of the evaluation method S1A. As shown in Fig. 4, the evaluation method S1A includes steps S11A to S15A.

[0031] In step S11A, the acquisition unit 11A acquires embeddings of natural language sentences included in each training data piece of the training data set DS1, using the embedding layer L1 included in the language processing model M2.

[0032] FIG. 5 is a schematic diagram showing an example of embedding acquired in step S11A. The training data set DS1 shown in FIG. 5 includes training data fragments D1 to D8. For example, the training data fragment D1 includes a pair of natural language sentence labels "Label 1" and "Text 1". The label "Label 1" indicates the classification of the natural language sentence "Text 1". The embedding E1 is an embedding obtained by inputting the natural language sentence "Text 1" into the embedding layer L1 of the language processing model M2, and each element is represented by a numerical value. The training data fragments D2 to D8 and the embeddings E2 to E8 are also described in the same manner as the training data fragment D1 and the embedding E1. Hereinafter, for the sake of simplicity, the "embedding acquired from the natural language sentence associated with the label" is also described as the "embedding associated with the label" or the like.

[0033] In step S12A shown in Fig. 4, the generating unit 12A generates image data by converting each element of a matrix including each embedding as a row or column into a pixel value. Fig. 6 is a flow diagram showing a detailed flow of the generating process in step S12A. As shown in Fig. 6, the generating process (step S12A) includes steps S21 to S26.

[0034] In step S21, the generating unit 12A generates a matrix including each embedding as a row or column. Here, an example of generating a matrix including each embedding as a row will be described below. Note that the case of creating a matrix including each embedding as a column will be described in the same manner in the following description by replacing "row" and "column" with each other.

[0035] In step S22, the generation unit 12A rearranges the multiple rows based on the labels. In step S23, when there are multiple embeddings associated with the same label, the generation unit 12A may rearrange the multiple rows corresponding to the multiple embeddings based on a first representative value calculated from each row. In this way, the multiple rows with the same label are rearranged based on the first representative value.

[0036] Specific examples of the first representative value include, but are not limited to, the average, the variance, and a value obtained by dimensional compression. In addition to the first representative value, a third representative value of a type different from the first representative value may be used. For example, the generation unit 12A may perform rearrangement by referring to the label, the first representative value, and the third representative value in this order. In this case, multiple rows having the same label and first representative value are rearranged based on the third representative value. Note that the number of types of representative values ​​to be referred to next to the label in order to rearrange multiple rows is not limited to one or two, and may be three or more. In addition, the order in which multiple types of representative values ​​are referred to in order to rearrange rows is not limited to the above-mentioned example.

[0037] In step S24, the generation unit 12A rearranges the multiple columns included in the matrix. As an example, the generation unit 12A rearranges the multiple columns based on the second representative value calculated from each column.

[0038] A specific example of the second representative value is the same as the first representative value described in step S22. The first representative value and the second representative value may be the same type or different types. In addition to the second representative value, a fourth representative value of a type different from the second representative value may be used. For example, the generation unit 12A may refer to the second representative value and the fourth representative value in this order for multiple columns to perform rearrangement. In this case, multiple columns having the same second representative value are rearranged based on the fourth representative value. Note that the number of types of representative values ​​referred to in order to rearrange multiple columns is not limited to one or two, and may be three or more. Also, the order in which multiple types of representative values ​​are referred to in order to rearrange columns is not limited to the above example.

[0039] The processing of steps S23 and S24 is not limited to being performed in this order, and may be performed in the reverse order or in parallel. Also, the processing of steps S23 and S24 does not necessarily have to be performed. In other words, it is sufficient that the matrix is ​​rearranged at least based on the labels.

[0040] FIG. 7 is a schematic diagram showing an example of a matrix rearranged based on labels. As shown in FIG. 7, matrix A is a matrix with 8 rows and 7 columns including rows corresponding to the embeddings E1 to E8 shown in FIG. 5, respectively. Note that since the embeddings E1 to E8 are assumed to be seven-dimensional here, the number of columns is 7. These multiple rows are rearranged based on the labels "label 1", "label 2", and "label 3" associated with the embeddings corresponding to each row. In this example, rearrangement based on the first representative value of the multiple rows and rearrangement based on the second representative value of the multiple columns are not performed.

[0041] In step S25 shown in FIG. 6, the generation unit 12A assigns pixel values ​​representing a predetermined number of gradations between multiple colors to the minimum to maximum elements of the matrix. For example, the multiple colors may be two colors, or three or more colors. The minimum and maximum values ​​that each element of the embedding obtained from the embedding layer L1 can take are determined according to the pre-learning model M1. FIG. 8 is a schematic diagram showing a specific example of pixel value assignment. In FIG. 8, the minimum value of the matrix element is "-1" and the maximum value is "1".

[0042] Mapping example 1 shows an example in which white is assigned to the minimum value "-1" and black is assigned to the maximum value "1", and pixel values ​​(so-called grayscale pixel values) that represent a predetermined number of gradations from white to black are assigned to numerical values ​​from -1 to 1. Mapping example 1 shows an example in which white to black are represented in 11 gradations, but the number of gradations is not limited to this. For example, the number of gradations from white to black may be 256 gradations.

[0043] Moreover, mapping example 2 shows an example in which blue is assigned to the minimum value "-1", white is assigned to the intermediate value "0", and red is assigned to the maximum value "1". In this example, pixel values ​​expressing blue to white in a predetermined number of gradations are assigned to numerical values ​​from -1 to 0, and pixel values ​​expressing white to red in a predetermined number of gradations are assigned to numerical values ​​from 0 to 1. Mapping example 2 shows an example in which blue to white are expressed in 6 gradations and white to red are expressed in 6 gradations, thereby expressing the whole in 11 gradations, but the number of gradations is not limited to this. For example, if blue to white are expressed in 256 gradations and white to red are expressed in 256 gradations, pixel values ​​of 512 gradations are assigned from the minimum value "-1" to the maximum value "1".

[0044] Note that the colors used for allocating pixel values ​​are not limited to the two or three colors shown in Fig. 8, but may be four or more colors. Also, the types of colors used for allocation are not limited to the above-mentioned examples.

[0045] In step S26 shown in FIG. 6, the generating unit 12A converts each element of the matrix into a pixel value according to the pixel value allocation calculated in step S25. In this way, image data is generated. FIG. 9 is a schematic diagram showing an example of image data generated from a matrix including embeddings in each row. In FIG. 9, image data IMG1 is grayscale image data generated by performing a conversion process according to mapping example 1 shown in FIG. 8 on each element included in matrix A shown in FIG. 7. The image data IMG1 has 8 training data pieces as the number of pixels in the vertical direction, and 7 embedding dimensions as the number of pixels in the horizontal direction. In FIG. 9, the numerical values ​​included in the cells indicating each pixel indicate the elements of matrix A corresponding to the pixel for ease of understanding, and do not indicate that the image data IMG1 includes a figure having a numerical shape.

[0046] In the image data generated in this way, the number of vertical pixels is the number of training data pieces, and the number of horizontal pixels is the number of dimensions of embedding. The generating unit 12A may perform the process of step S13A and subsequent steps using image data obtained by enlarging the generated image data. For example, in the example of FIG. 9, image data having a vertical and horizontal number of pixels of 8×7 may be enlarged twice to 16×14. This may facilitate the detection process of the first boundary compared to the case where the image data is not enlarged. In addition, when enlarging the image data, the aspect ratio does not necessarily need to be maintained. For example, when the number of vertical pixels is smaller (or larger) than the number of horizontal pixels, the vertical enlargement ratio may be larger (or smaller) than the horizontal enlargement ratio. The magnification ratio when enlarging the image data is not limited to the above example. This concludes the detailed description of step S12A shown in FIG. 6.

[0047] In step S13A shown in Fig. 6, the detection unit 13A detects a first boundary based on a change in pixel value as a feature of the image data. Fig. 10 is a flow diagram showing a detailed flow of the detection process in step S13A. As shown in Fig. 10, the detection process (step S13A) includes steps S31 to S33.

[0048] In step S31, the detection unit 13A detects candidates for the first boundary based on changes in pixel values ​​in the image data. Here, the first boundary to be detected is a boundary that matches a linear second boundary described later. Therefore, the first boundary to be detected is linear. Therefore, for example, a known technique for detecting a straight line based on changes in pixel values ​​from image data can be adopted for the process of detecting candidates for the first boundary. Known techniques include, but are not limited to, LSD (Line Segment Detector) and Hough transformation. The detection unit 13A detects one or more line segments as candidates for the first boundary.

[0049] In step S32, the detection unit 13A deletes candidates (other than parallel lines) that are determined not to be parallel to the row direction of the image data from among the one or more detected candidates for the first boundary. For example, the detection unit 13A may determine that a candidate whose angle with the row direction is greater than θ is not parallel to the row direction, with an allowable range of inclination up to an angle θ with respect to the row direction, and delete it from the candidates. As a result, all of the one or more candidates for the first boundary are parallel to the row direction. This is because the first boundary to be detected is a boundary that matches a second boundary (described later) that is parallel to the row direction corresponding to the embedding.

[0050] In step S33, when a parallel line parallel to one side of the image data includes one or more first boundary candidates and the total length of the one or more first boundary candidates is equal to or greater than a predetermined ratio of the length of the side, the detection unit 13A detects the parallel line as a first boundary. For example, the predetermined ratio may be 10% of the number of pixels on one side in the row direction of the image data (corresponding to the number of dimensions of embedding when the image data is not enlarged). For example, if the number of dimensions of embedding is 700, the parallel line is detected as a first boundary if the total length of the first boundary candidates included in the parallel line is equal to or greater than 70 pixels.

[0051] FIG. 11 is a schematic diagram showing an example of a first boundary detected in image data. In FIG. 11, candidates a11 and a12 of the first boundary are included in a parallel line A1 parallel to the row direction of the image data IMG1. The total length of the candidates a11 and a12 is equal to or greater than a predetermined ratio (for example, 10%) of the length d of one side parallel to the row direction of the image data IMG1. Therefore, the detection unit 13A detects the parallel line A1 as the first boundary A1. In FIG. 11, candidate a13 of the first boundary is included in a parallel line A2, and the length of candidate a13 is equal to or greater than a predetermined ratio of the length d of one side. Therefore, the parallel line A2 is detected as the first boundary A2. Similarly, the parallel line A3 including candidate a14 is detected as the first boundary A3, and the parallel line A4 including candidate a15 is detected as the first boundary A4.

[0052] In addition, in step S31, the detection unit 13A may detect a line segment having a length equal to or greater than a predetermined ratio of the length of one side parallel to the row direction as a candidate for the first boundary. In this case, the process of step S33 can be omitted. This concludes the detailed description of step S13A shown in FIG. 6.

[0053] In step S14A shown in Fig. 6, the evaluation unit 14A evaluates the quality of the embedding layer based on the comparison result between the first boundary and the second boundary. Fig. 12 is a flow diagram showing a detailed flow of the evaluation process in step S14A. As shown in Fig. 12, the evaluation process (step S14A) includes steps S41 to S44.

[0054] In step S41, the evaluation unit 14A determines whether or not each of the one or more first boundaries detected in step S13A matches any of the one or more second boundaries. Here, an example is described in which the image data is generated from a matrix including embeddings as rows. Therefore, the second boundary indicates a boundary between rows in the image data that correspond to embeddings whose labels are different from each other. A specific example of the process of step S41 will be described with reference to FIG. 11.

[0055] In the image data IMG1 shown in FIG. 11, the first to third rows correspond to an embedding associated with label 1. The fourth to sixth rows correspond to an embedding associated with label 1. The seventh to eighth rows correspond to an embedding associated with label 1. Therefore, the line indicating the gap between the third row corresponding to label 1 and the fourth row corresponding to label 2 is the second boundary B1. The line indicating the gap between the sixth row corresponding to label 2 and the seventh row corresponding to label 3 is the second boundary B2. The evaluation unit 14A determines that there is no second boundary that matches the first boundary A1. Furthermore, the evaluation unit 14A determines that the first boundary A2 matches the second boundary B1. Furthermore, the evaluation unit 14A determines that there is no second boundary that matches the first boundary A3. Furthermore, the evaluation unit 14A determines that the first boundary A4 matches the second boundary B2.

[0056] In step S42, the evaluation unit 14A calculates the recall. Here, the recall is the ratio of the number of first boundaries determined to match the second boundaries to the number of second boundaries. In the example of FIG. 11, the number of first boundaries determined to match is the ratio of the number of first boundaries determined to match the second boundaries to the number of second boundaries. The number of first boundaries is two, A2 and A4, and the number of second boundaries is two, B1 to B2. Therefore, the recall rate is calculated as 1.

[0057] In step S43, the evaluation unit 14A calculates the detection magnification. Here, the detection magnification is the ratio of the number of first boundaries to the number of second boundaries. In the example of Fig. 11, the number of first boundaries is four, A1 to A4, and the number of second boundaries is two, B1 to B2. Therefore, the detection magnification is calculated as 2.

[0058] In step S44, the evaluation unit 14A refers to the recall rate and the detection rate, and evaluates the quality of the embedding layer based on the judgment criteria. For example, the judgment criteria may be a criterion that both the recall rate is equal to or greater than a first threshold and the detection rate is equal to or less than a second threshold. In this case, the evaluation result may indicate whether the quality is good or bad depending on whether the judgment criteria are satisfied. When 1 is applied as the first threshold and 50 is applied as the second threshold, in the example of FIG. 11, the recall rate of 1 and the detection rate of 2 both satisfy the judgment criteria, so that the quality is judged to be good. The judgment criteria are not limited to the above example. The evaluation result is not limited to the two values ​​of good and bad, and may be expressed by three or more levels (for example, good, normal, bad, etc.) or a score indicating the quality. This concludes the detailed description of step S14A shown in FIG. 6.

[0059] In step S15A shown in FIG. 6, the output unit 15A outputs one or both of the evaluation result by the evaluation unit 14A and the image data generated by the generation unit 12A to the display device 3. For example, the evaluation result to be output may include information indicating whether the quality of the embedding layer is good or bad. An example of information indicating that the quality of the embedding layer is good is text information such as "The embedding layer of this language processing model has been determined to be appropriate." An example of information indicating that the quality of the embedding layer is poor is text information such as "The embedding layer of this language processing model has been determined to be inappropriate. Please check the training dataset and consider changing the pre-learning model." The evaluation result to be output may also include information indicating the recall rate and the detection magnification. The evaluation result to be output may also include information indicating the judgment criteria used to judge whether the quality of the embedding layer is good or bad.

[0060] Moreover, lines indicating the first boundary and the second boundary may be superimposed on the image data to be output. Moreover, the first boundary superimposed on the image data to be output may be in a display manner in which it is possible to distinguish whether or not it matches the second boundary. Moreover, the second boundary superimposed on the image data to be output may be in a display manner in which it is possible to distinguish whether or not it matches the first boundary. The distinguishable display manner may be, for example, a manner in which the color, thickness, type of line, etc. are different. This concludes the explanation of the evaluation method S1A.

[0061] (A variation of evaluation method S1A) The evaluation method S1A can be modified as follows. When the evaluation device 1A determines that the evaluation result of the embedding layer quality obtained by executing steps S11A to S14A for the first time (hereinafter referred to as the first evaluation result) does not satisfy the judgment criterion, the evaluation device 1A may execute steps S11A to S14A for the second time. In steps S11A to S14A for the second time, the same training dataset DS1 as in the first time and a language processing model M2-1 different from that in the first time are used. As described above, the language processing model M2-1 may be a model generated using the same training dataset DS1 based on a pre-learning model M1-1 different from the pre-learning model M1. By executing steps S11A to S14A for the second time, an evaluation result of the embedding layer quality (hereinafter referred to as the second evaluation result) is obtained.

[0062] In this modification, after executing step S14A for the second time, the evaluation unit 14A judges whether the second evaluation result satisfies the judgment criterion. If the second evaluation result does not satisfy the judgment criterion like the first evaluation result, the evaluation unit 14A evaluates that the quality of the training data set DS1 does not satisfy the criterion. In other words, the cause of poor performance of the language processing models M2 and M2-1 is narrowed down to the training data set DS1.

[0063] In addition, when the first evaluation result does not satisfy the judgment criterion, the evaluation unit 14A may execute steps S11A to S14A multiple times (for example, the second time, the third time, ...) without being limited to the second time. For example, when the first evaluation result does not satisfy the judgment criterion, the evaluation unit 14A may obtain two or more evaluation results (a second evaluation result, a third evaluation result, ...) using the same training data set DS1 and two or more different language processing models (M2-1, M2-2, ...). For example, the multiple language processing models (M2, M2-1, M2-2, ...) may be obtained by fine-tuning a plurality of pre-learning models (M1, M1-1, M1-2, ...) different from each other using the same training data set DS1. In this case, the evaluation unit 14A may evaluate the quality of the training data set DS1 based on the statistics of three or more evaluation results (a first evaluation result, a second evaluation result, a third evaluation result, ...). For example, when a predetermined ratio or more of the three or more evaluation results do not satisfy the standard, the evaluation unit 14A may evaluate that the quality of the training dataset DS1 does not satisfy the standard.

[0064] In this case, the output unit 15A may output information indicating that the quality of the training data set DS1 does not meet the standard in step S15A. An example of such information is text information such as "It has been determined that none of the embedding layers of multiple language processing models generated using the same training data set is appropriate. The cause may be the training data set. Please check."

[0065] Furthermore, when the first evaluation result satisfies the judgment criterion, the evaluation device 1A may execute step S15A in the same manner as described above without executing the second or subsequent times. Furthermore, the evaluation device 1A may execute the second or subsequent times regardless of the first evaluation result.

[0066] (Specific examples) Here, a specific example of executing the evaluation method S1A will be described with reference to Fig. 13 to Fig. 14. In this specific example, Japanese BERT is used as the pre-training model M1, and a language processing model M2 generated using a training dataset DS consisting of 2749 training data pieces is targeted. In this example, the embedding output by the pre-training model M1 is 768-dimensional, so the embedding obtained from the embedding layer L1 is also 768-dimensional.

[0067] Fig. 13 is a diagram showing image data generated in step S12A in this specific example. In Fig. 13, image data IMG101-IMG107 are generated from matrices whose rows contain 768-dimensional embeddings obtained from 2749 pieces of training data, respectively, and therefore have a pixel count of 2749 x 768. In addition, image data IMG101-IMG107 are generated by performing conversion processing on the matrices according to mapping example 2 shown in Fig. 8, and therefore are three-color color scale image data.

[0068] Here, the image data IMG101 is image data generated from a matrix in which rows have been rearranged based on the labels. In the image data IMG101, no rearrangement of rows and columns has been performed other than the rearrangement of rows based on the labels.

[0069] The image data IMG102 to IMG103 are generated from matrices in which the rows are rearranged based on the labels, and the columns are also rearranged. In the image data IMG102, the columns are rearranged based on the average of each column (an example of a second representative value). In the image data IMG103, the columns are rearranged based on the variance of each column (an example of a second representative value).

[0070] The image data IMG104 to IMG106 are generated from matrices in which rows are rearranged based on the label, as well as multiple rows with the same label. In the image data IMG104, multiple rows with the same label are rearranged based on the average of each row (an example of a first representative value). In the image data IMG105, multiple rows with the same label are rearranged based on the variance of each row (an example of a first representative value).

[0071] In image data IMG106, multiple rows with the same label are rearranged based on a value (an example of a first representative value) calculated from each row using t-distributed Stochastic Neighbor Embedding (t-SNE). Hereinafter, the value calculated using t-SNE is also referred to as the t-SNE value. In this example, the t-SNE value is a value obtained by compressing 768-dimensional rows to one dimension using t-SNE. However, when rearranging using the t-SNE value, the number of dimensions after compression and which element among them is used for rearrangement are not limited to the above example.

[0072] Although not shown in Fig. 12, rearrangement based on the t-SNE value may be applied not only to rows but also to columns. In other words, image data may be generated from a matrix in which, in addition to rearrangement of rows based on labels, rearrangement of columns is performed based on values ​​(an example of second representative values) calculated from each column by t-SNE.

[0073] The image data IMG107 is generated from a matrix in which, in addition to rearranging the rows based on the labels, multiple rows with the same labels are rearranged based on the variance of each row (an example of a first representative value), and columns are rearranged based on the average of each column (an example of a second representative value).

[0074] In addition, when both row rearrangement and column rearrangement are performed to generate image data, the combination of the first representative value and the second representative value is not limited to the above example. For example, although not shown in Fig. 12, image data may be generated from a matrix in which multiple rows with the same label are rearranged based on one of the three types of "average, variance, t-SNE value" and columns are rearranged based on one of the three types of "average, variance, t-SNE value".

[0075] FIG. 14 is a diagram showing the first boundary and the second boundary in this specific example. In FIG. 14, the line segment shown inside the image data IMG107 indicates a candidate for the first boundary. In this example, the first boundaries A101 to A117 are detected based on such a candidate for the first boundary. In addition, the second boundaries B101 to B103 on the image data IMG107 indicate lines between rows with different labels. Among the first boundaries A101 to A117, the first boundary A102 matches the second boundary B101. The first boundary A103 matches the second boundary B102. The first boundary A117 matches the second boundary B103. The recall rate is calculated as "1" as the ratio of the number of matches "3" to the number of second boundaries "3". The detection magnification rate is calculated as approximately "5.7" as the ratio of the number of first boundaries "17" to the number of second boundaries "3". When the criteria used are "recall rate of 1 or more and detection rate of 50 or less," the quality of the embedding layer L1 is evaluated as good.

[0076] (Examples) The evaluation device 1A according to this exemplary embodiment can be used, for example, to narrow down the cause of poor performance of a language processing model that classifies electronic medical records in an electronic medical record system for medical use.

[0077] (Effects of this exemplary embodiment) As described above, in the evaluation device 1A according to this exemplary embodiment, in addition to the same configuration as that of exemplary embodiment 1, the language processing model is a model that executes a classification task, the training data pieces include information associating natural language sentences with labels indicating the classification of the natural language sentences, the generation unit 12A generates image data from a matrix in which each of the multiple embeddings is arranged as a row or column in order based on the labels, the detection unit 13A detects a first boundary based on changes in pixel values ​​as a feature of the image data, and the evaluation unit 14A evaluates the quality of the embedding layer based on the comparison result between the first boundary and a second boundary indicating between rows or columns corresponding to embeddings in the image data whose labels differ from each other.

[0078] Therefore, according to the present exemplary embodiment, when the performance of a language processing model performing a classification task is poor and the quality of the embedding layer included in the language processing model is poor, it is found that the training data set, the pre-learning model, or the learning algorithm is likely to be the cause. Also, when the quality of the embedding layer is good, it is considered that the quality of the classification layer is poor, and it is found that the hyperparameters are likely to be the cause. As a result, when the performance of a language processing model performing a classification task is poor, the cause can be narrowed down.

[0079] In addition, in the evaluation device 1A according to the present exemplary embodiment, the evaluation unit 14A is configured to evaluate the quality of the embedding layer based on a recall ratio indicating the ratio of the number of first boundaries determined to match the second boundaries to the number of second boundaries. Therefore, according to the present exemplary embodiment, in addition to the above-mentioned effects, the quality of the embedding layer can be evaluated with higher accuracy.

[0080] In addition, in the evaluation device 1A according to the present exemplary embodiment, the evaluation unit 14A is configured to evaluate the quality of the embedding layer based on a detection magnification indicating the ratio of the number of first boundaries to the number of second boundaries. Therefore, according to the present exemplary embodiment, in addition to the above-mentioned effects, the quality of the embedding layer can be evaluated with higher accuracy.

[0081] In addition, in the evaluation device 1A according to the present exemplary embodiment, the generating unit 12A generates the above-mentioned image data from a matrix in which a plurality of rows included in the above-mentioned matrix are rearranged based on a first representative value calculated from each element of each row. Therefore, according to the present exemplary embodiment, in addition to the above-mentioned effects, the first boundary that matches the second boundary can be detected with higher accuracy.

[0082] In addition, in the evaluation device 1A according to the present exemplary embodiment, the generating unit 12A generates image data from a matrix in which a plurality of columns included in the above-mentioned matrix are rearranged based on a second representative value calculated from each element of each column. Therefore, according to the present exemplary embodiment, in addition to the above-mentioned effects, the first boundary that matches the second boundary can be detected with higher accuracy.

[0083] In addition, in the evaluation device 1A according to this exemplary embodiment, the first evaluation result is the evaluation result of the quality of the embedding layer obtained by applying the language processing model M2 as the language processing model and operating the acquisition unit 11A, the generation unit 12A, the detection unit 13A, and the evaluation unit 14A. In addition, the second evaluation result is the evaluation result of the quality of the embedding layer obtained by applying a language processing model M2-1 different from the language processing model M2 as the language processing model and operating the acquisition unit 11A, the generation unit 12A, the detection unit 13A, and the evaluation unit 14A. In addition, the evaluation unit 14A is configured to evaluate that the quality of the training data set DS1 does not meet the standard when neither the first evaluation result nor the second evaluation result obtained using the same training data set DS1 meets the standard. Therefore, according to this exemplary embodiment, the cause of poor performance of the language processing model can be narrowed down to the training data set. The effect of obtaining the above can be obtained.

[0084] In addition, in the evaluation device 1A according to this exemplary embodiment, the generating unit 12A converts each element of the matrix into a pixel value by assigning pixel values ​​that represent a range between a plurality of colors in a predetermined number of gradations to the minimum to maximum values ​​of the elements of the matrix. Therefore, according to this exemplary embodiment, in addition to the above-mentioned effects, the first boundary that matches the second boundary can be detected with higher accuracy.

[0085] Moreover, the evaluation device 1A according to this exemplary embodiment further includes an output unit 15A that outputs one or both of the evaluation result by the evaluation unit 14A and the image data generated by the generation unit 12A. Therefore, according to this exemplary embodiment, in addition to the above-mentioned effects, the user can recognize the evaluation result of the quality of the embedding layer.

[0086] In addition, in the evaluation device 1A according to this exemplary embodiment, the detection unit 13A detects, as the first boundary, a candidate parallel to the row or column direction corresponding to the embedding in the image data, among the candidates of the first boundary detected based on the change in pixel value. Therefore, according to this exemplary embodiment, in addition to the above-mentioned effects, the detection of the first boundary that does not match the second boundary can be reduced.

[0087] In addition, in the evaluation device 1A according to this exemplary embodiment, the detection unit 13A is configured to detect a parallel line as a first boundary when the total length of one or more parallel candidates included in a parallel line parallel to one side of the image data is equal to or greater than a predetermined ratio of the length of the side. Therefore, according to this exemplary embodiment, in addition to the above-mentioned effects, it is possible to reduce the detection of a first boundary that does not match the second boundary.

[0088] [Modifications] In the above-described exemplary embodiment 2, an example has been described in which the language processing model is a model that executes a classification task. However, the language processing model is not limited to a classification task, and may be a model that executes other language processing tasks.

[0089] Furthermore, in the above-described exemplary embodiment 2, the evaluation device 1A has been described as being included in the evaluation system 10, but the evaluation device 1A does not necessarily have to be included in the evaluation system 10. For example, the evaluation device 1A does not have to be able to access the training dataset DS1 itself and the embedding layer L1 itself, as long as the evaluation device 1A can access a storage device in which embeddings generated using the embedding layer L1 for each of the training data pieces included in the training dataset DS1 are stored.

[0090] [Software implementation example] A part or all of the functions of the evaluation device 1, 1A (hereinafter, referred to as the device) may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0091] In the latter case, the device is realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 15. The computer C has at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the device. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the device.

[0092] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0093] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0094] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0095] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0096] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects. (Appendix 1) An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A generating means for generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or a column into a pixel value; A detection means for detecting a characteristic of the image data; evaluation means for evaluating the quality of the embedding layer based on characteristics of the image data; 23. An evaluation device comprising:

[0097] (Appendix 2) the language processing model is a model for performing a classification task; the training data fragment includes information associating the natural language sentence with a label indicating a classification of the natural language sentence; the generating means generates the image data from the matrix in which each of the plurality of embeddings is arranged as a row or a column in an order based on the label; the detection means detects a first boundary based on a change in pixel value as a feature of the image data; The evaluation means evaluates quality of the embedding layer based on a comparison result between a second boundary indicating a boundary between rows or columns corresponding to embeddings in which the labels are different from each other in the image data and the first boundary. 2. The evaluation device according to claim 1.

[0098] (Appendix 3) the evaluation means evaluates quality of the embedding layer based on a recall indicating a ratio of the number of the first boundaries determined to match the second boundaries to the number of the second boundaries. 3. An evaluation device as described in appended claim 2.

[0099] (Appendix 4) the evaluation means evaluates quality of the embedding layer based on a detection magnification indicating a ratio of the number of the first boundaries to the number of the second boundaries. 4. The evaluation device according to claim 2 or 3.

[0100] (Appendix 5) the generating means generates the image data from a matrix obtained by rearranging a plurality of rows included in the matrix based on a first representative value calculated from each element of each row. 5. The evaluation device according to claim 1 .

[0101] (Appendix 6) the generating means generates the image data from a matrix obtained by rearranging a plurality of columns included in the matrix based on a second representative value calculated from each element of each column. 6. The evaluation device according to claim 1 ,

[0102] (Appendix 7) applying a first language processing model as the language processing model, and operating the acquisition means, the generation means, the detection means, and the evaluation means to obtain an evaluation result of the quality of the embedding layer as a first evaluation result; applying a second language processing model different from the first language processing model as the language processing model, and operating the acquisition means, the generation means, the detection means, and the evaluation means to obtain an evaluation result of the quality of the embedding layer as a second evaluation result; The evaluation means includes: When neither the first evaluation result nor the second evaluation result obtained using the same plurality of training data pieces meets a predetermined standard, the quality of the plurality of training data pieces is evaluated as not meeting the standard. 7. The evaluation device according to any one of claims 1 to 6.

[0103] (Appendix 8) the generating means converts each element of the matrix into a pixel value by assigning pixel values ​​representing a predetermined number of gradations between a plurality of colors to elements of the matrix ranging from a minimum value to a maximum value. 8. The evaluation device according to claim 1 .

[0104] (Appendix 9) Further comprising an output unit for outputting one or both of the evaluation result by the evaluation unit and the image data generated by the generation unit. 9. The evaluation device according to any one of appendixes 1 to 8.

[0105] (Appendix 10) the detection means detects, as the first boundary, a candidate that is parallel to a row or column direction corresponding to the embedding in the image data, from among the candidates for the first boundary detected based on the change in pixel value; 5. The evaluation device according to claim 2,

[0106] (Appendix 11) the detection means detects a parallel line as the first boundary when a total length of one or more parallel candidates included in a parallel line parallel to one side of the image data is equal to or greater than a predetermined ratio to the length of the side; 11. The evaluation apparatus of claim 10.

[0107] (Appendix 12) A method of evaluation executed by at least one processor, comprising: the at least one processor: Obtaining embeddings generated using an embedding layer included in a language processing model for each natural language sentence included in a plurality of training data pieces; generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value; detecting features of the image data; assessing a quality of the embedding layer based on characteristics of the image data; and (c) evaluation methods;

[0108] (Appendix 13) Computer, An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A generating means for generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or a column into a pixel value; A detection means for detecting a characteristic of the image data; and a program that functions as an evaluation means for evaluating the quality of the embedding layer based on the characteristics of the image data. [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows.

[0109] (Appendix 1) at least one processor, the at least one processor comprising: An acquisition process of acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A generation process for generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or a column into a pixel value; A detection process for detecting features of the image data; an evaluation process for evaluating the quality of the embedding layer based on characteristics of the image data; An evaluation device that performs the above.

[0110] (Appendix 2) the language processing model is a model for performing a classification task; the training data fragment includes information associating the natural language sentence with a label indicating a classification of the natural language sentence; In the generating process, the at least one processor generates the image data from the matrix in which each of the plurality of embeddings is arranged as a row or a column in an order based on the label; In the detection process, the at least one processor detects a first boundary based on a change in pixel value as a feature of the image data; In the evaluation process, the at least one processor evaluates the quality of the embedding layer based on a comparison result between a second boundary indicating a boundary between rows or columns in the image data corresponding to embeddings in which the labels are different from each other and the first boundary. 2. The evaluation device according to claim 1.

[0111] (Appendix 3) In the evaluation process, the at least one processor evaluates quality of the embedding layer based on a recall ratio indicating a ratio of a number of the first boundaries determined to match the second boundaries to a number of the second boundaries. 3. An evaluation device as described in appended claim 2.

[0112] (Appendix 4) In the evaluation process, the at least one processor evaluates quality of the embedding layer based on a detection magnification indicating a ratio of the number of the first boundaries to the number of the second boundaries. 4. The evaluation device according to claim 2 or 3.

[0113] (Appendix 5) In the generation process, the at least one processor generates the image data from a matrix in which a plurality of rows included in the matrix are rearranged based on a first representative value calculated from each element of each row. 5. The evaluation device according to claim 1 .

[0114] (Appendix 6) In the generation process, the at least one processor generates the image data from a matrix obtained by rearranging a plurality of columns included in the matrix based on a second representative value calculated from each element of each column. 6. The evaluation device according to claim 1 ,

[0115] (Appendix 7) applying a first language processing model as the language processing model, and performing the acquisition process, the generation process, the detection process, and the evaluation process to evaluate the quality of the embedding layer, the evaluation result being a first evaluation result; applying a second language processing model different from the first language processing model as the language processing model, and performing the acquisition process, the generation process, the detection process, and the evaluation process to obtain an evaluation result of quality of the embedding layer as a second evaluation result; In the evaluation process, the at least one processor When neither the first evaluation result nor the second evaluation result obtained using the same plurality of training data pieces meets a predetermined standard, the quality of the plurality of training data pieces is evaluated as not meeting the standard. 7. The evaluation device according to any one of claims 1 to 6.

[0116] (Appendix 8) In the generation process, the at least one processor converts each element of the matrix into a pixel value by assigning pixel values ​​representing a predetermined number of gradations between a plurality of colors to elements of the matrix ranging from a minimum value to a maximum value. 8. The evaluation device according to claim 1 .

[0117] (Appendix 9) The at least one processor: further executing an output process for outputting one or both of the evaluation result obtained by the evaluation process and the image data generated by the generation process. 9. The evaluation device according to any one of appendixes 1 to 8.

[0118] (Appendix 10) In the detection process, the at least one processor detects, as the first boundary, a candidate that is parallel to a row or column direction corresponding to the embedding in the image data, among the candidates of the first boundary detected based on the change in the pixel value. 5. The evaluation device according to claim 2,

[0119] (Appendix 11) In the detection process, the at least one processor detects a parallel line as the first boundary when a total length of one or more parallel candidates included in a parallel line parallel to one side of the image data is equal to or greater than a predetermined ratio of the length of the side. 11. The evaluation apparatus of claim 10.

[0120] The evaluation device according to any one of Supplementary Notes 1 to 11 in Supplementary Note 3 may further include a memory, and the memory may store a program for causing the processor to execute the acquisition process, the generation process, the detection process, and the evaluation process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0121] 1. 1A Evaluation Device 2 Learning device 3 Display device 10 Rating System 11, 11A Acquisition Department 12, 12A generation section 13, 13A Detector 14, 14A Evaluation Section 15A output section C1 Processor C2 Memory

Claims

1. An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A generating means for generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or a column into a pixel value; A detection means for detecting a characteristic of the image data; evaluation means for evaluating the quality of the embedding layer based on characteristics of the image data; 23. An evaluation device comprising:

2. the language processing model is a model that performs a classification task; the training data fragment includes information associating the natural language sentence with a label indicating a classification of the natural language sentence; the generating means generates the image data from the matrix in which each of the plurality of embeddings is arranged as a row or a column in an order based on the label; the detection means detects a first boundary based on a change in pixel value as a feature of the image data; the evaluation means evaluates quality of the embedding layer based on a comparison result between a second boundary indicating a boundary between rows or columns corresponding to embeddings in which the labels are different from each other in the image data and the first boundary; The evaluation device according to claim 1 .

3. the evaluation means evaluates quality of the embedding layer based on a recall ratio indicating a ratio of the number of the first boundaries determined to match the second boundaries to the number of the second boundaries. The evaluation device according to claim 2 .

4. the evaluation means evaluates quality of the embedding layer based on a detection magnification indicating a ratio of the number of the first boundaries to the number of the second boundaries. The evaluation device according to claim 2 or 3.

5. the generating means generates the image data from a matrix obtained by rearranging a plurality of rows included in the matrix based on a first representative value calculated from each element of each row. The evaluation device according to claim 1 .

6. the generating means generates the image data from a matrix obtained by rearranging a plurality of columns included in the matrix based on a second representative value calculated from each element of each column. The evaluation device according to claim 1 .

7. a first language processing model is applied as the language processing model, and an evaluation result of the quality of the embedding layer obtained by operating the acquisition means, the generation means, the detection means, and the evaluation means is set as a first evaluation result; applying a second language processing model different from the first language processing model as the language processing model, and operating the acquisition means, the generation means, the detection means, and the evaluation means to obtain an evaluation result of the quality of the embedding layer as a second evaluation result; The evaluation means includes: When neither the first evaluation result nor the second evaluation result obtained using the same plurality of training data pieces satisfies a predetermined standard, the quality of the plurality of training data pieces is evaluated as not satisfying the standard. The evaluation device according to claim 1 .

8. Further comprising an output unit for outputting one or both of the evaluation result by the evaluation unit and the image data generated by the generation unit. The evaluation device according to claim 1 .

9. A method of evaluation executed by at least one processor, comprising: The at least one processor: Obtaining embeddings generated using an embedding layer included in a language processing model for each natural language sentence included in a plurality of training data pieces; generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or column into a pixel value; detecting features of the image data; assessing a quality of the embedding layer based on characteristics of the image data; and (c) evaluation methods;

10. Computer, An acquisition means for acquiring embeddings generated for each natural language sentence included in a plurality of training data pieces using an embedding layer included in a language processing model; A generating means for generating image data by converting each element of a matrix including each of the plurality of embeddings as a row or a column into a pixel value; A detection means for detecting a characteristic of the image data; and a program that functions as an evaluation means for evaluating the quality of the embedding layer based on the characteristics of the image data.

Citation Information

Patent Citations

  • Model evaluation method and computer system

    JP2023019341A