Training device, evaluation device, training method, and program

Abstract

A training device according to one aspect of the present disclosure trains a model that evaluates text, in a document format, with respect to a prescribed issue. The training device includes: a first user interface provision unit that provides a first user interface capable of at least designating data for training the model; and a training unit that trains the model on the basis of the training data and index values representing the degree of contribution to the quality of the model.

Description

Training device, evaluation device, training method, evaluation method, and program

[0001] The present disclosure relates to a training device, an evaluation device, a training method, an evaluation method, and a program.

[0002] Techniques for supporting the evaluation of text written in sentence format have been known for some time. For example, Patent Literature 1 discloses a technique for grading a report written in sentence format using natural language processing. In addition, with the recent advances in machine learning technology, techniques for grading written answers and the like using machine learning models such as neural networks have also been researched.

[0003] Patent No. 6943999

[0004] However, in Patent Document 1, for example, a learning model using machine learning technology is utilized to present the evaluation basis points in accordance with the evaluation results, but it does not go so far as to indicate the reliability of the evaluation basis points, and as a result, the reliability and evaluation accuracy of the evaluation basis points and evaluation results are insufficient, and therefore it is unable to adequately support the evaluation of text written in sentence format.

[0005] The present disclosure has been made in consideration of the above points, and provides a technology that realizes quality assurance of evaluations by presenting evaluations of text written in prose together with the reliability of the evaluation grounds, thereby providing more support than ever before. Furthermore, quality assurance includes the provision of a technology that can easily realize quality improvement when the quality is low.

[0006] A training device according to one aspect of the present disclosure is a training device that trains a model that evaluates document-formatted text for a specified task, and includes a first UI providing unit that provides a first UI that allows at least specification of training data for the model, and a training unit that trains the model based on the training data and an index value that represents the contribution of the training data to the quality of the model.

[0007] A technique is provided that can support the evaluation of text written in prose form while ensuring the quality of the evaluation process.

[0008] 1 is a diagram illustrating an example of the overall configuration of a scoring system according to an embodiment; FIG. 2 is a diagram illustrating an example of the hardware configuration of a scoring device according to an embodiment; FIG. 3 is a diagram illustrating an example of the hardware configuration of a user terminal according to an embodiment; FIG. 4 is a diagram illustrating an example of the functional configuration of a scoring device according to an embodiment; FIG. 5 is a diagram illustrating an example of the functional configuration of a user terminal according to an embodiment; FIG. 6 is a diagram illustrating an example of screen transitions; FIG. 7 is a diagram illustrating an example of a menu screen; FIG. 8 is a diagram illustrating an example of a question management screen; FIG. 9 is a diagram illustrating an example of a model management screen; FIG. 10 is a diagram illustrating an example of a model creation screen; FIG. 11 is a diagram illustrating an example of a model verification screen; FIG. 12 is a diagram illustrating an example of a consistency verification support screen for scoring criteria;

[0009] An embodiment of the present invention will be described below.

[0010] <SAS (Automated Short Answer Scoring)> In the following, as an example, a written answer representing an answer to a task or problem as text written in sentence format is assumed, and a case where this written answer is evaluated by scoring is assumed. A task for performing such scoring is generally called a written answer automatic scoring task (SAS). In recent years, research has been conducted on realizing SAS using machine learning technology. Research has also been conducted on estimating the parts (words, phrases, etc.) that a machine learning model uses as the basis for scoring using a technique called Feature Attribution (hereinafter referred to as feature attribution). In the embodiment described below, a written answer automatic scoring task using feature attribution is also assumed, and this task is formulated below. Note that a written answer may be called, for example, a written answer, a written answer, or simply an answer or answer.

[0011] Here, the needs of SAS users have been explained as follows: Previously, the needs included automated grading without any time lag after answers were given, and grading support that reduces the cost and ensures the specialized staff required for grading large-scale exams and mock exams held at schools and cram schools. In recent years, however, with the practical application of automated grading, there has been an increasing need for more objective quality assurance regarding these automated grading and grading support systems. In other words, there is an increasing need for new quality assurance information regarding automated grading results to further improve trust and satisfaction in automated grading. Note that SAS users include, for example, schools and teachers who provide and grade test questions, cram schools and teachers, students and their parents who answer questions, and software companies and developers that provide SAS services.

[0012] However, the text written in a sentence format is not limited to a written answer, and the embodiments described below can be similarly applied to any text written in a sentence format for some assignment or problem. For example, the embodiments described below can be similarly applied to text such as business reports, daily sales reports, and reports.

[0013] For example, in the case of a sales daily report, the sales status of products at each retail store where the company's products are sold can be entered in descriptive form for each input field of predetermined items on a portable tablet device, and the input information can be automatically evaluated with the aim of improving sales performance. The predetermined items correspond to descriptive questions, and the input information corresponds to the answers to the descriptive questions. Also, instead of information about retail stores, it can also be sales information about existing customers (business partners) and new customers.

[0014] A written answer to a given problem (or assignment) x = (x １ , x ２ , ..., x Ｔ ), the score (graded score) predicted by the scoring model (hereinafter simply referred to as the model) f in the automatic scoring task of written answers is defined as s∈{0, 1..., S}. ｔ is the t-th word (or character) included in the written answer, and T is the length of the written answer. Note that the value of T may differ depending on the written answer.

[0015] In addition, in the feature attribution method, x = (x １ , x ２ , ..., x Ｔ ) is the model output s, the input sequence x for the score s １ , x ２ , ..., x Ｔ The contribution of (i.e., the index value representing the importance) of １ , e ２ , ..., e Ｔ ) Each contribution e ｔ The larger the value, the greater the impact on the score. ｔ In the following, the explanation series e is expressed as 0<e ｔ <1 and e １ +e ２ +...+e Ｔ = 1.

[0016] At this time, e ｍａｘ = max {e １ , e ２ , ..., e Ｔ}, and e ｍａｘ -e ｔ<th when r ｔ = 1, otherwise r ｔ = 0 and r = (r １ , r ２ , ..., r Ｔ ) is defined as follows: where th is a predetermined threshold value. １ , r ２ , ..., r Ｔ ) represents the basis location when model f outputs the scoring score, and will be referred to as the basis label or simply label hereinafter. Furthermore, hereinafter, the basis label r will be represented as "^r" by adding a hat "^" which indicates that it is an estimated value predicted by the model. Similarly, the scoring score s will be represented as "^s" by adding a hat "^" which indicates that it is an estimated value predicted by the model. Note that, although the hat "^" is technically placed directly above r or s, it will be placed immediately before r or s in the text of this specification.

[0017] <Training Data> When training a model, a set of training data (training dataset) is provided, which includes, for example, a written answer, a scoring item score assigned by a human grader according to each scoring item included in a predetermined scoring standard, the sum of these scoring item scores, and a basis label (hereinafter referred to as a gold label) that the grader used as the basis for scoring each scoring item. Here, the sum of the scoring item scores is the overall scoring score for the written answer. Hereinafter, a model that predicts the overall scoring score for a written answer is referred to as an "overall scoring model," and a model that predicts a vector whose elements are the scoring scores for each scoring item of the written answer is referred to as an "individual scoring model."

[0018] When training an overall scoring model, the sum of the scoring item scores becomes the grader's scoring score, so the sum of the scoring item scores is called the gold score. On the other hand, when training an individual scoring model, the scoring item scores become the grader's scoring score, so the scoring item scores become the gold score.

[0019] The training data described above is merely an example, and is not limited thereto. The training data may include any data necessary for model training. For example, when training an overall scoring model, the training data may not include a score for each scoring item. Similarly, when training an individual scoring model, the training data may not include a total score for each scoring item.

[0020] Additionally, one or more predetermined grading criteria are provided along with the training data (or may be separate from the training data) to be used for grading the written answers. A grading criterion (or rubric) is information that, for example, consists of one or more grading items, and each grading item defines a criterion for adding or subtracting a certain number of points if a certain word, phrase, or phrase is included in the written answer. Note that a human grader is also called a grader, and the grading criteria are also called a rubric.

[0021] In addition, when the gold label is r, if r = 0, we consider that the written answer sheet has not been assigned a gold label. This is because when a human grader grades a written answer sheet, a gold score is always obtained, but a gold label is not necessarily obtained.

[0022] Although the term "training" is used below, training may also be called, for example, "learning."

[0023] <Plausibility and Faithfulness> We introduce two concepts, plausibility and faithfulness, to improve the scoring accuracy of a model and explain its reliability (quality). The evaluation indexes for evaluating these two concepts will be described later.

[0024] Validity is a concept that expresses whether the model is able to focus on the same evidence points as a human grader. In other words, validity is a concept that expresses the consistency between the evidence label ^r output by the model and the gold label r.

[0025] The concept of "plausibility" is well known, but in the case of automatic marking of texts written in prose, the technical challenge is to further improve users' trust and satisfaction in automatic marking. Applying the concept of "plausibility" to describe the consistency between the evidence points focused on by a human marker and the evidence points focused on by the model is a new technical idea, which will provide new quality assurance information for automatic marking and improve users' trust and satisfaction in automatic marking. Furthermore, as users gain a better understanding of automatic marking, automatic marking technology for written answers will become more widespread and its use will be promoted, which will improve the efficiency of burdensome education-related work and prevent increases in the labor and costs of marking work due to a shortage of specialized personnel.

[0026] On the other hand, faithfulness is a concept that indicates whether the model scores by focusing on the evidence portion. In other words, faithfulness is a concept that indicates the consistency between the model's evidence label ^r and the scoring score ^r.

[0027] <Example of overall configuration> Fig. 1 is a diagram showing an example of the overall configuration of a scoring system 1 according to one embodiment. As shown in Fig. 1, the scoring system 1 according to one embodiment includes a scoring device 10 and one or more user terminals 20. Furthermore, the scoring device 10 and each user terminal 20 are communicatively connected via a communication network 30 including, for example, the Internet.

[0028] The scoring device 10 trains models and performs scoring using the trained models. The scoring device 10 also provides various screens to the user terminal 20 as a UI (user interface) for the user to train models and perform scoring using the trained models. The scoring device 10 is realized, for example, by a general-purpose server or a group of servers.

[0029] The user terminal 20 accepts various information for model training, grading, and the like, and transmits this information to the grading device 10. Here, users who use the user terminal 20 include users who primarily train models (including retraining) (hereinafter also referred to as model creators) and users who primarily request grading of written answers (hereinafter also referred to as model users). Model creators are those who prepare training datasets, such as model developers, academic staff (including not only school teachers but also cram school instructors), and model administrators. On the other hand, model users are those who prepare written answers, such as students who write written answers and academic staff (including not only school teachers but also cram school instructors) who collect written answers from students. However, the same user may be both a model creator and a model user. The user terminal 20 may be implemented by various devices, such as a PC (personal computer), smartphone, tablet device, or wearable device.

[0030] The overall configuration example of the scoring system 1 shown in Figure 1 is an example and is not limited to this. For example, in the example shown in Figure 1, model training and scoring are performed by the same scoring device 10, but model training may be performed by a device different from the scoring device 10. In this case, the device that performs model training may be called, for example, a "training device" or a "learning device." Furthermore, the scoring device 10 may be called, for example, an "evaluation device."

[0031] <Hardware Configuration Example> An example of the hardware configuration of the scoring device 10 and the user terminal 20 according to an embodiment will be described.

[0032] <Scoring Device 10> Fig. 2 is a diagram showing an example of the hardware configuration of the scoring device 10 according to one embodiment. As shown in Fig. 2, the scoring device 10 according to one embodiment has an external I / F 11, a communication I / F 12, a RAM (Random Access Memory) 13, a ROM (Read Only Memory) 14, an auxiliary storage device 15, and a processor 16. Each of these pieces of hardware is connected to each other via a bus 17 so as to be able to communicate with each other.

[0033] The external I / F 11 is an interface with an external device such as a recording medium 11a. The scoring device 10 can read from and write to the recording medium 11a via the external I / F 11. Examples of the recording medium 11a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0034] The communication I / F 12 is an interface for connecting the scoring device 10 to the communication network 30. The RAM 13 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 14 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 15 is a storage device (storage device) such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory. The processor 16 is an arithmetic device such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit).

[0035] The scoring device 10 according to one embodiment has the hardware configuration shown in Fig. 2 and is therefore capable of implementing various processes described below. Note that the hardware configuration shown in Fig. 2 is merely an example, and the hardware configuration of the scoring device 10 is not limited to this. For example, the scoring device 10 may have multiple auxiliary storage devices 15 or multiple processors 16, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0036] <User Terminal 20> Fig. 3 is a diagram illustrating an example of the hardware configuration of a user terminal 20 according to one embodiment. As shown in Fig. 3, the user terminal 20 according to one embodiment includes an input device 21, a display device 22, an external I / F 23, a communication I / F 24, a RAM 25, a ROM 26, an auxiliary storage device 27, and a processor 28. Furthermore, each of these pieces of hardware is connected to each other via a bus 29 so as to be able to communicate with each other.

[0037] The input device 21 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 22 is, for example, a display, a display panel, etc.

[0038] The external I / F 23 is an interface with an external device such as a recording medium 23a. The user terminal 20 can read from and write to the recording medium 23a via the external I / F 23. Examples of the recording medium 23a include a flexible disk, a CD, a DVD, an SD memory card, and a USB memory card.

[0039] The communication I / F 24 is an interface for connecting the user terminal 20 to the communication network 30. The RAM 25 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 26 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 27 is a storage device (storage device) such as an HDD, SSD, or flash memory. The processor 28 is an arithmetic device such as a CPU.

[0040] The user terminal 20 according to one embodiment has the hardware configuration shown in Fig. 3 and is thereby able to perform various processes described below. Note that the hardware configuration shown in Fig. 3 is merely an example, and the hardware configuration of the user terminal 20 is not limited to this. For example, the user terminal 20 may have multiple auxiliary storage devices 27 or multiple processors 28, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0041] <Functional Configuration Example> A functional configuration example of the scoring device 10 and the user terminal 20 according to an embodiment will be described.

[0042] <Scoring Device 10> Figure 4 is a diagram showing an example of the functional configuration of the scoring device 10 according to one embodiment. As shown in Figure 4, the scoring device 10 according to one embodiment has a UI providing unit 101, a data acquiring unit 102, a training unit 103, an evaluation unit 104, and a storage unit 105. The UI providing unit 101, the data acquiring unit 102, the training unit 103, and the evaluation unit 104 are realized, for example, by processing in which one or more programs installed in the scoring device 10 are executed by the processor 16 or the like. Furthermore, the storage unit 105 is realized, for example, by the auxiliary storage device 15 or the like.

[0043] The UI providing unit 101 provides various screens as UIs for the user to train a model and score using a trained model to the user terminal 20. The UI providing unit 101 also receives various pieces of information specified by user operations on these various screens from the user terminal 20.

[0044] The data acquisition unit 102 acquires training data sets used for model training, etc., and performs preprocessing on each training data set included in the training data sets. The data acquisition unit 102 also acquires evaluation data used for grading. The evaluation data is data including written answers to be graded by the trained model.

[0045] The training unit 103 trains a model using the training dataset preprocessed by the data acquisition unit 102. The training unit 103 also verifies consistency with the trained model using the training data included in this training dataset. Note that this consistency verification is possible even if a gold label is not assigned to the training data. Therefore, this is particularly effective when the training data included in the training dataset is not assigned a gold label or when there is extremely little training data assigned a gold label.

[0046] The evaluation unit 104 uses the evaluation data acquired by the data acquisition unit 102 and the trained model to score the written answer included in this evaluation data. The evaluation unit 104 also calculates various evaluation index values ​​related to the trained model.

[0047] The storage unit 105 stores various data (for example, trained models, training data used for consistency verification, etc.).

[0048] <<User Terminal 20>> Fig. 5 is a diagram showing an example of the functional configuration of the user terminal 20 according to one embodiment. As shown in Fig. 5, the user terminal 20 according to one embodiment has a UI unit 201 and a storage unit 202. The UI unit 201 is realized, for example, by a process in which one or more programs installed in the user terminal 20 are executed by the processor 28 or the like. The storage unit 202 is realized, for example, by the auxiliary storage device 27 or the like.

[0049] The UI unit 201 displays various screens provided by the scoring device 10 on a display device 22 such as a display. The UI unit 201 also accepts various information input by the user on these screens using the input device 21 and transmits the information to the scoring device 10.

[0050] The storage unit 202 stores various data (for example, training datasets used for model training, scoring criteria, etc.).

[0051] <Screen Transition> The following describes the transition relationship between screens provided to the user terminal 20 by the UI providing unit 101 of the scoring device 10.

[0052] 6 is a diagram showing an example of screen transitions. However, the screen transitions described below represent the main transition relationships between screens, and the transition relationships between the screens are not limited to these. For example, there may be screen transitions not shown in FIG. 6.

[0053] As shown in FIG. 6 , for example, when a user logs in to the system by entering a user ID, password, etc., the UI unit 201 displays a menu screen 1100 on the display device 22 of the user terminal 20. If the logged-in user is a model creator, the menu screen 1100 can be transitioned to a question management screen 1200. Subsequently, transitions are possible in the following order: question management screen 1200 → model management screen 1300 → model creation screen 1400 → model verification screen 1500. Furthermore, from the model verification screen 1500, it is possible to return to the question management screen 1200 or transition to a consistency verification support screen 1600 for the scoring criteria. Here, the question management screen 1200 is a screen for managing questions in which models are used, the model management screen 1300 is a screen for managing models used in the questions, and the model creation screen 1400 is a screen for specifying training data, scoring criteria, etc. used in model creation (training) for the questions. The model verification screen 1500 is a screen for checking the verification results of a trained model, and the consistency verification support screen 1600 for the scoring criteria is a screen for supporting consistency verification for the scoring criteria. These screens are mainly used when creating a model.

[0054] On the other hand, if the logged-in user is a model user (or may be a model creator), it is possible to transition from the menu screen 1100 to the grading screen 1700. After that, it is possible to transition from the grading screen 1700 to the grading result screen 1800. The grading screen 1700 is a screen for creating a written answer and requesting grading of that written answer, and the grading result screen 1800 is a screen for checking the grading results of the written answer. These screens are mainly used when grading.

[0055] Below, we will first explain the screen used when creating a model and the processes for model training and model verification using the training data specified on that screen. Then we will explain the screen used when using the model and the processes for grading written answers created on that screen.

[0056] <When creating a model> <Specific examples of each screen> First, we will explain the menu screen 1100, the question management screen 1200, the model management screen 1300, the model creation screen 1400, the model verification screen 1500, and the consistency verification support screen 1600. These screens are provided to the user terminal 20 by the UI providing unit 101 of the scoring device 10, and are displayed by the UI unit 201 of the user terminal 20.

[0057] Menu Screen 1100 FIG. 7 is a diagram showing an example of the menu screen 1100. As shown in FIG. 7, the menu screen 1100 includes an AI scoring button 1101 and an AI scoring management button 1102. The AI ​​scoring button 1101 is a UI component for transitioning to the scoring screen 1700. The AI ​​scoring management button 1102 is a UI component for transitioning to the question management screen 1200. Note that if the logged-in user is a model user, the AI ​​scoring management button 1102 may be hidden, or the AI ​​scoring management button 1102 may not be selectable. However, this is just an example, and it is sufficient if control is exercised so that transition to the question management screen 1200 is possible only if the logged-in user is a model creator.

[0058] Question Management Screen 1200 Fig. 8 is a diagram showing an example of the question management screen 1200. As shown in Fig. 8, the question management screen 1200 includes a question display field 1210 and a new creation button 1220. The question display field 1210 is a display field that displays at least a question and information about a model for scoring a written answer for that question. For example, the question display field 1201 of the question management screen 1200 shown in Fig. 8 includes columns such as test name 1211, question number 1212, model name 1213, validity 1214, scoring accuracy 1215, setting change 1216, and confirmation 1217.

[0059] For example, in the first row of the question display column 1210 included in the question management screen 1200 shown in FIG. 8 , the test name 1211 is set to "A," the question number 1212 is set to "Question A-1," the model name 1213 is set to "Model A," the validity 1214 is set to "78," and the scoring accuracy 1215 is set to "74." This means that a scoring model with the model name "Model A" will be used to score a written answer sheet for question number "Question A-1" with test name "A." This also means that when the validity (plausibility) of this model A is evaluated using a predetermined evaluation index (described below), the value is "78," and the scoring accuracy is "74." The same applies to the other rows of the question display column 1210.

[0060] Additionally, the setting change 1216 in the question display column 1210 is a column for changing or deleting the model in the corresponding row. The user can change the model set in the corresponding row by selecting "change" in the setting change 1216. Similarly, the user can delete the model set in the corresponding row by selecting "delete" in the setting change 1216.

[0061] Furthermore, confirmation 1217 in question display column 1210 is a column for testing the model set in the corresponding row. The user can actually test the model set in the corresponding row by selecting the "Test" button in confirmation 1217. For example, when the user selects the "Test" button in a certain row, the screen can transition to grading screen 1700, which allows the user to create a written answer to the question set in that row and request grading of the written answer.

[0062] The new creation button 1220 is a UI component for adding a new row to the question display column 1210. For example, when a user wants to add a new test or a new question, the user can add a new row to the question display column 1210 by selecting the new creation button 1220.

[0063] Model Management Screen 1300 Fig. 9 is a diagram showing an example of a model management screen 1300. The model management screen 1300 shown in Fig. 9 is displayed, for example, when "Change" is selected in the setting change 1216 of the problem display field 1210 included in the model management screen 1300 shown in Fig. 8. As shown in Fig. 9, the model management screen 1300 includes a model display field 1310, a new creation button 1320, and a setting button 1330.

[0064] The model display field 1310 is a display field that displays a list of models that can be set for a row for which "Change" is selected in the setting change 1216 of the question display field 1210. For example, the model display field 1310 of the model management screen 1300 shown in Fig. 9 has columns such as Selection 1311, Model Name 1312, Validity 1313, Scoring Accuracy 1314, Creation Date and Time 1315, Total Number of Training Cases 1316, Number of Cases with Gold Label 1317, and Detailed Display 1318.

[0065] For example, in the first row of the model display field 1310 included in the model management screen 1300 shown in FIG. 9 , the model name 1312 is set to "Model A," the validity 1313 is set to "78," the scoring accuracy 1314 is set to "74," the creation date and time 1315 is set to "2022 / 1 / 14," the total number of training cases 1316 is set to "100," and the number of Gold-labeled cases 1317 is set to "35." This indicates that when the validity (Plausibility) of the scoring model with the model name "Model A" is evaluated using a predetermined evaluation index (described below), the value is "78," the scoring accuracy is "74," and the creation date and time is January 14, 2022. This also indicates that Model A was trained with 100 pieces of training data, of which 35 pieces were gold-labeled. The same applies to the other rows of the model display field 1310.

[0066] Furthermore, selection 1311 in the model display field 1310 is a column for selecting a corresponding row. Furthermore, detail display 1318 in the model display field 1310 is a column for checking the verification results of the model set in the corresponding row. The user can check the verification results of the model set in the corresponding row by selecting the "details" button in detail display 1318. For example, when the user selects the "details" button for a certain row, the screen can transition to a model verification screen 1500, which displays the verification results of the model set in that row.

[0067] The new creation button 1320 is a UI component for creating a new model. When the user selects the new creation button 1320, the screen transitions to a model creation screen 1400.

[0068] The setting button 1330 is a UI component for setting the model of the row selected in the selection 1311 of the model display field 1310 for the row for which "change" has been selected in the setting change 1216 of the question display field 1210. The user can set the desired model for the desired question by selecting the desired model in the selection 1311 of the model display field 1310 and then selecting the setting button 1330.

[0069] Model Creation Screen 1400 Fig. 10 is a diagram showing an example of a model creation screen 1400. The model creation screen 1400 shown in Fig. 10 is displayed, for example, when the new creation button 1320 included in the model management screen 1300 shown in Fig. 9 is selected. As shown in Fig. 10, the model creation screen 1400 includes a training dataset selection button 1410, an evaluation criterion selection button 1420, a division ratio setting field 1430, a model name setting field 1440, a field 1450 for setting whether or not to verify consistency with the scoring criteria, and a model creation execution button 1460.

[0070] The training dataset selection button 1410 is a UI component for selecting a training dataset to be used for model training. By selecting the training dataset selection button 1410, the user can select a desired training dataset from among the training datasets stored in the storage unit 202 of the user terminal 20, for example, and transmit (upload) the training dataset to the scoring device 10. Information (e.g., file name, etc.) of the training dataset selected by the training dataset selection button 1410 is displayed in a selected training dataset display field 1411.

[0071] The evaluation criterion selection button 1420 is a UI component for selecting an evaluation criterion to be used in model training. By selecting the evaluation criterion selection button 1420, the user can select a desired evaluation criterion from the evaluation criteria stored in the storage unit 202 of the user terminal 20, for example, and transmit (upload) the evaluation criterion to the scoring device 10. Information about the evaluation criterion selected by the evaluation criterion selection button 1420 (for example, a file name, etc.) is displayed in a selected evaluation criterion display field 1421.

[0072] The division ratio setting field 1430 is a UI component for dividing the training dataset selected with the evaluation criterion selection button 1420 into a training dataset used for updating model parameters, a validation dataset used for adjusting hyperparameters, etc., and a test dataset used for verifying scoring accuracy. The user can set the ratios for training, validation, and test using the division ratio setting field 1430. Note that, hereinafter, of the training datasets included in the training dataset, those designated as "training" will also be referred to as "parameter update training data," those designated as "validation training data," and those designated as "test training data."

[0073] The model name setting field 1440 is a UI component for setting the model name. The scoring criterion consistency verification setting field 1450 is a UI component for setting whether or not to execute a scoring criterion consistency verification support process (described below) using the parameter update training data. By checking the consistency verification setting field 1450, the user can execute the scoring criterion consistency verification support process using the parameter update training data. Note that if the consistency verification setting field 1450 is checked, the parameter update training data is stored in the memory unit 105 of the scoring device 10 even after model training.

[0074] The model creation execution button 1460 is a UI component for starting execution of a model training process (described later). When the model creation execution button 1460 is selected, a trained model is created, and then a model verification screen 1500 is displayed.

[0075] Model Verification Screen 1500 Fig. 11 is a diagram showing an example of a model verification screen 1500. The model verification screen 1500 shown in Fig. 11 is displayed, for example, when the model creation execution button 1460 included in the model creation screen 1400 shown in Fig. 10 is selected. As shown in Fig. 11, the model verification screen 1500 includes a validity display field 1510, a scoring accuracy display field 1520, a verification result display field 1530, a registration button 1540, a test button 1550, and a consistency verification button 1560 for the scoring criteria.

[0076] The validity display field 1510 displays an evaluation index value (described later) that indicates the validity of the trained model. The scoring accuracy display field 1520 displays an evaluation index value (described later) that indicates the scoring accuracy of the trained model.

[0077] The verification result display field 1530 displays the results of verifying a trained model using several sampled test training data. For example, the verification result display field 1530 included in the model verification screen 1500 shown in FIG. 11 has columns including a sample 1531, an answer and label 1532, a scoring score 1533, a scoring confidence level 1534, and a manual scoring score 1535. The sample 1531 is the sample number of the sampled test training data. The answer and label 1532 displays a gold label for the written answer included in the test training data in the upper row and a rationale label predicted by the model for the written answer in the lower row. The scoring score 1533 displays the scoring score predicted by the model for the written answer. The scoring confidence level 1534 displays a scoring confidence level (described below) that indicates the confidence level (likelihood) of the scoring score. The manual scoring score 1535 displays a gold score. This allows the user to compare and confirm the ground labels predicted by the model with the gold labels for some sampled test training data, or to compare and confirm the scoring scores with the human scoring scores while referring to the scoring confidence. Therefore, the user can refer to the verification result display field 1530, for example, to determine whether retraining is necessary or to perform data maintenance such as adding a new gold label.

[0078] In this way, by introducing the concept of plausibility, which looks at the consistency between the evidence points focused on by human graders and the evidence points focused on by the model, into a completely new quality assurance system for automatic grading, model creators or users (such as teachers who create essay questions or developers who provide SAS services) can evaluate the validity of the model based on objective evidence and retrain the model as necessary, thereby providing an automatic SAS grading service that students can trust and feel satisfied with.

[0079] The register button 1540 is a UI component for registering a trained model. When the register button 1540 is selected by the user, the trained model is stored in the storage unit 105 of the scoring device 10.

[0080] The test button 1550 is a UI component for testing a trained model. The user can test the trained model by selecting the test button 1550. For example, when the user selects the test button 1550, the screen transitions to a grading screen 1700 for creating a written answer to a certain question and requesting grading of the written answer.

[0081] The scoring criteria consistency verification button 1560 is a UI component for executing a scoring criteria consistency verification support process (described below) using the training data used during model training (i.e., parameter update training data) and displaying a scoring criteria consistency verification support screen 1600. By selecting the scoring criteria consistency verification button 1560, the user can execute the scoring criteria consistency verification support process and display the scoring criteria consistency verification support screen 1600. Note that the scoring criteria consistency verification button 1560 may be displayed only when a check mark is placed in the consistency verification enable / disable setting field 1450 included in the model creation screen 1400 shown in FIG. 10 , for example.

[0082] Consistency verification support screen 1600 for scoring criteria Fig. 12 is a diagram showing an example of a consistency verification support screen 1600 for scoring criteria. The consistency verification support screen 1600 shown in Fig. 12 is displayed when, for example, the consistency verification button 1560 for scoring criteria included in the model verification screen 1500 shown in Fig. 11 is selected. As shown in Fig. 12, the consistency verification support screen 1600 includes a verification result display field 1610, a scoring accuracy display field 1620, a switch button 1630, a download button 1640, and a back button 1650.

[0083] The verification result display field 1610 displays the results of the "consistency verification support process" (described later) performed using the parameter update training data. In this consistency verification support process, the explanation sequences corresponding to the parameter update training data are divided into multiple clusters (five clusters, 0 to 4, in the example shown in FIG. 12 ). As a result, as shown in FIG. 12 , it is possible to display, for each cluster, a heat map in which the written answers included in the parameter update training data belonging to that cluster are colored by the values ​​of the corresponding explanation sequences. This allows the user to check, for example, whether the trained model is scoring answers based on parts that are inconsistent with the scoring criteria. In other words, the user can check, for example, whether the trained model is scoring answers based on "spurious clues" and giving high scores to answers that do not meet the scoring criteria. This is because clustering makes it possible to efficiently check whether scoring based on "spurious clues" occurs in clusters. As a result, the user can discern the quality of the scoring model. In addition to this, the user can also check the grading criteria and the like to see which cluster the written answer sheets belong to and which grading criteria they conform to.

[0084] The scoring accuracy display field 1620 displays an evaluation index value (described later) that indicates the scoring accuracy of the trained model. The switching button 1630 is a UI component for switching the display contents of the verification result display field 1610 and the scoring accuracy display field 1620 for each of the overall scoring model and the individual scoring model, for example, when both the overall scoring model and the individual scoring model are being trained.

[0085] The download button 1640 is a UI component for downloading the verification results displayed in the verification result display field 1610 and saving them in the storage unit 202 of the user terminal 20. This allows the user to perform data maintenance, such as using the downloaded verification results to assign a gold label to parameter update training data that does not have a gold label assigned to it. Note that the training data after such data maintenance can be used, for example, for retraining to further improve scoring accuracy.

[0086] The back button 1650 is a UI component for returning to the model verification screen 1500, for example.

[0087] <Model Training Process> Hereinafter, the model training process that is executed when, for example, the model creation execution button 1460 is selected on the model creation screen 1400 shown in FIG. 10 will be described with reference to FIG.

[0088] First, the data acquisition unit 102 acquires a training data set (step S101). That is, the data acquisition unit 102 acquires the training data set uploaded from the user terminal 20.

[0089] Next, the data acquisition unit 102 performs preprocessing on the training dataset acquired in step S101 (step S102). For example, as preprocessing, the data acquisition unit 102 divides the training dataset into parameter update training data, validation training data, and test training data according to a value set in a division ratio setting field 1430 included in the model creation screen 1400 shown in FIG. 10 . Furthermore, for example, the data acquisition unit 102 converts the parameter update training data, validation training data, and test training data into a format that can be input to the model. Note that converting into a format that can be input to the model means, for example, dividing the text represented by the written answer into certain units (word units, token units, character units, etc.) according to an input format that can be input to the model.

[0090] An example of the data structure of certain training data after preprocessing is shown in Fig. 14. The example shown in Fig. 14 is an example in which the text represented by a written answer is divided into both morpheme units and character units.

[0091] "Mecab" is a text that has been divided into morpheme units by a morphological analysis engine called Mecab (Reference 1).

[0092] "A," "B," "C," "D," and "E" represent annotations (gold labels) for the morphemes that are the basis for scoring for each scoring item. In other words, in light of the scoring item, 1 is assigned to the part that is the basis for the score, and 0 is assigned to the other parts, and the i-th annotation corresponds to the i-th morpheme. The letters A to E are linked to the scoring items defined in the scoring criteria.

[0093] "Char" is text divided into character units.

[0094] "C_A," "C_B," "C_C," "C_D," and "C_E" are annotations (gold labels) for characters that serve as the basis for scoring for each scoring item. In other words, 1 is assigned to the part that serves as the basis for the scoring item, and 0 is assigned to the other parts, and the i-th annotation corresponds to the i-th character. The letters A to E are linked to the scoring items defined in the scoring criteria.

[0095] "Id" is a serial number that identifies the training data within the training dataset.

[0096] "Score" is the score given to the entire written answer. This score is the sum of the scores for each grading item, which will be described later. However, if the sum is negative, it will be set to 0.

[0097] "A_Score," "B_Score," "C_Score," "D_Score," "E_Score," "Miss_Score," and "EOD_Score" are the grading item unit scores for each grading item. A_Score to E_Score are the grading item unit scores scored using the point-adding items (grading items) defined in the grading criteria. On the other hand, Miss_Score and EOD_Score are the grading item unit scores scored using the point-deducting items (grading items) defined in the grading criteria.

[0098] In the example shown in Figure 14, the text represented by the written answer is divided into both morpheme units and character units, but it may be divided into only one of them depending on the format that can be input to the model. The data structure shown in Figure 14 is an example and is not limited to this. The format of the training data after preprocessing can be converted into various formats as long as it is in a format that can be input to the model. Various preprocessing steps can also be performed (for example, a process of aligning the lengths of the morpheme units or character units when the written answer is divided).

[0099] Below, the written answers included in the training data after preprocessing are expressed as x = (x １ , x ２ , ..., x Ｔ ) In the following, the model to be trained is defined as f, the gold score as s, and the gold label as r = (r １ , r ２ , ..., r Ｔ )

[0100] Here, when training an individual scoring model, if the number of scoring items defined in the scoring criteria is K, the individual scoring model is f = (f １ , ..., f Ｋ ) and so the model f of a certain k-th scoring item ｋ Let f be the training case. The gold score s should be the score unit score of the scoring item, and the gold label r should be the gold label for the score unit score of the scoring item.

[0101] On the other hand, when training an overall scoring model, the sum of the scores for each scoring item (i.e., the score for the entire written answer) can be used as the gold score s. The gold label r can be obtained by taking the logical sum of the gold label elements for each scoring item. Specifically, the gold label for the scoring item score of the kth scoring item can be calculated as r. （ｋ） = (r １ （ｋ） , r ２ （ｋ） , ..., r Ｔ （ｋ） ) and then r ｔ （１） ~r ｔ （Ｋ） The logical sum up to r ｔ Then, r = (r １ , r ２ , ..., r Ｔ ) should be designated as the Gold Label.

[0102] Although various machine learning models can be used as model f, the following mainly assumes a model consisting of an embedding layer, one biLSTM layer, one attention layer, and a Softmax layer. This model f is generated by inputting a written answer x = (x １ , x ２ , ..., x Ｔ ) is embedded in the embedding layer emb(·) to create a distributed representation x ｔ '=emb(x ｔ ), and then the hidden state (h １ , ..., h Ｔ )=biLSTM(x １ ', x ２ ',...,x Ｔ Then, the contribution e ｔ ａ =Attention(h １ , ..., h ｔ ) and then the contribution e ｔ ａ and hidden state h ｔ The weighted sum h = e １ ａh １ +...+e Ｔ ａ h Ｔ The scoring score is predicted by ^s = softmax(Wh + b). The parameters of the biLSTM layer, Attention layer, and Softmax layer are the training targets. The dimensions of the distributed representation and hidden state can be set appropriately. Note that this contribution e ｔ ａ (or its calculation method) is also called Attention Weight. ｔ ａ The reason why Attention Weight is used as the weight is that it is believed that Attention Weight contributes to validity.

[0103] However, the above model f is merely an example and is not limiting. For example, BERT or the like may be used instead of the biLSTM layer.

[0104] Next, the training unit 103 trains the model f using the training data for parameter update included in the preprocessed training data set (step S103). Here, in order to train the model f, the following loss function L ｓｃｏｒｅ , L ａｔｔｎ , L ｉｎｇｒａｄ Introduce.

[0105] Here, N is the number of training data for parameter update (for example, the number of training data for parameter update included in one batch), and CE(·,·) is the cross-entropy error. s is the gold score, and ^s is the graded score predicted by the model. Note that the "(n)" added to the upper right of s represents the gold score included in the nth training data for parameter update, and the "(n)" added to the upper right of ^s represents the graded score predicted from the written answer sheet x included in the nth training data for parameter update.

[0106] Here, e ｔ ｉｎｇｒａｄ is the contribution calculated by a method called Integrated Gradients (Reference 2). ｔThe "(n)" in the upper right corner of e represents the gold label included in the n-th parameter update training data (or the gold label obtained from the gold label included in the n-th parameter update training data). ｔ ｉｎｇｒａｄ is calculated as follows:

[0107] Here, x ｂ is an input sample that serves as a reference for comparison, called a baseline, and is, for example, a zero vector.

[0108] At this time, the training unit 103 uses one of the following (1) to (4) as a loss function and trains the model f using an existing optimization method (e.g., gradient method, etc.) so as to minimize the value of this loss function.

[0109] (1) L ｓｃｏｒｅ (2) L ｓｃｏｒｅ +L ａｔｔｎ (3) L ｓｃｏｒｅ +L ｉｎｇｒａｄ (4) L ｓｃｏｒｅ +L ａｔｔｎ +L ｉｎｇｒａｄ It is possible to decide which of the above (1) to (4) to adopt as the loss function as appropriate, but it is preferable to adopt any of (2) to (4) using the gold label r, and it is even more preferable to adopt either (2) or (4).

[0110] Following step S103, the training unit 103 determines whether to end the training (step S104). Various conditions can be used to end the training, such as when the objective function (loss function) has converged or when training has been performed for a predetermined number of epochs.

[0111] If it is not determined in step S103 that the training is to be ended, the training unit 103 returns to step S103 and trains the model f again. For example, the training unit 103 selects the next batch of training data for parameter update from the preprocessed training data set, and trains the model f again.

[0112] On the other hand, if it is determined in step S103 that the training is to be terminated, the training unit 103 stores the trained model f in the storage unit 105 (step S105). At this time, the training unit 103 may also store, in the storage unit 105, for example, a scoring standard corresponding to the training data set.

[0113] In the above model training process, the contribution e is calculated using a technique called Integrated Gradients. ｔ ｉｎｇｒａｄ However, other contributions used in feature attribution methods may be calculated. For example, contributions calculated by methods such as Saliency Map (Reference 3) or Input X Gradient (Reference 4) may be used. ｔ ｉｎｇｒａｄ The reason why Integrated Gradients (or Saliency Map or Input X Gradient) are used as the input gradient is that they are believed to contribute to fidelity.

[0114] <Calculation of various evaluation index values> Using the trained model trained in the above model training process, various evaluation index values ​​representing the validity, faithfulness, scoring accuracy, scoring confidence, etc. of the model can be calculated.

[0115] Validity The evaluation unit 104 can calculate an evaluation index value called the agreement of justification cue (AJC) (Reference 5) as an evaluation index value representing the validity of the trained model f, using the test training data included in the preprocessed training dataset. The AJC is an index value that calculates the degree of agreement between the gold label r and the ground label ^r predicted by the trained model f as an f1 score. That is, r ｔ = 1 and ^r ｔ = 1 ｔ The number of TP, r ｔ = 1 and ^r ｔ = 0 ｔ The number of FN, r ｔ= 0 and ^r ｔ = 1 ｔ The number of FP, r ｔ = 0 and ^r ｔ = 0 ｔ The f1 score was calculated using the number of TN.

[0116] Faithfulness: The evaluation unit 104 can calculate an evaluation index value called the remove ratio (Reference 6) as an evaluation index value representing the faithfulness of the trained model f, using the test training data included in the preprocessed training dataset.

[0117] Scoring Accuracy: The evaluation unit 104 can calculate an evaluation index value representing the scoring accuracy of the trained model f using the test training data included in the preprocessed training dataset. As such an evaluation index value, for example, the quadratic weighted kappa coefficient (QWK) or the root mean squared error (RMSE) can be adopted.

[0118] Scoring confidence: The evaluation unit 104 can calculate an evaluation index value that represents the scoring confidence for the test training data using the test training data included in the preprocessed training dataset. For example, an evaluation index value called a trust score (Reference 7) can be used as such an evaluation index value.

[0119] <<Consistency Verification Support Process for Scoring Criteria>> The following describes the consistency verification support process for scoring criteria that is executed when, for example, a consistency verification button 1560 for scoring criteria is selected on the model verification screen 1500 shown in Fig. 11, with reference to Fig. 15. The following describes the case where the consistency verification support process for scoring criteria is executed for a certain trained model f.

[0120] First, the training unit 103 acquires the parameter updating training data used in training the trained model f from the parameter updating training data stored in the storage unit 105 (step S201).

[0121] Next, the training unit 103 calculates an explanation series of contributions using the descriptive answer x included in the parameter update training data acquired in step S201 and the trained model f (step S202). Here, the contributions may be calculated using, for example, Integrated Gradients. Hereinafter, the total number of update training data is N, and the descriptive answer x included in the n-th update training data is （ｎ） The explanatory series calculated from e （ｎ） However, the contribution may be calculated by a method other than Integrated Gradients.

[0122] Next, as a preprocessing step for clustering, the training unit 103 performs the following steps: （ｎ） (Step S203) （ｎ） Since the lengths may differ, preprocessing is performed to make the lengths uniform.

[0123] Then, the training unit 103 performs the preprocessing on each explanation sequence e in step S203. （ｎ）are clustered (step S204). For example, a method called SpRAy (Reference 8) may be used for this. The clustering method used in SpRAy may be any method, such as the k-means method. The number of clusters k in the k-means method can be determined arbitrarily, but may also be determined by the user from the eigenvalues ​​of the Laplacian matrix calculated by SpRAy. For example, the UI providing unit 101 may provide the user terminal 20 with a number-of-clusters specification screen 1900 shown in FIG. 16. The number-of-clusters specification screen 1900 shown in FIG. 16 includes an eigenvalue display field 1901 that displays the eigenvalues ​​of the Laplacian matrix calculated by SpRAy. The user can set the desired number of clusters k in the k-means method by referring to the eigenvalue display field 1901 and then selecting the OK button 1903. The number of clusters k can be set to, for example, the i at which the difference between the i-1th eigenvalue and the i-th eigenvalue becomes gentle when the eigenvalues ​​are arranged in ascending order (more specifically, the i at which the difference between the i-1th eigenvalue and the i-th eigenvalue becomes less than a certain threshold).

[0124] As a result, the explanation series are classified into k clusters. As a result, the written answer sheets belonging to each cluster and the corresponding explanation series are obtained for each cluster, and it is possible to display each written answer sheet as a heat map colored by the value of the explanation series in the verification result display field 1610 included in the consistency verification support screen 1600 shown in FIG.

[0125] <During Scoring> <Specific Examples of Each Screen> First, we will explain the scoring screen 1700 and the scoring result screen 1800. These screens are provided to the user terminal 20 by the UI providing unit 101 of the scoring device 10 and displayed by the UI unit 201 of the user terminal 20.

[0126] Scoring Screen 1700 Fig. 17 is a diagram showing an example of a scoring screen 1700. As shown in Fig. 17, the scoring screen 1700 includes a question display field 1710, an answer input field 1720, a scoring button 1730, and a question transition button 1740.

[0127] The question display field 1710 is a display field where the question (problem statement and question text) is displayed. The answer input field 1720 is a UI component where an answer to the question can be entered as text (i.e., a UI component for creating a written answer). The grading button 1730 is a UI component for executing an evaluation process (described below) for the written answer entered in the answer input field 1720. After creating a written answer in the answer input field 1720, the user can request grading of the written answer by selecting the grading button 1730. By selecting the grading button 1730, the written answer is graded, and then a grading result screen 1800 is displayed.

[0128] The question transition button 1740 is a UI component for transitioning to the next question or the previous question. The user can use the question transition button 1740 to transition to a desired question.

[0129] The scoring screen 1700 may include a model name display field 1750 that displays the model name of the model used to score the written answer. This model name display field 1750 may be displayed only if the logged-in user is the model creator. Furthermore, if the logged-in user is the model creator, the model used to score the written answer may be changed or selected from a list of available models.

[0130] Scoring result screen 1800 Fig. 18 is a diagram showing an example of a scoring result screen 1800. The scoring result screen 1800 shown in Fig. 18 is displayed, for example, when the scoring button 1730 included in the scoring screen 1700 shown in Fig. 17 is selected. As shown in Fig. 18, the scoring result screen 1800 includes a scoring result display field 1810, a scoring item display field 1820, and a scoring evaluation certainty level display field 1830.

[0131] The scoring result display field 1810 is a display field that displays the scoring result for the entire question (i.e., the total of the scoring item unit scores). The scoring item display field 1820 is a display field that displays the scoring confidence for each scoring item and each scoring item unit score. The scoring evaluation confidence display field 1830 is a display field that displays the scoring confidence for the scoring result for the entire question (a value evaluated by plausibility (FIG. 8), as well as, for example, the average scoring confidence for the scoring item unit scores, an overall score that takes into account the variance of the confidence for the scoring item unit scores, etc.). This allows the user to know the scoring result for the written answer sheet and its scoring confidence.

[0132] Note that, while the above describes the "overall scoring model and overall scoring score for the written answer sheet" and the "individual scoring model and scoring item unit score" as one embodiment of the present invention, the "overall scoring model and overall scoring score for the written answer sheet" basically become new quality assurance information for automatic scoring based on plausibility, and can improve the user's trust and satisfaction with automatic scoring, so the "individual scoring model and scoring item unit score" are not necessarily required. If there is an additional user need to see the scoring item unit scores in addition to the overall score, this is quality assurance information that is selectively provided as needed, and increasing the options for information provided to the user improves convenience.

[0133] The scoring result screen 1800 may also include a model name display field 1840 that displays the model name of the model used to score the written answer. The model name display field 1840 may also be configured to be displayed only when the logged-in user is the model creator.

[0134] <Evaluation Process> The evaluation process executed when, for example, the scoring button 1730 is selected on the scoring screen 1700 shown in FIG. 17 will be described below with reference to FIG. 19. In the following, the trained model is designated as f. For example, when the evaluation process is executed using an overall scoring model, this may be designated as f. On the other hand, for example, when the evaluation process is executed using an individual scoring model, the model for the kth scoring item may be designated as f = f. ｋ17, the following steps S301 to S305 may be executed for each k. In the following, it is assumed that, in response to the selection of the scoring button 1730 included in the scoring screen 1700 shown in FIG. 17, the evaluation data representing the written answer entered in the answer input field 1720 and the information representing the test name and question number of the question corresponding to the scoring screen 1700 are transmitted from the user terminal 20 to the scoring device 10.

[0135] First, the data acquisition unit 102 acquires the corresponding trained model f from the storage unit 105 (step S301). That is, the data acquisition unit 102 acquires, for example, the trained model f set for the test name and question number received from the user terminal 20 from the storage unit 105.

[0136] Next, the data acquisition unit 102 acquires the evaluation data received from the user terminal 20 (step S302).

[0137] Next, the data acquiring unit 102 performs preprocessing on the evaluation data acquired in step S302 (step S303). That is, the data acquiring unit 102 converts the evaluation data into a format that can be input to the trained model f.

[0138] Next, the evaluation unit 104 evaluates (grades) the written answer represented by the preprocessed evaluation data using the preprocessed evaluation data and the trained model f (step S305). That is, when the written answer represented by the preprocessed evaluation data is x, the evaluation unit 104 predicts the graded score using ^s = f(x). Note that if f is an overall grading model, ^s represents the graded score for the entire question. On the other hand, if f is an individual grading model that predicts the graded score for a certain k-th grading item, ^s represents the grading item unit score for the k-th grading item.

[0139] Then, the evaluation unit 104 stores the score obtained in step S305 in the storage unit 105.

[0140] <<Casting to a Human Grader>> For example, in step S305 above, the evaluation unit 104 may predict the scoring score and calculate the scoring confidence level. Furthermore, at this time, if the scoring confidence level is smaller than a predetermined threshold τ, the evaluation unit 104 may cast (send) the scoring of the written answer sheet to a human grader. This allows the written answer sheet to be graded by a human grader when the scoring confidence level is low, thereby preventing a decline in scoring quality.

[0141] The threshold value τ can be determined by various methods. For example, it can be determined by using validation training data divided for validation purposes during training of the model.

[0142] For example, let x be the training data for validation, s be the score predicted from the training data for validation by the trained model, and C(x, s) be the confidence of this score. Also, let P be the whole of (x, s). Furthermore, P τ = {(x, ^s)∈P | C(x, ^s) ≧τ}, P ｈ = {(x, s ｈ )∈P|C(x, ^s)<τ}, where s ｈ is the score given by the person who graded it. ｆ =P τ ∪P ｈ It is conceivable to determine the maximum τ at which the sum of the errors between the gold score and the threshold is equal to or less than a predetermined allowable value ε.

[0143] <Performance Evaluation> The performance evaluation of the trained model described above will be described below. In this performance evaluation, QWK was used as the evaluation index value representing the scoring accuracy, AJC was used as the evaluation index value representing the validity, and Remove Ratio was used as the evaluation index value representing the faithfulness. In addition, the above four patterns (1) to (4) were used as the loss function. The contribution degree e ｔ ａ The Attention Weight is used to calculate the contribution e ｔ ｉｎｇｒａｄFor the calculation, either Saliency Map, Input X Gradient, or Integrated Gradient was used. A dataset called RIKEN-SAA was used as the training dataset. In addition, a method (Random) that generates contributions from a uniform distribution was used as the baseline.

[0144] The performance evaluation results are shown below.

[0145] Here, unsup represents the case where the above (1) is used as the loss function, attn represents the case where the above (2) is used as the loss function, igrad represents the case where the above (3) is used as the loss function, and attn&igrad represents the case where the above (4) is used as the loss function.

[0146] As shown in Table 1 above, all of the trained models obtained in the above embodiment have high scoring accuracy, with attn & igrad being particularly high. Furthermore, as shown in Table 2 above, all of the trained models obtained in the above embodiment have high AJC compared to the baseline, with attn being particularly high. Furthermore, as shown in Table 2 above, all of the trained models obtained in the above embodiment have low Remove Ratios compared to the baseline, and it can be seen that a particularly low Remove Ratio is obtained when the contribution is calculated using Integrated Gradients. Note that the lower the Remove Ratio, the better the model's performance.

[0147] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0148] The SAS automatic scoring service realized by the present invention is provided via the cloud or on-premise, depending on the user's choice. Taking the cloud as an example, multiple automatic scoring applications (software) available to the user may be provided in the scoring device 10, and the user may select the application to be used from among the multiple applications displayed on the screen of the user terminal 20 using a selection means on the user terminal 20. Specifically, for example, three versions of the application may be provided: a Basic version without a plausibility quality assurance function, an Advanced version with a plausibility quality assurance function, and a Hi-Advanced version with a plausibility quality assurance function and a quality improvement function through model retraining. This allows for use cases that do not necessarily require plausibility quality assurance, depending on the purpose of use, and provides a SAS automatic scoring service tailored to the purpose. SAS service users (schools, cram schools) or model creators (teachers creating essay-style questions, developers providing SAS services, etc.) can select from the three versions that best suit their purpose. Students may be allowed to choose the version. In this example, as the version increases, the functionality becomes more advanced and the usage fee increases, so allowing users to select the appropriate application based on cost-effectiveness according to their purpose of use will improve the overall convenience of the SAS automatic grading service. On-premise provision is basically the same as cloud provision.

[0149] Furthermore, as described above, the text written in the sentence format of the present invention is not limited to written answers, and the above embodiments can be similarly applied to any text written in the sentence format for some assignment or problem. The present invention is highly versatile as an automatic scoring technology (automatic evaluation technology) for text written in the sentence format, and can be shown with objective evidence that it is as reliable as human evaluation.

[0150] <Modifications> Modifications 1 and 2 of the above embodiment will now be described.

[0151] Variation 1: Instead of the k-means method, hierarchical clustering may be used as the clustering method in step S204 of the consistency verification support process shown in FIG. 15 . An example of a consistency verification support screen for the scoring criteria when hierarchical clustering is used as the clustering method is shown in FIG. 20 . The verification result display field 1610A of the consistency verification support screen 1600A shown in FIG. 20 includes a dendrogram display field 1611A that displays a dendrogram obtained by hierarchical clustering, and heat map display fields 1612A to 1619A that display heat maps in which the written answer sheets included in the parameter update training data belonging to each cluster are colored by the corresponding explanation series values. Note that the number in parentheses in the dendrogram display field 1611A indicates the number of parameter update training data belonging to the cluster. Furthermore, each of the heat map display fields 1612A to 1619A displays a heat map of three lines of written answer sheets.

[0152] This allows users to view not only the heat map for their written answers but also the dendrogram, making it easier to understand the relationships between clusters.

[0153] Each of the heat map display fields 1612A to 1619A displays a heat map of three lines of written answers, but the user can, for example, select a field to expand the heat map of the written answers belonging to the cluster corresponding to that heat map display field. For example, when a selection operation is performed on the heat map display field 1613A, the heat map of the written answers belonging to the cluster corresponding to the heat map display field 1613A is expanded, as shown in Fig. 21. This makes it possible to check the heat maps of all written answers belonging to that cluster.

[0154] Variation 2: In Variation 1 described above, gold labels (i.e., rationale labels specified in annotations) may also be displayed along with the heat map for the written answer. Figure 22 shows an example of a consistency verification support screen in which gold labels are displayed along with the heat map for the written answer. The verification result display field 1610B of the consistency verification support screen 1600B shown in Figure 22 includes a dendrogram display field 1611B and heat map / gold label display fields 1612B-1613B in which heat maps and gold labels are displayed for the written answers included in the parameter update training data belonging to each cluster, colored by the corresponding explanation series values. The heat maps are output by the trained model f using the written answers included in the parameter update training data as input. The example shown in Figure 22 also shows an example in which the heat maps and gold labels for the written answers belonging to the cluster corresponding to the heat map / gold label display field 1613B are displayed.

[0155] This allows the user to compare the heat map with the gold label and identify areas where the explanation sequence deviates from the gold label. For example, in the heat map gold label display field 1613B in the example shown in Figure 22, the gold label is "Western," while the value of the "different" area in the heat map is high. Therefore, it can be confirmed that this area is used for scoring, and there is a concern about a decrease in reliability.

[0156] <Summary> As described above, the scoring system 1 according to the above embodiment can evaluate text written in a sentence format with high accuracy and reliability (quality) compared to conventional techniques. Furthermore, the scoring system 1 according to the above embodiment provides various UIs for performing this evaluation to the user terminal 20. Therefore, the scoring system 1 according to the above embodiment can support the evaluation of text written in a sentence format while guaranteeing the quality of the evaluation process.

[0157] This application is based on basic application No. 2023-033131 filed in Japan on March 3, 2023, the entire contents of which are incorporated herein by reference.

[0158] [References] Reference 1: MeCab: Yet Another Part-of-Speech and Morphological Analyzer, Internet <URL: https: / / taku910.github.io / mecab / > Reference 2: Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. March 2017. Reference 3: Simonyan, K., Vedaldi, A., Zisserman, A.: Deep Inside Convolutional Networks: Visualizing Image Classification Models and Saliency Maps. ArXiv abs / 1312.6034, 8 pages (2014) Reference 4: Shrikumar, A., Greenside, P., Shcherbina, A., Kundaje, A.: Not Just a Black Box: Learning Important Features Through Propagating Activation Differences. arXiv:1605.01713, (2017) Reference 5: Mizumoto, T., Ouchi, H., Isobe, Y., Reisert, P., Nagata, R., Sekine, S., Inui, K.: Analytic Score Prediction and Justification Identification in Automated Short Answer Scoring. In: ACL. pp. 316-325 (2019). Reference 6: Serrano, S., Smith, NA: Is Attention Interpretable? In: ACL. pp. 2931-2951 (2019). https: / / doi.org / 10.18653 / v1 / P19-1282 Reference 7: Funayama, H., Sasaki, S., Matsubayashi, Y., Mizumoto, T., Suzuki, J., Mita, M., Inui, K.: Preventing critical scoring errors in short answer scoring with confidence estimation. In: ACL-SRW. pp. 237{243. Association for Computational Linguistics, Online (Jul 2020). https: / / doi.org / 10.18653 / v1 / 2020.acl-srw.32 Reference 8: Sebastian Lapuschkin, Stephan Waldchen, Alexander Binder, Gr´egoire Montavon, Wojciech Samek, and Klaus-Robert Muller. Unmasking clever hans predictors and assessing what machines really learn. Nat. Commun., Vol. 10, No. 1, p. 1096, March 2019.

[0159] 1 Scoring system 10 Scoring device 11 External I / F 11a Recording medium 12 Communication I / F 13 RAM 14 ROM 15 Auxiliary storage device 16 Processor 17 Bus 20 User terminal 21 Input device 22 Display device 23 External I / F 23a Recording medium 24 Communication I / F 25 RAM 26 ROM 27 Auxiliary storage device 28 Processor 29 Bus 30 Communication network 101 UI providing unit 102 Data acquisition unit 103 Training unit 104 Evaluation unit 105 Memory unit 201 UI unit 202 Memory unit

Claims

1. A training device for training a model to evaluate document-format text for a given task, A first UI providing unit that provides a first UI that can specify at least the training data for the model and whether or not to perform consistency verification with respect to the scoring criteria for the task, A training unit that trains the model based on the training data and an index value representing the degree of contribution to the quality of the model, If the execution of the consistency verification is specified, a second UI providing unit provides a second UI that visualizes the contribution of each predetermined unit contained in the text as a heat map when the text contained in the training data is evaluated by the trained model trained by the training unit, A training device having the following features.

2. The first UI provision unit described above is: The training apparatus according to claim 1, further providing a third UI that displays the evaluation accuracy of the trained model trained by the training unit and an evaluation index value that evaluates the quality of the trained model.

3. The training data includes gold labels indicating the supporting locations in the text when a human evaluator evaluates the text. The aforementioned quality includes validity, which represents the consistency between the basis for which the model evaluated the text and the basis for which the gold label represents. The aforementioned training unit, The training apparatus according to claim 1 or 2, which trains the model based on the training data and the index values ​​that contribute to the validity.

4. The training device according to claim 3, wherein the index value contributing to the validity is a weight calculated in the attention mechanism layer included in the model.

5. The aforementioned quality includes fidelity, which represents the consistency between the basis for the model's evaluation of the text and the evaluation value obtained by the model. The aforementioned training unit, Furthermore, the training device according to claim 1 or 2, which trains the model based on the index values ​​that contribute to the fidelity.

6. The training apparatus according to claim 5, wherein the index value contributing to fidelity is a contribution calculated by Integrated Gradients, Salience Map, or Input X Gradient.

7. An evaluation device for evaluating text in document format for a given task, An evaluation unit that uses a model to evaluate the text to evaluate the consistency between first information indicating the supporting locations in the text when the model evaluates the text and second information indicating the supporting locations in the text when a human evaluates the text, A third UI providing unit provides a fourth UI that displays, with respect to evaluation values ​​representing the evaluation by the evaluation unit, at least an evaluation value for the entire task and evaluation values ​​for each item constituting the task, An evaluation device having the following features.

8. The evaluation device according to claim 7, wherein the evaluation value is a value indicating validity that represents the consistency between the first information and the second information, which is the correct answer for the basis location used during training of the model.

9. A computer that trains a model to evaluate document-format text for a given task, A first UI provisioning procedure that provides a first UI that can specify at least the training data for the model and whether or not to perform consistency verification with respect to the scoring criteria for the task, A training procedure for training the model based on the aforementioned training data and an index value representing the degree of contribution to the quality of the model, If the execution of the consistency verification is specified, a second UI provision procedure provides a second UI that visualizes the contribution of each predetermined unit contained in the text as a heat map when the text contained in the training data is evaluated by the trained model trained by the training procedure, A training method to perform this task.

10. A computer that evaluates text in document format for a given task, An evaluation procedure that uses a model to evaluate the text, and evaluates the consistency between first information indicating the supporting locations in the text when the model evaluates the text, and second information indicating the supporting locations in the text when a human evaluates the text, A third UI provision procedure provides a fourth UI that displays, with respect to evaluation values ​​representing the evaluation according to the evaluation procedure, at least an evaluation value for the entire task and an evaluation value for each item constituting the task, An evaluation method for performing this task.

11. A computer trained to evaluate a document-format text for a given task, A first UI provisioning procedure that provides a first UI that can specify at least the training data for the model and whether or not to perform consistency verification with respect to the scoring criteria for the task, A training procedure for training the model based on the aforementioned training data and an index value representing the degree of contribution to the quality of the model, If the execution of the consistency verification is specified, a second UI provision procedure provides a second UI that visualizes the contribution of each predetermined unit contained in the text as a heat map when the text contained in the training data is evaluated by the trained model trained by the training procedure, A program that executes the command.

12. A computer that evaluates text in document format for a given task, An evaluation procedure that uses a model to evaluate the text, and evaluates the consistency between first information indicating the supporting locations in the text when the model evaluates the text, and second information indicating the supporting locations in the text when a human evaluates the text, A third UI provision procedure provides a fourth UI that displays, with respect to evaluation values ​​representing the evaluation according to the evaluation procedure, at least an evaluation value for the entire task and an evaluation value for each item constituting the task, A program that executes the command.