Ability estimation program, ability estimation device, and ability estimation method

Abstract

To efficiently estimate the ability of an answerer from an answer to a question.SOLUTION: A computer infers a correspondence relation between each of a plurality of description contents included in an answer by free description of an answerer to a question and each of a plurality of evaluation items included in an evaluation criterion for evaluating the answer by using a correspondence relation inference model generated by first machine learning. The computer infers evaluation results of the answer relative to each of the plurality of evaluation items from the answer and the correspondence relation using an answer evaluation results inference model generated by second machine learning. The computer estimates the ability of the answerer based on evaluation results of the answer for each of the plurality of evaluation items to output estimation results of the ability of the answerer.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a capability estimation technique. [Background technology]

[0002] Aptitude tests such as the Synthetic Personality Inventory (SPI) assess the degree of suitability of test takers for specific activities such as occupations. Such aptitude tests often use a one-question, one-answer format.

[0003] Regarding technology for evaluating the work capabilities of members of an organization, a capability evaluation device is known that, when evaluating a subject's work capabilities, reduces the burden placed on the subject compared to conventional methods and makes it possible to capture subtle changes in the subject (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2022 / 224661 Brochure Summary of the Invention [Problem to be solved by the invention]

[0005] It is difficult to estimate the potential abilities of test takers in a question-and-answer format aptitude test. On the other hand, in a one-to-many evaluation test format that asks for free-form responses to questions, it is easy to estimate potential abilities from the responses, but the workload of the evaluator who evaluates the responses increases.

[0006] This problem is not limited to aptitude tests and potential ability tests, but arises when inferring various abilities of respondents from their answers to questions.

[0007] In one aspect, the present invention aims to efficiently estimate the ability of a respondent from the answer to a question. [Means for solving the problem]

[0008] In one proposal, the ability estimation program causes a computer to execute the following processes.

[0009] The computer uses the correspondence inference model generated by the first machine learning to infer correspondence between each of the multiple descriptions included in the respondent's free-form response to the question and each of the multiple evaluation items included in the evaluation criteria for evaluating the response.

[0010] The computer infers an evaluation result of the answer for each of the multiple evaluation items from the answer and the corresponding relationship using the answer evaluation result inference model generated by the second machine learning. The computer estimates the ability of the answerer based on the evaluation result of the answer for each of the multiple evaluation items, and outputs the estimated result of the ability of the answerer. Effect of the Invention

[0011] According to one aspect, the ability of a respondent can be efficiently inferred from the answers to the questions. [Brief description of the drawings]

[0012] [Figure 1] FIG. 13 is a diagram showing a one-to-many evaluation test based on free description. [Diagram 2] FIG. 2 is a functional configuration diagram of the ability estimation device according to the embodiment. [Diagram 3] 13 is a flowchart of a first ability estimation process. [Figure 4] FIG. 1 is a configuration diagram of a capability estimation system. [Diagram 5] FIG. 1 is a diagram showing questions included in a test and evaluation items for the questions. [Figure 6] FIG. 11 is a diagram showing a second ability estimation process. [Figure 7] 2 is a functional configuration diagram of a capability estimation device included in the capability estimation system. FIG. [Figure 8]FIG. 1 shows answer 1 to question 1 and evaluation items. [Figure 9] FIG. 13 is a diagram showing the number of correspondence relationships C(i, j). [Figure 10] FIG. 13 is a diagram showing the distribution of evaluation results E(i, j). [Figure 11] FIG. 13 is a diagram showing the importance of evaluation results E(i, j). [Figure 12] 13 is a flowchart of a training process. [Figure 13A] 13 is a flowchart (part 1) of a second ability estimation process. [Figure 13B] 13 is a flowchart (part 2) of the second ability estimation process. [Figure 14] FIG. 2 is a hardware configuration diagram of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0014] The Data Scientist (DS) aptitude test service identifies the characteristics of candidates based on the results of a written test that evaluates their potential as a DS. Using the identified characteristics of the candidates, human resources are utilized to accelerate digital transformation (DX). Human resource utilization to accelerate DX includes employee reskilling and human resource assessment support based on the customer's DX perspective.

[0015] Human potential abilities are abilities that at first glance are difficult to assess objectively with numerical values, or abilities that indirectly indicate such abilities. In the case of the DS Aptitude Diagnostic Service, the potential abilities that represent aptitude as a DS are the direct subject of assessment, but since it is difficult to measure such potential abilities, indirect abilities required as aptitude are defined. In the DS Aptitude Diagnostic Service, indirect abilities include problem-solving ability, logical thinking ability, and communication ability. By assessing indirect abilities, it is possible to grasp the characteristics of the examinee.

[0016] However, in a one-question, one-answer test format, the range of answers is narrow, so the degree of freedom is low and it is difficult to bring out characteristics such as the limits of the examinee's potential. For this reason, it is difficult to measure the depth of the examinee's potential.

[0017] In order to bring out the characteristics of potential abilities, it is preferable to use a one-to-many evaluation test format that asks for free-form responses to questions, rather than a one-question-one-answer test format. Free-form responses have a high degree of freedom, making it easier to bring out the characteristics of potential abilities.

[0018] FIG. 1 shows an example of a one-to-many assessment test based on free writing. The test in FIG. 1 includes Question 1 and Question 2, and test takers write free-form answers to Question 1 and Question 2, respectively. Answer 1 is written for Question 1, and Answer 2 is written for Question 2. Each answer i (i=1, 2) includes multiple written contents. Each written content corresponds to, for example, a paragraph, sentence, or clause included in Answer i.

[0019] The evaluator grades each answer i based on the evaluation items prepared in advance for each question i (i=1, 2). For question 1, evaluation items 1-1 and 1-2 are prepared, and for question 2, evaluation items 2-1 and 2-2 are prepared.

[0020] The evaluator checks whether or not there is a description corresponding to evaluation item ij (j=1,2) among the multiple descriptions included in answer i. Next, if there is a description corresponding to evaluation item ij, the evaluator checks whether the description is appropriate. If the description corresponding to evaluation item ij is appropriate, the evaluator assigns a score of 1, and if the description corresponding to evaluation item ij is inappropriate or there is no description corresponding to evaluation item ij, the evaluator assigns a score of 0.

[0021] Next, the evaluator calculates the sum of the scores for each of the evaluation items i-1 and i-2 as the score P(i) for answer i, and estimates the examinee's indirect ability based on the scores P(1) and P(2).

[0022] However, when the evaluator manually grades the free-form responses, the evaluator's workload increases and the efficiency of the task of estimating the examinee's ability decreases. If the number of examinees, the number of questions, or the number of evaluation items increases, the task efficiency decreases further.

[0023] In addition, test design, including the structure of questions and the method of evaluating answers, is difficult and often depends on the evaluator's knowledge and experience. For this reason, it is rare for the questions and evaluation methods in the initial design to be perfect. Therefore, it is desirable to initially set up a large number of evaluation items and then review each evaluation item according to the estimated results of the examinees' abilities. However, if the evaluator manually reviews the evaluation items, the workload of the evaluator increases.

[0024] 2 shows an example of the functional configuration of an ability estimation device according to an embodiment. The ability estimation device 201 in FIG. 2 includes a first inference unit 211, a second inference unit 212, an estimation unit 213, and an output unit 214.

[0025] Fig. 3 is a flowchart showing an example of a first ability estimation process performed by the ability estimation device 201 in Fig. 2. The correspondence inference model is generated by a first machine learning, and the answer evaluation result inference model is generated by a second machine learning.

[0026] First, the first inference unit 211 uses a correspondence inference model to infer a correspondence between each of a plurality of written contents contained in the respondent's free-form response to a question and each of a plurality of evaluation items contained in the evaluation criteria for evaluating the response (step 301).

[0027] Next, the second inference unit 212 infers the evaluation result of the answer for each of the multiple evaluation items from the answer and the corresponding relationship using the answer evaluation result inference model (step 302).

[0028] Next, the estimation unit 213 estimates the answerer's ability based on the evaluation results of the answers to each of the multiple evaluation items (step 303).Then, the output unit 214 outputs the estimation results of the answerer's ability (step 304).

[0029] According to the ability estimation device 201 of FIG. 2, the ability of the respondent can be efficiently estimated from the answer to the question.

[0030] Fig. 4 shows an example of the configuration of a capability estimation system including the capability estimation device 201 of Fig. 2. The capability estimation system of Fig. 4 includes a capability estimation device 401 and a storage device 402. The capability estimation device 401 corresponds to the capability estimation device 201 of Fig. 2.

[0031] A capacity estimation device 401 communicates with a storage device 402 via a communication network 403. The communication network 403 is, for example, a wide area network (WAN) or a local area network (LAN).

[0032] The storage device 402 stores a training data set for machine learning and answer data of examinees, and transmits the training data set or the answer data to the ability estimation device 401 in response to a request from the ability estimation device 401.

[0033] The ability estimation device 401 generates multiple models for ability estimation by performing machine learning using a training data set received from the storage device 402. Then, the ability estimation device 401 estimates the abilities of examinees from the answer data received from the storage device 402 using the multiple models.

[0034] The answer data is obtained by conducting a free-form test. The conducted test may be a test in the DS Aptitude Test Service, and the estimated ability may be an ability that indirectly indicates the examinee's potential ability.

[0035] Figure 5 shows examples of questions included in a test and the evaluation items for the questions. The test in Figure 5 includes questions 1 to I (I is an integer equal to or greater than 1), and test takers write free-form answers i to each question i (i = 1 to I). Test takers are an example of respondents.

[0036] The answer data of the examinee includes answers 1 to I. Each answer i includes a plurality of written contents. Each written content corresponds to, for example, a paragraph, a sentence, or a clause included in answer i.

[0037] The answer data is scored based on the evaluation criteria. The evaluation criteria includes multiple evaluation items ij (i = 1 to I, j = 1 to J(i)). J(i) is an integer of 1 or more determined according to question i. Each answer i to question i is scored using evaluation items i-1 to iJ(i).

[0038] 6 shows an example of the second ability estimation process performed by the ability estimation device 401. The ability estimation device 401 infers a set of correspondence relationships C(i,j) (i=1 to I, j=1 to J(i)) between each description included in each answer i of the answer data and each evaluation item ij included in the evaluation criteria.

[0039] The correspondence relationship C(i,j) includes information for identifying the description content corresponding to the evaluation item ij among the multiple description contents included in the answer i. Two or more description contents included in the answer i may correspond to the evaluation item ij, and the description content corresponding to the evaluation item ij may not be included in the answer i.

[0040] Next, the ability estimation device 401 infers a set of evaluation results E(i,j) of the answer i for each evaluation item ij from each answer i and each correspondence C(i,j). The evaluation results E(i,j) indicate, for example, the degree to which the description content corresponding to the evaluation item ij satisfies the conditions stipulated for the evaluation item ij. The evaluation results E(i,j) may indicate whether or not the description content corresponding to the evaluation item ij satisfies the conditions stipulated for the evaluation item ij.

[0041] Next, the ability estimation device 401 uses the set of evaluation results E(i,j) to estimate a set (k=1 to K) of scores S(k) for each of the examinee's abilities A(1) to A(K) (K is an integer equal to or greater than 1). For example, in the case of a test in a DS aptitude diagnostic service, the ability A(k) may be an indirect ability such as problem-solving ability, logical thinking ability, communication ability, critical thinking ability, metacognitive ability, etc.

[0042] Fig. 7 shows an example of a functional configuration of the ability estimation device 401 in Fig. 4. The ability estimation device 401 in Fig. 7 includes an acquisition unit 711, a training unit 712, an inference unit 713, an inference unit 714, an estimation unit 715, an evaluation unit 716, an output unit 717, and a storage unit 718.

[0043] 2. An inference unit 713, an inference unit 714, an estimation unit 715, and an output unit 717 correspond to the first inference unit 211, the second inference unit 212, the estimation unit 213, and the output unit 214 in FIG. 2, respectively.

[0044] The ability estimation device 401 operates in a training mode and an operation mode. In the training mode, the acquisition unit 711 transmits a data request R1 for requesting a training data set for machine learning to the storage device 402, receives a training data set 721 from the storage device 402, and stores it in the memory unit 718.

[0045] The training data set 721 includes a plurality of training data, each of which includes an answer i (i=1 to I) in free text form for each question i. Each training data further includes a set of correct answer correspondences C(i,j) (i=1 to I, j=1 to J(i)), a set of correct answer evaluation results E(i,j) (i=1 to I, j=1 to J(i)), and a set of correct answer scores S(k) (k=1 to K).

[0046] A set of correct answer correspondences C(i,j) is attached to each answer i. The correct answer evaluation results E(i,j) are attached to the evaluation item ij for each answer i. A set of correct answer scores S(k) is attached to the set of evaluation results E(i,j).

[0047] The training unit 712 generates a correspondence inference model 722 by performing a first machine learning using a training data set 721, and stores the correspondence inference model 722 in the storage unit 718. The correspondence inference model 722 is a trained machine learning model. In the first machine learning, each answer i included in each of the multiple training data is used as an input to the machine learning model, and a set of correct answer correspondences C(i,j) added to each answer i is used as a correct answer label.

[0048] Next, the training unit 712 generates an answer evaluation result inference model 723 by performing a second machine learning using the training data set 721, and stores the answer evaluation result inference model 723 in the storage unit 718. The answer evaluation result inference model 723 is a trained machine learning model. In the second machine learning, each answer i and each correspondence C(i,j) included in each of the multiple training data are used as inputs to the machine learning model, and the evaluation result E(i,j) of the correct answer added to the evaluation item ij for the answer i is used as a correct answer label.

[0049] A machine learning model for natural language processing is used as the correspondence inference model 722 and the answer evaluation result inference model 723. The machine learning model for natural language processing, for example, converts words or sentences included in the text data into feature vectors and performs a process of comparing two pieces of text data by calculating the similarity between the feature vectors.

[0050] As the machine learning model of natural language processing, for example, a neural network such as a Transformer series, Word2vec, or Doc2vec is used. The Transformer series neural network may be a BERT (Bidirectional Encoder Representations from Transformers) model. As the machine learning model of natural language processing, in addition to a neural network, other machine learning models such as a support vector machine or logistic regression can also be used.

[0051] Next, the training unit 712 generates a capability estimation model 724 by performing a third machine learning using the training data set 721, and stores the capability estimation model 724 in the storage unit 718. The capability estimation model 724 is a trained machine learning model. In the third machine learning, a set of evaluation results E(i,j) is used as an input for the machine learning model, and a set of correct scores S(k) added to the set is used as a correct label.

[0052] A machine learning model that estimates the score S(k) from the evaluation result E(i,j) is used as the ability estimation model 724. As the machine learning model that estimates the value, a neural network, linear regression, regression tree, etc. are used.

[0053] In the operation mode, the acquisition unit 711 transmits a data request R2 requesting answer data of an evaluation target to the storage device 402, receives answer data 725 from the storage device 402, and stores it in the memory unit 718. The answer data 725 includes free-form answers i (i=1 to I) to each question i.

[0054] Next, the inference unit 713 infers a set of correspondences C(i,j) (i=1 to I, j=1 to J(i)) from the response data 725 using the correspondence inference model 722. Then, the inference unit 713 stores in the storage unit 718 correspondence information 726 including the set of inferred correspondences C(i,j).

[0055] Fig. 8 shows an example of answer 1 to question 1 and evaluation items. Answer 1 in Fig. 8 includes description 801 to description 803. Description 801 is "The overall trend is toward a decrease," description 802 is "A in particular has been declining year by year," and description 803 is "On the other hand, B peaked at one point and has since been declining."

[0056] The evaluation items for question 1 include evaluation item 1-1, evaluation item 1-2, and evaluation item 1-3. Evaluation item 1-1 is "Are there any references based on a time series?", evaluation item 1-2 is "Are the differences in the trends between A and B pointed out?", and evaluation item 1-3 is "Are the factors behind the differences in the trends between A and B mentioned?"

[0057] In this case, description content 801 corresponds to evaluation item 1-1, and description content 802 and description content 803 correspond to evaluation item 1-2. Therefore, information specifying description content 801 is inferred as correspondence relationship C(1,1), and information specifying description content 802 and description content 803 is inferred as correspondence relationship C(1,2). However, since answer 1 does not contain description content corresponding to evaluation item 1-3, information indicating that no description content exists is inferred as correspondence relationship C(1,3).

[0058] Next, the inference unit 714 infers a set of evaluation results E(i, j) from the answer data 725 and the correspondence information 726 using the answer evaluation result inference model 723. Then, the inference unit 714 stores in the memory unit 718 answer evaluation results 727 including the set of inferred evaluation results E(i, j).

[0059] In the example of Figure 8, the description 801 "The overall trend is toward a decrease" identified by the correspondence relationship C(1,1) represents the "mention based on a time series axis" of the evaluation item 1-1. Therefore, a score of 1 is inferred as the evaluation result E(1,1). This evaluation result E(1,1) indicates that the description corresponding to the evaluation item 1-1 satisfies the condition stipulated in the evaluation item 1-1.

[0060] Additionally, description 802 "· In particular, A has been declining year by year" and description 803 "· Meanwhile, B peaked once and has been declining since then" identified by correspondence C(1,2) represent the "trend difference between A and B" in evaluation item 1-2. Therefore, a score of 1 is inferred as evaluation result E(1,2). This evaluation result E(1,2) indicates that the description corresponding to evaluation item 1-2 meets the conditions stipulated in evaluation item 1-2.

[0061] However, since there is no description content specified by the correspondence relationship C(1,3), a score of 0 is inferred as the evaluation result E(1,3). This evaluation result E(1,3) indicates that answer 1 does not meet the conditions specified in evaluation items 1-3.

[0062] Next, the estimation unit 715 estimates a set of scores S(k) from the answer evaluation results 727 using the ability estimation model 724. Then, the estimation unit 715 stores an ability estimation result 728 including the set of estimated scores S(k) in the memory unit 718. The output unit 717 outputs the ability estimation result 728 as an estimation result of the examinee's ability.

[0063] According to the ability estimation device 401 of FIG. 7, there is no need for an evaluator to manually score each answer i included in the answer data 725, and the ability estimation result 728 is automatically generated, thereby making it possible to efficiently estimate the abilities of examinees.

[0064] The evaluator can easily estimate the degree to which the examinee has ability A(k) from each score S(k) included in the ability estimation result 728. For example, in the case of a test in the DS aptitude diagnostic service, it becomes possible to evaluate the potential abilities of the examinee by estimating indirect abilities such as problem-solving ability, logical thinking ability, communication ability, critical thinking ability, and metacognitive ability.

[0065] In the operation mode, the ability estimation device 401 acquires answer data 725 of each of a plurality of examinees from the storage device 402, and generates correspondence information 726, answer evaluation results 727, and ability estimation results 728 for each examinee.

[0066] The evaluation unit 716 evaluates the evaluation criteria using correspondence information 726 calculated from the answer data 725 of each examinee, answer evaluation results 727, and ability estimation results 728, generates evaluation criterion evaluation results 729, and stores them in the memory unit 718. The output unit 717 outputs the evaluation criterion evaluation results 729.

[0067] The evaluation criterion evaluation results 729 include evaluation result F1, evaluation result F2, and evaluation result F3. Evaluation result F1 is an evaluation result obtained by evaluating the evaluation criteria using the correspondence relationship information 726 of each of the multiple examinees. Evaluation result F2 is an evaluation result obtained by evaluating the evaluation criteria using the answer evaluation results 727 of each of the multiple examinees. Evaluation result F3 is an evaluation result obtained by evaluating the evaluation criteria using the answer evaluation results 727 and ability estimation results 728 of each of the multiple examinees.

[0068] Each of the evaluation results F1, F2, and F3 includes, for example, candidate information indicating evaluation items ij that are candidates for deletion among the evaluation items ij included in the evaluation criteria.

[0069] For example, for each evaluation item ij, the evaluation unit 716 determines the number N(i,j) of correspondence relationships C(i,j) that indicate that one or more description contents of answer i correspond to evaluation item ij among the correspondence relationships C(i,j) included in the correspondence relationship information 726 for each of the multiple examinees. Then, the evaluation unit 716 compares N(i,j) with a threshold value T1, and if N(i,j) is smaller than the threshold value T1, records the evaluation item ij corresponding to N(i,j) in the evaluation result F1 as a candidate for deletion.

[0070] When N(i,j) is smaller than the threshold T1, the number of answers i that include description contents corresponding to the evaluation item ij is small, so that even if the evaluation item ij is deleted, it is determined that the effect on the examinee's ability estimation result 728 is small. In this case, the grading is simplified by deleting the evaluation item ij, so that the ability estimation process can be made more efficient based on the correspondence relationship C(i,j).

[0071] Figure 9 shows an example of the number N(i,j) of correspondences C(i,j). The total number represents the number N(1,j) of correspondences C(1,j) inferred for each evaluation item 1-j for answer 1. Therefore, the total number of correspondences C(1,j) is equal to or less than the total number of examinees.

[0072] The average confidence level represents the average confidence level of the correspondence C(1,j) for each of the examinees. The confidence level of the correspondence C(1,j) is an index showing the reliability of the inferred correspondence C(i,j) and is output from the correspondence inference model 722.

[0073] For example, when T1=10, the total number of evaluation items 1-2 is 2, which is less than T1, so evaluation item 1-2 is recorded as a candidate for deletion in evaluation result F1. The total number of evaluation items 1-1 is 200, which is more than T1, so evaluation item 1-1 is not recorded as a candidate for deletion.

[0074] The evaluation unit 716 obtains, for example, for each evaluation item ij, a distribution D(i,j) of the evaluation results E(i,j) included in the answer evaluation results 727 of each of the multiple examinees. Then, the evaluation unit 716 performs a statistical test to determine whether or not D(i,j) is a predetermined distribution, and if D(i,j) is not the predetermined distribution, records the evaluation item ij corresponding to D(i,j) in the evaluation result F2 as a candidate for deletion. The predetermined distribution may be a normal distribution, and the statistical test may be a Shapiro-Wilk test.

[0075] In the Shapiro-Wilk test, if the test statistic for the null hypothesis that D(i,j) is normally distributed is less than the significance level, D(i,j) is determined to not be normally distributed. On the other hand, if the test statistic is greater than the significance level, D(i,j) is determined to be normally distributed.

[0076] If D(i,j) is not normally distributed, the evaluation result E(i,j) corresponding to the evaluation item ij is biased, and the evaluation item ij is determined to be inappropriate for estimating the ability of the examinee. In this case, the grading can be simplified by deleting the evaluation item ij, and the ability estimation process can be made more efficient based on the evaluation result E(i,j).

[0077] FIG. 10 shows an example of the distribution D(i,j) of the evaluation result E(i,j). FIG. 10(a) shows an example of the distribution D(1,1) of the evaluation result E(1,1) for the evaluation item 1-1. The horizontal axis represents the score of the evaluation result E(1,1), and the vertical axis represents the number of examinees who were given the corresponding score. In this example, the score of the evaluation result E(i,j) is an integer in the range of 0 to 1000. Since D(1,1) in FIG. 10(a) is determined to be a normal distribution, the evaluation item 1-1 is not recorded as a candidate for deletion.

[0078] Figure 10(b) shows an example of distribution D(1,2) of evaluation result E(1,2) for evaluation item 1-2. Since D(1,2) in Figure 10(b) is determined not to be a normal distribution, evaluation item 1-2 is recorded in evaluation result F2 as a candidate for deletion.

[0079] The evaluation unit 716 calculates the importance G(i,j) of the evaluation result E(i,j) for each evaluation item ij using, for example, the answer evaluation results 727 and the ability estimation results 728 of each of the multiple examinees. The importance G(i,j) represents the degree of influence that the evaluation result E(i,j) has on the ability estimation result 728.

[0080] As the importance G(i,j), for example, permutation importance is used. Permutation importance is calculated by the following procedure.

[0081] (P1) The evaluation unit 716 sets the set of scores S(k) included in the ability estimation results 728 of each of the multiple examinees as a reference value for the ability estimation results.

[0082] (P2) The evaluation unit 716 randomly shuffles one of the evaluation results E(i,j) included in the answer evaluation results 727 of each of the multiple examinees between examinees. This generates a new set of evaluation results E(i,j) for each of the multiple examinees.

[0083] (P3) The evaluation unit 716 uses the ability estimation model 724 to estimate a new set of scores S(k) from the set of evaluation results E(i,j) for each of the multiple examinees generated in (P2).

[0084] (P4) The evaluation unit 716 calculates the difference between the set of scores S(k) of each of the examinees estimated in (P3) and the reference value set in (P1). As the difference, RMSE (Root Mean Squared Error) or the like is used.

[0085] (P5) The evaluation unit 716 repeats the processes (P2) to (P4) for each evaluation result E(i,j) and calculates the difference when each evaluation result E(i,j) is shuffled.

[0086] (P6) The evaluation unit 716 calculates the importance G(i,j) of the evaluation result E(i,j) based on the difference when each evaluation result E(i,j) is shuffled. For example, the evaluation unit 716 calculates an index that increases as the difference becomes larger as the importance G(i,j). The evaluation unit 716 may calculate the absolute value of the difference as the importance G(i,j).

[0087] Next, the evaluation unit 716 compares G(i,j) with a threshold T2, and if G(i,j) is smaller than the threshold T2, records the evaluation item ij corresponding to G(i,j) as a candidate for deletion in the evaluation result F3.

[0088] When G(i,j) is smaller than the threshold T2, deleting the evaluation item ij has little effect on the examinee's ability estimation result 728. In this case, deleting the evaluation item ij simplifies the grading, so that the ability estimation process can be made more efficient based on the evaluation result E(i,j) and the score S(k).

[0089] FIG. 11 shows an example of the importance G(i,j) of evaluation result E(i,j). The length of the horizontal bar for each evaluation item ij represents the importance G(i,j) of evaluation result E(i,j). In this example, the importance G(1,2) of evaluation result E(1,2) is smaller than T2, so evaluation item 1-2 is recorded in evaluation result F3 as a candidate for deletion. The importance of evaluation result E(1,1) and evaluation result E(1,3) is greater than T2, so evaluation item 1-1 and evaluation item 1-3 are not recorded as candidates for deletion.

[0090] In this way, by evaluating each evaluation item ij of the evaluation criteria using the correspondence relationship information 726, answer evaluation result 727, and ability estimation result 728 of each of the multiple examinees, it is possible to identify evaluation items ij that can be deleted. By deleting the identified evaluation items ij from the evaluation criteria, the evaluator can simplify the evaluation criteria and make the test more efficient. The evaluator can also improve the evaluation criteria by adding another evaluation item ij in place of the deleted evaluation item ij. This makes it more efficient to review the evaluation criteria.

[0091] The evaluation unit 716 may generate an evaluation criteria evaluation result 729 that includes only one or two of the evaluation result F1, the evaluation result F2, or the evaluation result F3.

[0092] Fig. 12 is a flowchart showing an example of a training process performed by the ability estimation device 401 in Fig. 7 in the training mode. First, the acquisition unit 711 acquires a training data set 721 from the storage device 402 by transmitting a data request R1 to the storage device 402 (step 1201).

[0093] Next, the training unit 712 generates a correspondence inference model 722 by performing a first machine learning using the training data set 721 (step 1202). Next, the training unit 712 generates an answer evaluation result inference model 723 by performing a second machine learning using the training data set 721 (step 1203). Next, the training unit 712 generates a capability estimation model 724 by performing a third machine learning using the training data set 721 (step 1204).

[0094] 13A and 13B are flowcharts showing an example of the second ability estimation process performed by the ability estimation device 401 of Fig. 7 in the operation mode. First, the acquisition unit 711 acquires the response data 725 from the storage device 402 by transmitting a data request R2 to the storage device 402 (step 1301).

[0095] Next, the inference unit 713 uses the correspondence inference model 722 to infer a set of correspondences C(i,j) from the response data 725, and generates correspondence information 726 including the set of correspondences C(i,j) (step 1302).

[0096] Next, the evaluation unit 716 evaluates the evaluation criteria using the correspondence relationship information 726 of each of the examinees, including the generated correspondence relationship information 726 (step 1303).Then, the evaluation unit 716 checks whether or not there is an evaluation item ij that is a candidate for deletion (step 1304).If there is an evaluation item ij that is a candidate for deletion (step 1304, YES), the evaluation unit 716 records the evaluation item ij in the evaluation result F1 (step 1305).

[0097] Next, the inference unit 714 uses the answer evaluation result inference model 723 to infer a set of evaluation results E(i, j) from the answer data 725 and the correspondence information 726, and generates answer evaluation results 727 including the set of evaluation results E(i, j) (step 1306).

[0098] Next, the evaluation unit 716 evaluates the evaluation criteria using the answer evaluation results 727 of each of the examinees, including the generated answer evaluation results 727 (step 1307).Then, the evaluation unit 716 checks whether or not there is an evaluation item ij that is a candidate for deletion (step 1308).If there is an evaluation item ij that is a candidate for deletion (step 1308, YES), the evaluation unit 716 records the evaluation item ij in the evaluation result F2 (step 1309).

[0099] Next, the estimation unit 715 uses the ability estimation model 724 to estimate a set of scores S(k) from the answer evaluation results 727, and generates an ability estimation result 728 including the set of scores S(k) (step 1310).

[0100] Next, the evaluation unit 716 evaluates the evaluation criteria using the answer evaluation results 727 and ability estimation results 728 of each of the multiple examinees, including the generated answer evaluation results 727 and ability estimation results 728 (step 1311).Then, the evaluation unit 716 checks whether or not there is an evaluation item ij that is a candidate for deletion (step 1312).If there is an evaluation item ij that is a candidate for deletion (step 1312, YES), the evaluation unit 716 records the evaluation item ij in the evaluation result F3 (step 1313).

[0101] Next, the evaluation unit 716 generates an evaluation criterion evaluation result 729 including the evaluation result F1, the evaluation result F2, and the evaluation result F3 (step 1314), and the output unit 717 outputs the ability estimation result 728 and the evaluation criterion evaluation result 729 (step 1315).

[0102] If there is no evaluation item ij that is a candidate for deletion in step 1304 (step 1304, NO), the ability estimation device 401 performs the processes from step 1306 onwards. If there is no evaluation item ij that is a candidate for deletion in step 1308 (step 1308, NO), the ability estimation device 401 performs the processes from step 1310 onwards. If there is no evaluation item ij that is a candidate for deletion in step 1312 (step 1312, NO), the ability estimation device 401 performs the processes from step 1314 onwards.

[0103] The configurations of the ability estimation device 201 in Fig. 2 and the ability estimation device 401 in Fig. 7 are merely examples, and some of the components may be omitted or changed depending on the application or conditions of the ability estimation device. For example, in the ability estimation device 401 in Fig. 7, if the correspondence inference model 722, the answer evaluation result inference model 723, and the ability estimation model 724 are generated by an external device, the training unit 712 can be omitted. If the evaluation criteria are not evaluated, the evaluation unit 716 can be omitted.

[0104] The configuration of the ability estimation system in FIG. 4 is merely an example, and some of the components may be omitted or changed depending on the application or conditions of the ability estimation system.

[0105] The flowcharts in Figures 3, 12, 13A, and 13B are merely examples, and some of the processes may be omitted or changed depending on the configuration or conditions of the ability estimation device. For example, in the ability estimation device 401 in Figure 7, if the correspondence inference model 722, the answer evaluation result inference model 723, and the ability estimation model 724 are generated by an external device, the training process in Figure 12 can be omitted.

[0106] When generating an evaluation criterion evaluation result 729 that does not include evaluation result F1, the processes of steps 1303 to 1305 in Fig. 13A can be omitted. When generating an evaluation criterion evaluation result 729 that does not include evaluation result F2, the processes of steps 1307 to 1309 in Fig. 13A can be omitted. When generating an evaluation criterion evaluation result 729 that does not include evaluation result F3, the processes of steps 1311 to 1313 in Fig. 13B can be omitted.

[0107] When the evaluation criteria are not evaluated, the processes of steps 1303 to 1305 and steps 1307 to 1309 in FIG. 13A and steps 1311 to 1313 in FIG. 13B can be omitted.

[0108] The questions and evaluation items shown in Figure 1 are merely examples, and the number of questions and evaluation items varies depending on the test. The questions and evaluation items shown in Figures 5 and 6 are merely examples, and evaluation item ij for any question i may be omitted. The answers and evaluation items shown in Figure 8 are merely examples, and the answers and evaluation items vary depending on the test and the examinee.

[0109] The number of correspondence relationships C(i,j) shown in Fig. 9, the distribution of evaluation results E(i,j) shown in Fig. 10, and the importance of evaluation results E(i,j) shown in Fig. 11 are merely examples. The number of correspondence relationships C(i,j), the distribution of evaluation results E(i,j), and the importance of evaluation results E(i,j) change depending on the test.

[0110] FIG. 14 shows an example of the hardware configuration of an information processing device (computer) used as the ability estimation device 201 in FIG. 2 and the ability estimation device 401 in FIG.

[0111] 14 includes a CPU (Central Processing Unit) 1401, a memory 1402, an input device 1403, and an output device 1404. The information processing device further includes an auxiliary storage device 1405, a medium drive device 1406, a network connection device 1407, and a GPU (Graphics Processing Unit) 1408. These components are hardware, and are connected to each other via a bus 1409.

[0112] The memory 1402 is, for example, a semiconductor memory such as a Read Only Memory (ROM) or a Random Access Memory (RAM), and stores programs and data used in processing. The memory 1402 may operate as the storage unit 718 in FIG.

[0113] The CPU 1401 (processor) operates as the first inference unit 211, the second inference unit 212, and the estimation unit 213 in FIG.

[0114] 7 by executing a program using the memory 1402. The CPU 1401 may also operate as the acquisition unit 711, the inference unit 713, the inference unit 714, the estimation unit 715, and the evaluation unit 716 by executing a program using the memory 1402.

[0115] The input device 1403 is, for example, a keyboard, a pointing device, etc., and is used to input instructions or information from a user or operator. The output device 1404 is, for example, a display device, a printer, etc., and is used to output inquiries or instructions to a user or operator and processing results. The output device 1404 may operate as the output unit 214 in Fig. 2 or the output unit 717 in Fig. 7, and the processing results may be the ability estimation result 728 and the evaluation criterion evaluation result 729.

[0116] The auxiliary storage device 1405 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 1405 may be a hard disk drive or a solid state drive (SSD). The information processing device can store programs and data in the auxiliary storage device 1405 and use them by loading them into the memory 1402. The auxiliary storage device 1405 may operate as the storage unit 718 in FIG. 7.

[0117] The medium drive device 1406 drives the portable recording medium 1410 and accesses the recorded contents. The portable recording medium 1410 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, etc. The portable recording medium 1410 may be a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, etc. A user or operator can store programs and data in the portable recording medium 1410 and load them into the memory 1402 for use.

[0118] Thus, the computer-readable recording medium that stores the program and data used in the processing is a physical (non-transitory) recording medium such as the memory 1402, the auxiliary storage device 1405, or the portable recording medium 1410.

[0119] The network connection device 1407 is a communication interface circuit that is connected to the communication network 403 and performs data conversion associated with communication. The information processing device can receive programs and data from an external device via the network connection device 1407 and load them into the memory 1402 for use. The network connection device 1407 may operate as the output unit 214 in FIG. 2 or the output unit 717 in FIG. 7.

[0120] The GPU 1408 is a computing circuit that performs information processing. The GPU 1408 may operate as the training unit 712 in FIG.

[0121] It is not necessary for the information processing device to include all of the components in Fig. 14, and some of the components may be omitted or changed depending on the purpose or conditions of the information processing device. For example, if an interface with a user or operator is not required, the input device 1403 and the output device 1404 may be omitted. If the portable recording medium 1410 or the communication network 403 is not used, the medium drive device 1406 or the network connection device 1407 may be omitted.

[0122] Although the disclosed embodiments and their advantages have been described in detail, it will be appreciated that those skilled in the art may make various modifications, additions and omissions therein without departing from the scope of the invention as clearly set forth in the claims.

[0123] The following supplementary notes are further disclosed regarding the embodiment described with reference to FIGS. (Appendix 1) Using the correspondence inference model generated by the first machine learning, infer a correspondence between each of a plurality of description contents included in the respondent's free-text response to the question and each of a plurality of evaluation items included in the evaluation criteria for evaluating the response; inferring an evaluation result of the answer for each of the plurality of evaluation items from the answer and the correspondence relationship using an answer evaluation result inference model generated by the second machine learning; estimating the ability of the respondent based on an evaluation result of the answer to each of the plurality of evaluation items; outputting the estimation result of the respondent's ability; A capability estimation program for causing a computer to execute a process. (Appendix 2) Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; evaluating the evaluation criteria based on the correspondence between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items; outputting the evaluation result of the evaluation criteria; 2. The ability estimation program according to claim 1, further comprising causing the computer to execute a process. (Appendix 3) Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; using the answer evaluation result inference model, inferring an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; Evaluating the evaluation criteria based on an evaluation result of each of the answers of the plurality of respondents to each of the plurality of evaluation items; outputting the evaluation result of the evaluation criteria; 2. The ability estimation program according to claim 1, further comprising causing the computer to execute a process. (Appendix 4) Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; using the answer evaluation result inference model, inferring an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; estimating the ability of each of the plurality of respondents based on an evaluation result of the answer of each of the plurality of respondents to each of the plurality of evaluation items; evaluating the evaluation criteria based on an evaluation result of the answer of each of the plurality of respondents to each of the plurality of evaluation items and an estimation result of the ability of each of the plurality of respondents; outputting the evaluation result of the evaluation criteria; 2. The ability estimation program according to claim 1, further comprising causing the computer to execute a process. (Appendix 5) A capability estimation program according to any one of appendices 2 to 4, characterized in that the evaluation results of the evaluation criteria include candidate information indicating evaluation items that are candidates for deletion among the plurality of evaluation items. (Appendix 6) a first inference unit that uses a correspondence inference model generated by the first machine learning to infer a correspondence between each of a plurality of description contents included in a free-form response by a respondent to a question and each of a plurality of evaluation items included in an evaluation criterion for evaluating the response; a second inference unit that infers an evaluation result of the answer for each of the plurality of evaluation items from the answer and the correspondence relationship by using an answer evaluation result inference model generated by a second machine learning; an estimation unit that estimates the ability of the respondent based on an evaluation result of the answer to each of the plurality of evaluation items; an output unit that outputs the estimation result of the respondent's ability; A capability estimation device comprising: (Appendix 7) An evaluation unit that evaluates the evaluation criteria, The first inference unit uses the correspondence inference model to infer a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items; the evaluation unit evaluates the evaluation criteria based on the correspondence between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items; The ability estimation device according to claim 6, wherein the output unit outputs an evaluation result of the evaluation criteria. (Appendix 8) An evaluation unit that evaluates the evaluation criteria, The first inference unit uses the correspondence inference model to infer a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items; the second inference unit infers, using the answer evaluation result inference model, an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; the evaluation unit evaluates the evaluation criteria based on evaluation results of answers of each of the plurality of respondents to each of the plurality of evaluation items; The ability estimation device according to claim 6, wherein the output unit outputs an evaluation result of the evaluation criteria. (Appendix 9) An evaluation unit that evaluates the evaluation criteria, The first inference unit uses the correspondence inference model to infer a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items; the second inference unit infers, using the answer evaluation result inference model, an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; the estimation unit estimates a capability of each of the plurality of respondents based on an evaluation result of the answer of each of the plurality of respondents to each of the plurality of evaluation items; the evaluation unit evaluates the evaluation criteria based on an evaluation result of an answer of each of the plurality of respondents to each of the plurality of evaluation items and an estimation result of an ability of each of the plurality of respondents; The ability estimation device according to claim 6, wherein the output unit outputs an evaluation result of the evaluation criteria. (Appendix 10) A capability estimation device according to any one of appendices 7 to 9, characterized in that the evaluation results of the evaluation criteria include candidate information indicating evaluation items that are candidates for deletion among the plurality of evaluation items. (Appendix 11) Using the correspondence inference model generated by the first machine learning, infer a correspondence between each of a plurality of description contents included in the respondent's free-text response to the question and each of a plurality of evaluation items included in the evaluation criteria for evaluating the response; inferring an evaluation result of the answer for each of the plurality of evaluation items from the answer and the correspondence relationship using an answer evaluation result inference model generated by the second machine learning; estimating the ability of the respondent based on an evaluation result of the answer to each of the plurality of evaluation items; outputting the estimation result of the respondent's ability; A method for estimating ability, characterized in that the processing is executed by a computer. (Appendix 12) Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; evaluating the evaluation criteria based on the correspondence between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items; outputting the evaluation result of the evaluation criteria; 12. The ability estimation method according to claim 11, further characterized in that the computer executes processing. (Appendix 13) Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; using the answer evaluation result inference model, inferring an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; Evaluating the evaluation criteria based on an evaluation result of each of the answers of the plurality of respondents to each of the plurality of evaluation items; outputting the evaluation result of the evaluation criteria; 12. The ability estimation method according to claim 11, further characterized in that the computer executes processing. (Appendix 14) Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; using the answer evaluation result inference model, inferring an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; estimating the ability of each of the plurality of respondents based on an evaluation result of the answer of each of the plurality of respondents to each of the plurality of evaluation items; evaluating the evaluation criteria based on an evaluation result of the answer of each of the plurality of respondents to each of the plurality of evaluation items and an estimation result of the ability of each of the plurality of respondents; outputting the evaluation result of the evaluation criteria; 12. The ability estimation method according to claim 11, further characterized in that the computer executes processing. (Appendix 15) A method for estimating ability described in any one of Appendices 12 to 14, characterized in that the evaluation results of the evaluation criteria include candidate information indicating evaluation items that are candidates for deletion among the multiple evaluation items. [Explanation of symbols]

[0124] 201, 401 Capacity estimation device 211 1st Reasoning Part 212 Second Reasoning Part 213, 715 Estimation Department 214, 717 Output section 402 Storage device 403 Communication Network 711 Acquisition Department 712 Training Department 713, 714 Reasoning part 716 Evaluation Department 718 Storage section 721 Training Data Set 722 Correspondence Inference Model 723 Answer Evaluation Result Inference Model 724 Ability Estimation Model 725 Response Data 726 Correspondence Information 727 Answer Evaluation Results 728 Ability estimation result 729 Evaluation criteria Evaluation results 801~803 Description 1401 CPU 1402 Memory 1403 Input Device 1404 Output Device 1405 Auxiliary storage device 1406 Media Drive Unit 1407 Network connection device 1408 GPU 1409 Bus 1410 Portable recording media

Claims

1. Using the correspondence inference model generated by the first machine learning, infer a correspondence between each of a plurality of description contents included in the respondent's free-text response to the question and each of a plurality of evaluation items included in the evaluation criteria for evaluating the response; using an answer evaluation result inference model generated by the second machine learning, inferring an evaluation result of the answer for each of the plurality of evaluation items from the answer and the correspondence relationship; estimating the ability of the respondent based on an evaluation result of the answer to each of the plurality of evaluation items; outputting the estimation result of the respondent's ability; A capability estimation program for causing a computer to execute a process.

2. Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; evaluating the evaluation criteria based on the correspondence between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items; outputting the evaluation result of the evaluation criteria; 2. The ability estimation program according to claim 1, further comprising instructions to cause the computer to execute a process.

3. Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; using the answer evaluation result inference model, inferring an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; Evaluating the evaluation criteria based on an evaluation result of each of the answers of the plurality of respondents to each of the plurality of evaluation items; outputting the evaluation result of the evaluation criteria; 2. The ability estimation program according to claim 1, further comprising instructions to cause the computer to execute a process.

4. Using the correspondence inference model, a correspondence between each of a plurality of description contents included in the free-text responses of each of a plurality of respondents to the question and each of the plurality of evaluation items is inferred; using the answer evaluation result inference model, inferring an evaluation result of each of the answers of the plurality of respondents for each of the plurality of evaluation items from the correspondence relationship between each of the plurality of description contents included in the answer of each of the plurality of respondents and each of the plurality of evaluation items and from the answer of each of the plurality of respondents; estimating the ability of each of the plurality of respondents based on an evaluation result of the answer of each of the plurality of respondents to each of the plurality of evaluation items; evaluating the evaluation criteria based on an evaluation result of the answer of each of the plurality of respondents to each of the plurality of evaluation items and an estimation result of the ability of each of the plurality of respondents; outputting the evaluation result of the evaluation criteria; 2. The ability estimation program according to claim 1, further comprising instructions to cause the computer to execute a process.

5. 5. The ability estimating program according to claim 2, wherein the evaluation result of the evaluation criteria includes candidate information indicating evaluation items that are candidates for deletion among the plurality of evaluation items.

6. a first inference unit that uses a correspondence inference model generated by the first machine learning to infer a correspondence between each of a plurality of description contents included in a free-form response by a respondent to a question and each of a plurality of evaluation items included in an evaluation criterion for evaluating the response; a second inference unit that infers an evaluation result of the answer for each of the plurality of evaluation items from the answer and the correspondence relationship by using an answer evaluation result inference model generated by a second machine learning; an estimation unit that estimates the ability of the respondent based on an evaluation result of the answer to each of the plurality of evaluation items; an output unit that outputs the estimation result of the respondent's ability; A capability estimation device comprising:

7. Using the correspondence inference model generated by the first machine learning, infer a correspondence between each of a plurality of description contents included in the respondent's free-text response to the question and each of a plurality of evaluation items included in the evaluation criteria for evaluating the response; using an answer evaluation result inference model generated by the second machine learning, inferring an evaluation result of the answer for each of the plurality of evaluation items from the answer and the correspondence relationship; estimating the ability of the respondent based on an evaluation result of the answer to each of the plurality of evaluation items; outputting the estimation result of the respondent's ability; A method for estimating ability, characterized in that the processing is executed by a computer.

Images