Test result analysis system and method

The test result analysis system provides detailed performance insights by classifying and analyzing test questions and answers, allowing test takers to identify areas for improvement before official results, enhancing study effectiveness.

JP7593618B2Active Publication Date: 2024-12-03FORESIGHT CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2020216656
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-12-25
Publication Date
2024-12-03
Estimated Expiration
2040-09-08

AI Technical Summary

Technical Problem

Existing systems fail to provide instant analysis of test results before pass/fail announcements, limiting test takers' ability to gauge their performance and identify areas for improvement.

Method used

A test result analysis system that collects answer data, estimates pass/fail status, classifies questions by level and type, and analyzes answers of presumed unsuccessful candidates based on successful candidates' data, providing detailed performance insights before official results are released.

Benefits of technology

Enables test takers to receive objective and informative analysis of their performance, identifying strengths and weaknesses before official results, facilitating targeted study and improvement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593618000001
    Figure 0007593618000001
  • Figure 0007593618000002
    Figure 0007593618000002
  • Figure 0007593618000003
    Figure 0007593618000003
Patent Text Reader

Abstract

To provide a technology that enables test results to be analyzed even before the results are announced. [Solution] The test result analysis system has an answer data collection unit that collects answer data indicating the answers to each question submitted in a test containing multiple questions by multiple test takers; a pass / fail estimation unit that estimates the pass / fail status of each of the multiple test takers based on the answer data of the multiple test takers; and an answer analysis unit that analyzes the answers to the questions of presumed failers who are presumed to have failed based on the answer data of presumed passers who are presumed to have passed.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to techniques for analyzing test takers' test results before the results are announced. [Background technology]

[0002] In qualification exams, there is a strong demand from test-takers to know their scores and chances of passing before the results are announced, and what they need to know if they do not pass. However, the correct answers and passing grades for the actual exam are not usually made public. Therefore, educational institutions that provide courses and content for studying for the exam publish their own model answers and expected passing grades on their websites immediately after the actual exam. By viewing these websites, test-takers can predict their scores and chances of passing.

[0003] Patent Document 1 discloses a system for predicting the rank in a test and the possibility of passing. The system in Patent Document 1 stores various questions for a mock test in a database, and in response to a user's request to select a question, a server extracts the relevant question from the database and displays it on the display of the corresponding user terminal, allowing the user to take the mock test. In addition, the user can obtain information on the predicted rank in the mock test and the pass / fail outcome from the system. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2002-72857 A Summary of the Invention [Problem to be solved by the invention]

[0005] The system of Patent Document 1 predicts the possibility of passing the actual exam from the results of the mock exam. The mock exam measures the level of the learner's academic ability at the time of the mock exam using test questions and answers prepared by the educational institution itself, and does not analyze the results of the actual exam based on the answers given in the actual exam. Therefore, after taking the mock exam, the examinee can obtain information that will be useful for studying until the actual exam, but it does not satisfy the need to obtain information on the results of the actual exam as soon as possible.

[0006] One object of the present disclosure is to provide a technique that allows for analysis of test results even before the results are announced. [Means for solving the problem]

[0007] A test result analysis system according to one aspect of the present disclosure includes an answer data collection unit that collects answer data indicating answers to each question submitted in a test including a plurality of questions by a plurality of test takers, a pass / fail estimation unit that estimates the pass / fail status of each of the plurality of test takers based on the answer data of the plurality of test takers, and an answer analysis unit that analyzes the answers to the questions of presumed failing test takers who are presumed to have failed, based on the answer data of presumed passing test takers who are presumed to have passed. Effect of the Invention

[0008] According to one aspect of the present disclosure, multiple test takers are divided into presumed passers and presumed failers, and the answers to each question of the presumed failers are analyzed based on the answer data of the presumed passers, making it possible to analyze the test results even before the pass / fail results are announced. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a test result analysis system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a block diagram showing a computer system according to an embodiment of the present invention. [Diagram 3] 4 is a flowchart showing a test result analysis process executed by the test result analysis system. [Figure 4] 13 is a flowchart showing a question classification process. [Diagram 5] 1 is a table showing examples of question types. [Figure 6] FIG. 13 is a diagram showing an example of a true / false question. [Figure 7] FIG. 13 is a diagram showing an example of a multiple-choice question. [Figure 8] FIG. 13 is a diagram illustrating an example of a combination problem. [Figure 9] FIG. 13 is a diagram showing an example of a multiple-choice question. [Figure 10] FIG. 13 is a diagram showing an example of a calculation problem. [Figure 11] FIG. 13 is a diagram illustrating an example of a quantity problem. [Figure 12] FIG. 13 is a diagram showing an example of a descriptive question. [Figure 13] 1 is a table showing examples of problem levels. [Figure 14] FIG. 11 is a diagram illustrating an example of problem information. [Figure 15] 13 is a flowchart showing an example of a prerequisite knowledge determination process. [Figure 16] FIG. 11 is a diagram illustrating an example of field information. [Figure 17] 13 is a flowchart showing an example of an answer format identification process. [Figure 18] 13 is a flowchart showing an example of a problem level evaluation process. [Figure 19] 13 is a flowchart of an evaluation process by answer format. [Figure 20] FIG. 11 is a diagram showing an example of question level determination information. [Figure 21] FIG. 13 is a diagram showing an example of judgment logic information for the multiple-choice question comprehensive evaluation process. [Figure 22] 13 is a flowchart showing an example of a multiple-choice question comprehensive evaluation process. [Diagram 23] 13 is a flowchart showing an example of an answer analysis process. [Figure 24] FIG. 11 is a diagram illustrating an example of criteria information. [Diagram 25] FIG. 13 is a diagram showing an example of a subject-specific determination result. [Figure 26] 13 is a table showing the analysis results of the problem-specific analysis. [Figure 27] 13 is another table showing the analysis results of the problem-specific analysis. [Figure 28] 13 is yet another table showing the analysis results of the problem-specific analysis. [Figure 29] 13 is yet another table showing the analysis results of the problem-specific analysis. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] 1 is a block diagram showing a test result analysis system according to the present embodiment. The test result analysis system 10 is a system that makes it possible to analyze the test results of examinees in a test with a pass / fail result before the pass / fail result is announced.

[0012] The exams referred to here are, for example, exams that are held periodically. Examples of exams that are held periodically, such as once a year, include national exams and other qualification exams, and entrance exams for various schools. There are many other exams that are held periodically, such as once a year or several times a year. In many cases, candidates who fail continue to study and take the next exam again. Therefore, analyzing the results of an exam is beneficial for candidates who failed, especially in terms of their studies for the next exam.

[0013] The exam contains multiple questions. If the total score obtained by answering the questions correctly exceeds the pass mark, the candidate passes. If the total score falls below the pass mark, the candidate fails. For example, the Real Estate Transaction Agent Qualification Exam is a test format in which candidates answer 50 questions in two hours.

[0014] It is empirically known that there are questions in the exam that you must answer correctly to pass, and questions that will have little impact on your success even if you cannot answer them correctly. It is said that you can pass the exam if you can reliably answer questions at the same level that other test takers answer correctly, and that the shortcut to passing is to acquire the ability to reliably answer questions at that level.

[0015] Referring to FIG. 1, a test result analysis system 10 includes a pass grade acquisition unit 11 , an answer data collection unit 12 , a pass / fail estimation unit 13 , a question classification unit 14 , and an answer analysis unit 15 .

[0016] The pass line acquiring unit 11 acquires the minimum number of questions (number of correct answers) that must be answered correctly to pass the exam as the pass line. If the exam contains 50 questions and 35 or more questions are answered correctly to pass, the pass line is 35. For example, the pass line acquiring unit 11 may be given a predetermined pass rate (for example, an average) based on the pass rate in previous years, and determine the pass line based on the number of examinees from whom answer data was obtained and the pass rate. If answer data is obtained from 100 examinees and the pass rate is 20%, the pass line acquiring unit 11 may determine the pass line so that the top 20 examinees pass.

[0017] The answer data collection unit 12 collects answer data from multiple examinees. The answer data is data indicating the answers to each question submitted by the examinee in the test. For example, the answer data collection unit 12 provides a web page including a form for the examinee to input their answers so that the examinee can access it from an information processing terminal (not shown) such as a personal computer, tablet terminal, or smartphone, and acquires the answer data entered from the web page. It is not necessary to obtain answer data from all examinees who took the test. However, obtaining answer data from as many examinees as possible increases the likelihood of obtaining good analysis results.

[0018] The pass / fail estimation unit 13 estimates the pass / fail status of each of the examinees based on the collected answer data of the examinees. By comparing the number of correct answers of the examinee with the pass mark, it is possible to determine whether the examinee is estimated to have passed or failed. Here, an examinee estimated to have passed will be referred to as an estimated passer, and an examinee estimated to have failed will be referred to as an estimated failer.

[0019] The question classification unit 14 classifies the questions included in the test into a plurality of question levels based on the correct answer rate in the answer data of the presumed successful candidates. The question classification unit 14 calculates the correct answer rate for each question in the answer data of the presumed successful candidates, and determines which question level each question belongs to by comparing the correct answer rate with a threshold value. The details of the question levels will be described later.

[0020] The question classification unit 14 further classifies the questions included in the test into a plurality of question types based on the answer format. The question classification unit 14 identifies the answer format of each question based on the question sentence of each question included in the question data, and determines the question type of each question according to the answer format. The types of answer formats and the method of identifying the answer format of each question will be described later.

[0021] The answer analysis unit 15 analyzes the answers given to each question by the presumed failing candidates based on the answer data of the presumed passing candidates. According to this embodiment, multiple examinees are divided into presumed passing candidates and presumed failing candidates, and the answers given to each question by the presumed failing candidates are analyzed based on the answer data of the presumed passing candidates, so that the test results can be analyzed before the pass / fail results are announced.

[0022] For example, if the test includes questions on multiple subjects, the answer analysis unit 15 calculates the standard deviation of the presumed failing candidates for each subject based on the answer data of the presumed passing candidates. The average score of the presumed passing candidates will be a standard deviation of 50.

[0023] In addition, the answer analysis unit 15 calculates the presumptive successful candidates' correct answer rate for each question in the answer data of the presumptive successful candidates, which is the proportion of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, and analyzes the answer to that question of the presumptive unsuccessful candidates being analyzed based on the presumptive successful candidates' correct answer rate for that question.

[0024] According to this embodiment, the answers of the presumed unsuccessful candidates to each question are analyzed based on the correct answer rate of the presumed successful candidates, so that it is possible to perform an analysis that takes into account the percentage of correct answers of the presumed successful candidates before the results are announced. Note that the pass / fail estimate of the examinee does not necessarily need to be accurate. Even if the pass / fail cannot be accurately determined, the correct answer rate of the top candidates shows the same tendency as the correct answer rate of the successful candidates in the actual test, so that an analysis that takes into account the tendency of the successful candidates can provide useful information.

[0025] For example, the answer analysis unit 15 analyzes the answers of the presumed failing person to the question based on the question level of each question classified by the question classification unit 14 and the correct answer rate of the presumed passer. The analysis here includes calculating the number of correct answers and / or the correct answer rate of the presumed failing person for each question level. A question that many presumed passers answer correctly is likely to be a question that cannot be answered correctly in order to pass. For example, if the correct answer rate of the presumed passers is 80% or more, it is likely to be a question that must be answered correctly in order to pass. If such a question is not answered correctly by the presumed failing person to be analyzed, the academic ability of the presumed failing person may be insufficient, or the question may be a problem that the presumed failing person is not good at. The answer analysis unit 15 presents the analysis result to the presumed failing person. For example, the answer analysis unit 15 displays the correct answer rate for each question level of the presumed failing person to be analyzed in a web page in a table or graph format based on the analysis result.

[0026] According to this embodiment, the answers of the presumed failing candidates to each question are analyzed based on the degree to which it is required to answer the question correctly in order to pass the exam and the percentage of correct answers of the presumed passers for that question. Therefore, before the results are announced, an effective analysis can be performed that takes into account the degree to which it is required to answer the question correctly in order to pass the exam and the percentage of correct answers of the presumed passers.

[0027] Furthermore, for example, the answer analysis unit 15 further analyzes the answers of the presumed failing students to the questions based on the question type and the correct answer rate of the presumed failing students classified by the question classification unit 14. The analysis here includes calculating the number of correct answers and / or the correct answer rate of the presumed failing students for each question type. The answer analysis unit 15 presents the analysis results to the presumed failing students. For example, based on the analysis results, the answer analysis unit 15 displays the correct answer rate for each question type for the presumed failing students being analyzed in a tabular or graph format on a web page.

[0028] According to this embodiment, the answers of the presumed successful applicants to each question are analyzed based on the format of the answer to that question and the percentage of correct answers of the presumed successful applicants to that question, so that an effective analysis can be performed before the results are announced, taking into consideration the format of the answer to that question and the percentage of correct answers of the presumed successful applicants to that question.

[0029] In addition, if the target exam includes questions on multiple subjects, the answer analysis unit 15 may calculate the standard deviation of presumed failing candidates for the entire exam and for each subject therein based on the answer data of presumed passing candidates.

[0030] According to this embodiment, in an exam which includes multiple subjects and the pass / fail decision is based on the total score of those subjects, the standard deviation for each subject of the presumed failing candidates is calculated based on the answer data of the presumed passing candidates, so that the performance of the presumed failing candidates in each subject can be objectively analyzed.

[0031] The test result analysis system 10 of this embodiment can be realized by one or more computers.

[0032] Fig. 2 is a block diagram showing a computer system according to this embodiment. The computer system 20 is a computer that realizes the test result analysis system 10 shown in Fig. 1. The computer system 20 has, as hardware, a processor 21, a main memory 22, a storage device 23, a communication device 24, an input device 25, and a display device 26, which are connected to a bus 27.

[0033] The storage device 23 is a device for storing data in a writable and readable manner. The pass rate data, answer data, correct answer data, question data, and analysis results shown in FIG.

[0034] Processor 21 is a processor that reads data stored in storage device 23 into main memory 22 and executes processing of the software program using main memory 22. Processor 21 realizes pass grade acquisition unit 11, answer data collection unit 12, pass / fail estimation unit 13, question classification unit 14, and answer analysis unit 15 shown in FIG.

[0035] The communication device 24 is a device that transmits and receives data to and from an external device such as an information processing terminal via a communication line (not shown) such as a communication network. For example, answer data is input via the communication device 24 and stored in the storage device 23.

[0036] The input device 25 is a device that receives information input by an operator using a keyboard, mouse, or the like, and the input information is used for software processing by the processor 21. When the operator performs an operation on the input device 25 to instruct a desired process, the processor 21 executes the process instructed by that operation.

[0037] The display device 26 is a device that displays image and text information on a display screen in accordance with software processing by the processor 21.

[0038] FIG. 3 is a flowchart showing the test result analysis process executed by the test result analysis system.

[0039] In step 101, the answer data collection unit 12 acquires answer data for each question written on the answer sheet in the actual test from multiple examinees. The answer data collection unit 12 displays an input screen of the self-grading site on a web page. When multiple examinees access the self-grading site and input the answers written on the answer sheet in the actual test, the answer data collection unit 12 records the input answers as answer data.

[0040] In step 102, the pass / fail estimation unit 13 predicts the pass / fail status of each of the examinees based on the collected answer data of the examinees. For example, the pass / fail estimation unit 13 compares correct answer data indicating the correct answers to each question given in advance with the answer data obtained from the examinee, determines which questions the examinee answered correctly and which questions he or she answered incorrectly, and calculates the number of questions answered correctly. The pass / fail estimation unit 13 then compares the number of questions answered correctly with a predetermined pass line, and estimates that the examinee has passed if the number of questions answered correctly is equal to or greater than the pass line, and estimates that the examinee has failed if the number of questions answered correctly is below the pass line.

[0041] In step 103, the question classification unit 14 classifies the questions included in the test. Here, the process of classifying the questions included in the test is referred to as a question classification process.

[0042] 4 is a flowchart showing the question classification process, which classifies test questions according to two criteria: question type and question level.

[0043] Figure 5 is a table showing examples of question types.

[0044] The question types are type (1) true / false questions and type (2) multiple-choice knowledge questions.

[0045] Type (3) is a multiple-choice degree case question, type (4) is a combination question, type (5) is a multiple-choice case law question, type (6) is a multiple-choice question, type (7) is a multiple-choice degree case question, type (8) is a calculation question, type (9) is a quantity question, and type (10) is a written question.

[0046] The test targeted in this embodiment includes, as typical answer formats, true / false questions, multiple choice questions, combination questions, multiple choice questions, calculation questions, quantity questions, and written questions. Furthermore, the multiple choice questions include legal precedent questions, case questions, and knowledge questions. Furthermore, the case questions in the multiple choice questions have different levels of difficulty depending on the number of subjects that appear in the options, so they are divided into low-level case questions and high-level case questions with different levels of difficulty.

[0047] The question type (answer format) is something that you need to be aware of when preparing to pass the exam, and at the same time, since the ease of arriving at the correct answer varies depending on the answer format, it can also be a factor in determining the level of the question.

[0048] The format of the question text is determined to some extent depending on the answer format of the question. Therefore, the question classification unit 14 can determine which answer format the target question has by analyzing the question text of the target question in the question data and determining whether it meets the predetermined conditions for each answer format.

[0049] Below are examples of questions with each answer format.

[0050] FIG. 6 is a diagram showing an example of a true / false question.

[0051] A true / false question is a multiple-choice question that asks whether a certain item is true or equivalent, or false or equivalent. Figure 6 shows an example of a true / false question that asks whether a given sentence is true or appropriate, or false or equivalent. True / false questions have the characteristic that the answer is multiple-choice and that the question contains the word "true" or a specified word equivalent thereto, and / or "false" or a specified word equivalent thereto. In the true / false question shown in Figure 6, "appropriate" is an example of a word equivalent to "true". Also, "inappropriate" is an example of a word equivalent to "false".

[0052] FIG. 7 is a diagram showing an example of a multiple-choice question.

[0053] A multiple-choice question has a predetermined number of sentences (options) as options, and asks the testee to select one correct answer (correct option) from among the options. The number of options (options) can be, for example, four, five, or six, depending on the test. FIG. 7 shows, as an example, a multiple-choice question that asks the testee to select which of five option sentences (options) is correct. The correct option becomes the correct option. Another example is a multiple-choice question that asks the testee to select one incorrect option from multiple options. In that case, the incorrect option becomes the correct option. A multiple-choice question is characterized in that the question contains multiple options, and the options themselves become answer options.

[0054] In addition, multiple-choice questions in law-related exams include knowledge questions, case questions, and legal precedent questions. Case questions are questions that ask about specific cases. Case questions are characterized by the fact that the question text contains words (case-subject terms) that indicate a specific subject that appears in the case question. Typical case-subject terms that appear in law exams include A, B, C, D, Company A, Company B, Company C, and Company D. Legal precedent questions are questions that ask about understanding of specific legal precedents. Legal precedent questions are characterized by the fact that the question text contains words or combinations of words that are specific to specific legal precedents. Knowledge questions are questions that ask about knowledge of specific matters. Questions that do not fall into either legal precedent or case questions can be presumed to be knowledge questions.

[0055] FIG. 8 is a diagram illustrating an example of a combination problem.

[0056] A combination question has multiple combinations of multiple sentences as options, and asks the participant to select from the options the combination of sentences that meets the conditions given in the question. The combinations mentioned here include cases where there is zero or one combination. Figure 8 shows, as an example, a combination question in which the participant is asked to select the incorrect combination from the options for three items A, B, and C, and four combinations 1, 2, 3, and 4 are given as options. A combination question is characterized by having multiple sentences in the question, and each option being a combination of zero or more sentences.

[0057] FIG. 9 is a diagram showing an example of a multiple-choice question.

[0058] A multiple-choice question is a question in which the question has multiple blanks, and multiple words that fit the blanks are given as options in addition to the question, and the questioner is asked to select a word that fits each blank from among the options. Figure 9 shows an example of a question in which the question has four blanks, A, B, C, and D, and the questioner is asked to select a word that fits the blanks from 20 options, 1 to 20. Multiple-choice questions are characterized by having a question containing multiple blanks and a section that lists candidate words that fit the blanks as options.

[0059] FIG. 10 is a diagram showing an example of a calculation problem.

[0060] A calculation problem is a problem in which the correct answer is derived by performing a numerical calculation according to the conditions of the problem statement. As an example, FIG. 10 shows a problem in which four amounts, 1 to 4, are given as options, and the amount requested in the problem statement is calculated and the option that matches the calculated amount is selected. A calculation problem is characterized in that each answer option contains a numerical value, and the problem statement also contains the numerical value used in the calculation.

[0061] FIG. 11 is a diagram showing an example of a quantity problem.

[0062] A counting problem is a problem in which a question has multiple sentences, the number of multiple sentences is given as a choice, and the number of sentences that meet the condition given in the question is selected from the choices. The number here includes 0. FIG. 11 shows an example of a counting problem in which a question has three sentences, A, B, and C, and four choices, 1 (one), 2 (two), 3 (three), and 4 (none), and the questioner is asked to select from the choices how many of the three sentences are incorrect. A counting problem is characterized in that each of the multiple choices has a number ranging from 0 to the number of sentences. The symbol here can be anything that identifies each sentence, such as numbers, alphabets, katakana, hiragana, kanji, and figures.

[0063] FIG. 12 is a diagram showing an example of a descriptive question.

[0064] A descriptive question is a question that requires a candidate to write about a given matter based on the question text. FIG. 12 shows, as an example, a descriptive question that requires a candidate to write about the matters required by the question text in a sentence of about a given number of characters. A descriptive question is characterized by the inclusion of a given word or phrase that requires a sentence to be written. "Please write" in the question text shown in FIG. 12 is an example of a given word or phrase that requires a sentence to be written. Other examples include "please write," "please discuss," and "please explain."

[0065] FIG. 13 is a table showing examples of problem levels. There are four problem levels, A, B, C, and D, which are classified in order of increasing degree of need to correctly answer the target problem in order to pass the test. Level D is a problem that is assumed to be passable even if not answered correctly. Level C is a problem that is assumed to have a probability of correct answering of approximately 1 / 2. Level B is a difficult question that is assumed to have to be answered correctly in order to pass. Level A is an easy question that is assumed to have to be answered correctly in order to pass. The problem classification unit 14 determines which level the target problem belongs to.

[0066] In this way, the approach required in studying to pass the exam differs depending on the four problem levels: problems that are expected to be passable even without answering correctly, problems where the probability of getting the correct answer is expected to be around 1 / 2, difficult questions that must be answered correctly in order to pass, and easy questions that must be answered correctly in order to pass.

[0067] In qualification exams and other such tests, questions that are said to be extremely difficult or unusual are sometimes asked, and almost all test-takers, including those who pass, are unable to solve them. Treating such questions as if they were other questions and studying them may not be effective in helping you pass the exam. Level D questions are low priority questions that you can pass even if you do not get the answer right. The other questions, levels A to C, are questions that require a high level of study and must be answered correctly in order to pass. When studying questions that require a high level of study, the three levels provide useful information for studying and exam guidance tailored to the test-takers' academic abilities.

[0068] The three-level division, excluding level D, which has a low study priority and allows you to pass even if you don't get the answers right, is an example of an effective number of levels for studying step by step. The three levels from A to C provide useful information for studying step by step.

[0069] Returning to FIG. 4, first, in step 201, the question classification unit 14 executes a prior knowledge discrimination process. The prior knowledge discrimination process is a process for discriminating the field of prior knowledge required to solve a target question. This process specifies which field the target question presupposes, that is, which field the question tests for knowledge in. If a test includes multiple subjects, it is also possible to discriminate the subject of the target question by discriminating the field of the target question. Details of the prior knowledge discrimination process will be described later with reference to FIG. 15.

[0070] Next, in step 202, the question classification unit 14 executes a question text length calculation process. The question text length calculation process is a process for determining the question text length of the target question. The question text length is the length of the question text. The unit for expressing the question text length is not particularly limited, and may be expressed, for example, as the number of characters, the number of words, the number of lines, etc. When expressed as the number of characters, the question classification unit 14 only needs to count the number of characters included in the question text.

[0071] 14 is a diagram showing an example of question information. In the question information 30, a relevant field 32, a question length 33, an answer format (question type) 34, and a question level 35 are registered in association with a question number 31 of the question. In this step 202, in the question information 30, a question length 33 is registered in association with the question number 31 of the target question.

[0072] Next, in step 203, the question classification unit 14 executes an answer format identification process. The answer format identification process is a process for identifying the answer format, i.e., the question type, of the target question. The details of the answer format identification process will be described later with reference to Fig. 17. This process identifies the answer format of the target question.

[0073] Next, in step 204, the question classification unit 14 executes a question level evaluation process. The question level evaluation process is a process for evaluating the question level of the target question. For example, the question level of each question may be determined. The expression of the question level is not particularly limited, and may be expressed, for example, by a numerical score, ranking, or the like. In this embodiment, the question level is expressed by a ranking of levels A to D. The difficulty level increases from level A to level D, but the impact on passing the exam decreases. The question level can be determined based on the field of prerequisite knowledge of the target question, the question length, and the answer format. Details of the question level evaluation process will be described later with reference to FIG. 18.

[0074] FIG. 15 is a flowchart showing an example of the premise knowledge determination process.

[0075] The question classification unit 14 extracts field-related terms contained in the target question by acquiring and analyzing the target question text from the question data.

[0076] The field-related terms are terms that appear in questions in each field. The question classification unit 14 manages field information that predetermines the field-related terms for each field. FIG. 16 is a diagram showing an example of field information. The field information 40 is a table in which the number of terms 42, the field-related terms 43, and the knowledge difficulty 44 are registered in association with a field 41. The number of terms 42 indicates the number of terms related to the corresponding field. The field-related terms 43 indicate terms related to the corresponding field. The knowledge difficulty 44 indicates the difficulty of acquiring knowledge in the corresponding field. Here, as an example, the knowledge difficulty is shown on a 10-point scale. For example, four field-related terms 43, namely, "abb", "acc", "add", and "aee", are set in the field "1". The knowledge difficulty of the field "1" is "4". Similarly, seven field-related terms 43, namely, "bcc", "bdd", "bee", "bff", "bgg", "bhh", and "bii", are set in the field "2". The knowledge difficulty of the field "2" is "7".

[0077] Specifically, the question classification unit 14 compares the words appearing in the target question with the field-related terms associated with each field in the field information 40, and extracts the field-related terms of each field appearing in the target question.

[0078] Next, the question classification unit 14 calculates a domain score as an index of the degree of possibility that the question belongs to each domain based on the comparison result in step 301 (step 302). As an example, the domain score for domain X is calculated by formula (1).

[0079] Field score S for field X X =(ST / N X )×N m (1)

[0080] Here, ST is a constant value that is common to all domains. Nx is the total number of domain-related terms set in domain X, i.e., the number of terms is 42. Nm is the number of domain-related terms in domain X included in the target problem.

[0081] In order to keep the total score constant between each field, the constant value ST is divided by the number of terms Nx in the field X and the value is distributed equally to each field-related term B33 as the score of each field-related term B33, and the scores of the field-related terms B33 included in the target question are added up to become the field score for the field X of the target question. For example, if the target question contains two field-related terms B33, "abb" and "acc", which are related to field A, the field score S1 for field "1" of the target question is S1 = (ST / 4) × 2 = ST / 2.

[0082] Next, the question classification unit 14 identifies the field (applicable field) to which the target question applies based on the field score of each field calculated in step 302 (step 303). For example, a field whose field score exceeds a predetermined threshold may be determined as the applicable field of the target question. The applicable field of the target question may be multiple.

[0083] Next, the problem classification unit 14 registers the relevant field of the target problem identified in step 303 in the problem information 30. In this step 303, the relevant field 32 is registered in association with the problem number 31 of the target problem in the problem information 30 illustrated in FIG.

[0084] FIG. 17 is a flowchart showing an example of the answer format identification process.

[0085] First, in step 401, the question classification unit 14 determines whether the target question is a true / false question. The true / false question illustrated in FIG. 6 has a feature that the answer is a multiple-choice question and the question text contains the word "correct" or a predetermined word equivalent thereto, and / or "incorrect" or a predetermined word equivalent thereto. The question classification unit 14 analyzes the question text of the target question and determines whether the question text of the target question contains the characteristics of a true / false question. If the question text contains the characteristics of a true / false question, the target question can be determined to be a true / false question. If the target question is a true / false question, the question classification unit 14 proceeds to step 409 and registers "true / false question" in the answer format 34 corresponding to the question number 31 of the target question in the question information 30.

[0086] If the target question is not a true / false question in step 401, then in step 402 the question classification unit 14 determines whether or not the target question is a multiple-choice question. The multiple-choice question shown in FIG. 7 has a feature in that the question text contains multiple options, and the options themselves are answer options. The question classification unit 14 determines whether or not the question text of the target question contains the characteristics of a multiple-choice question. If the question text contains the characteristics of a multiple-choice question, then the target question can be determined to be a multiple-choice question. If the target question is a multiple-choice question, then the question classification unit 14 proceeds to step 403.

[0087] In step 403, the problem classification unit 14 judges whether each option of the target problem, which is a multiple-choice question, is a case-based question option, a case-specific question option, or a knowledge-based question option. A judgment question option has a feature that a word or a combination of words specific to a specific case is included in the question text. A case-specific question has a feature that a case-based term is included in the question text. The problem classification unit 14 judges the target problem to be a case-based question if the question text contains the characteristics of a case-based question, judges the target problem to be a case-based question if the question text contains the characteristics of a case-based question, and judges the target problem to be a knowledge question if the question text does not contain the characteristics of a case-based question or the characteristics of a case-based question. Thereafter, the problem classification unit 14 proceeds to step 409 and registers "multiple-choice question" in the answer format 34 corresponding to the question number 31 of the target problem in the problem information 30. The problem classification unit 14 also records whether each option of the multiple-choice question is a case-based question option, a case-specific question option, or a knowledge-based question option.

[0088] If the target question is not a multiple-choice question in step 402, then in step 404 the problem classification unit 14 determines whether or not the target question is a combination question. The combination question illustrated in FIG. 8 has a feature that the question has multiple optional sentences, and each of the options is a combination of zero or more optional sentences. The problem classification unit 14 determines whether or not the question text of the target question contains the characteristics of a combination question. If the question text contains the characteristics of a combination question, then the target question can be determined to be a combination question. If the target question is a combination question, then the problem classification unit 14 proceeds to step 409, and registers "combination question" in the answer format 34 corresponding to the question number 31 of the target question in the problem information 30.

[0089] If the target question is not a combination question in step 404, then in step 405 the question classification unit 14 determines whether the target question is a multiple-choice question. The multiple-choice question illustrated in FIG. 9 has a feature that it has a question statement including multiple blanks and a section that lists candidate words that fit the blanks as options. The question classification unit 14 determines whether the question statement text of the target question contains the characteristics of a multiple-choice question. If the question statement text contains the characteristics of a multiple-choice question, then the target question can be determined to be a multiple-choice question. If the target question is a multiple-choice question, then the question classification unit 14 proceeds to step 409 and registers "multiple-choice question" in the answer format 34 corresponding to the question number 31 of the target question in the question information 30.

[0090] If the target problem is not a multiple choice problem in step 405, then in step 406 the problem classification unit 14 determines whether or not the target problem is a calculation problem. The calculation problem illustrated in FIG. 10 has a feature in that each answer option contains a numerical value, and the problem statement also contains a numerical value used in calculation. The problem classification unit 14 determines whether or not the problem statement text of the target problem contains the characteristics of a calculation problem. If the problem statement text contains the characteristics of a calculation problem, then the target problem can be determined to be a calculation problem. If the target problem is a calculation problem, then the problem classification unit 14 proceeds to step 409, and registers "calculation problem" in the answer format 34 corresponding to the problem number 31 of the target problem in the problem information 30.

[0091] If the target problem is not a calculation problem in step 406, then in step 407 the problem classification unit 14 determines whether or not the target problem is a quantity problem. The quantity problem illustrated in FIG. 11 has a feature that each of the multiple options has a number ranging from 0 to the number of options. The problem classification unit 14 determines whether or not the problem text of the target problem contains the characteristics of a quantity problem. If the problem text contains the characteristics of a quantity problem, then the target problem can be determined to be a quantity problem. If the target problem is a quantity problem, then the problem classification unit 14 proceeds to step 409, and registers "quantity problem" in the answer format 34 corresponding to the problem number 31 of the target problem in the problem information 30.

[0092] If the target question is not a quantity question in step 407, then in step 408 the problem classification unit 14 determines whether or not the target question is a written question. The written question illustrated in FIG. 12 is characterized by including a predetermined phrase that requires a written description. The problem classification unit 14 determines whether or not the problem text of the target question includes the characteristics of a written question. If the problem text includes the characteristics of a written question, then the target question can be determined to be a written question. If the target question is a written question, then the problem classification unit 14 proceeds to step 409, and registers "written question" in the answer format 34 corresponding to the question number 31 of the target question in the problem information 30.

[0093] If the target question is not a written question in step 408, the question classification unit 14 outputs an error message indicating that the answer format of the target question does not correspond to any of the above.

[0094] FIG. 18 is a flowchart showing an example of the problem level evaluation process.

[0095] First, in step 501, the question classification unit 14 evaluates the frequency with which the target question is asked. If the period during which a question has not been asked from the field corresponding to the prerequisite knowledge of the target question exceeds a predetermined period threshold, the target question is determined to be level D. Generally, many examinees tend to exclude fields that have not been asked for a long time from their study targets or lower their priority. Therefore, it is expected that many examinees will not be able to answer questions from fields that have not been asked for a long time because they do not know the prerequisite knowledge. It can be assumed that questions from such fields have little impact on pass / fail.

[0096] Next, in step 502, the problem classification unit 14 evaluates the knowledge difficulty of the field of the target problem. The field of the target problem can be known by referring to the problem information 30 illustrated in FIG. 14. The field of the target problem is not limited to one, and may be multiple. The knowledge difficulty of the field can be known by referring to the field information 40 illustrated in FIG. 16. As illustrated in FIG. 16, for each field 41, a knowledge difficulty 44 indicating the difficulty of acquiring knowledge in the field 41 is set in advance. As an example, the knowledge difficulty of each field may be set in advance by a lecturer or the like. Alternatively, the knowledge difficulty may be automatically set based on the amount of the field in the teaching material. For example, a field with a large number of pages of text may be set to have a high knowledge difficulty. Alternatively, a field with a long e-learning time may be set to have a high knowledge difficulty.

[0097] If the period during which the target question has not been asked from the relevant field is equal to or less than the period threshold, the question classification unit 14 sets the question level of the target question according to the knowledge difficulty of the relevant field. Since the difficulty of acquiring knowledge in the relevant field of the target question is reflected in the question level in this way, it is possible to provide information that reflects the knowledge difficulty that is useful for studying to acquire knowledge in the field to be acquired.

[0098] Specifically, the question classification unit 14 classifies the question level of the target question into one of levels A, B, and C by comparing the maximum value of the relevant field of the target question with a predetermined knowledge difficulty threshold. As described above, level C is a question that is assumed to have a probability of correct answering of about 1 / 2, level B is a difficult question that is assumed to have to be answered correctly to pass, and level A is an easy question that is assumed to have to be answered correctly to pass. The knowledge difficulty threshold is set in advance so that the target question is appropriately classified into those levels A, B, and C. Specifically, it is preferable to lower the question level if the knowledge difficulty is high. This is because it is assumed that the higher the knowledge difficulty, the higher the possibility that many examinees do not have knowledge in that field.

[0099] Note that the problem level set here is not final. As will be described later, it will be adjusted based on the problem length and answer format.

[0100] Next, in step 503, the problem classification unit 14 evaluates the problem text length of the target problem. Specifically, when the problem level set for the target problem is level A, if the problem text length of the target problem exceeds a predetermined problem text length threshold, the problem classification unit 14 adjusts the problem level of the target problem to level B. Even if a problem is set from a field with low knowledge difficulty, if the problem text is long, the number of places in the problem text to which the prerequisite knowledge is associated increases, making it difficult to grasp the intention of the question setter. On the other hand, problems in fields with high knowledge difficulty of prerequisite knowledge are different in nature, such as a large amount of knowledge to be known as a prerequisite and complex mutual relationships. For problems in fields with high knowledge difficulty of prerequisite knowledge, if a problem level that matches the knowledge difficulty is set in advance, the complexity of the relationship between the knowledge and the problem has already been taken into consideration, so even if the problem text is longer, the difficulty does not necessarily increase sharply and the problem level increases. In light of such nature, in this embodiment, only for target problems of level A, if the problem text length exceeds a predetermined problem text length threshold, the problem level is changed to level B to improve the appropriateness of the problem level.

[0101] Next, in step 504, the problem classification unit 14 executes an evaluation process by answer format. The evaluation process by answer format is a process for evaluating the problem level of a target problem according to the answer format of the target problem. There are various answer formats for problems. The difficulty of deriving the correct answer varies depending on the answer format. Depending on the answer format, there are problems for which it is difficult to derive the correct answer unless all related knowledge is accurately understood and memorized. On the other hand, there are also problems for which the correct answer can be derived by a process of elimination or the like if some related knowledge is understood and memorized. Such differences in answer formats can be reflected in the problem level.

[0102] FIG. 19 is a flowchart of the evaluation process by answer format.

[0103] In step 601, the question classification unit 14 determines whether or not the target question is a multiple-choice question. Whether or not the target question is a multiple-choice question can be determined from the processing result of the answer format identification processing.

[0104] If the target question is a multiple-choice question, the question classification unit 14 evaluates each option included in the multiple-choice question in step 602. At that time, the question classification unit 14 determines the question level of each option based on the option type of the option. The option type of each option can be known from the processing result of the answer format processing. The question classification unit 14 refers to preset question level determination information to identify the question level corresponding to the option type of each option.

[0105] 20 is a diagram showing an example of question level determination information. The question classification unit 14 is a table in which question levels 52 are registered in association with the answer format of the target question or the option type 51 of the option. If the target question is a multiple-choice question, a question level is set for each option type, and if it is not a multiple-choice question, a question level is set for each answer format.

[0106] In the example of FIG. 20, the problem level of the knowledge question options in the multiple-choice questions is set to A or above. The problem level of the low-level case question options in the multiple-choice questions is set to B or above. The problem level of the case law question options in the multiple-choice questions is set to B or above. The problem level of the advanced case question options in the multiple-choice questions is set to C or above.

[0107] If the question level set for the option is lower than the question level "o" set as "o or higher" in the question level determination information 50, the question level for the option is changed to "o." For example, if the option is a low-level example question option for a multiple-choice question and its question level is set to A, the question level is changed to B.

[0108] Unlike knowledge question options that test whether or not you know the prerequisite knowledge, case question options require you to fit the prerequisite knowledge into the question, so they are not easy questions that can be solved just by knowing the prerequisite knowledge. Therefore, case question options are set at a lower problem level than knowledge question options to ensure their validity. Also, among the case question options, high-level case question options, which have many subjects appearing, have more places to fit in than low-level case question options, which have few subjects appearing, so the problem level is lowered to ensure validity. In light of these characteristics, in this embodiment, low-level case question options are set at level B and high-level case question options are set at level C to ensure the validity of the problem level.

[0109] Next, the question classification unit 14 executes a multiple-choice question comprehensive evaluation process in step 603. The multiple-choice question comprehensive evaluation process is a process for setting the problem level of a target problem that is a multiple-choice question based on the problem levels of each option included in the target problem. The multiple-choice question comprehensive evaluation process will be described in detail later with reference to Figs. 21 and 22.

[0110] If the target question is not a multiple-choice question in step 601, the question classification unit 14 evaluates the question level of the target question in step 604. At that time, the question classification unit 14 refers to the preset question level determination information 50 to identify the question level corresponding to the answer format of the target question.

[0111] In the example of FIG. 20, the problem level of true / false questions is set to A or above. The problem level of matching questions is set to B or above. The problem level of multiple choice questions is set to B or above. The problem level of calculation questions is set to C or above. The problem level of quantity questions is set to C or above. The problem level of written questions is set to C or above.

[0112] If the problem level set for the target problem is lower than the problem level "O" set as "O or higher" in the problem level determination information 50, the problem level of the target problem is changed to "O." For example, if the target problem is a combination problem and its problem level is set to A, the problem level is changed to B.

[0113] That is, if the problem level set by the prerequisite knowledge and question length of the target problem is level A and the target problem is a combination problem or multiple choice problem, problem classification unit 14 changes the target problem to level B. If the problem level set by the prerequisite knowledge and question length of the target problem is level A or level B and the target problem is a calculation problem, a quantity problem, or a written question, problem classification unit 14 changes the target problem to level C. It is possible to provide an appropriate problem level adjusted according to the answer format.

[0114] Fig. 21 is a diagram showing an example of judgment logic information for the multiple-choice question comprehensive evaluation process Fig. 22 is a flowchart showing an example of the multiple-choice question comprehensive evaluation process.

[0115] The multiple-choice question comprehensive evaluation is a process for setting the problem level of a multiple-choice question by a comprehensive evaluation based on the problem level of each option. The comprehensive evaluation is performed using the decision logic information shown in FIG.

[0116] In a multiple-choice question, if it is not possible to make a correct decision for all options with certainty, the correct answer is arrived at by narrowing down the options using a process of elimination. Therefore, the ease of arriving at the correct answer for a multiple-choice question depends on how easily the options can be narrowed down using the process of elimination. In this embodiment, the degree to which the options can be narrowed down corresponds to the overall evaluation.

[0117] 21, in the judgment logic information 60, a condition (condition content 62) for judging whether or not the option for a target question is applicable is associated with the problem level (overall evaluation B 63) of the target question that satisfies the condition, and a condition number 61 is assigned to the condition. In this example, the judgment is made in ascending order of the condition number 61.

[0118] In the case of condition number = 1, the problem level of the target problem as a comprehensive evaluation is A for the condition content "includes an option that is level A and a correct answer." In the case of condition number = 2, the problem level of the target problem as a comprehensive evaluation is A for the condition content "includes an option that is level A and has one less incorrect answer than the total number of options." In the case of condition number = 3, the problem level of the target problem as a comprehensive evaluation is B for the condition content "does not meet narrowing condition 2, includes an option that is level B and a correct answer." In the case of condition number = 4, the problem level of the target problem as a comprehensive evaluation is B for the condition content "does not meet narrowing conditions 1 and 3, includes an option that is level A or B and has one less incorrect answer than the total number of options." In the case of condition number = 5, the problem level of the target problem as a comprehensive evaluation is C for the condition content "does not meet narrowing conditions 2 and 4, includes an option that is level C and a correct answer." In the case of condition number 6, the problem level as the overall evaluation of the target problem that corresponds to the condition "does not meet any of the filtering conditions 1 to 5, and contains level A or B options that are incorrect, and the number of options is two less than the total number of options," is C. In the case of condition number 7, the problem level as the overall evaluation of the target problem that corresponds to the condition "does not meet any of the filtering conditions 1 to 6," is D.

[0119] The multiple-choice question comprehensive evaluation process in FIG. 22 is realized by a series of flowcharts.

[0120] First, in step 701, the problem classification unit 14 determines whether or not the target problem satisfies narrowing-down condition 1. If the target problem satisfies narrowing-down condition 1, the problem classification unit 14 sets the problem level of the target problem to A in step 703.

[0121] If the target problem does not meet narrowing condition 1, then in step 702, the problem classification unit 14 determines whether the target problem meets narrowing condition 2. If the target problem meets narrowing condition 2, in step 703, the problem classification unit 14 sets the problem level of the target problem to A.

[0122] If the target problem does not meet narrowing condition 2, then in step 704, the problem classification unit 14 determines whether the target problem meets narrowing condition 3. If the target problem meets narrowing condition 3, in step 706, the problem classification unit 14 sets the problem level of the target problem to B.

[0123] If the target problem does not meet narrowing condition 3, then in step 705, the problem classification unit 14 determines whether the target problem meets narrowing condition 4. If the target problem meets narrowing condition 4, in step 706, the problem classification unit 14 sets the problem level of the target problem to B.

[0124] If the target problem does not meet narrowing condition 4, then in step 707, the problem classification unit 14 determines whether the target problem meets narrowing condition 5. If the target problem meets narrowing condition 5, in step 709, the problem classification unit 14 sets the problem level of the target problem to C.

[0125] If the target problem does not meet narrowing condition 5, then in step 708, the problem classification unit 14 determines whether the target problem meets narrowing condition 6. If the target problem meets narrowing condition 6, in step 709, the problem classification unit 14 sets the problem level of the target problem to C.

[0126] If the target question does not satisfy narrowing condition 6, the question classification unit 14 sets the question level of the target question to D in step 710.

[0127] The process of FIG. 22 can also be organized as follows.

[0128] The problem classification unit 14 sets the target problem to level A if the target problem satisfies narrowing condition 1 that the multiple-choice question contains options that are level A and correct answers. The problem classification unit 14 sets the target problem to level A if the target problem satisfies narrowing condition 2 that the multiple-choice question contains options that are level A and incorrect answers one less than the number of options. The problem classification unit 14 sets the target problem to level B if the target problem does not satisfy narrowing condition 2 and satisfies narrowing condition 3 that the multiple-choice question contains options that are level B and correct answers. The problem classification unit 14 sets the target problem to level B if the target problem does not satisfy narrowing condition 1 or narrowing condition 3 and satisfies narrowing condition 4 that the multiple-choice question contains options that are level A and incorrect answers or level B and incorrect answers one less than the total number of options. If the target question does not fall under either of the narrowing conditions 2 or 4, and falls under narrowing condition 5 that the multiple-choice question contains an option that is level C and a correct answer, the question classification unit 14 sets the target question to level C. If the target question does not fall under any of the narrowing conditions 1 to 5, and falls under narrowing condition 6 that the multiple-choice question contains options that are level A and an incorrect answer or level B and an incorrect answer, the number of which is two less than the total number of options, the question classification unit 14 sets the target question to level C. If the target question does not fall under any of the narrowing conditions 1 to 6, the question classification unit 14 sets the target question to level D. For a multiple-choice question, the problem level of the entire multiple-choice question and each option can be provided as information useful for studying for passing the exam.

[0129] Returning to FIG. 3, in step 104, the answer analysis unit 15 executes an answer analysis process.

[0130] FIG. 23 is a flowchart showing an example of the answer analysis process.

[0131] First, in step 801, the answer analysis unit 15 performs an analysis of the test results by subject.

[0132] The answer analysis unit 15 calculates the standard deviation of the presumed failing students for each subject included in the test based on the answer data of the presumed passing students. The average score of the presumed passing students is the standard deviation of 50. The answer analysis unit 15 then judges the performance of each subject based on the standard deviation of each subject by referring to predetermined judgment criteria information.

[0133] FIG. 24 is a diagram illustrating an example of the criteria information.

[0134] In the judgment criteria information 70, judgments 72 and measures 73 are set in correspondence with deviation value ranges 71. Judgments 72 are information expressing the performance of the test in the subject in question divided into multiple stages. There are grades from A to D, with A being the best performance and D being the worst performance. Measures 73 are information indicating measures to be taken in future studies for each grade. The judgments and measures for each subject are provided to the examinee (presumed to fail).

[0135] If the deviation value range 71 is 55 or above, the judgment 72 is A, and the measure 73 indicates that the examinee does not need to take any special measures for the subject and can simply maintain the current situation. If the deviation value range 71 is 50 or more and less than 55, the judgment 72 is B, and the measure 73 indicates that the examinee needs to take measures for the subject and should make an effort to make it their strong subject. If the deviation value range 71 is 45 or more and less than 50, the judgment 72 is C, and the measure 73 indicates that the examinee needs to take measures for the subject and is required to make an effort to aim for an average score or above. If the deviation value range 71 is less than 45, the judgment 72 is D, and the measure 73 indicates that the examinee needs to take considerable measures for the subject.

[0136] The examinee can know which of the multiple subjects included in the test he or she did well and which subjects he or she did poorly in. Generally, data such as the score distribution of presumed successful candidates in a test is not made public, but according to this embodiment, presumed unsuccessful candidates can obtain objective information that compares the score distribution of presumed successful candidates with their own score distribution.

[0137] Fig. 25 is a diagram showing an example of the subject-specific assessment results. Fig. 25 shows the subject-specific assessment results of Mr. A in the real estate agent examination. The subject-specific assessment results 80 show, for each subject 81, the score allocation 82, the average score of the estimated successful candidates 83, Mr. A's score 84, and the subject-specific assessment 85.

[0138] In the real estate license examination, for subject 81, Real Estate License Law, the score is 20 points, the average score 83 of presumptive passers is 18.5 points, Mr. A's score is 20 points, and the subject-specific assessment 85 is A. For subject 81, Legal Restrictions, the score is 8 points, the average score 83 of presumptive passers is 5.5 points, Mr. A's score is 2 points, and the subject-specific assessment 85 is D. For subject 81, Other Laws and Regulations, the score is 8 points, the average score 83 of presumptive passers is 5.5 points, Mr. A's score is 3 points, and the subject-specific assessment 85 is C. For subject 81, Rights and Relations, the score is 14 points, the average score 83 of presumptive passers is 9.5 points, Mr. A's score is 10 points, and the subject-specific assessment 85 is B.

[0139] In qualification examinations, etc., when the examinee's performance is judged immediately after the examination, it is usually judged subjectively by the examinee. In contrast, according to this embodiment, the examinee can objectively judge the performance of the examination. In addition, in this embodiment, the standard score of the presumed failing students is calculated based on the average score of the presumed passing students for each subject, and the judgment is made, so that the performance of the presumed failing students in each subject can be known by comparing with the presumed passing students.

[0140] Returning to FIG. 23, in step 802, the answer analysis unit 15 performs an analysis of the test results by question.

[0141] The answer analysis unit 15 calculates the rate of correct answers of the presumed successful candidates for each question in the answer data of the presumed successful candidates, analyzes the answers of the presumed unsuccessful candidates to that question based on the rate of correct answers of the presumed successful candidates for each question, processes the analysis results, and outputs them in table format.

[0142] FIG. 26 is a table showing the analysis results of the problem-specific analysis.

[0143] 26 shows the results of analyzing A's answers to each question. In table 90 shown in FIG. 26, for each of the 50 questions included in the test, subject 92, field 93, correct / incorrect answers 94, estimated passer's correct answer rate 95, question level 96, and question type 97 are recorded in association with question number 91.

[0144] The answer analysis unit 15 records whether the answer determined by the pass / fail estimation unit 13 is correct or incorrect in the correct / incorrect field 94 of the table 90. In FIG. 26, correct answers are indicated by circles (◯) and incorrect answers are indicated by crosses (×). The answer analysis unit 15 records the subject, field, question level, and question type of the question identified by the question classification unit 14 in the subject 92, field 93, question level 96, and question type 97 of the table 90, respectively. Furthermore, the answer analysis unit 15 records the calculated estimated passer's correct answer rate in the estimated passer's correct answer rate 95 of the table 90. For example, for question number 1, the subject is rights-related, the field is expression of intent, and Mr. A's correct answer is correct.

[0145] Fig. 27 is another table showing the analysis results by question. In table A0 shown in Fig. 27, the number of questions A2, the number of questions answered correctly by Mr. A A3, and Mr. A's correct answer rate A4 are recorded in association with the range A1 of the estimated passer's correct answer rate.

[0146] The range A1 of estimated successful candidates' correct answer rates is set as follows: 0% or more and less than 10%, 10% or more and less than 20%, 20% or more and less than 30%, 30% or more and less than 40%, 40% or more and less than 50%, 50% or more and less than 60%, 60% or more and less than 70%, 70% or more and less than 80%, 80% or more and less than 90%, and 90% or more and less than 100%.

[0147] The answer analysis unit 15 identifies, among the questions in the test, those questions for which the estimated pass rate falls within each of the ranges mentioned above, calculates the number of those questions (say N1), the number of questions that Mr. A answered correctly (say N2), and N1 / N2, and records these in the number of questions A2, the number of questions that Mr. A answered correctly A3, and Mr. A's correct answer rate A4, respectively.

[0148] In order to pass the exam, it is important to ensure that candidates correctly answer the questions that should be answered correctly. Looking at table A0 in Figure 27, the percentage of correct answers by predicted successful candidates is higher in the upper range. Therefore, in order to pass, it is necessary to answer questions included in the upper range with a high percentage of correct answers, but in the example in Figure 27, we can see that Mr. A is not able to do this.

[0149] In addition, the answer analysis unit 15 calculates the percentage of correct answers of the presumed successful candidates for each question in the answer data of the presumed successful candidates, analyzes the answers of the presumed unsuccessful candidates to the questions based on the percentage of correct answers of the presumed successful candidates for each question and the question level, processes the analysis results, and outputs them in table format.

[0150] Fig. 28 is yet another table showing the analysis results of the question-by-question analysis. In table B0 shown in Fig. 28, the number of questions B2, the number of questions answered correctly by Mr. A B3, the correct answer rate B4 of Mr. A, and the correct answer rate B5 of the presumed successful candidates are recorded in association with the question level B1.

[0151] The answer analysis unit 15 calculates the number of questions included in the test that correspond to each level (N3), the number of questions that Mr. A answered correctly (N4), N4 / N3, and the correct answer rate of the presumed successful candidate (P1), and records these in the number of questions B2, the number of questions that Mr. A answered correctly B3, Mr. A's correct answer rate B4, and the correct answer rate of the presumed successful candidate B5, respectively. P1 is calculated using the following formula (1).

[0152] P1 = (the cumulative number of people who answered correctly to each question at a given level) / (N3 × (the number of people who answered correctly)) (1)

[0153] Looking at table B0 in Figure 28, A's correct answer rate B4 is lower than the correct answer rate B5 of the presumed successful candidates at levels A, B, and C. On the other hand, A's correct answer rate B4 is higher than the correct answer rate B5 of the presumed successful candidates at level D. Level D questions are questions that are assumed to be passable even without correct answers, and can be said to be questions that are difficult to answer correctly based on ability. The reason why 9A's correct answer rate B4 at level D is higher than the correct answer rate B5 of the presumed successful candidates is likely due to luck. A is required to be able to obtain a correct answer rate B4 higher than the correct answer rate B5 of the presumed successful candidates at levels A, B, and C.

[0154] The answer analysis unit 15 calculates the percentage of correct answers of presumed successful candidates for each question in the answer data of the presumed successful candidates, analyzes the answers of the presumed unsuccessful candidates to that question based on the percentage of correct answers of the presumed successful candidates for each question and the question type, processes the analysis results, and outputs them in table format.

[0155] Fig. 29 is yet another table showing the analysis results by question. In table C0 in Fig. 29, the number of questions C2, the number of questions that Mr. A answered correctly C3, Mr. A's correct answer rate C4, and the correct answer rate C5 of the presumed successful candidates are recorded in association with the question type C1.

[0156] The answer analysis unit 15 calculates the number of questions that fall into each question type among the questions included in the test (N5), the number of questions that A answered correctly (N6), N6 / N5, and the correct answer rate of the presumed successful candidate (P2), and records these in the number of questions C2, the number of questions that A answered correctly C3, the correct answer rate of A C4, and the correct answer rate of the presumed successful candidate C5, respectively. P2 is calculated using the following formula (2).

[0157] P2 = (the cumulative number of people who answered correctly to each question of a given type) / (N5 × (the number of people who answered correctly)) (2)

[0158] In order to pass the exam, it is important to avoid any problem type that is difficult for the student, and it is required to obtain the same correct answer rate for all problem types as those who passed. Looking at table C0 in Figure 29, Mr. A's correct answer rate C4 for multiple-choice case study questions and number of items questions is lower than the correct answer rate C5 of the estimated successful candidates. Mr. A should make an effort to increase his correct answer rate for multiple-choice case study questions and number of items questions, with the goal of obtaining a correct answer rate higher than the correct answer rate C5 of the estimated successful candidates for all problem types.

[0159] The present embodiment described above includes the following items. However, the items included in the present embodiment are not limited to the following items.

[0160] The test result analysis system includes an answer data collection unit 12 that collects answer data indicating answers to each question submitted by multiple examinees who took a test including multiple questions, a pass / fail estimation unit 13 that estimates the pass / fail status of each of the multiple examinees based on the answer data of the multiple examinees, and an answer analysis unit 15 that analyzes the answers to the questions of presumed failers who are presumed to have failed based on the answer data of the presumed passers who are presumed to have passed. According to this embodiment, the multiple examinees are divided into presumed passers and presumed failers, and the answers to each question of the presumed failers are analyzed based on the answer data of the presumed passers, so that the test results can be analyzed before the pass / fail results are announced.

[0161] In addition, the answer analysis unit 15 calculates the presumptive successful candidate correct answer rate, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates, and analyzes the answers of the presumptive unsuccessful candidates to the question based on the presumptive successful candidate correct answer rate for that question. According to this aspect, the answers of the presumptive unsuccessful candidates to each question are analyzed based on the presumptive successful candidates' correct answer rate, so that an analysis can be performed that takes into account the percentage of presumptive successful candidates who will answer correctly before the results are announced.

[0162] In addition, the answer analysis unit 15 divides the correct answer rate of presumed passers into a plurality of ranges, and for questions belonging to each range, calculates the number of questions that the presumed failer to be analyzed answered correctly and / or the ratio of that number to the number of questions belonging to that range. According to this embodiment, it is possible to analyze the test results by comparing the correct answer rate of presumed passers with the number of correct answers and / or the correct answer rate of the presumed failer to be analyzed.

[0163] The test result analysis system further includes a problem classification unit 14 that classifies a plurality of questions into a plurality of problem levels that indicate the degree to which the question needs to be answered correctly in order to pass the test, and an answer analysis unit 15 analyzes the answer of the presumed failing person to the question based on the problem level of the question and the correct answer rate of the presumed passer. According to this embodiment, the answer of the presumed failing person to each question is analyzed based on the degree to which the question needs to be answered correctly in order to pass the test and the correct answer rate of the presumed passer for the question, so that an effective analysis can be performed before the pass / fail result is announced, taking into consideration the degree to which the question needs to be answered correctly in order to pass the test and the percentage of the presumed passers who answered correctly.

[0164] In addition, for each of a plurality of question levels, the answer analysis unit 15 calculates the correct answer rate of the presumed failing person by dividing the number of questions corresponding to that question level that the presumed failing person answered correctly by the number of questions corresponding to that question level, and calculates the correct answer rate of the presumed passing person by dividing the cumulative value of the number of presumed passing people who answered each question corresponding to that question level correctly by the value obtained by multiplying the number of questions corresponding to that question level and the number of presumed passing people, thereby making it possible to compare the correct answer rates of the presumed failing people and the presumed passing people.

[0165] The test result analysis system further includes a question classification unit 14 that classifies the multiple questions into multiple question types according to the format of the answers to the test, and an answer analysis unit 15 analyzes the answers to the questions of the presumed failing persons based on the question type and the correct answer rate of the presumed passing persons. According to this embodiment, the answers to each question of the presumed failing persons are analyzed based on the format of the answer to the question and the correct answer rate of the presumed passing persons for that question, so that an effective analysis can be performed before the pass / fail result is announced, taking into consideration the format of the answer to the question and the percentage of the presumed passing persons who answered the question correctly.

[0166] In addition, the answer analysis unit 15 calculates the correct answer rate of the presumed failing person by dividing the number of questions that correspond to that question type and that the presumed failing person answered correctly by the number of questions that correspond to that question level, for each of the multiple question types, and calculates the correct answer rate of the presumed passing person by dividing the cumulative value of the number of presumed passing people who answered each question that corresponds to that question type correctly by a value obtained by multiplying the number of questions that correspond to that question type and the number of the presumed passing people, thereby making it possible to compare the correct answer rates of the presumed failing people and the presumed passing people.

[0167] The test result analysis system further includes a problem classification unit 14 that classifies a plurality of questions into a plurality of problem levels that indicate the degree to which the question needs to be answered correctly in order to pass the test, and a plurality of problem types that correspond to the answer format for the test, and the answer analysis unit 15 analyzes the answers of the presumed failing persons to the questions based on the problem level, problem type, and the correct answer rate of the presumed passing persons. According to this embodiment, before the result of passing or failing is announced, an effective analysis can be performed that takes into consideration the degree to which the question needs to be answered correctly in order to pass the test, the format of the answer to the question, and the percentage of the presumed passing persons who answered the question correctly.

[0168] In addition, the question classification unit 14 determines the prerequisite knowledge required to solve the question, calculates the question length, which is the length of the question sentence, identifies the answer format for the question, determines the question level based on the prerequisite knowledge, the question length, and the answer format, determines the question type based on the answer format, and uses the question level and question type to analyze the answers of the predicted failing candidates.

[0169] Moreover, the test includes questions on multiple subjects, and the answer analysis unit 15 calculates the deviation value of the presumed failing student for each subject based on the answer data of the presumed passing student. According to this embodiment, in a test that includes multiple subjects and the pass / fail is determined by the total score of the subjects, the deviation value of the presumed failing student for each subject is calculated based on the answer data of the presumed passing student, so that the performance of the presumed failing student for each subject can be objectively analyzed.

[0170] The above-described embodiments of the present invention are merely illustrative examples of the present invention, and are not intended to limit the scope of the present invention to these embodiments. Those skilled in the art can implement the present invention in various other forms without departing from the scope of the present invention. [Explanation of symbols]

[0171] 10... test result analysis system, 11... pass grade acquisition section, 12... answer data collection section, 13... pass / fail estimation section, 14... question classification section, 15... answer analysis section, 20... computer system, 21... processor, 22... main memory, 23... storage device, 24... communication device, 25... input device, 26... display device

Claims

1. A test result analysis system having a memory and a processor, The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, using a specific presumed failing individual as an analysis subject, identifies questions for which the presumed passing individual's correct answer rate is equal to or greater than a predetermined threshold and which the analysis subject has not answered correctly. Test result analysis system.

2. A test result analysis system having a memory and a processor, The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, with a specific estimated failing person as an analysis subject, presents whether or not the analysis subject answered each question correctly together with the estimated passing person's correct answer rate. Test result analysis system.

3. A test result analysis system having a memory and a processor, The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, Classifying questions included in the test into a plurality of attributes; Using specific estimated failing candidates as analysis subjects, for each attribute, calculate the estimated passer's correct answer rate for questions classified into the attribute, and the number of questions correctly answered by the analysis subject relative to the number of questions classified into the attribute and / or the ratio of said number; Test result analysis system.

4. A test result analysis system having a memory and a processor, The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, Identifying questions for which the rate of correct answers of the estimated successful candidates is below a predetermined threshold; Test result analysis system.

5. A test result analysis system having a memory and a processor, The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor calculates the deviation scores of the presumed failing candidates based on the answer data of the presumed successful candidates so that the average score of the presumed successful candidates is a deviation score of 50. Test result analysis system.

6. A method for analyzing test results executed by a computer having a memory and a processor, comprising: The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, using a specific presumed failing individual as an analysis subject, identifies questions for which the presumed passing individual's correct answer rate is equal to or greater than a predetermined threshold and which the analysis subject has not answered correctly. How to analyze test results.

7. A method for analyzing test results executed by a computer having a memory and a processor, comprising: The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, with a specific estimated failing person as an analysis subject, presents whether or not the analysis subject answered each question correctly together with the estimated passing person's correct answer rate. How to analyze test results.

8. A method for analyzing test results executed by a computer having a memory and a processor, comprising: The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, Classifying questions included in the test into a plurality of attributes; Using specific estimated failing candidates as analysis subjects, for each attribute, calculate the estimated passer's correct answer rate for questions classified into the attribute, and the number of questions correctly answered by the analysis subject relative to the number of questions classified into the attribute and / or the ratio of said number; How to analyze test results.

9. A method for analyzing test results executed by a computer having a memory and a processor, comprising: The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor, Identifying questions for which the rate of correct answers of the estimated successful candidates is below a predetermined threshold; How to analyze test results.

10. A method for analyzing test results executed by a computer having a memory and a processor, comprising: The memory stores answer data indicating answers to each question submitted by a plurality of examinees who have taken an exam including a plurality of questions, The processor, Predicting the pass / fail status of each of the plurality of examinees based on the answer data of the plurality of examinees; Calculate the rate of correct answers of presumptive successful candidates, which is the ratio of presumptive successful candidates who answered the question correctly to the total number of presumptive successful candidates, for each question in the answer data of the presumptive successful candidates who are presumed to have passed; Analyzing the answers to the questions of the presumed failed candidates who are presumed to have failed based on the correct answer rate of the presumed successful candidates for the questions; The processor calculates the deviation scores of the presumed failing candidates based on the answer data of the presumed successful candidates so that the average score of the presumed successful candidates is a deviation score of 50. How to analyze test results.

Citation Information

Patent Citations

  • Method and system for performing simulated examination while utilizing communication network

    JP2002072857A

  • Quickly obtaining system of examinee information and test result information

    JP2002123167A

  • Education support system

    JP2005164810A

  • Learning information processor, learning information processing method, and computer program

    JP2007248773A

  • Information processing apparatus

    JP2019117552A