Apparatus, program and method for evaluating respondents to multiple choice questions with no correct answer

The evaluation device scores respondent answers based on reference person distributions to assess consistency, addressing the challenge of evaluating multiple-choice questions without correct answers and ensuring minority opinions are included, enhancing the reliability of evaluations.

JP7590779B2Active Publication Date: 2024-11-27YAMAGUCHI UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022172848
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-10-28
Publication Date
2024-11-27
Estimated Expiration
2042-10-28

AI Technical Summary

Technical Problem

Evaluating respondents to multiple-choice questions without correct answers is challenging due to subjectivity and variability in interpretations, making it difficult to assess consistency and accuracy of answers, and existing methods like majority voting overlook minority opinions.

Method used

An evaluation device and method that scores respondent answers by comparing them to the distribution of answers from multiple reference persons, using a respondent evaluation value calculated based on the total number of selections and the maximum total number of selections for each option, reflecting the consistency of respondent answers with reference person distributions.

Benefits of technology

Enables effective evaluation of respondents by quantifying the consistency of their answers with reference person distributions, ensuring that minority opinions are considered, and excluding questions with divided reference person opinions, thereby improving the reliability of assessments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590779000001
    Figure 0007590779000001
  • Figure 0007590779000002
    Figure 0007590779000002
  • Figure 0007590779000003
    Figure 0007590779000003
Patent Text Reader

Abstract

To provide an evaluation device, evaluation program, and evaluation method that evaluate an answerer on the basis of an answer of the answerer to a question with multiple choices that has no correct answer.SOLUTION: An evaluation device (1) according to the present invention evaluates an answerer by comparing an answer of the answerer to a question with multiple choices that has no correct answer with an answer of each of multiple reference persons to the question with multiple choices and grading the answer of the answerer. The evaluation device includes an answerer evaluation value calculation part (8) which calculates an answerer evaluation value being an index for the evaluation of the answerer on the basis of the answerer choice selected as the answer of the answerer in the multiple choices included in the question with multiple choices and the selection total number being the total number of reference persons selected as a reference person choice for each of the multiple choices. The answerer evaluation value calculation part calculates the answerer evaluation value on the basis of the selection total number corresponding to the answerer choice and the maximum selection total number in the selection total numbers for each of the multiple choices.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an evaluation device, an evaluation program, and an evaluation method for evaluating respondents to multiple choice questions that have no correct answer. [Background technology]

[0002] A multiple choice question is a question in which the respondent is asked to select one or more answers from multiple options included in each question, and the answers are scored (given a score) to evaluate the respondent. In other words, the respondent of the multiple choice question is the person being evaluated. Multiple choice questions are broadly divided into questions with a correct answer, such as exam questions, and questions without a correct answer, such as surveys and medical interviews.

[0003] In the case of multiple choice exam questions, the multiple options include one or more options given as correct answers. That is, respondents to exam questions with correct answers are scored and evaluated based on whether the answer option they selected is the correct answer or not. In the field of statistical inference, this is classified as a problem with an external criterion, or in the field of machine learning, as a supervised problem.

[0004] On the other hand, multiple-choice questions in the case of questionnaires and medical interviews do not have a correct answer. In other words, the multiple options that make up each question in a multiple-choice question do not include the option given as the correct answer. Therefore, it is more difficult for respondents to evaluate a questionnaire or medical interview than it is to evaluate the aforementioned test questions. In the field of statistical inference, this is classified as a problem without an external standard, or in the field of machine learning, as an unsupervised problem.

[0005] Here, for example, consider a case where a medical student studying Chinese medicine is learning how to diagnose a patient's tongue (tongue diagnosis). The medical student interprets an image of the tongue (tongue image) of a patient who is a case that is used as a teaching material, and diagnoses whether the patient has a disease or not based on the size and color of the tongue. The patient who is a case that is used as a teaching material is a patient who has been confirmed to have a disease or not. For example, a multiple-choice question on tongue diagnosis may have eight questions about the size and color of the tongue for a given tongue image, and each question may be composed of multiple options. However, the interpretation of the size and color of the tongue is easily influenced by the reader's subjectivity, based on the reader's knowledge and experience. Therefore, for example, when multiple Chinese medicine specialists read the same tongue image, the interpretation results of the tongue size and color of each specialist may not necessarily be the same, that is, the options selected by each specialist may not necessarily be the same. In other words, the interpretation results of the specialists may differ, in other words, even specialists may have different opinions. This is not limited to tongue diagnosis, but the same is true for the interpretation of pathological images, and is unavoidable in diagnoses that involve the doctor's subjectivity. When the results of the specialist's interpretation are different, it is difficult to evaluate the results of the medical student's interpretation of the same tongue image as that interpreted by the specialist by a simple method of whether or not it matches the results of the specialist's interpretation. In this way, it is more difficult for a respondent to evaluate a multiple-choice question that does not have a correct answer than to evaluate a multiple-choice question that has a correct answer.

[0006] As a method of evaluating respondents for multiple choice questions that have no correct answer, the so-called majority vote is considered. The majority vote is a method of evaluating respondents by setting up a provisional correct answer and replacing a question that has no correct answer with a question that does have a correct answer. However, evaluation by majority vote has a problem with the legitimacy of the evaluation results.

[0007] Furthermore, evaluation by majority vote has the problem that only the majority opinion is reflected in the evaluation, and not the minority opinion, that is, the minority opinion is ignored in the evaluation.

[0008] Furthermore, the inventors obtained and analyzed the answers to 125 cases of tongue images read by five Chinese medicine specialists to answer multiple choice questions consisting of eight questions. The specialists who read the tongue images selected one option as the reading result from among the options for each question (size, color, etc.). As a result of the analysis, it was found that only 44 cases had a provisional correct answer obtained by majority vote among the five specialists in all eight questions. In other words, 81 cases, which is about 65% of the 125 cases, were not used for the evaluation by majority vote. In other words, evaluation by majority vote is not necessarily effective for any multiple choice questions.

[0009] There have been proposed inventions relating to the evaluation of respondents to multiple choice questions (for example, see Patent Document 1).

[0010] This document discloses a method for calculating the degree of similarity between a first examinee who underwent a medical checkup and one or more second examinees (groups) who underwent the same test. The degree of similarity calculated based on the results of the medical checkup is used to diagnose the health condition, etc., of the first examinee (whether or not the first examinee suffers from a particular disease, and whether or not the first examinee is likely to suffer from the disease in the future, etc.). In other words, the degree of similarity enables appropriate medical treatment to be provided to the first examinee based on the health condition, etc. of the second examinee (who is known to suffer from a particular disease, etc.) who is highly similar to the first examinee.

[0011] Here, the test in this document can be considered as a multiple choice question. That is, the test item is a question, and if the test result is qualitative data (symbolic data), it becomes the option as it is, or if it is quantitative data (numerical data), each of the multiple categories to which the test result may belong is defined as an option. Then, based on the test result of the second examinee, the probability that the second examinee exists for each category (option) is obtained and the similarity is calculated. That is, for example, for a test item with three categories "x", "y", and "z" defined as the test results, the test results of 10 second examinees are 5 people with "x", 4 people with "y", and 1 person with "z". In this case, the probability that the second examinee exists in each category is "0.50" for "x", "0.40" for "y", and "0.10" for "z". Here, when the test result of the first examinee is "y", the similarity between the first examinee and the 10 second examinees is calculated as "0.40".

[0012] Also, an invention has been proposed regarding classification of respondents (answerers) to multiple choice questions (see, for example, Patent Document 2).

[0013] This document discloses a method for calculating distance, which is an index showing the consistency of multiple specific options (incorrect answer options) selected by multiple respondents from among the options (including correct answer options and multiple incorrect answer options) included in each question of a multiple-choice question. In other words, the distance showing the consistency of incorrect answer options of multiple respondents for a multiple-choice question with a correct answer classifies multiple respondents who answered the multiple-choice question. In other words, respondents who are classified into the same group are classified as people who have a common tendency to make errors. As a result, for example, a trainer who provides guidance to the respondents can identify groups of respondents who have a common tendency to make errors and provide common guidance to avoid making errors.

[0014] Note that this document discloses not only its application to multiple-choice questions with correct answers, but also its application to multiple-choice questions without correct answers. This document also discloses that even in multiple-choice questions without correct answers, the consistency of specific options may be meaningful in the field. For example, a group of respondents who answer the same option in a given group may be given a certain interpretation by an expert in the field. In other words, in this document, the evaluation of respondents who answered each question in a multiple-choice question without a correct answer is performed using cluster analysis. [Prior art documents] [Patent documents]

[0015] [Patent Document 1] Patent Publication No. 2021-26190 [Patent Document 2] Patent Publication No. 2022-70413 Summary of the Invention [Problem to be solved by the invention]

[0016] An object of the present invention is to provide an evaluation device, an evaluation program, and an evaluation method that can evaluate an answerer based on the scoring results of answers to multiple-choice questions that have no correct answer. [Means for solving the problem]

[0017] The evaluation device of the present invention is an evaluation device that evaluates a respondent by scoring the answer of the respondent through a comparison between respondent options, which are the respondent's answer to a multiple-choice question that has no correct answer, and benchmark options, which are answers of each of a plurality of benchmarks to the multiple-choice question, and comprises a memory unit that stores respondent option information that identifies the respondent option selected as the respondent's answer from among a plurality of options included in the multiple-choice question, and a total number of selections, which is the total number of benchmarks selected as the benchmark option for each of the plurality of options, and a respondent evaluation value calculation unit that calculates a respondent evaluation value, which is an index of the respondent's evaluation, based on the respondent option information and the total number of selections for each of the plurality of options, and is characterized in that the respondent evaluation value calculation unit calculates the respondent evaluation value based on the total number of selections corresponding to the respondent option and the maximum total number of selections among the total numbers of selections for each of the plurality of options. Effect of the Invention

[0018] The present invention can evaluate respondents based on the results of scoring their answers to multiple choice questions that have no correct answer. [Brief description of the drawings]

[0019] [Figure 1] 1 is a block diagram showing an embodiment of an evaluation device according to the present invention; [Diagram 2] FIG. 2 is a network configuration diagram showing an embodiment of the evaluation device. [Diagram 3] FIG. 4 is a schematic diagram showing an example of an answerer evaluation value calculated by the evaluation device. [Figure 4] FIG. 2 is a schematic diagram showing examples of multiple choice questions stored in a storage unit of the evaluation device. [Diagram 5] FIG. 13 is a schematic diagram showing examples of answers to each question in a multiple choice question for each benchmark subject, which are stored in the storage unit. [Figure 6] FIG. 13 is a schematic diagram showing an example of the total number of selections and the base-subject probability for each of a plurality of options constituting each question of a multiple-choice question stored in the storage unit. [Figure 7]4 is a schematic diagram showing examples of answers to each question of a multiple choice question for each respondent stored in the storage unit. FIG. [Figure 8] 13 is a histogram showing an example of a reference person probability calculated based on the total number of selections for each of the plurality of options. [Figure 9] 13 is a histogram showing another example of the reference person probability calculated based on the total number of selections for each of the plurality of options. [Figure 10] 13 is a schematic diagram of an answer input screen showing an example of input by a respondent to each question in a multiple choice question. FIG. [Figure 11] 13 is a schematic diagram of an evaluation result output screen showing an example of an output of an evaluation result of a respondent for a multiple choice question. FIG. [Figure 12] 13 is a schematic diagram of an answer input screen showing an example of input by another respondent to each question of a multiple choice question. FIG. [Figure 13] 13 is a schematic diagram of an evaluation result output screen showing an example of an output of an evaluation result of another respondent for a multiple choice question. FIG. [Figure 14] FIG. 13 is a schematic diagram showing an example of the base subject probability for each of a plurality of options constituting each question of a multiple choice question stored in the storage unit. [Figure 15] FIG. 13 is a schematic diagram showing an example of the probability of each candidate for each question in the multiple choice questions stored in the storage unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An evaluation apparatus, an evaluation program, and an evaluation method according to the present invention will be described below with reference to the accompanying drawings.

[0021] This embodiment will be described by taking as an example a case where a medical student is evaluated using the interpretation results of a tongue image, which is the answer. The tongue image has been interpreted by five Chinese medicine specialists. The medical student's interpretation results are evaluated in relation to the interpretation results of the five specialists. The tongue image interpretation results are shown as one option selected as an answer from five options included in question 1 regarding the size of the tongue, and one option selected as an answer from six options included in question 2 regarding the color of the tongue. That is, the options of the question regarding the tongue image and its size, and the options of question 2 regarding the tongue image and its color are examples of options in each question of a multiple-choice question with no correct answer in the present invention. The five Chinese medicine specialists are examples of the reference persons in the present invention. Other examples of the reference persons include (1) a group of patients with a definite diagnosis, (2) a group of healthy people, and (3) the past respondent himself / herself. The medical student is an example of the respondent in the present invention. Randomized multiple choice questions (interviews) are particularly effective for mental health care for past respondents. Here, randomized multiple choice questions are questions in which both the order in which questions appear and the order of options within each question are randomly changed, resulting in multiple questions that are essentially the same but have different apparent wording. Using these randomized questions, it is possible to detect unconscious changes in the respondent's mind as a sign of consistency between the respondent's past answers.

[0022] ● Evaluation device configuration ● FIG. 1 is a functional block diagram showing an embodiment of an evaluation device according to the present invention (hereinafter referred to as "the device").

[0023] This device is realized by an information processing device such as a personal computer. This device executes an evaluation program according to the present invention (hereinafter referred to as "this program"). This program running on this device cooperates with the hardware resources of this device to realize an evaluation method according to the present invention (hereinafter referred to as "this method").

[0024] The hardware resources of this device are, for example, processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), etc. The processor executes instructions written in this program to realize each of the means provided in this device, which will be described later.

[0025] When an information processing device (not shown) different from this device executes this program, the information processing device functions in the same manner as this device and realizes this method.

[0026] The device 1 comprises a memory unit 2, an option display unit 3, a reference person response receiving unit 4, a total selection calculation unit 5, a reference person probability calculation unit 6, a respondent response receiving unit 7, and a respondent evaluation value calculation unit 8.

[0027] The storage unit 2 stores the program, information used by the device 1 to realize the method, etc. Details of the information used by the device 1 to realize the method will be described later.

[0028] The types of storage units in the present invention include, for example, portable storage media such as HDDs (Hard Disk Drives), SSDs (Solid State Drives), flash memories, and other non-transitory storage media, as well as RAMs (Random Access Memory) and other temporary storage media.

[0029] The option display unit 3 displays a plurality of options constituting each question of the multiple choice question on a referee terminal operated by a referee and on a respondent terminal operated by a respondent.

[0030] FIG. 2 is a network configuration diagram showing an embodiment of the present device 1. As shown in FIG. The device 1 is communicatively connected to the reference terminals D11-D15 and the respondent terminals D21, D22 via a communication network N. The communication between the device 1 and each terminal is realized, for example, by the device 1 operating as a so-called web server and each terminal operating as a so-called web client.

[0031] The communication network N is a computer communication network such as the Internet or a LAN (Local Area Network).

[0032] Each of the terminals D11-D15, D21, and D22 is realized by a personal computer or a smartphone.

[0033] Here, the reference person terminals D11-D15 are operated by five reference persons, respectively. The respondent terminals D21, D22 are operated by two respondents, respectively. Here, the reference person IDs of the five reference persons are A1-A5, and the respondent IDs of the two respondents are B1, B2. The reference person ID and respondent ID are information used by the device 1 to identify the reference person and the respondent, and are assigned in advance to the device 1, for example (the same applies to other IDs described below).

[0034] The referent answer receiving unit 4 receives referent option information from each of the referent terminals D11-D15 operated by the referent. The referent option information is information that identifies the referent option. The referent option is an option selected as the referent's answer to each question of the multiple choice question. The content of the referent option and the method of selecting it will be described in detail later.

[0035] The total number of choices calculation unit 5 calculates the total number of choices selected by the reviewer for each of the multiple choice options included in each question of the multiple choice question as the total number of choices. The details of the total number of choices and the calculation method thereof will be described later.

[0036] The standard candidate probability calculation unit 6 calculates the standard candidate probability for each of the multiple choice options included in each question of the multiple choice question. The content of the standard candidate probability and the calculation method thereof will be described in detail later.

[0037] The answerer answer receiving unit 7 receives answerer option information from each of the answerer terminals D21 and D22. The answerer option information is information that identifies an answerer option. An answerer option is an option selected as an answer by an answerer for each question of a multiple choice question. The contents of the answerer options and the method of selecting them will be described in detail later.

[0038] The answerer evaluation value calculation unit 8 calculates an answerer evaluation value based on the total number of selections for each of the multiple choice options constituting each question of the multiple choice question and the answerer option information. The answerer evaluation value is the result of scoring the answer of the answerer to each question of the multiple choice question, and is an index of the answerer's evaluation. The content of the answerer evaluation value and the calculation method thereof will be described in detail later.

[0039] Respondent evaluation score FIG. 3 is a schematic diagram showing an example of the respondent evaluation value calculated by the device 1. As shown in FIG. The figure shows three examples of the distribution of answers when 10 benchmark subjects answer a question consisting of four options, "X1," "X2," "X3," and "X4."

[0040] In Figure (a), the distribution of answers of the standard subjects when 6 of 10 standard subjects select option "X1" as the standard subject option and 4 select option "X2" as the standard subject option is estimated by a histogram in which the options are intervals and the heights of the intervals represent the standard subject probability. In other words, the histogram is a probability density function of the distribution, and the total number of selections of option "X1" is "6", the total number of selections of option "X2" is "4", and the total number of selections of options "X3" and "X4" is "0". The total number of selections for each of multiple options is the total number of standard subjects who selected that option, and there are cases where the total number of selections is zero. In addition, the standard subject probability for each of multiple options is a probability defined as the ratio of the number of standard subjects who selected that option to the total number of standard subjects. The standard probability of option "X1" is "0.60" (=6 / 10), the standard probability of option "X2" is "0.40" (=4 / 10), and the standard probability of options "X3" and "X4" is "0.00" (=0 / 10).

[0041] Here, when the histogram of the standard answer is shown in Figure 1(a) and the respondent selects option "X2", the respondent evaluation value, which is the scoring result of the respondent's answer (selection of option "X2"), is 0.40 / 0.60, which is the ratio of the standard probability of "X2", which is the option selected by the respondent, "0.40", to the option with the largest total number of selections out of the four options, i.e., the maximum standard probability, "0.60".

[0042] Figure (b) shows the distribution of answers of the benchmark subjects when, out of 10 benchmark subjects, 5 benchmark subjects select option "X1" as the benchmark subject option, 4 benchmark subjects select option "X2" as the benchmark subject option, and 1 benchmark subject selects option "X3" as the benchmark subject option. In other words, the total number of choices for option "X1" is "5", the total number of choices for option "X2" is "4", the total number of choices for option "X3" is "1", and the total number of choices for option "X4" is "0". In other words, the benchmark subject probability of option "X1" is "0.50", the benchmark subject probability of option "X2" is "0.40", the benchmark subject probability of option "X3" is "0.10", and the benchmark subject probability of option "X4" is "0.00".

[0043] Here, when the histogram of the standard respondent's answers is shown in Figure (b) and the respondent selects option "X2," the respondent evaluation value, which is the scoring result of the respondent's answer (selection of option "X2"), is 0.40 / 0.50, which is the ratio of the standard respondent probability of "0.40" for option "X2" selected by the respondent to the maximum standard respondent probability of "0.50" among the four options.

[0044] Figure (c) shows the distribution of answers of the benchmark subjects when, among 10 benchmark subjects, 4 benchmark subjects select option "X2" as the benchmark subject option, 3 benchmark subjects select option "X3" as the benchmark subject option, and 3 benchmark subjects select option "X4" as the benchmark subject option. In other words, the total number of choices for option "X1" is "0", the total number of choices for option "X2" is "4", the total number of choices for option "X3" is "3", and the total number of choices for option "X4" is "3". In other words, the benchmark subject probability of option "X1" is "0.00", the benchmark subject probability of option "X2" is "0.40", the benchmark subject probability of option "X3" is "0.30", and the benchmark subject probability of option "X4" is "0.30".

[0045] Here, when the histogram of the standard respondent's answers is shown in Figure (c) and the respondent selects option "X2," the respondent evaluation value, which is the scoring result of the respondent's answer (selection of option "X2"), is 0.40 / 0.40, which is the ratio of the standard respondent probability of "X2," the option selected by the respondent, to the maximum standard respondent probability of "0.40" among the four options.

[0046] In this way, the method of scoring answers in the present invention and the answerer evaluation value, which is the scoring result, are defined as follows. L: Number of multiple choice questions (1 is fine) L ≥ 1 ·K j : Number of options in question j (The number of options may vary for each question.) ·R j (k): The kth answer for question j j=1~L, k=1~K j n: total number of references ·n j (k): Option R in question j j Total number of people who selected (k) (total number of selections) n=n j (1)+···+n j (K j ) p j (k): Option R in question jj (k)'s standard probability p j (k)=n j (k) / n p j (1)+···+p j (K j )=1 ·s i,j : In question j, respondent i chooses option R j When (k) is selected Respondent i's evaluation score for question j s i,j =p j (k) / max[p j (1),...,p j (K j )] 0≦s i,j <= 1 ·S i : Average evaluation score of respondent i for multiple choice questions S i =(s i,1 +···+s i,L ) / L 0≦S i <= 1

[0047] The respondent's respondent evaluation value, which is the result of scoring using the answers to each question in the multiple choice question defined above, will be "1" when the option selected as the respondent's answer matches the option selected by the greatest number of criteria persons. On the other hand, when the option selected as the respondent's answer differs from the option selected by the greatest number of criteria persons, if there are no criteria persons who selected the option selected as the respondent's answer, the respondent evaluation value will be "0," and if there is at least one criteria person who selected the option selected as the respondent's answer, the respondent evaluation value will not be "0," as shown in Figure 3. The respondent's evaluation value based on the scoring of the answers to the multiple choice question (all questions) is the average respondent evaluation value.

[0048] In this way, the answerer evaluation value in the present invention indicates, so to speak, the degree of agreement of the answerer's answer with the distribution of answers of the reference person. In other words, the higher the degree of agreement between the answerer's answer and the distribution of answers of the reference person, the higher the answerer evaluation value of the answerer.

[0049] The respondent evaluation value in the present invention is calculated based on the distribution of the answers of the reference person, and is completely different from the "similarity" in the above-mentioned Patent Document 1, the "distance" in Patent Document 2, or evaluation by majority vote. The present invention is characterized in that the distribution of the answers of the reference person is represented by a histogram in which the options are intervals and the height of the intervals represents the probability of the reference person, and the respondent is evaluated by focusing on the agreement between the answer of the respondent and the distribution of the answers of the reference person.

[0050] - Inputting the respondent's answers to each question in multiple choice questions and outputting the respondent's evaluation results using the answers FIG. 4 is a schematic diagram showing examples of multiple choice questions stored in the storage unit 2 of the device 1. As shown in FIG. The figure shows that a question ID, a question, an option ID, and options are stored in the storage unit 2 in association with each other.

[0051] The question ID is information that the device 1 uses to identify each question in the multiple choice question.

[0052] The question is information indicating the question content of the multiple choice question.

[0053] The option ID is information that the device 1 uses to identify each option constituting each question in a multiple choice question.

[0054] The figure shows, for example, that question ID "Q1", question 1 "Size", option ID "C11" and option "Normal", option ID "C12" and option "Large", option ID "C13" and option "Weak", option ID "C14" and option "Not applicable", and option ID "C15" and option "Cannot be evaluated" are stored in association with each other in memory unit 2.

[0055] Here, the phrase "a plurality of pieces of information are stored in association with each other" means that the present device 1 can search the storage unit 2 using one piece of information among the plurality of pieces of information, thereby reading out the other pieces of information from the storage unit 2 (hereinafter the same). That is, for example, the present device 1 can search the storage unit 2 using the question ID "Q1", thereby reading out from the storage unit 2 the question 1 "size", the option ID "C11", the option "normal", and the like, which are stored in association with the question ID "Q1".

[0056] FIG. 5 is a schematic diagram showing examples of answers to multiple choice questions for each of five assumed reference subjects, stored in the storage unit 2, for each reference subject, with respect to tongue images. The figure shows that a reference person ID, a tongue ID, a question ID, and an option ID of the reference person's option are stored in the storage unit 2 in association with each other.

[0057] The reference person ID is information used by the device 1 to identify the reference person.

[0058] The tongue ID is information that the device 1 uses to identify the tongue image stored in the storage unit 2.

[0059] For example, the figure shows that a reference person with a reference person ID of "A1" selected an option with an option ID of "C11" as an answer to a question ID of "Q1" for a tongue image with a tongue ID of "T1".

[0060] FIG. 6 is a schematic diagram showing an example of the total number of selections and the base subject probability for each of a plurality of options constituting each question of a multiple choice question stored in the storage unit 2. In FIG.

[0061] As described above, the total number of selections is information about each of the multiple choice options contained in each question in the multiple choice question to which the subject has answered, and is the total number of subject candidates who selected that option among those who answered each question in the multiple choice question.

[0062] The probability of being the reference person for each of the multiple options is calculated as the ratio of the total number of reference people who selected that option (total number of selections) to the total number of reference people, as defined above.

[0063] The figure shows that the tongue ID, question ID, option ID, total number of selections, and the probability of the reference person are stored in association with each other. That is, for example, the figure shows that, of the five options as answers to the question ID "Q1" of each of the five reference people for the tongue image of the tongue ID "T1", the total number of reference people who selected option ID "C11" is "2", and the reference person probability of this option is "0.40", the total number of reference people who selected option ID "C12" is "0", and the reference person probability of this option is "0.00", the total number of reference people who selected option ID "C13" is "3", and the reference person probability of this option is "0.60", the total number of reference people who selected option ID "C14" is "0", and the reference person probability of this option is "0.00", and the total number of reference people who selected option ID "C15" is "0", and the reference person probability of this option is "0.00".

[0064] FIG. 7 is a schematic diagram showing examples of answers to each multiple choice question for each respondent stored in the storage unit 2. As shown in FIG.

[0065] The figure shows that the respondent ID, tongue ID, question ID, and option ID of the respondent option are stored in association with each other in the storage unit 2. That is, for example, the figure shows that the respondent with the respondent ID "B1" selected the option with the option ID "C13" as the answer to the question ID "Q1" for the tongue image with the tongue ID "T1".

[0066] FIG. 8 is a histogram showing an example of the probability of a reference person calculated based on the total number of selections for each of a plurality of options.

[0067] The figure shows that the total number of choices for each of the five options for question 1, "size," - "normal," "overweight," "emaciated," "not applicable," and "unassessable" - was "2," "0," "3," "0," and "0." The figure shows that the reference probability for the option "normal" was "0.40" (=2 / 5), and the reference probability for the option "emaciated" was "0.60" (=3 / 5).

[0068] FIG. 9 is a histogram showing another example of the reference person probability calculated based on the total number of selections for each of a plurality of options.

[0069] The figure shows that the total number of choices for each of the six options in question 2, "Color," namely, "Pale white," "Pale pink," "Deep red," "Purple," "Other," and "None," are "0," "3," "0," "2," "0," and "0." The figure shows that the standard probability of the option "Pale pink" is "0.60" (=3 / 5), and the standard probability of the option "Purple" is "0.40" (=2 / 5).

[0070] FIG. 10 is a schematic diagram of an answer input screen showing an example of an answer input by a respondent with an answerer ID of "B1" to a tongue image with an tongue ID of "T1".

[0071] The figure shows that the answer input screen R11 is displayed on the display device of the answerer terminal D21 operated by the answerer with the answerer ID "B1." The information included in the answer input screen R11 includes the answerer ID, the tongue image of the tongue ID "T1," options for each of the two questions, and the input button image I1.

[0072] The device 1 uses the option display unit 3 to read out each question of the two multiple choice questions stored in the memory unit 2, and generates an answer input screen R11. The device 1 uses the option display unit 3 to transmit the generated answer input screen R11 to the respondent terminal D21. As a result, the answer input screen R11 is displayed on the respondent terminal D21.

[0073] Here, the tongue image of tongue ID "T1" is an image stored in the storage unit 2. The tongue image displayed on the answer input screen R11 is an image that has been interpreted in advance by a reference person. In other words, the reference person who interpreted the tongue image of tongue ID "T1" displayed on the answer input screen R11 has already answered each question of the two multiple choice questions displayed on the answer input screen R11.

[0074] That is, the present device 1 uses the option display unit 3 to display the answer input screen R11 on each of the reference person terminals D11-D15. The answers of the reference person entered on the answer input screen R11 are transmitted from each of the reference person terminals D11-D15 to the present device 1 as reference person option information. The present device 1 uses the reference person answer receiving unit 4 to store the reference person option information received from each of the reference person terminals D11-D15 in the storage unit 2 in association with the reference person ID (FIG. 5). The present device 1 uses the total number of choices calculation unit 5 to calculate the total number of choices for each option of each question in the multiple choice question, based on the reference person option information stored in the storage unit 2 in association with the reference person ID. The present device 1 uses the reference person probability calculation unit 6 to calculate the reference person probability for each of the multiple options, based on the total number of choices for each option of each question in the multiple choice question.

[0075] In this way, before the answer input screen R11 is displayed on the respondent terminal D21, the total number of selections (Figure 6) and the base respondent probability (Figures 8 and 9) for each of the multiple choice options in each question of the multiple choice question are calculated by the total number of selections calculation unit 5 and the base respondent probability calculation unit 6 of the device 1 and stored in the memory unit 2.

[0076] The respondent, while viewing the answer input screen R11, inputs answers to each of the multiple choice questions displayed on the screen. Fig. 10 shows that the option "thin" has been selected and input as the answer to question 1 "size". The same figure shows that the option "light pink" has been selected and input as the answer to question 2 "color".

[0077] When the respondent selects the input button image I1, the respondent's answer entered on the answer input screen R11 is transmitted as answerer option information from the respondent terminal D21 to the present device 1. The present device 1 uses the respondent answer receiving unit 7 to store the answerer option information received from the respondent terminal D21 in the memory unit 2 in association with the respondent ID as the answer of the respondent with the respondent ID "B1" displayed on the answer input screen R11 (FIG. 7).

[0078] The display device of the respondent terminal D21 is, for example, a touch panel display device. In other words, the respondent selects an option (inputs an answer) and selects an input button image I1 by touching the touch panel display device.

[0079] FIG. 11 is a schematic diagram of an evaluation result output screen showing an example of an output of an evaluation result of a respondent with a respondent ID of "B1" for a tongue image with a tongue ID of "T1".

[0080] The figure shows that the evaluation result output screen R12 is displayed on the display device of the respondent terminal D21 operated by the respondent with the respondent ID "B1." Information included in the evaluation result output screen R12 includes the respondent ID and the respondent's evaluation result (100 points).

[0081] The respondent's evaluation result is calculated by the device 1 when it receives the respondent choice information. When the device 1 receives the respondent choice information, it uses the respondent evaluation value calculation unit 8 to calculate a respondent evaluation value based on the respondent choice information and the total number of selections. The device 1 transmits an evaluation result output screen R12 including the calculated respondent evaluation value to the respondent terminal D21. The respondent can understand the evaluation result for his or her answer by viewing the evaluation result output screen R12 displayed on the display device of the respondent terminal D21.

[0082] Here, the evaluation result displayed on the evaluation result output screen R12 is the above-mentioned average respondent evaluation value. The average respondent evaluation value is calculated as follows based on the above-mentioned definition.

[0083] The answer selected by the respondent with ID "B1" was "thin" for question 1 "size" and "light pink" for question 2 "color" (Figure 10).

[0084] On the other hand, in the responses of the five standard subjects, the total number of selections for the option "lean" in question 1 "size" was "3," and the standard subject probability was "0.60." Among the options for question 1 "size," the option with the highest standard subject probability was "lean" (Figure 8). Therefore, the respondent evaluation value (s B1,1 ) is "1.00" (=0.60 / 0.60).

[0085] In addition, in the responses of the five standard subjects, the total number of selections for the option "pale pink" in question 2 "color" was "3," and the standard subject probability was "0.60." Among the options for question 2 "color," the option with the highest standard subject probability was "pale pink" (Figure 9). Therefore, the respondent evaluation value (s B1,2 ) is "1.00" (=0.60 / 0.60).

[0086] From the above, the average respondent evaluation score (S B1 ) is "1.00" (=(1.00+1.00) / 2).

[0087] The evaluation results included in the evaluation result output screen R12 are the average respondent evaluation value (S B1 ) is converted into 100 points and the result is displayed as "100 points."

[0088] FIG. 12 is a schematic diagram of an answer input screen showing an example of an answer input by a respondent with an answerer ID of "B2" to a tongue image with an tongue ID of "T1".

[0089] The figure shows an answer input screen R21 displayed on an answerer terminal D22 operated by an answerer with an answerer ID of "B2." The figure shows that the answer options selected by the answerer with an answer ID of "B2" are "normal" for question 1 "size" and "deep red" for question 2 "color."

[0090] On the other hand, in the responses of the five standard subjects, the total number of selections for the option "normal" in question 1 "size" was "2," and the standard subject probability was "0.40." Among the options for question 1 "size," the option with the highest standard subject probability was "lean," and the standard subject probability for the option "lean" was "0.60" (Figure 8). Therefore, the respondent evaluation value (s B2,1 ) is "0.67" (=0.40 / 0.60).

[0091] In addition, in the responses of the five standard subjects, the total number of selections for the option "deep red" in question 2 "color" was "0", and the standard subject probability was "0.00". Among the options for question 2 "color", the option with the highest standard subject probability was "pale pink", and the standard subject probability for the option "pale pink" was "0.60" (Figure 9). Therefore, the respondent evaluation value (s B2,2 ) is "0.00" (=0.00 / 0.60).

[0092] From the above, the average respondent evaluation score (S B2 ) is "0.34" (=(0.67+0.00) / 2).

[0093] FIG. 13 is a schematic diagram of an evaluation result output screen showing an example of an output of an evaluation result of a respondent with a respondent ID of "B2" for a tongue image with a tongue ID of "T1".

[0094] This figure shows the evaluation result output screen R22 displayed on the respondent terminal D22. B2) converted to 100 points, the result "34 points" is displayed.

[0095] ●Summary● According to the embodiment described above, the answer of the respondent to each question of the multiple choice question that has no correct answer is evaluated using the respondent evaluation value that indicates the correspondence between the answer of the respondent to each question of the multiple choice question and the distribution of the answers of the benchmark person. In other words, the higher the correspondence between the answer of the respondent and the distribution of the answers of the benchmark person, the higher the evaluation of the respondent.

[0096] The respondent evaluation value in the present invention, calculated using the distribution of the answers of such reference persons, is an index that is completely different from the "similarity" in the above-mentioned Patent Document 1 and the "distance" in Patent Document 2. Furthermore, the evaluation based on the respondent evaluation value in the present invention differs from the evaluation based on the majority vote in that the opinion of the minority is also reflected in the evaluation.

[0097] ●Evaluation of multiple choice questions● The device 1 may evaluate each of the multiple choice questions answered by the criterion person based on the criterion person's answer to the multiple choice question.

[0098] FIG. 14 is a schematic diagram showing an example of the base subject probability for each of a plurality of options constituting each question of a multiple choice question stored in the storage unit 2. As shown in FIG. The figure shows the options R that make up question j. j (k) and the reference probability p j That is, for example, the option R1(1) of question 1 and the reference person probability p1(1) are stored in the storage unit 2 in association with each other.

[0099] Here, the probability of the standard for each option p jAs defined above, (k) is calculated as the ratio of the total number of the criterion subjects who selected that option to the total number of the criterion subjects. That is, for example, when the criterion subject probability for each of the multiple options constituting each question of a multiple choice question is approximately the same for each option, it means that the opinions of the criterion subjects are divided. Therefore, for example, a question that divides the opinions of the criterion subjects (for example, the tongue image, or the question and its options in the embodiment described above) may not be appropriate to be used as a question for evaluating the respondent.

[0100] Therefore, the present device 1 may use a question evaluation unit (not shown) to evaluate questions based on the reference person probability for each of multiple options constituting the question answered by the reference person, that is, to determine whether or not to adopt the question as a question to be answered by the respondent. For example, the question evaluation unit may determine whether or not each question includes an option with a reference person probability of a predetermined value (e.g., 0.60) or more, and determine only questions including options with a reference person probability of a predetermined value or more (i.e., the maximum total number of options is a predetermined number or more) as good questions to be answered by the respondent because the opinions of the reference people are unified to a certain extent. In other words, the option display unit 3 of the present device 1 displays only the questions determined by the question evaluation unit as questions to be answered by the respondent on the respondent terminal.

[0101] In this way, by evaluating each question in a multiple choice question based on the probability of the judges, questions on which the judges' opinions are divided are excluded from the questions to be answered by the respondent.

[0102] ● Evaluation by the benchmark ● The device 1 may evaluate each subject based on the subject's own answers to each question in the multiple choice questions.

[0103] FIG. 15 is a schematic diagram showing an example of the probability of each benchmark subject for each question in the multiple choice questions stored in the storage unit 2. As shown in FIG. The figure shows the reference person ID and the probability p j(k) are stored in the storage unit 2 in association with each other. That is, for example, the reference person ID “A1” and the reference person probability p1(k A1 ) are stored in association with each other in the storage unit 2.

[0104] Here, the probability of the standard for each option p j As defined above, (k) is calculated as the ratio of the total number of reference persons who selected that option to the total number of reference persons. In other words, for example, a reference person who selected an option with an extremely low reference person probability value means that his / her opinion differs from that of many other reference persons. In particular, the answers of a reference person who selected options with an extremely low reference person probability value in many questions out of multiple questions may not be appropriate as a reference person answer used to evaluate respondents.

[0105] Therefore, the present device 1 uses a benchmark evaluation unit (not shown) to evaluate the benchmark based on the benchmark probability for each of the multiple choice questions that constitute each question to which the benchmark has answered. The evaluation of the benchmark is, for example, a ranking of the benchmarks based on the magnitude of the benchmark probability.

[0106] Furthermore, the present device 1 may use a reference person selection unit (not shown) to select a reference person from among multiple reference person candidates. That is, the reference person selection unit evaluates the reference person candidates based on the reference person candidate probability for each of multiple options constituting the question to which the reference person candidate has answered, and selects a specific reference person candidate as the reference person. Here, the reference person candidate probability for each of multiple options is a probability defined as the ratio of the number of reference person candidates who selected that option to the total number of reference person candidates, similar to the definition of the reference person probability described above.

[0107] The present device 1 displays each question of the multiple choice question on a reference candidate terminal operated by the reference candidate. The present device 1 receives, from the reference candidate terminal, reference candidate candidate option information that identifies a reference candidate option selected as the reference candidate's answer to each question. The present device 1 calculates the reference candidate candidate probability for each of the multiple options constituting the question based on the reference candidate candidate option information of each of the multiple reference candidate candidates.

[0108] The reference person selection unit determines whether or not to select each reference person candidate who has answered a question as the reference person based on the reference person candidate probability of the option selected for each question by the reference person candidate. In other words, the reference person selection unit divides a set (a set of reference person candidates) having multiple reference person candidates as elements into multiple sets, and determines only the reference person candidates belonging to a specific set as the reference person.

[0109] For example, if the reference person probability of a reference person candidate in the set of reference person candidates for each question is not equal to or less than a predetermined value (e.g., 0.10), the reference person selection unit selects the reference person candidate as the reference person. In other words, the reference person selection unit divides the set of reference person candidates across all questions into a subset of reference person candidates whose reference person candidate probability is equal to or less than a predetermined value for even one question, and a subset of reference person candidates whose reference person candidate probability is not equal to or less than the predetermined value for all questions. Then, the reference person selection unit determines as the reference person only those reference person candidates that belong to the subset of reference person candidates whose reference person candidate probability is equal to or less than the predetermined value for all questions.

[0110] Alternatively, for example, if the average value of the reference candidate probability for all questions of the reference candidate is not equal to or less than a predetermined value, the reference candidate selection unit selects the reference candidate as the reference candidate. In other words, the reference candidate selection unit divides the set of reference candidate candidates into a subset of reference candidate candidates whose average value of the reference candidate probability is not equal to or less than a predetermined value and a subset of reference candidate candidates whose average value of the reference candidate probability is equal to or less than a predetermined value. Then, the reference candidate selection unit determines as the reference candidate only those reference candidate candidates that belong to the subset of reference candidate candidates whose average value of the reference candidate probability is equal to or less than a predetermined value.

[0111] In this way, by evaluating the reference person based on the reference person probability, reference people whose opinions differ from many other reference people are identified. Also, by evaluating the reference person candidates based on the reference person candidate probability, answers of reference person candidates whose opinions differ from many other reference person candidates are excluded from the respondent's evaluation.

[0112] Features of the device, program, and method The features of the present device, the present program, and the present method described thus far will be summarized below.

[0113] Features of this device This device is An evaluation device (e.g., evaluation device 1) that evaluates an answerer by scoring the answerer's answer by comparing an answerer option selected as an answer by the answerer to a multiple-choice question having no correct answer with a criterion option selected as an answer by each of a plurality of criterion users to the multiple-choice question, Answer choice information that identifies the answer choice selected as an answer by the answerer from among a plurality of options included in the multiple choice question; The total number of selections (e.g., n j (k)) and A storage unit (e.g., storage unit 2) that stores the Based on the answerer option information and the total number of selections for each of the multiple options, an answerer evaluation value (e.g., s i,j A respondent evaluation value calculation unit (e.g., the respondent evaluation value calculation unit 8) that calculates and the answerer evaluation value calculation unit calculates the answerer evaluation value based on the total number of selections corresponding to the answerer option and the maximum total number of selections among the total numbers of selections for each of the plurality of options; It is characterized by:

[0114] In this device, The answerer evaluation value calculation unit expressing a distribution of the total number of selections for each of the options as a histogram in which the options are intervals and the reference person probability calculated from the total number of selections for each of the options is a value for each interval; Calculating the answerer evaluation value using the histogram. It can also be something like that.

[0115] In this device, the answerer evaluation value calculation unit calculates, as the answerer evaluation value, a ratio of the total number of selections corresponding to the answerer option to the maximum total number of selections among the total numbers of selections for each of the plurality of options; It can also be something like that.

[0116] This device is Connect multiple terminals via a communication network, The plurality of terminals include A reference terminal (e.g., reference terminals D11-D15) operated by each of the plurality of reference persons; A respondent terminal (e.g., respondent terminals D21 and D22) operated by the respondent; Including, a reference person response receiving unit (e.g., reference person response receiving unit 4) that receives reference person option information that identifies the reference person option of each of the reference persons from the plurality of options from the reference person terminal operated by each of the plurality of reference persons and stores the information in the storage unit; A total selection number calculation unit (for example, a total selection number calculation unit 5) that calculates the total number of selections for each of the plurality of options based on the reference person option information; An option display unit (e.g., option display unit 3) that displays the multiple options on the respondent terminal; a respondent answer receiving unit (e.g., the respondent answer receiving unit 7) that receives the respondent option information from the respondent terminal and stores it in the storage unit; and the option display unit displays the plurality of options on the respondent terminal only when the maximum total number of options among the total number of options selected for each of the plurality of options is equal to or greater than a predetermined number. It can also be something like that.

[0117] In this device, The storage unit is Answer choice information that identifies the answer choice for each of the multiple choice questions; the total number of selections for each of the multiple choice options included in each question of the multiple choice questions; Remember, The answerer evaluation value calculation unit calculating the respondent evaluation value for each of the multiple choice questions based on the total number of selections corresponding to the respondent options and the maximum total number of selections among the total numbers of selections for each of the multiple options; Based on the calculated respondent evaluation value for each question of the multiple choice questions, an average respondent evaluation value (e.g., S i ) to calculate It can also be something like that.

[0118] This device is a criterion selection unit for selecting the criterion candidate from a plurality of criterion candidate candidates who have answered each of the plurality of questions in the multiple choice question; and The reference person selection unit, Criterion candidate option information that identifies a criterion candidate option selected as an answer by the criterion candidate to each question of the multiple choice questions; determining whether to select the reference candidate as the reference person based on the It can also be something like that.

[0119] In this device, The reference person selection unit, Dividing the plurality of reference candidate candidates into a plurality of sets based on the reference candidate option information for each question of the multiple choice questions for each of the plurality of reference candidate candidates; determining the reference person candidate belonging to a specific set among the plurality of sets as the reference person; It can also be something like that.

[0120] ●Features of this program This program includes: Making an information processing device function as the present device; It is characterized by:

[0121] Features of this method The method comprises: A method executed by an evaluation device for evaluating a respondent by scoring the answer of the respondent by comparing an answer option, which is an answer of the respondent to a multiple-choice question having no correct answer, with a referee option selected as an answer of each of a plurality of referees to the multiple-choice question, the method comprising: The evaluation device includes: Connecting to a respondent terminal operated by the respondent via a communication network, Answer choice information that identifies the answer choice selected as an answer by the answerer from among a plurality of options included in the multiple choice question; A total number of selections, which is a total number of the reference persons selected as the reference person option for each of the multiple options; A storage unit for storing the Equipped with The evaluation device, receiving the respondent option information from the respondent terminal and storing it in the storage unit; calculating an answerer evaluation value, which is an index of an evaluation of the answerer, based on the answerer option information and the total number of selections for each of the plurality of options; and the answerer evaluation value is calculated based on the total number of selections corresponding to the answerer option and the maximum total number of selections among the total numbers of selections for each of the plurality of options; It is characterized by: [Explanation of symbols]

[0122] 1. Evaluation device 2 Storage section 3 Option display section 4. Reference Person Response Reception Department 5. Selection total calculation section 6. Reference probability calculation section 7. Respondent Response Reception Department 8. Respondent evaluation value calculation section

Claims

1. 1. An evaluation device for evaluating an answerer by scoring the answerer's answer by comparing an answerer option selected as an answer by the answerer to a multiple-choice question having no correct answer with a criterion option selected as an answer by each of a plurality of criterion users to the multiple-choice question, comprising: Answer choice information that identifies the answer choice selected as an answer by the answerer from among a plurality of options included in the multiple choice question; A total number of selections, which is a total number of the reference persons selected as the reference person option for each of the multiple options; A storage unit that stores the an answerer evaluation value calculation unit that calculates an answerer evaluation value, which is an index of an evaluation of the answerer, based on the answerer option information and the total number of selections for each of the multiple options; and the answerer evaluation value calculation unit calculates, as the answerer evaluation value, a ratio of the total number of selections corresponding to the answerer option to the maximum total number of selections among the total numbers of selections for each of the plurality of options; An evaluation device comprising:

2. The answerer evaluation value calculation unit expressing a distribution of the total number of selections for each of the plurality of options as a histogram in which the options are set as intervals and a reference person probability calculated as a ratio of the total number of selections for each of the options to a total number of the plurality of reference persons is set as a value for each of the intervals; Calculating the answerer evaluation value using the histogram. The evaluation device according to claim 1.

3. Connect multiple terminals via a communication network, The plurality of terminals include A reference person terminal operated by each of the plurality of reference persons; A respondent terminal operated by the respondent; Including, a reference person response receiving unit that receives, from the reference person terminal operated by each of the plurality of reference persons, reference person option information that identifies the reference person option of each of the plurality of reference persons from the plurality of options, and stores the information in the storage unit; a total selection number calculation unit that calculates the total number of selections for each of the plurality of options based on the reference person option information; an option display unit that displays the multiple options on the respondent terminal; a respondent answer receiving unit that receives the respondent option information from the respondent terminal and stores it in the storage unit; and the option display unit displays the plurality of options on the respondent terminal only when the maximum total number of options among the total number of options for each of the plurality of options is equal to or greater than a predetermined number. The evaluation device according to claim 1.

4. The storage unit is the answerer choice information that identifies the answerer choice for each of the multiple choice questions; the total number of selections for each of the multiple choice options included in each question of the multiple choice questions; Remember, The answerer evaluation value calculation unit calculating the respondent evaluation value for each of the multiple choice questions based on the total number of selections corresponding to the respondent options and the maximum total number of selections among the total numbers of selections for each of the multiple options; calculating an average answerer evaluation value of the answerers for the multiple choice questions based on the calculated answerer evaluation value for each question of the multiple choice questions; The evaluation device according to claim 1.

5. a criterion selection unit for selecting the criterion candidate from a plurality of criterion candidate candidates who have answered each of the plurality of questions in the multiple choice question; and The reference person selection unit, Criterion candidate option information that identifies a criterion candidate option selected as an answer by the criterion candidate to each question of the multiple choice questions; determining whether to select the reference candidate as the reference person based on the The evaluation device according to claim 1.

6. The reference person selection unit, Dividing the plurality of reference candidate candidates into a plurality of sets based on the reference candidate option information for each question of the multiple choice questions for each of the plurality of reference candidate candidates; determining the reference person candidate belonging to a specific set among the plurality of sets as the reference person; The evaluation device according to claim 5.

7. A computer is caused to function as the evaluation device according to claim 1.

13. An evaluation program comprising:

8. A method executed by an evaluation device for evaluating a respondent by scoring the answer of the respondent by comparing an answer option, which is an answer of the respondent to a multiple-choice question having no correct answer, with a referee option selected as an answer of each of a plurality of referees to the multiple-choice question, the method comprising: The evaluation device includes: Connecting to a respondent terminal operated by the respondent via a communication network, Answer choice information that identifies the answer choice selected as an answer by the answerer from among a plurality of options included in the multiple choice question; A total number of selections, which is a total number of the reference persons selected as the reference person option for each of the multiple options; A storage unit for storing the Equipped with The evaluation device, receiving the respondent option information from the respondent terminal and storing it in the storage unit; calculating an answerer evaluation value, which is an index of an evaluation of the answerer, based on the answerer option information and the total number of selections for each of the plurality of options; and the answerer evaluation value is a ratio of the total number of selections corresponding to the answerer option to the maximum total number of selections among the total numbers of selections for each of the multiple options; 13. An evaluation method comprising:

Citation Information

Patent Citations

  • Question and answer service method

    JP2010073217A

  • Questionnaire support device, questionnaire support system, questionnaire support method, and questionnaire support program

    JP2018106275A

  • Information processing device, information processing program and information processing method

    JP2021026190A

  • Information processing device, information processing program and information processing method

    JP2022070413A