Answer information evaluation device and method

Abstract

When presenting question information to a user and acquiring first answer information of the user to the question information to evaluate the same, one aspect of the present invention: classifies the first answer information into any of a plurality of predefined types; from among a plurality of additional question information items created in advance and respectively corresponding to the plurality of types, selects an additional question information item corresponding to the type into which the first answer information has been classified; and presents the additional question information item to the user. Then, said one aspect of the present invention: acquires second answer information of the user to the additional question information item; evaluates consistency between the acquired second answer information and the first answer information; and, on the basis of the evaluation result of the consistency, determines whether an erroneous answer to the first answer information is equivalent to a first erroneous answer, in which the question information was examined but could not be understood, or a second erroneous answer, which is considered to have been answered without examining the question information.

Description

Answer information evaluation device and method

[0001] One aspect of the present invention relates to a response information evaluation device and management method used to evaluate response information to, for example, an academic survey.

[0002] In recent years, a method of distributing tests on websites and collecting responses has become widely used for academic research and other purposes. This method allows the survey to be conducted without preparing a venue or a facilitator, making it possible to conduct large-scale surveys with little effort and cost.

[0003] However, when responding to a test without supervision by the test setter, such as participating in an academic survey, where the respondent will not suffer any disadvantage as a result of their answers, they may answer without sufficient consideration in order to reduce the effort required to answer. This behavior is called "Satisficing" and is one of the factors that leads to a decline in the quality of response data.

[0004] In contrast to this, for example, Non-Patent Document 1 uses a method in which a response field that appears to be a general questionnaire response method such as a Likert scale is presented, and the respondent is instructed to respond in an unusual way. Using this method, if the respondent answers incorrectly, it is possible to consider that the respondent did not read the response instructions sufficiently and is a "Satisfied" respondent.

[0005] Daniel M. Oppenheimer, Tom Meyvis, Nicolas Davidenko, “Instructional manipulation checks: Detecting satisficing to increase statistical power”, Journal of Experimental Social Psychology, Volume 45, Issue 4,2009,Pages 867-872.

[0006] However, the method described in Non-Patent Document 1 does not take into consideration cases where respondents cannot understand the content of the answer instructions even though they have read them, and therefore cannot distinguish between respondents who are "Satisfied" and respondents who lack the ability to understand the answer instructions. For this reason, when excluding answers that are deemed "Satisfied," incorrect answers that should have been kept may also be excluded, which may result in biased answer information.

[0007] This invention was made with the above-mentioned circumstances in mind, and aims to provide a technology that makes it possible to distinguish between incorrect answers that are likely to be "Satisfied" and incorrect answers that are due to a failure to understand the question, thereby improving the quality of answer information.

[0008] In order to solve the above problem, one aspect of an answer information management device or method according to the present invention presents question information to a user, and when acquiring and evaluating first answer information from the user to the question information, classifies the first answer information into one of a plurality of predefined types, and selects additional question information corresponding to the classified type from a plurality of additional question information pre-created for each type and presents it to the user. Then, second answer information from the user to the additional question information is acquired, and consistency between the acquired second answer information and the first answer information is evaluated, and based on the evaluation result of the consistency, it is determined whether an incorrect answer to the first answer information corresponds to a first incorrect answer that is inferred to be due to the user having considered the question information but not understanding it, or a second incorrect answer that is inferred to be due to the user having answered without considering the question information.

[0009] According to one aspect of the present invention, additional question information corresponding to the type of first answer information is presented to the user, and based on the evaluation result of the consistency between the user's second answer information to this additional question information and the first answer information, it is determined whether the incorrect answer corresponds to a first incorrect answer, which is presumed to be a first incorrect answer given because the user considered the question information but was unable to understand it, or a second incorrect answer, which is presumed to be a careless answer given without considering the question information, i.e., an answer that is likely to be "Satisfiing." As a result, it is possible to exclude only answers that are likely to be "Satisfiing" from the first answer information, thereby improving the quality of the first answer information.

[0010] In other words, according to one aspect of the present invention, it is possible to distinguish between incorrect answers that are likely to be "Satisfied" and incorrect answers that are due to a failure to understand the question, thereby providing a technology that improves the quality of answer data.

[0011] FIG. 1 is a diagram showing an example of the configuration of an answer information evaluation system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the hardware configuration of an answer information evaluation device used in the system shown in FIG. 1. FIG. 3 is a block diagram showing an example of the software configuration of an answer information evaluation device used in the system shown in FIG. 1. FIG. 4 is a diagram used to explain an example of an additional question information creation process, showing an example of an important point group indicating types of answer patterns to questions and their classification conditions. FIG. 5 is a diagram used to explain an example of an additional question information creation process, showing an example of additional question information created by type. FIG. 6 is a flowchart showing an example of the processing procedure and processing content of the answer information evaluation process executed by the control unit of the answer information evaluation device shown in FIG. 3. FIG. 7A is a flowchart showing an example of the processing procedure and processing content of the first half of the additional question information presentation process in the answer information evaluation process shown in FIG. 6. FIG. 7B is a flowchart showing an example of the processing procedure and processing content of the second half of the additional question information presentation process in the answer information evaluation process shown in FIG. 6. FIG. 8A is a flowchart showing an example of the processing procedure and processing content of the first half of the answer type determination process in the answer information evaluation process shown in FIG. 6. Fig. 8B is a flowchart showing an example of the processing procedure and processing content of the middle part of the answer type determination processing in the answer information evaluation processing shown in Fig. 6. Fig. 8C is a flowchart showing an example of the processing procedure and processing content of the latter part of the answer type determination processing in the answer information evaluation processing shown in Fig. 6. Fig. 9 is a diagram for explaining an example of the answer type determination processing.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the configuration of an answer information evaluation system according to one embodiment of the present invention.

[0014] The answer information evaluation system according to one embodiment includes an answer information evaluation device CS, which enables communication of information data via a network NW between the answer information evaluation device CS and user terminals UT1 to UTn used by multiple users who are respondents, and an administrator terminal MT used by a system administrator.

[0015] The user terminals UT1 to UTn and the administrator terminal MT are, for example, mobile terminals such as smartphones or personal computers. The user terminals UT1 to UTn receive and display the question information and additional question information transmitted from the answer information evaluation device CS, and also return user answer information to the question information and additional question information to the answer information evaluation device CS. The administrator terminal MT transmits various setting information and control information input by the system administrator to the answer information evaluation device CS, and receives evaluation results of the answer information transmitted from the answer information evaluation device CS.

[0016] (2) Response Information Evaluation Device CS FIGS. 2 and 3 are block diagrams showing an example of the hardware configuration and software configuration of the response information evaluation device CS, respectively.

[0017] The answer information evaluation device CS is composed of a server computer located on the Web or the cloud, for example. The answer information evaluation device CS includes a control unit 1 having a hardware processor, and a storage unit including a program storage unit 2 and a data storage unit 3, and a communication I / F unit 4, which are connected to the control unit 1 via a bus 5.

[0018] The communication I / F unit 4 transmits and receives information data to and from the user terminals UT1 to UTn and the administrator terminal MT in accordance with a communication protocol defined by the network NW.

[0019] The program storage unit 2 is, for example, a combination of a non-volatile memory such as a HDD (Hard Disk Drive) or SSD (Solid State Drive) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a ROM (Read Only Memory), and stores application programs necessary for executing various processes related to the first embodiment of the present invention, in addition to middleware such as an OS (Operating System).

[0020] The data storage unit 3 is a combination of a non-volatile memory such as an HDD or SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and the storage area includes an answer pattern storage unit 31, a type and important point storage unit 32, an additional question information storage unit 33, an answer information storage unit 34, and a judgment information storage unit 35. The storage area also includes a buffer area for temporarily storing data when the control unit 1 executes various processes.

[0021] The answer pattern storage unit 31 stores in advance a plurality of expected answer patterns for test question information. The type / important point storage unit 32 stores in advance a plurality of types of the answer patterns and a group of important points indicating the classification conditions thereof.

[0022] The additional question information storage unit 33 is used to store additional question information created for each of the multiple answer pattern types. The answer information storage unit 34 is used to store user answer information to question information.

[0023] The determination information storage unit 35 is used to store information representing the determination result of the type of answer information. In one embodiment, two types of answer information are defined: an answer corresponding to "Satisfied," which may be an answer given by the user without considering the question information, and other answers.

[0024] The control unit 1 has the processing functions necessary to realize one embodiment of this invention, including an additional question information creation processing unit 11, an answer information collection processing unit 12, an additional question information presentation processing unit 13, an answer type determination processing unit 14, and an answer evaluation processing unit 15.

[0025] Each of the processing units 11 to 15 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 15 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0026] The additional question information creation processing unit 11 classifies the multiple types of incorrect answer patterns stored in the answer pattern storage unit 31 into multiple types based on the important points for each type stored in the type / important point storage unit 32, and creates additional question information corresponding to the incorrect answer patterns for each classified type.The created additional question information is then stored in the additional question information storage unit 33 in association with type identification information (type ID).

[0027] The response information collection processing unit 12 distributes question information designated by, for example, a system administrator or the like to the user terminals UT1 to UTn from the communication I / F unit 4, and receives answer information (hereinafter also referred to as first answer information) input by users in response to the question information from the user terminals UT1 to UTn via the communication I / F unit 4. Then, the received first answer information is stored in the response information storage unit 34 in association with the identification information (user ID) of the user who sent it.

[0028] The additional question information presentation processing unit 13 first calculates a score S (first score) indicating the possibility of “Satisficing” for each of the first answer information stored in the answer information storage unit 34, and determines whether the calculated score S is higher than a threshold value a (first threshold value).

[0029] The additional question information presentation processing unit 13 then determines the type of the first answer information whose score S is determined to be higher than the preset threshold a, in accordance with the important points for each type stored in the type / important point storage unit 13. Then, the additional question information presentation processing unit 13 reads out at least one additional question information corresponding to the determined type from the additional question information storage unit 33, and transmits the read out additional question information from the communication I / F unit 4 to the user terminals UT1 to UTn that are the senders of the first answer information.

[0030] An example of the calculation process of the score S and the type determination process performed by the additional question information presentation processing unit 13 will be described in the operation example.

[0031] The answer type determination processing unit 14 first receives the user's answer information (hereinafter also referred to as second answer information) to the above additional question information from the user terminals UT1 to UTn via the communication I / F unit 4, compares the received second answer information with multiple answer patterns stored in the answer pattern storage unit 31, selects an answer pattern that does not contradict the second answer information, and generates an answer set.

[0032] The answer type determination processing unit 14 then calculates a score Ca (second score) indicating the consistency between the answer set and the first answer information, and compares the calculated consistency score Ca with a preset threshold value s (second threshold value) to determine whether the type of the first answer information is an answer corresponding to "Satisfiing" or another answer. Then, a label indicating the type determination result is assigned to the first answer information, and the label is stored in the determination information storage unit 35.

[0033] An example of the process of generating the answer set and the process of determining consistency using the consistency score Ca performed by the answer type determination processing unit 14 will be described in the operation example.

[0034] The answer evaluation processing unit 15 excludes answer information corresponding to "Satisficing" from the first answer information based on the labeled first answer information stored in the determination information storage unit 35. Then, a predetermined evaluation process is performed based on the first answer information after the exclusion process, and information representing the evaluation result is transmitted from the communication I / F unit 4 to, for example, the administrator terminal MT.

[0035] (Example of Operation) Next, an example of operation of the response information evaluation device CS configured as above will be described.

[0036] (1) Creation of Additional Question Information In one embodiment, additional question information is prepared to determine whether the first answer information provided by the user in response to a question is an incorrect answer equivalent to “Satisfied,” a careless answer given without considering the question, or is another incorrect answer.

[0037] Therefore, the control unit 1 of the answer information evaluation device CS creates additional question information as follows under the control of the additional question information creation processing unit 11, prior to evaluation of the answer information.

[0038] That is, the additional question information creation processing unit 11 reads patterns of incorrect answers from the answer pattern storage unit 31, and classifies the read incorrect answer patterns into one of a plurality of types based on the important points for each type stored in the type / important point storage unit 32. Then, the additional question information creation processing unit 11 creates at least one piece of additional question information for each type of classified incorrect answer, and stores the created additional question information in the additional question information storage unit 33 in association with the type ID.

[0039] 4 shows an example of question information and predefined types and important point groups for answers to this question information. In this example, Type 1 and Type 2 are defined for expected answer patterns to the question information asking about the expression of a process that causes a state change, and Important Point Group 1 and Important Point Group 2 are defined for classifying answer patterns into Type 1 or Type 2.

[0040] Then, additional question information 1-1, 1-2 and additional question information 2-1, 2-2 are created corresponding to each of the above incorrect answer types 1 and 2, as shown in Fig. 5. These additional question information include content that asks whether or not the subject understands the gist of the question depending on the pattern of the incorrect answer.

[0041] (2) Evaluation of Answer Information to Questions FIG. 6 is a flowchart showing an example of the procedure and content of the answer information evaluation process executed by the control unit 1 of the answer information evaluation device CS.

[0042] (2-1) Collection of Answer Information When the control unit 1 of the answer information evaluation device CS receives, for example, an instruction to start evaluation sent from the administrator terminal MT in response to an operation by the system administrator in step S10, first, in step S20, under the control of the answer information collection processing unit 12, it distributes question information to the user and collects the corresponding answer information as follows.

[0043] For example, the response information collection processing unit 12 receives question information prepared by the system administrator from the administrator terminal MT, and distributes the received question information to the respondent's user terminals UT1 to UTn from the communication I / F unit 4. Then, the response information collection processing unit 12 receives the response information (first response information) entered by the user in response to the above-mentioned question information from each of the user terminals UT1 to UTn via the communication I / F unit 4, and stores the received first response information in the response information storage unit 34 in association with the user ID of the sender.

[0044] (2-2) Presentation of additional question information When the process of collecting the first answer information is completed, the control unit 1 of the answer information evaluation device CS proceeds to step S30, and under the control of the additional question information presentation processing unit 13, executes the process of presenting additional question information to users who are likely to be “Satisfied” and acquiring their answer information as follows.

[0045] 7A and 7B are flowcharts showing an example of the processing procedure and processing content of the additional question information presentation processing executed by the additional question information presentation processing unit 13. FIG.

[0046] (2-2-1) Determining the Likelihood of "Satisfied" Based on the Score S First, in step S301, the additional question information presentation processing unit 13 reads the first answer information of each user from the answer information storage unit 34. Then, in step S302, the additional question information presentation processing unit 13 calculates a score S indicating the likelihood of "Satisfied" for each of the read first answer information.

[0047] One example of a method for calculating the score S is to use the number of options Nc in the answer information for one question, the number of times the same option appears Nsc in the answer information for one question, the number of times the same option appears Nsca in each answer information for all questions, and a position score Pc that indicates the placement position of the option in the question.

[0048] The above Nc, Nsc, Nsca, and Pc are all elements that reflect the degree to which the user has omitted effort in answering the question, and the score S is set according to the number of elements. For example, if the number of selections Nc is small, such as when the user omits the selection operation of some options, the score S is set to a higher value as the number decreases. Also, if the number of selections Nsc of the same option is large, such as when the user repeatedly selects the same option for a single question, the score S is set to a higher value as the number increases. The same is true for Nsca. Also, if the number of times an option in an easy-to-select position is selected is large, such as when only options in easy-to-select positions are selected among the options in the question, the score S is set to a higher value as the number increases.

[0049] Specifically, the additional question information presentation processor 13 may calculate the score S by weighting and adding the above Nc, Nsc, Nsca, and Pc, or by averaging the weighted sum, i.e., the weighted average. The calculation formulas are as follows: S = αNc + βNsc + γNsca + δPc S = (αNc + βNsc + γNsca + δPc) / 4

[0050] The additional question information presentation processing unit 13 compares the score S calculated by the above formula with a threshold value a in step S303. If the score S is equal to or less than the threshold value a, the first answer information is determined to be an answer that is unlikely to be "Satisfied," and the subsequent processing related to the presentation of additional question information is not performed.

[0051] (2-2-2) Determination of Type of First Answer Information In contrast, if the score S is greater than the threshold value a, the additional question information presentation processing unit 13 determines that the first answer information is likely to be “Satisfied” and continues to perform the presentation processing of additional question information.

[0052] That is, the additional question information presentation processing unit 13 first selects the i=0th type in step S304 and determines whether the first answer information that is determined to have the possibility of being "Satisfiing" corresponds to the i=0th type.

[0053] For example, the additional question information presentation processing unit 13 first reads the corresponding important point group from the type and important point storage unit 32 in step S305, and selects the j=0th important point from the read important point group in step S306. Then, it determines in step S307 whether the j=0th important point is included in the first answer information.

[0054] If the result of this determination is that the j=0th important point is included in the first answer information, the additional question information presentation processing unit 13 determines in step S308 whether the number j of the important point has reached the total number M of important points included in the important point group. If the result of this determination is that the number j of the important point has not reached the total number M, the additional question information presentation processing unit 13 sets j to 1 in step S309 and selects the next important point. Then, returning to step S307, the additional question information presentation processing unit 13 determines whether the j=1th important point is included in the answer information. Thereafter, the additional question information presentation processing unit 13 repeatedly determines whether all important points included in the important point group are included in the answer information in the same manner.

[0055] On the other hand, when the number j of the selected important point reaches the total number M of important points included in the important point group, the additional question information presentation processing unit 13 assigns a label indicating the type i to the first answer information in step S310.

[0056] When the process of comparing the first answer information with type i is completed, the additional question information presentation processor 13 subsequently determines in step S311 whether type i has reached the total number of types N. If the result of this determination is that type i has not reached the total number of types N, the additional question information presentation processor 13 selects the (i+1)th type by setting i=i+1 in step S312, and returns to step S305 to execute a series of type determination processes in steps S305 to S312.

[0057] In addition, if, as a result of the comparison with all types N, it is determined that the first answer information does not fall into any of the types, the additional question information presentation processing unit 13 assigns an indeterminable label to the first answer information in step S313 and terminates the processing without performing any further processing related to the presentation of additional question information.

[0058] (2-2-3) Setting Additional Question Information and Acquiring Answer Information After the type determination process described above is completed, the additional question information presentation processing unit 13 then executes the setting process of additional question information as follows.

[0059] That is, in step S315, the additional question information presentation processing unit 13 first reads out additional question information corresponding to the type label assigned to the first answer information from the additional question information storage unit 33. Then, the kth additional question information among the read out additional question information is transmitted from the communication I / F unit 4 to the user terminal that transmitted the first answer information. In this way, the kth additional question information is presented to the user.

[0060] If the user returns answer information to the additional question information, the additional question information presentation processing unit 13 acquires the answer information as second answer information via the communication I / F unit 4 in step S316 and passes the acquired second answer information to the answer type determination processing unit 14.

[0061] Next, in step S317, the additional question information presentation processing unit 13 determines whether the number k of the additional question information has reached the total number O of additional question information. If the result of this determination is that k has not reached O, that is, if unsent additional selected information remains, in step S318, k = k + 1 is set, and the process returns to step S315 to select the next (k + 1)th additional question information and transmit it to the user terminal. Then, in step S316, the additional question information presentation processing unit 13 obtains second answer information returned from the user terminal in response to the above additional question information.

[0062] Thereafter, in the same manner, the additional question information presentation processing unit 13 repeatedly executes the process of transmitting additional question information one by one to the user terminal and receiving second answer information in response thereto.

[0063] (2-3) Determining the Type of Answer Information When the process of acquiring second answer information for the additional question information is completed, the control unit 1 of the answer information evaluation device CS proceeds to step S40. Then, under the control of the answer type determination processing unit 14, a process is executed to determine whether the first answer information is an answer corresponding to "Satisfiing" or something else, based on the consistency between the first answer information and the second answer information, as follows:

[0064] 8A, 8B, and 8C are flowcharts showing an example of the procedure and content of the determination process executed by the answer type determination processing unit 14. FIG.

[0065] (2-3-1) Generation of a set of answer patterns consistent with the first answer information In step S401, the answer type determination processing unit 14 first reads all answer patterns from the answer pattern storage unit 31. At the same time, in step S403, the answer type determination processing unit 14 acquires a second set of answer information for the additional question information from the additional question information presentation processing unit 13.

[0066] The answer type determination processing unit 14 then sets i=0 in step S404, and selects the i=0th second answer information from the second answer information group in step S405. Then, in step S406, the answer type determination processing unit 14 selects all answer patterns that do not contradict the i=0th second answer information from the plurality of answer patterns, and generates a set A0 of the selected answer patterns in step S407.

[0067] Subsequently, in step S408, the answer type determination processing unit 14 determines whether or not the above-mentioned i has reached the number N of answer information included in the second answer information group. If the result of this determination is that i has not reached N, the answer type determination processing unit 14 sets i=i+1 in step S409, selects the (i+1)th second answer information in step S405, and generates a set Ai+1 of answer patterns that do not contradict the (i+1)th second answer information in steps S406 to S408.

[0068] Thereafter, the answer type determination processing section 14 repeatedly executes the process of generating an answer pattern set Ai corresponding to each piece of second answer information N included in the second answer information group in the same manner.

[0069] (2-3-2) Determining Consistency The answer type determination processing unit 14 then executes a process of determining consistency between the first answer information and the second answer information as follows.

[0070] 9 illustrates an example of consistency determination processing, showing an example in which a user responds to question information including a question image, a question statement, and multiple options by providing first answer information ta, and then responds to additional question information 1-1 and 1-2 by providing second answer information 1 and 2. Note that the answer pattern is expressed as [i,ii] ([i,ii] ≠ [ii,i]), where i and ii represent the options arranged in the question.

[0071] That is, first, in step S410, the answer type determination processing unit 14 resets the values ​​of C1, C2, and C3, which are elements that determine the consistency score Ca. Here, C1 indicates the number of answer pattern sets that include the first answer information ta, C2 indicates the proportion of the number of elements in the intersection of each answer pattern set to the number of elements in the union, and C3 indicates whether the first answer information ta is included in the intersection of all answer pattern sets.

[0072] The answer type determination processing unit 14 initializes the number j of the answer pattern set Aj in step S411, and then determines in step S412 whether the answer pattern set Aj includes the first answer information ta. If the result of this determination is that the answer pattern set Aj includes the first answer information ta, the answer type determination processing unit 14 sets C1 = C1 + 1 in step S413. That is, it counts up C1. Note that if the answer pattern set Aj does not include the first answer information ta, it does not count up C1.

[0073] The answer type determination processing unit 14 then sets k=j+1 in step S414, and calculates C2 in step S415 using the following formula: C2=C2+(|Aj ∩Ak | / |Aj ∪Ak |).

[0074] Next, in step S416, the answer type determination processing unit 14 determines whether k has reached N. If the result of this determination is that k has not yet reached N, then in step S417, k = k + 1 is set, and C2 is calculated for the next answer pattern set Ak+1 in steps S414 to S417. Thereafter, the answer type determination processing unit 14 similarly repeats the process of calculating C2 for each answer pattern set Ak until k reaches N.

[0075] Next, in step S418, the answer type determination processing unit 14 determines whether or not the answer pattern set number j has reached the total number N of said sets. If the result of this determination is that the answer pattern set number j has not reached N, the answer type determination processing unit 14 sets j=j+1 in step S419 to select the next answer pattern set Aj+1, and returns to step S412 to execute the process of calculating C1 and C2 in steps S412 to S419. Thereafter, in the same manner, the answer type determination processing unit 14 repeatedly executes the process of calculating C1 and C2 for each answer pattern set Aj until j reaches N.

[0076] Next, in step S420, the answer type determination processor 14 determines whether (A0 ∩A1 ∩ ... ∩AN)εta. That is, it determines whether the first answer information ta is included in the intersection set of all answer pattern sets. If the result of this determination is that the first answer information ta is included in (A0 ∩A1 ∩ ... ∩AN), the answer type determination processor 14 sets C3 = 1 in step S421. Note that if the first answer information ta is not included in (A0 ∩A1 ∩ ... ∩AN), C3 is not counted up.

[0077] Next, in step S422, the answer type determination processing unit 14 calculates a consistency score Ca based on the previously calculated C1, C2, and C3. Here, examples of calculating the consistency score Ca include weighting and adding C1, C2, and C3, or calculating the average (weighted average) of the weighted and added values. Ca = αC1 + βC2 + γC3 Ca = (αC1 + βC2 + γC3) / 3

[0078] Finally, in step S423, the answer type determination processing unit 14 compares the consistency score Ca with a threshold value s to determine whether the consistency score Ca is higher than the threshold value s. If the result of this determination is that the consistency score Ca is higher than the threshold value s, the answer type determination processing unit 14 determines in step S424 that the first answer information ta does not correspond to "Satisficing." On the other hand, if the consistency score Ca is equal to or lower than the threshold value s, the answer type determination processing unit 14 determines in step S425 that the first answer information ta corresponds to "Satisficing."

[0079] Then, the answer type determination processing unit 14 stores the determination result in the determination information storage unit 35 in association with the first answer information ta.

[0080] The above consistency determination process will be explained using the example shown in Fig. 9. For example, if the weighting factors are set to α = 1, β = 0.2, and γ = 1, the consistency score Ca calculated for example answer 1 to additional question 1-1 is Ca = αC1 + βC2 + γC3 = 2 + 1 / 135 + 1 = 3.01.

[0081] Furthermore, the consistency score Ca calculated for example answer 2 to additional question 1-2 is Ca=αC1+βC2+γC3=1+5 / 29+0=1.03.

[0082] If the threshold value s is set to 2, the example answer 1 is determined to be an incorrect answer that does not correspond to "Satisficing" because the consistency score Ca is higher than the threshold value s. On the other hand, the example answer 2 is determined to be an incorrect answer that corresponds to "Satisficing" because the consistency score Ca is equal to or lower than the threshold value s.

[0083] (3) Evaluation of Answer Information In step S50, the control unit 1 of the answer information evaluation device CS determines whether or not processing for all of the first answer information has been completed. When the processing is completed, the control unit 1 proceeds to step S60, and evaluates the first answer information under the control of the answer evaluation processing unit 15.

[0084] That is, the answer evaluation processing unit 15 first excludes answer information that is determined to correspond to “Satisficing” from the first answer information group stored in the answer information storage unit 34 based on the determination information stored in the determination information storage unit 35. Then, the answer evaluation processing unit 15 executes a predetermined evaluation process on the first answer information after the above exclusion, and transmits the obtained evaluation information from the communication I / F unit 4 to, for example, the administrator terminal MT.

[0085] (Effects) As described above, in one embodiment, additional question information is created and stored in advance for each type of incorrect answer pattern based on a group of important points associated with the pattern. In this state, when first answer information is acquired from each user in response to question information, a score S indicating the likelihood of "Satisfiing" is calculated for each acquired first answer information, and whether the first answer information is an incorrect answer that is likely to be "Satisfiing" is determined based on the calculated score S. If the first answer information is determined to be likely to be "Satisfiing," the type of the first answer information is determined, and additional question information corresponding to that type is presented to the user, and second answer information for the additional question information is acquired. Next, a consistency score Ca indicating the consistency between the acquired second answer information and the first answer information is calculated, and based on the consistency score Ca, it is determined whether the first answer information is an incorrect answer that corresponds to "Satisfiing" or an incorrect answer that does not correspond to "Satisfiing." Then, based on the result of the determination, an incorrect answer corresponding to "Satisfied" is excluded from the first answer information group.

[0086] Therefore, according to one embodiment, it is possible to determine whether the first answer information to the first question is an incorrect answer corresponding to "Satisficing" or an incorrect answer not corresponding to "Satisficing," and it is possible to exclude only the incorrect answer corresponding to "Satisficing" from the first answer information group. As a result, it is possible to leave the incorrect answers not corresponding to "Satisficing" in the first answer information, thereby enabling a highly accurate evaluation of the first answer information group.

[0087] Furthermore, when the additional question information is presented, it is determined whether the first answer information is an incorrect answer that may be "Satisfied," and the additional question information is sent only to users who have answered an incorrect answer that may be "Satisfied." This makes it possible to improve the efficiency of the process from the presentation of the additional question information to the subsequent process of determining the answer type, and also reduces the burden on users who are not likely to be "Satisfied."

[0088] Furthermore, when calculating the consistency score Ca between the first answer information and the second answer information to the additional question information, the weighted sum or weighted average of the three elements C1, C2, and C3 is used, so that the consistency can be determined with high accuracy, and thereby it becomes possible to accurately determine whether the answer information is an incorrect answer corresponding to "Satisfiing."

[0089] [Other Embodiments] (1) In one embodiment, when determining whether or not the first response information is likely to be “Satisfied,” a weighted addition or weighted average of Nc, Nsc, Nsca, and Pc is used. However, it is also possible to determine whether or not the first response information is likely to be “Satisfied” using at least one of Nc, Nsc, Nsca, and Pc.

[0090] (2) In one embodiment, when calculating the consistency score Ca between the first answer information and the second answer information, a weighted sum or weighted average of three elements C1, C2, and C3 was used, but consistency may also be determined using at least one of C1, C2, and C3.

[0091] (3) In the embodiment, the response information evaluation device CS is provided in a server computer. However, it may be provided in a personal computer such as an administrator terminal used by a system administrator.

[0092] (4) Furthermore, the processing functions of the answer information evaluation device CS may be distributed among multiple computers, and these multiple computers may work together to execute the answer evaluation process according to the present invention. In particular, the additional question information may be created in advance by a server computer or a terminal computer other than the answer information evaluation device, and the created additional question information may be acquired by the answer information evaluation device.

[0093] (5) In addition, various modifications can be made to the processing functions, processing procedures, processing contents, question information configurations, and answer pattern configurations of the answer information evaluation device CS without departing from the spirit of the present invention.

[0094] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0095] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0096] CS...Answer information evaluation device UT1 to UTn...User terminal MT...Administrator terminal NW...Network 1...Control unit 2...Program storage unit 3...Data storage unit 4...Communication I / F unit 5...Bus 11...Additional question information creation processing unit 12...Answer information collection processing unit 13...Additional question information presentation processing unit 14...Answer type determination processing unit 15...Answer evaluation processing unit 31...Answer pattern storage unit 32...Type / important point storage unit 33...Additional question information storage unit 34...Answer information storage unit 35...Determination information storage unit

Claims

1. An answer information evaluation device that presents question information to a user and acquires and evaluates the user's first answer information to the question information, comprising: a first processing unit that classifies the first answer information into one of a plurality of predefined types; a second processing unit that selects the additional question information corresponding to the classified type from a plurality of additional question information created in advance for each type and presents the selected additional question information to the user; and a third processing unit that acquires the user's second answer information to the additional question information, evaluates the consistency between the acquired second answer information and the first answer information, and determines, based on the evaluation result of the consistency, whether the first answer information corresponds to a first incorrect answer that is presumed to be a result of the user having considered the question information but not understanding it, or a second incorrect answer that is presumed to be an answer given without considering the question information.

2. The answer information evaluation device of claim 1, wherein the first processing unit, when a plurality of options are provided in the first answer information, calculates a first score indicating the likelihood that the first answer information is the second incorrect answer based on at least one of the number of selections for the options and the regularity of the selections, and, when the first score is equal to or greater than a first threshold, performs processing to classify the first answer information into one of the types.

3. The answer information evaluation device described in claim 1, wherein the third processing unit calculates a second score indicating the consistency based on at least one of the number of times the answer result of the first answer information is included in the set of answer results included in the second answer information and, when there are multiple sets, the proportion of the intersection of these sets in the union, and evaluates the consistency by comparing the calculated second score with a second threshold value.

4. A method for evaluating answer information, which presents question information to a user and acquires and evaluates the user's first answer information to the question information, using an information processing device, comprising the steps of: classifying the first answer information into one of a plurality of predefined types; selecting the additional question information corresponding to the classified type from a plurality of additional question information created in advance for each type, and presenting the selected additional question information to the user; acquiring the user's second answer information to the additional question information, evaluating the consistency between the acquired second answer information and the first answer information, and determining, based on the evaluation result of the consistency, whether the first answer information corresponds to a first incorrect answer, which is presumed to be an answer given because the user considered the question information but was unable to understand it, or a second incorrect answer, which is presumed to be an answer given without considering the question information.

Images