Device for grouping graded answer sheets in written exams, method for grouping graded answer sheets in written exams, and program

Abstract

To provide a grouping device for graded answer sheets in a written examination, which contributes to suppressing variations in the grade results. [Solution] The grouping device for graded answers in a written exam includes a reading unit that reads graded answer data in a written exam, a morphological analysis unit that performs morphological analysis on the answer data, a vectorization unit that vectorizes the results of the morphological analysis into answer sentence vectors, a grouping implementation unit that groups answer data that matches the rule definition and groups answer data that does not conform to the rule definition based on the answer sentence vectors to generate groups, a group feature analysis unit that analyzes statistical information on the features of the generated groups, a scoring result analysis unit that analyzes statistical information on the scoring results of each group, and an output unit that outputs statistical information on the group features and statistical information on the scoring results.

Description

[Technical Field]

[0001] The present invention relates to a grouping device for graded answer sheets in a written test, a grouping method for graded answer sheets in a written test, and a program. [Background technology]

[0002] The following documents can be cited regarding information processing devices for grading written exam answer sheets.

[0003] Patent Document 1 relates to detecting possible incorrectly marked answer sheets in exams and the like. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-123121 Summary of the Invention [Problem to be solved by the invention]

[0005] The following analysis is given by the inventor.

[0006] In the field of education, various tests are frequently conducted, and answers to test questions are graded to evaluate the test taker's level of understanding. Test questions include those that require test takers to select an answer using a multiple-choice sheet, but descriptive test questions, in which test takers are presented with problem-solving questions and are asked to write a written answer, are also widely used.

[0007] The answers to these essay-style questions are usually graded by a human being, such as a teacher. In exams with a large number of examinees, such as entrance exams and qualification exams, multiple graders grade the answers, which can lead to variations in the grading results depending on the grader, even when the answers are similar. Furthermore, there is a concern that variations in the grading results may occur when the same grader grades the answers of many examinees. If variations in the grading results occur depending on the grader, it becomes difficult to maintain fairness in the evaluation.

[0008] For this reason, when marking, in order to reduce variation in the marking results, the marking worker must take the time to align their thinking with that of other marking workers and review the marks that have already been made, which places a heavy burden on the marking worker.

[0009] Patent Document 1 is an example of a conventional technology for detecting potentially incorrectly scored answer sheets in exams and the like. However, it is necessary to create a classification model trained by machine learning or deep learning using answer sheet data. This requires the preparation of answer sheet data for model creation, adjustment of hyperparameters for machine learning or deep learning, and execution of learning processes. This poses a problem in that it takes time before the system can actually be used.

[0010] The present invention aims to provide a grouping device for graded answer sheets in a written test, a grouping method for graded answer sheets in a written test, and a program that contribute to reducing variation in the grading results. [Means for solving the problem]

[0011] According to a first aspect of the present invention, there is provided a reading unit that reads graded answer sheet data for a written test; a morphological analysis unit that performs morphological analysis on the answer sheet data; a vectorization unit that vectorizes the result of the morphological analysis into an answer sentence vector; a grouping implementation unit that groups the answer data that matches a rule definition and groups the answer data that does not conform to the rule definition based on the answer sentence vector, thereby generating groups; a group feature analysis unit that analyzes the generated statistical information of the group features; a scoring result analysis unit that analyzes statistical information of the scoring results of each of the groups; an output unit that outputs statistical information on the characteristics of the group and statistical information on the scoring results; A grouping device for graded answers to written exams can be provided.

[0012] According to a second aspect of the present invention, a computer Read the graded answer data for the written test, Morphological analysis of the answer data is performed; The result of the morphological analysis is vectorized into an answer sentence vector; The answer data that conforms to the rule definition is grouped, and the answer data that does not conform to the rule definition is grouped based on the answer sentence vector to generate groups; Analyzing the generated statistical information of the characteristics of the group; Analyzing the statistical information of the scoring results for each of said groups; outputting statistical information of the characteristics of the group and statistical information of the scoring results; A method for grouping graded answers for a written test can be provided. This method is linked to a specific machine, a computer, that executes the method.

[0013] According to a third aspect of the present invention, a computer is provided with: A process for reading the graded answer data for the written test; A process of performing morphological analysis on the answer sheet data; A process of vectorizing the results of the morphological analysis into an answer sentence vector; A process of grouping the answer data that matches the rule definition and grouping the answer data that does not conform to the rule definition based on the answer sentence vector to generate groups; analyzing the generated statistical information of the characteristics of the group; analyzing statistical information of the scoring results of each of the groups; A program can be provided that executes a process of outputting statistical information on the characteristics of the group and statistical information on the scoring results.

[0014] These programs can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present invention can also be embodied as a computer program product. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a grouping device for graded answer sheets in a written test, a grouping method for graded answer sheets in a written test, and a program that contribute to reducing variation in the grading results. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a grouping device for grouping graded answer sheets in a written examination according to the present disclosure and an example of the overall system configuration. [Figure 2] 1 is a block diagram showing an example of the configuration of a grouping implementation unit of a grouping device according to the present disclosure. [Figure 3] FIG. 10 is a diagram showing an example of the configuration of graded answer sheet data for a written test according to the present disclosure. [Figure 4] FIG. 10 is a diagram illustrating an example of a rule definition configuration according to the present disclosure. [Figure 5] FIG. 10 is a flowchart illustrating an example of a processing operation of the grouping device according to the present disclosure. [Figure 6]FIG. 10 is a flow diagram illustrating an example of a rule-based grouping process of the grouping device according to the present disclosure. [Figure 7] FIG. 10 is a flow diagram illustrating an example of a group analysis process of the grouping device according to the present disclosure. [Figure 8] FIG. 10 is a flowchart illustrating an example of a group feature analysis process performed by the grouping device according to the present disclosure. [Figure 9] FIG. 10 is a flow diagram illustrating an example of a scoring result analysis process of the grouping device according to the present disclosure. [Figure 10] FIG. 10 is a flow diagram illustrating an example of a regrouping process of the grouping device according to the present disclosure. [Figure 11] FIG. 10 is a flow chart illustrating an example of a regrouping analysis process of the grouping device according to the present disclosure. [Figure 12] 10A and 10B are diagrams illustrating an example of calculation of cos similarity and contribution degree in the grouping device according to the present disclosure. [Figure 13] 10 is a diagram illustrating an example of the relationship between answer sheet data and calculation of cosine similarity in a grouping device according to the present disclosure. FIG. [Figure 14] FIG. 1 is a diagram illustrating an example of an overview of a transitional relationship-considered group in a grouping device according to the present disclosure. [Figure 15] FIG. 10 is a diagram illustrating an example of an image of group division by a grouping device according to the present disclosure. [Figure 16] FIG. 10 is a diagram showing an example of a case where the space of answer data vectors is reduced to two dimensions. [Figure 17] FIG. 10 is a diagram showing an example of an analysis of the scoring results. [Figure 18] FIG. 10 is a diagram illustrating an example of an operation for calculating cosine similarity when incremental data is regrouped. [Figure 19] FIG. 10 is a diagram illustrating an example of an operation when a transitional relationship group is regrouped. [Figure 20] FIG. 10 is a diagram illustrating an example of an operation when a transitional relationship group is regrouped. [Figure 21] FIG. 10 is a diagram illustrating an example of an operation when a transitional relationship group is regrouped. [Figure 22] FIG. 10 is a diagram illustrating an example of an operation when a transitional relationship group is regrouped. [Figure 23] FIG. 10 is a diagram illustrating an example of an operation when a transitional relationship group is regrouped. [Figure 24] FIG. 10 is a diagram illustrating an example of an operation when a transitional relationship group is regrouped. [Figure 25] FIG. 1 is a diagram illustrating a configuration of a computer that constitutes a grouping device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] In this disclosure, the drawings may relate to one or more embodiments. In addition, each embodiment described below can be combined with other embodiments as appropriate, and the present invention is not limited to each embodiment.

[0018] First, an overview of one embodiment will be described with reference to the drawings. Note that the reference numerals in the drawings attached to this overview are attached to each element for convenience as an example to facilitate understanding, and are not intended to limit the present invention to the illustrated form. Furthermore, connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of the main signal (data) and do not exclude bidirectionality.

[0019] FIG. 1 is a block diagram showing an example of the configuration of a grouping device for grouping graded answer sheets in a written test according to the present disclosure and an example of the overall system configuration.

[0020] The overall system includes a graded answer sheet data storage unit 100, a grouping device 200, and a result storage unit 300.

[0021] The graded answer data storage unit 100 stores graded answer data 110 from a CBT (Computer Based Testing) test and answer data obtained by scanning a graded answer sheet 130 with a scanner 120. The answer data includes answers to essay questions, the graded results (scores), and the examinee's ID.

[0022] The grouping device 200 for graded answer sheets in a written test includes a reading unit 210 that reads answer sheet data, a morphological analysis unit 211, a vectorization unit 212, a grouping implementation unit 213, a group feature analysis unit 214, a graded result analysis unit 216, and an output unit 217.

[0023] The reading unit 210 reads the graded answer sheet data for the written test from the graded answer sheet data storage unit 100.

[0024] The morphological analysis unit 211 receives the answer sheet data, performs morphological analysis, and stores the morphological analysis results in the result storage unit 300 .

[0025] The vectorization unit 212 vectorizes the results of the morphological analysis into answer sentence vectors, and stores the vectorized results in the result storage unit 300.

[0026] The grouping implementation unit 213 reads the rule definition 140, groups the answer data that matches the rule definition 140, generates groups (i.e., generates intermediate grouping results (1) in accordance with the rule definition 140), and stores them in the result memory unit 300.

[0027] The rule definition 140 is, for example, a list of information corresponding to the grading criteria, such as NG (No Good) words that are inappropriate for the content of the answer and the amount of text.

[0028] The grouping implementation unit 213 groups the remaining answer sheet data that does not conform to the rule definition based on the answer sentence vector, generates groups, and saves the grouping interim results (1) in the result storage unit 300.

[0029] The group feature analysis unit 214 analyzes statistical information on the features of the groups generated by the grouping. The group feature analysis unit 214 analyzes words that are characteristic of each group and words that are common to multiple groups, and analyzes statistical information on the features of the grouped groups. The group feature analysis unit 214 stores the analysis results in the result storage unit 300 as grouping intermediate results (2).

[0030] The grading result analysis unit 216 analyzes statistical information on the grading results of each group. For example, it performs statistical analysis of the mean, median, standard deviation, maximum value, minimum value, interquartile range, etc. from the grading results (scores) of the answer sheet data of the grouping results to detect outliers.

[0031] The output unit 217 outputs statistical information on the characteristics of the group and statistical information on the scoring results. As an example, the output unit 217 generates and outputs output information to be output on the screen of the user terminal or the like.

[0032] The grouping device 200 may further include a group combination adjustment unit 215. When an increase in answer sheet data occurs, the group combination adjustment unit 215 adjusts the combination of the increased answer sheet data for each group of processed answer sheet data. As an example, the grouping results are stored in the result storage unit 300.

[0033] The data grouping of the present invention allows answer sentences to be grouped according to the grading results from a large amount of answer data without relying on human labor or prior learning, thereby significantly reducing the time required for data preparation for model creation and model learning.

[0034] As a result, the variation in the grading results can be reduced and the burden on the grader can be reduced. Furthermore, the present invention can also be used to grade answers given by test takers in practice exercises, or to help test takers improve their scores if their scores are low.

[0035] As described above, according to one embodiment of the present invention, it is possible to provide a grouping device for graded answer sheets in a written test, a grouping method for graded answer sheets in a written test, and a program that contribute to reducing variation in the grading results.

[0036] [First embodiment] Next, the first embodiment will be described in detail with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of a grouping device 200 for grouping graded answer sheets in a written test according to the present disclosure and an example of the overall system configuration. Fig. 2 is a block diagram showing an example of the configuration of a grouping implementation unit 213 of the grouping device 200 according to the present disclosure.

[0037] Fig. 3 is a diagram showing an example of the configuration of the graded answer data for a written exam according to the present disclosure. Referring to Fig. 3, the graded answer data for a written exam read by the answer data reading unit 210 from the graded answer data storage unit 100 includes answer data having an examinee ID, an answer sentence, and a graded result (score). Fig. 4 is a diagram showing an example of the configuration of the rule definition according to the present disclosure.

[0038] Next, an example of the processing operation of the grouping device according to the present disclosure will be described with reference to Fig. 5 to Fig. 9. Fig. 5 is a flow diagram showing an example of the processing operation of the grouping device according to the present disclosure. Referring to Fig. 5, the processing starts at step S100. The answer sheet data reading unit 210 of the grouping device 200 reads data from the graded answer sheet data storage unit (also referred to as DB) 100 (step S101). The format of the read data includes the examinee ID, answer sentence, and graded result (score), as shown in Fig. 3 as an example.

[0039] The morphological analysis unit 211 performs morphological analysis on the graded answer sheet data (step S102). If any of the resulting morphemes contains parts of speech (such as particles or auxiliary verbs) that are determined to be unnecessary, the parts of speech determined to be unnecessary may be eliminated. The results of the morphological analysis are stored in the result storage unit 300.

[0040] The vectorization unit 212 vectorizes the morpheme-processed answer sentence (step S103). At this time, weights may be assigned to specific words that must be included in the answer, as defined by the scoring criteria, etc. A small value may be assigned to a word that is determined to be unimportant for grouping, and a large value may be assigned to a word that is considered important. The weighted vector may be normalized. Any method may be used for vectorization, such as TF (Term Frequency), TF-IDF (Term Frequency-Inverse Document Frequency), BM25 (Best Matching 25), Word2Vec, or Doc2Vec. In the first embodiment, TF-IDF is used as an example. The vectorized answer sentence is referred to as an answer sentence vector. The vectorization results are stored in the result storage unit 300.

[0041] Next, the grouping unit 213 performs group analysis to perform grouping (step S104). Referring to Fig. 2, the grouping unit 213 includes a rule definition reading unit 220, a rule-based group division unit 221, a similarity calculation unit 222, a high-similarity answer data extraction unit 223, a transitional relationship-considered group division unit 224, a community detection unit 225, and a group representative answer data generation unit 226.

[0042] 6 is a flow diagram showing an example of a rule-based grouping process of the grouping device according to the present disclosure. In the grouping performed in step S104 of FIG. 5, as an example, the rule-based grouping process of FIG. 6 is first performed.

[0043] Referring to FIG. 6, the rule-based grouping process begins in step S110. The rule definition reading unit 220 reads the rule definition from a rule definition file that defines the grading criteria, etc. (step S111). FIG. 4 shows an example of a rule definition. A rule definition is data consisting of a rule ID in column 500 and a rule content in column 501. Multiple rules may be written across multiple lines, as in the case of rule ID=3 shown in FIG. 4. In this case, as an example, the rule content of each line is used as an AND condition to determine whether the rule definition of rule ID=3 is satisfied. In other words, if the rule definitions of all lines are satisfied, it is determined that the rule definition matches. A determination is made as to whether the answer data matches the defined rule. If the answer data matches the rule definition (Yes in step S112), it is assigned to a group by rule ID (step S113). After the answer data is assigned to groups, groups by rule ID are generated (step S114). Even if there is no answer data assigned to a rule ID, groups by rule ID are generated. The process ends at step 115.

[0044] In group analysis (step S104), answer data that does not match the rule definition (No in step S112) due to group assignment according to the rule definition shown in steps S110 to S115 of Figure 6 will, as an example, undergo the following processing in group analysis (step S104).

[0045] [What to do if no rule definition is matched] 7 is a flow diagram showing an example of processing performed in the group analysis (step S104) shown in FIG. 5 of the grouping device according to the present disclosure. The processing starts in step S120. The similarity calculation unit 222 calculates the similarity between all answer sheet data (step S121). That is, the similarity between all answer sentence vectors is calculated. At this time, any method such as cosine (cosine) similarity or Euclidean norm may be used to calculate the similarity, but in the first embodiment, cosine similarity will be used as an example.

[0046] FIG. 12 illustrates an example of calculation of cosine similarity and contribution by the grouping device according to the present disclosure. As an example, the cosine similarity is calculated between the TF-IDF vector of answer data 1 and the TF-IDF vector of answer data 2 in column 600. Columns 601 to 603 represent elements based on morphological analysis. For example, the product of the values ​​of each element of the TF-IDF vector of answer data 1 and the TF-IDF vector of answer data 2 is calculated as the contribution, and the cosine similarity is calculated as the sum of these contributions. Referring to FIG. 12, for example, if the answer sentence is "I don't know how to write the answer," the morphological analysis yields "answer," "description," and "method." The TF-IDF vector of answer data 1 and the TF-IDF vector of answer data 2 can be used as the centroid vector of group 1 and the centroid vector of group 2, respectively. FIG. 13 illustrates an example of the relationship between answer data and calculation of cosine similarity by the grouping device according to the present disclosure. Since there is no need to calculate the overlap between each answer data 1 to N and the similarity between each answer data and itself, the cosine similarity of the combinations in the shaded areas is calculated. Note that if the similarity is above a threshold, it is determined that there is a similar relationship.

[0047] 2 extracts answer data whose similarity between each answer data is equal to or greater than a threshold value (step S122). In the first embodiment, as an example, the answer data whose similarity between answer sentence vectors is equal to or greater than 0.4 is selected.

[0048] The transitional relationship consideration group division unit 224 in Fig. 2 groups the network of highly similar answer data extracted in step S104 in Fig. 5 as a transitional relationship consideration group (step S123). Even if there is a similarity relationship between answer data A and B, and a similarity relationship between answer data B and C, but there is no similarity relationship between answer data A and C, it is considered that there is a relationship between A and C, and they are classified as a transitional relationship consideration group. In other words, the network of similar relationships is considered as one group, a transitional relationship consideration group.

[0049] FIG. 14 is a diagram showing an example of an overview of a transitional relationship consideration group of a grouping device according to the present disclosure. The numerical values ​​shown on the lines connecting each answer data indicate the degree of similarity between the answer data. When answer data with a similarity equal to or greater than a threshold is extracted, a network like that shown in FIG. 14 is created. Here, there is no direct similarity relationship between answer data 2 and answer data 5, but they belong to the same network via answer data 1. A network with such an indirect similarity relationship is called a "transitional relationship consideration group."

[0050] The community detection unit 225 in FIG. 2 divides the transitive relationship consideration group using community detection (step S124). Community detection can be based on edge betweenness centrality or random walk, but the first embodiment will explain community detection based on a greedy algorithm as an example. If one transitive relationship consideration group becomes too large, it may have different meanings at both ends of the network. This is divided into communities using community detection, a technology that divides groups into communities based on the shape of a graph. The divided groups are considered to be final groups. FIG. 15 is a diagram showing an example of group division by the grouping device according to the present disclosure. Circles indicate answer data vectors, and lines indicate connections. Connections will be described later.

[0051] The group representative answer data generation unit 226 in FIG. 2 generates data that is representative of the answer data of the group and sets it as one answer data (step S125). The generation method may be to extract one representative case from the group, or to use a sentence summarized using LLM (Large Language Models). Alternatively, the centroid vector of the group may be calculated and the answer data closest to the centroid vector may be selected. The process ends at step S126.

[0052] Returning to Fig. 5, the group feature analysis unit 214 shown in Fig. 1 performs the group feature analysis in step S105 of Fig. 5, analyzing words that are characteristic of a specific group and words that are common to many groups. Fig. 8 is a flow diagram showing an example of the group feature analysis processing of the grouping device according to the present disclosure, and describes the details of the group feature analysis processing executed in step S105 of Fig. 5. The group feature analysis processing starts in step S130.

[0053] Referring to FIG. 8, a centroid vector for each group is calculated (step S131). The distance between groups is calculated from the centroid vector of each group (step S132). The contribution degree, which will be described later, is calculated (step S133). Based on the contribution degree, statistical information such as words that commonly appear between groups, words that differ greatly between groups, words that are unique to a group, and words that are common to all groups is output as the analysis result (step S134). FIG. 16 is a diagram showing an example of a case where the space of answer data vectors is reduced to two dimensions. FIG. 16 shows an example of a diagram displayed as statistical information.

[0054] Each point in Figure 16 represents an answer data vector, and points connected by lines indicate that they belong to the same group. The centroid vector of each group is represented by a square. The degree of contribution between centroid vectors is checked, and those with a high degree of contribution between groups are displayed as common words (common words). Words with a low degree of contribution to any group are displayed as group-specific words. In other words, dimensionality reduction is performed to display common words between groups and group-specific words.

[0055] The contribution is calculated by vectorizing the center of gravity of the group and calculating the contribution between the vectors (between center of gravity vectors). As an example, if the TF-IDF vector of each answer data shown in Figure 12 is used as the group center of gravity vector, each term in the inner product of the vectors when calculating the cosine similarity is the contribution of each word.

[0056] Words with a high contribution between groups are words that appear in common between the groups, and words with a contribution of 0 are words that appear only in one of the groups. Words that appear in only one of the groups are group-specific words. Depending on the analysis results, the vector weighting for each word as described above can be applied and grouping can be performed again. The same analysis can also be performed between communities.

[0057] Returning to Fig. 5, in the group feature analysis in step S105 in Fig. 5, another method of inter-group analysis performed by the group feature analysis unit 214 in Fig. 1 may be a method using, for example, an IDF (Inverse Document Frequency) vector. When calculating the group centroid vector, the IDF vector may be calculated. It can be analyzed that words with large IDF values ​​are words that exist only in a smaller number of groups, and words with small IDF values ​​are words that exist in many groups.

[0058] Referring to Fig. 5, the scoring result analysis unit 216 shown in Fig. 1 performs the scoring result analysis in step S106 in Fig. 5. Fig. 9 is a flow diagram showing an example of the scoring result analysis processing of the grouping device according to the present disclosure, which corresponds to the scoring result analysis in step S106 in Fig. 5. The scoring result analysis processing starts in step S140.

[0059] The grading result analysis unit 216 reads the grouping results from the result storage unit 300 (step S141). From the grading results (scores) of all answer sheet data within a group, statistical information such as the mean, median, standard deviation, maximum value, minimum value, and interquartile range is calculated (step S142). Next, outliers are detected from the calculated statistical information (step S143). In the first embodiment, a method using the interquartile range will be described as an example of a method for calculating outliers. Let Q1 be the first quartile (the boundary of the bottom 25% of data points), and Q3 be the third quartile (the boundary of the top 25% of data points). In this case, the interquartile range can be calculated as Q3 - Q1.

[0060] Using these, the boundary of the outlier is calculated using the following formula to detect whether an outlier exists. Lower outlier = Q1-1.5 x interquartile range Upper outlier = Q3 + 1.5 x interquartile range FIG. 17 is a diagram showing an example of the analysis of the scoring results, showing an example of detecting outliers of a specific group, that is, outliers above the upper limit and outliers below the lower limit.

[0061] Data points that exceed the above range are detected as outliers. If an outlier is detected, an outlier flag is set for the examinee ID of the target response data in the grouping results in the result storage unit 300, and the result is stored in the result storage unit 300, for example.

[0062] The output section 217 in FIG. 1 references the grouping results stored in the result storage section 300, and outputs the groups for which outlier flags have been set and the examinee ID information.

[0063] The data grouping of the first embodiment of the present invention allows answer sentences to be grouped according to the grading results from a large amount of answer data without relying on human intervention or prior learning. This significantly reduces the time required for preparing data for model creation and model learning. Furthermore, community detection allows for further division within groups to create accurate groups. Unlike existing clustering methods, grouping can be performed without specifying the number of clusters in advance.

[0064] According to the first embodiment of the present invention, the characteristics of the group can be visualized by calculating the center of gravity of the group and outputting statistical information of the center of gravity vector. From the characteristics, parameters can be set again and grouping can be performed again, resulting in more accurate grouping.

[0065] As a result, the variation in the grading results can be reduced and the burden on the grader can be reduced. Furthermore, the present invention can also be used to grade answers given by test takers in practice exercises, or to help test takers improve their scores if they receive low scores.

[0066] Therefore, according to the first embodiment of the present invention, it is possible to provide a grouping device for graded answer sheets in a written test, a grouping method for graded answer sheets in a written test, and a program that contribute to reducing variation in the grading results.

[0067] [Second embodiment] Next, a second embodiment will be described in detail with reference to the drawings. The second embodiment relates to incremental data screening. FIG. 10 is a flow diagram showing an example of a regrouping process of the grouping device according to the present disclosure. FIG. 11 is a flow diagram showing an example of a regrouping analysis process of the grouping device according to the present disclosure. Note that for an example of the configuration of the grouping device and an example of the overall system configuration, refer to FIG. 12, and for an example of the configuration of the grouping implementation unit of the grouping device, refer to FIG. 2.

[0068] After grouping is performed once using answer data, it may be necessary to respond to answers from a similar test (for example, to verify the validity of the results of solving past questions and scoring), or a test may be administered nationwide with scoring performed in each region, resulting in delays in the collection of the scoring results. In other words, it ensures the overall validity of the scoring content even when the timing of the collection and collection of answer data is delayed. Incremental data screening involves adding incremental data to existing answer data and regrouping appropriately.

[0069] 10, the re-grouping process begins in step S150. Next, the answer sheet data reading unit 210 in FIG. 1 reads incremental data from the graded answer sheet data storage unit (also referred to as DB) 100 (step S151). The incremental data refers to the graded answer sheet data that has been accumulated since the previous grouping was performed.

[0070] The morphological analysis unit 211 shown in Fig. 1 performs morphological analysis on the incremental answer sentence (step S152). Of the resulting morphemes, parts of speech (such as particles and auxiliary verbs) that are determined to be unnecessary may be eliminated, but they must be the same parts of speech as those eliminated during the morphological analysis of the existing data. The results of the morphological analysis are stored in the result storage unit 300.

[0071] The vectorization unit 212 vectorizes the morpheme-processed answer sentences (step S153). At this time, specific words may be weighted. If a word is determined to be unimportant for grouping, a small value is assigned, and if it is considered important, a large value is assigned. The weighted vector may be normalized. Any method such as TF, TF-IDF, BM25, Word2Vec, or Doc2Vec may be used for vectorization, but it must be the same method as the original answer sentence vector. In the following, TF-IDF is used as an example. Each vectorized answer sentence is referred to as an incremental answer sentence vector. The vectorization results are stored in the result storage unit 300.

[0072] Next, the grouping execution unit 213 performs a regrouping analysis (step S154).

[0073] Fig. 11 is a flow diagram showing an example of the regrouping analysis process of the grouping device according to the present disclosure. The regrouping analysis process will be described with reference to Fig. 11. The regrouping analysis process starts in step S160.

[0074] The grouping implementation unit 213 in FIG. 1 reads the answer sentence vector of the existing answer data from the result storage unit (also referred to as DB) 300 (step S161).

[0075] FIG. 18 is a diagram showing an example of the operation of calculating cosine similarity when incremental data is regrouped. In FIG. 18, it is assumed that a similarity check has already been performed between existing data indicated by reference numeral 1201. Next, the similarity calculation unit 222 in FIG. 2 calculates the similarity between all incremental data, as indicated by the shaded area indicated by reference numeral 1202 in FIG. 18 (step S162). That is, it calculates the similarity between all incremental answer sentence vectors. At this time, any similarity may be used, such as cosine similarity or Euclidean norm, but the same method as used for calculating the similarity of existing data described above must be used.

[0076] Next, as shown by reference numeral 1203, the similarity calculation unit 222 calculates the similarity between all incremental data and existing data (step S163). At this time, any similarity such as cosine similarity or Euclidean norm can be used, but it is necessary to use the same method as the similarity calculation of the existing data described above.

[0077] The high similarity answer data extraction unit 223 in Figure 2 extracts answer data whose similarity between each answer data is equal to or greater than a threshold value (step S164). The threshold value at this time must be the same as the threshold value for the existing data mentioned above. As an example, answer sentence vectors whose similarity is 0.4 or greater are targeted.

[0078] The transitional relationship consideration group division unit 224 groups the networks of the highly similar answer sheet data extracted in step S164 into transitional relationship consideration groups (step S165).

[0079] At this time, the following six types of similarity relationships can be considered between the incremental data and the existing data. (1) Incremental data that has not been grouped is combined with one existing group. (2) Incremental data that has not been grouped is combined with multiple existing groups. (3) None of the incremental data that has not been grouped is combined with any existing group. (4) The incremental data creating the group is combined with one of the existing groups. (5) The incremental data creating the group is combined with multiple existing groups. (6) None of the incremental data creating the group is combined with an existing group.

[0080] Here, "connected" refers to a case where the similarity between each answer data is equal to or greater than a threshold value, and "not connected" refers to a case where the similarity between each answer data is less than a threshold value, but this is not limited to this.

[0081] The group connection adjustment unit 215 in FIG. 1 adjusts connections for each of the six patterns (step S166). In the case of the similarity relationship (1), the incremental data connected to the existing group is added. In the case of the similarity relationships (3) and (6), since the data is not connected to the existing group, there is no need to consider regrouping and the data can be left as is. In the case of the similarity relationships (2), (4), and (5), it can be rephrased that one incremental data is connected to multiple groups. In this case, the incremental data is added to the group with the greater number of connections, and the connections with other groups are deleted. If there are groups with the same number of connections, the incremental data is added to the group with the connection with the higher similarity, and the connections with other groups are deleted. The group after this operation is considered to be the group to which the incremental data has been added. FIGS. 19 to 22 show the operation when incremental data is connected to an existing group.

[0082] 19 to 24 are diagrams showing an example of the operation when regrouping transitional relationship groups. Referring to Fig. 19, data 1321 of incremental data group 1320 is combined with data 1301 of existing group A 1300, and data 1322 of incremental data group 1320 is combined with data 1301 of existing group A 1300 and data 1311, data 1312, and data 1313 of existing group B 1310.

[0083] 20, data 1321 of incremental data group 1320 has one connection with existing group A 1300 and three connections with incremental data group 1320. Therefore, data 1321 is attributed to incremental data group 1320.

[0084] 21, data 1322 of incremental data group 1320 has one connection with existing group A 1300, three connections with existing group B 1310, and two connections with incremental data group 1320. Therefore, data 1322 is attributed to existing group B 1310.

[0085] Referring to FIG. 22, data 1321 of incremental data group 1320 belongs to incremental data group 1320, and data 1322 belongs to existing group B 1310.

[0086] Referring to Figure 23, if incremental data 1400 is linked to two different groups, for example, existing group 1 and existing group 2, then of the links 1401 and 1402, it is added to existing group 1, which has the link 1401 with the higher similarity.

[0087] Referring to FIG. 24, when the incremental data 1400 that has not been grouped is combined with only one existing group 1, for example, it is added to the existing group 1.

[0088] 2 divides the transitional relationship consideration group created in step S166 using community detection (step S167). Community detection can be based on edge betweenness centrality or random walk, but as an example, community detection based on a greedy algorithm will be described.

[0089] The group representative answer data generation unit 226 in Figure 2 generates data that is representative of the answer data of the group and sets it as one answer data (step S168). The generation method may be to extract one representative case from the group, or a sentence summarized by LLM. The centroid vector of the group may be calculated, and the answer data closest to the centroid vector may be selected.

[0090] The regrouping analysis process ends in step S169, and the process returns to step S155 in FIG.

[0091] Step S155 in FIG. 10 is a group feature analysis process, which corresponds to step S105 in FIG. 5 in which the group feature analysis unit 214 analyzes words that are characteristic of a specific group or words that are common to many groups. However, the detailed process content is the same as steps S121 to S124 in FIG. 7 described in the first embodiment, and therefore will not be described here in detail.

[0092] Although the explanation has focused on the processing flow of components such as the graded answer sheet data storage unit 100, the grouping device (grouping function unit) 200, and the result storage unit 300, the present invention may also be configured using a scoring support device, a scoring support system, a virtual server on the cloud, etc.

[0093] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of the present invention. For example, the network configurations, element configurations, and message expression formats shown in the drawings are examples to aid in understanding the present invention, and the present invention is not limited to the configurations shown in these drawings. Furthermore, "A and / or B" is used to mean at least either A or B.

[0094] Furthermore, the procedures shown in the first and second embodiments can be realized by a program that causes a computer (9000 in FIG. 25) that functions as a grouping device for graded answer sheets in a written exam according to the present invention to function as a grouping device for graded answer sheets in a written exam. Such a computer is exemplified by a configuration including a CPU (Central Processing Unit) 9010, a communication interface 9020, a memory 9030, and an auxiliary storage device 9040 in FIG. 25. That is, the CPU 9010 in FIG. 25 executes a control program for the grouping device for graded answer sheets in a written exam, and performs an update process for each calculation parameter stored in the auxiliary storage device 9040, etc.

[0095] The memory 9030 is a RAM (Random Access Memory), a ROM (Read Only Memory), or the like.

[0096] In other words, each part (processing means, function) of the grouping device for graded written test answer sheets shown in the first and second embodiments can be realized by a computer program that causes the processor of the computer to execute each of the above-mentioned processes using its hardware.

[0097] Finally, preferred embodiments of the present invention will be summarized. [First form] The grouping device for graded answer sheets of an essay-style test may include a reading unit that reads data of graded answer sheets of an essay-style test. The grouping device may include a morphological analysis unit that performs morphological analysis on the answer sheet data. The grouping device may include a vectorization unit that vectorizes the result of the morphological analysis into an answer sentence vector. The grouping device may include a grouping implementation unit that groups the answer data that matches a rule definition and groups the answer data that does not conform to the rule definition based on the answer sentence vector, thereby generating groups. The grouping device may include a group characteristic analysis unit that analyzes statistical information on the characteristics of the generated groups. The grouping device may include a score analysis unit that analyzes statistical information of the score results of each of the groups. The grouping device may include an output unit that outputs statistical information on the characteristics of the group and statistical information on the scoring results. [Second form] In the grouping device for graded answers in a written test described in the first embodiment, it is preferable that the grouping implementation unit calculates the similarity between all answer data that do not match the rule definition, and groups the answer data whose similarity is equal to or greater than a predetermined threshold value to generate the groups. [Third Form] In the grouping device for graded answer sheets in a written test according to the first aspect, it is preferable that the grouping implementation unit groups transitional relationship consideration groups into one of the groups. [Fourth Form] In the grouping device for graded answer sheets in a written test according to the second aspect, it is preferable that the grouping implementation unit calculates the similarity based on the answer sentence vector. [Fifth Form] In the grouping device for graded answer sheets in a written test according to the fourth aspect, it is preferable that the similarity is a cosine similarity or a Euclidean norm. [Sixth Form] In the grouping device for graded answer sheets of a written test according to the first aspect, it is preferable that the group characteristic analysis unit analyzes statistical information on words characteristic of each group and words common to a plurality of the groups. [7th form] The grouping device for graded answer sheets in a written test described in the first embodiment may further include a group combination adjustment unit that, when there is an increase in the answer sheet data, adjusts the combination of the increased answer sheet data for each of the groups of processed answer sheet data. [8th form] In the grouping device for grouping graded answer sheets for a written test according to the seventh aspect, the group combination adjustment unit If the non-grouped incremental data is combined with only one of the existing groups, add the incremental data to the existing group; When one incremental data item is combined with a plurality of the existing groups, the incremental data item is added to the existing group with the largest number of combinations. Preferably, the combination of the incremental answer data is adjusted. [9th Form] The method of grouping the graded answer sheets of the written test may involve the computer reading in the graded answer sheet data of the written test. The computer may perform a morphological analysis of the answer data. The computer may vectorize the result of the morphological analysis into an answer sentence vector. The computer may group the answer data that matches a rule definition, and group the answer data that does not conform to the rule definition based on the answer sentence vector, thereby generating groups. The computer may analyze the generated statistical information of the characteristics of the group. The computer may analyze statistical information of the scoring results for each of the groups. The computer may output statistical information on the characteristics of the group and statistical information on the scoring results. [10th Form] The program is The program may cause the computer to execute a process of reading graded answer sheet data for a written test. The program may cause the computer to execute a process of morphologically analyzing the answer sheet data. The program may cause the computer to execute a process of vectorizing the result of the morphological analysis into an answer sentence vector. The program may cause the computer to execute a process of grouping the answer data that matches the rule definition and grouping the answer data that does not conform to the rule definition based on the answer sentence vector, thereby generating groups. The program may cause the computer to execute a process of analyzing statistical information about characteristics of the generated group. The program may cause the computer to execute a process of analyzing statistical information of the scoring results of each of the groups. The program may cause the computer to execute a process of outputting statistical information on the characteristics of the group and statistical information on the scoring results. The ninth and tenth forms can be expanded into the second to eighth forms, just like the first form.

[0098] The disclosures of the above-cited patent documents are incorporated herein by reference. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the claims), and further based on the basic technical concept thereof. Furthermore, various combinations and selections of the various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the disclosure of the present invention. In other words, the present invention naturally embraces various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concept, including the claims. In particular, with regard to the numerical ranges set forth herein, any numerical value or subrange within that range should be construed as specifically set forth, even if not otherwise specified. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of the disclosure of the present invention, in accordance with the spirit of the present invention, are also deemed to be included in the disclosures of this application. [Explanation of symbols]

[0099] 100 Graded answer data storage section 110 Graded answer data from CBT exams 120 Scanner 130 Graded Answer Sheets 140 Rule Definitions 200 Grouping Device 210 Answer data reading section 211 Morphological analysis section 212 Vectorization Department 213 Grouping Implementation Department 214 Group Characteristics Analysis Department 215 Group Coordination Department 216 Scoring Results Analysis Department 217 Output section 220 Rule definition reading section 221 Rule-based group division 222 Similarity calculation part 223 High Similarity Answer Data Extraction Unit 224 Transitive Relation Consideration Group Division 225 Community Detection 226 Group Representative Answer Data Generation Unit 300 Result storage section 9000 computers 9010 CPU 9020 Communication Interface 9030 Memory 9040 Auxiliary storage device

Claims

1. a reading unit that reads graded answer data for a written examination; a morphological analysis unit that performs morphological analysis on the answer sheet data; a vectorization unit that vectorizes the result of the morphological analysis into an answer sentence vector; a grouping implementation unit that groups the answer data that matches a rule definition and groups the answer data that does not conform to the rule definition based on the answer sentence vector, thereby generating groups; a group feature analysis unit that analyzes the generated statistical information of the group features; a scoring result analysis unit that analyzes statistical information of the scoring results of each of the groups; an output unit that outputs statistical information on the characteristics of the group and statistical information on the scoring results; A grouping device for graded answers in written exams.

2. 2. A grouping device for grouping graded answers in a written test as described in claim 1, wherein the grouping implementation unit calculates the similarity between all answer data that do not match the rule definition, groups the answer data whose similarity is equal to or greater than a predetermined threshold, and generates the groups.

3. 2. The grouping device for grouping graded answer sheets in a written test according to claim 1, wherein the grouping implementation unit groups transitional relationship-considered groups into one of the groups.

4. 3. The grouping device for graded answer sheets in a written test according to claim 2, wherein the grouping implementation unit calculates the similarity based on the answer sentence vector.

5. 2. The grouping device for graded answer sheets in a written test according to claim 1, wherein said group characteristic analysis unit analyzes statistical information on words characteristic of each group and words common to a plurality of said groups.

6. 2. A grouping device for graded answer sheets in a written test as described in claim 1, further comprising a group linkage adjustment unit that, when there is an increase in the answer sheet data, adjusts the linkage of the increased answer sheet data for each of the groups of processed answer sheet data.

7. The group coupling adjustment unit If the non-grouped incremental data is combined with only one of the existing groups, add the incremental data to the existing group; When one incremental data item is combined with a plurality of the existing groups, the incremental data item is added to the existing group with the largest number of combinations.

7. The grouping device for graded answer sheets in a written test according to claim 6, wherein the combining of the incremented answer sheet data is adjusted.

8. The computer Read the graded answer data for the written test, Morphological analysis of the answer data is performed; The result of the morphological analysis is vectorized into an answer sentence vector; The answer data that conforms to the rule definition is grouped, and the answer data that does not conform to the rule definition is grouped based on the answer sentence vector to generate groups; Analyzing the generated statistical information of the characteristics of the group; Analyzing the statistical information of the scoring results for each of said groups; outputting statistical information of the characteristics of the group and statistical information of the scoring results; How to group graded answers for written exams.

9. On the computer, A process for reading the graded answer data for the written test; A process of performing morphological analysis on the answer sheet data; A process of vectorizing the results of the morphological analysis into an answer sentence vector; A process of grouping the answer data that matches the rule definition and grouping the answer data that does not conform to the rule definition based on the answer sentence vector to generate groups; analyzing the generated statistical information of the characteristics of the group; analyzing statistical information of the scoring results of each of the groups; A program that executes a process of outputting statistical information on the characteristics of the group and statistical information on the scoring results.

Images