A grouping device for graded written exam answers, a method for grouping graded written exam answers, and a program.

Abstract

This invention provides a grouping device for graded answer sheets of written examinations, which helps to reduce variations in grading results. [Solution] The grouping device for graded written exam answers includes a reading unit that reads graded written exam answer data, a morphological analysis unit that performs morphological analysis on the answer data, a vectorization unit that vectorizes the results of morphological analysis into answer text vectors, a grouping implementation unit that groups answer data that matches the rule definition and groups answer data that does not conform to the rule definition based on the answer text vector to generate groups, a group feature analysis unit that analyzes statistical information of the characteristics of the generated groups, a scoring result analysis unit that analyzes statistical information of the scoring results of each group, and an output unit that outputs statistical information of the group characteristics and statistical information of the scoring results.

Description

Technical Field

[0001] The present invention relates to a grouping device for descriptive test scored answers, a method for grouping descriptive test scored answers, and a program.

Background Art

[0002] Regarding an information processing device related to scoring of descriptive test scored answers, the following documents can be cited.

[0003] Patent Document 1 relates to detecting those that may have been mis-scored among the scoring of answers in tests and the like.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The following analysis is provided by the inventor of the present invention.

[0006] In the field of education, various tests are frequently conducted, such as scoring the answers of respondents to test questions to evaluate the understanding level of the respondents. As test questions, there are, for example, those in a format where answers are selected using a mark sheet, while on the other hand, descriptive question formats that present problem-solving questions to respondents and require them to create answers in text are also widely adopted.

[0007] Typically, the answers to these essay-style questions are graded by teachers or other human beings. In exams with a large number of test-takers, such as entrance exams and qualification exams, multiple graders are involved in the grading process. This has led to problems such as inconsistencies in grading results even when students have similar answers. Furthermore, there are concerns that inconsistencies in grading results may arise when the same grader grades the answers of many test-takers. And when inconsistencies in grading results occur among graders, it becomes difficult to maintain fairness in evaluation.

[0008] Therefore, in order to minimize variations in scoring results during the scoring process, scorers have to go to great lengths such as coordinating with other scorers and reviewing already scored results, which places a heavy burden on scorers involved in the scoring process.

[0009] Patent Document 1 is an example of a conventional technique for detecting potentially misgraded answers from exam papers and the like. However, it requires creating a classification model trained using machine learning or deep learning with the answer data. Therefore, it requires preparing answer data for model creation, adjusting hyperparameters for machine learning or deep learning, and executing the training process. As a result, there was a problem that it took time before actual use could begin.

[0010] The present invention aims to provide a grouping device for graded written exam answers, a method for grouping graded written exam answers, and a program that contribute to suppressing variations in scoring results. [Means for solving the problem]

[0011] According to a first aspect of the present invention, a reading unit reads data of graded written exam answers, A morphological analysis unit that performs morphological analysis on the aforementioned answer data, A vectorization unit that vectorizes the results of the morphological analysis into a solution text vector, A grouping implementation unit groups the answer data that matches the rule definition, and groups the answer data that does not conform to the rule definition based on the answer text vector to generate groups. A group feature analysis unit analyzes statistical information of the characteristics of the generated group, A scoring result analysis unit analyzes statistical information of the scoring results for each of the aforementioned groups, Includes an output unit that outputs statistical information on the characteristics of the group and statistical information on the scoring results. We can provide a grouping device for graded answer sheets from written examinations.

[0012] According to a second aspect of the present invention, a computer, Read the data of graded answer sheets for the written exam, The aforementioned answer data is subjected to morphological analysis, The results of the morphological analysis are vectorized into a solution text vector. The answer data that matches the rule definition is grouped, and the answer data that does not conform to the rule definition is grouped based on the answer text vector to generate groups. Analyze the statistical information of the characteristics of the generated group, Analyze the statistical information of the scoring results for each of the aforementioned groups. This includes outputting statistical information on the characteristics of the group and statistical information on the scoring results. We can provide a method for grouping graded answer sheets from written examinations. This method is tied to a specific machine, a computer, that performs the above procedure.

[0013] According to a third aspect of the present invention, a computer, The process of reading graded answer sheet data for written exams, The process involves performing morphological analysis on the aforementioned answer data, The process of vectorizing the results of the morphological analysis into a solution text vector, Group the answer data that conforms to the rule definition, and group the answer data that does not conform to the rule definition based on the answer text vector to generate groups, and Analyze the statistical information of the characteristics of the generated groups, and Analyze the statistical information of the scoring results of each group, and It is possible to provide a program that executes a process of outputting the statistical information of the characteristics of the group and the statistical information of the scoring results.

[0014] Note that these programs can be recorded on a computer-readable storage medium. The storage medium can be non-transitory such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, etc. The present invention can also be embodied as a computer program product.

Effects of the Invention

[0015] According to the present invention, it is possible to provide a descriptive test scored answer grouping device, a descriptive test scored answer grouping method, and a program that contribute to suppressing the variation in scoring results.

Brief Description of the Drawings

[0016] [Figure 1] It is a block diagram showing an example of the configuration of a descriptive test scored answer grouping device according to the present disclosure and an example of the overall system configuration. [Figure 2] It is a block diagram showing an example of the configuration of a grouping implementation unit of the grouping device according to the present disclosure. [Figure 3] It is a diagram showing an example of the configuration of descriptive test scored answer data according to the present disclosure. [Figure 4] It is a diagram showing an example of the configuration of a rule definition according to the present disclosure. [Figure 5] It is a flowchart showing an example of the processing operation of the grouping device according to the present disclosure. [Figure 6]This flowchart shows an example of the rule-based grouping process of the grouping device relating to this disclosure. [Figure 7] This flowchart shows an example of the group analysis process of the grouping device relating to this disclosure. [Figure 8] This flowchart shows an example of the group feature analysis process of the grouping device related to this disclosure. [Figure 9] This flowchart shows an example of the processing for analyzing the scoring results of the grouping device related to this disclosure. [Figure 10] This is a flowchart showing an example of the regrouping process of the grouping device relating to this disclosure. [Figure 11] This is a flowchart illustrating an example of the regrouping analysis process of the grouping device related to this disclosure. [Figure 12] This figure shows an example of how to calculate the cosine similarity and contribution of the grouping device relating to this disclosure. [Figure 13] This figure shows an example of the relationship between answer data and cosine similarity calculation of the grouping device related to this disclosure. [Figure 14] This figure shows an example of an overview of the grouping device and its consideration of transitional relationships related to this disclosure. [Figure 15] This figure shows an example of how the grouping device described in this disclosure can divide a group. [Figure 16] This figure shows an example of reducing the dimensionality of the answer data vector space to two dimensions. [Figure 17] This figure shows an example of an analysis of scoring results. [Figure 18] This figure shows an example of how to calculate cosine similarity when regrouping incremental data. [Figure 19] This figure shows an example of how to regroup transitional relationships. [Figure 20] This figure shows an example of how to regroup transitional relationships. [Figure 21] This figure shows an example of how to regroup transitional relationships. [Figure 22] This figure shows an example of how to regroup transitional relationships. [Figure 23] This figure shows an example of how to regroup transitional relationships. [Figure 24] This figure shows an example of how to regroup transitional relationships. [Figure 25] This diagram shows the configuration of the computer that constitutes the grouping device relating to this disclosure. [Modes for carrying out the invention]

[0017] In this disclosure, the drawings may be associated with one or more embodiments. Furthermore, each embodiment described below may be combined with other embodiments as appropriate, and the present invention is not limited to each embodiment.

[0018] First, an overview of one embodiment will be described with reference to the drawings. The reference numerals in the drawings attached to this overview are provided for convenience to aid understanding and are not intended to limit the present invention to the illustrated embodiment. Furthermore, the connecting lines between blocks in the drawings and other references in the following description include both bidirectional and unidirectional lines. Unidirectional arrows schematically represent the flow of the main signal (data) and do not exclude bidirectional flow.

[0019] Figure 1 is a block diagram showing an example of the configuration of the grouping device for graded written examination answers related to this disclosure, and an example of the overall system configuration.

[0020] The overall system includes a graded answer sheet data storage unit 100, a grouping device 200, and a result storage unit 300.

[0021] The graded answer sheet data storage unit 100 stores graded answer sheet data 110 from CBT (Computer Based Testing) exams, as well as answer data obtained by scanning graded answer sheets 130 with a scanner 120. The answer data includes the answer text for written questions, the scoring result (score), and the examinee ID.

[0022] The grouping device 200 for graded written exam answers includes a reading unit 210 for reading answer data, a morphological analysis unit 211, a vectorization unit 212, a grouping implementation unit 213, a group feature analysis unit 214, a grading result analysis unit 216, and an output unit 217.

[0023] The reading unit 210 reads the data of graded written exam answers from the graded answer data storage unit 100.

[0024] The morphological analysis unit 211 receives the answer data, performs morphological analysis, and stores the morphological analysis results in the result storage unit 300.

[0025] The vectorization unit 212 vectorizes the results of the morphological analysis into answer text vectors and stores the vectorization results in the result storage unit 300.

[0026] The grouping unit 213 reads the rule definition 140, groups the answer data that matches the rule definition 140 to generate a group (i.e., generates an intermediate grouping result (1) according to the rule definition 140), and stores it in the result storage unit 300.

[0027] Rule Definition 140 is, as an example, a list of information equivalent to scoring criteria, such as NG (No Good) words that are inappropriate for the content of the answer, and the length of the text.

[0028] The grouping unit 213 groups the remaining answer data that do not conform to the rule definition based on the answer text vector to generate groups and stores the grouping intermediate result (1) in the result storage unit 300.

[0029] The group feature analysis unit 214 analyzes statistical information of the characteristics of the groups generated by the grouping. The group feature analysis unit 214 analyzes words characteristic of each group and words common to multiple groups, and analyzes statistical information of the characteristics of the grouped groups. The group feature analysis unit 214 stores the analysis results as grouping intermediate results (2) in the result storage unit 300.

[0030] The scoring result analysis unit 216 analyzes the statistical information of the scoring results for each group. For example, it performs statistical analysis of the scoring results (scores) of the grouped answer data, such as the mean, median, standard deviation, maximum value, minimum value, and interquartile range, to detect outliers.

[0031] The output unit 217 outputs statistical information on the group's characteristics and statistical information on the scoring results. For example, the output unit 217 generates and outputs output information to be displayed on the user terminal screen, etc.

[0032] The grouping device 200 may further include a group combination adjustment unit 215. The group combination adjustment unit 215 adjusts the combination of the incremented answer data for each group of processed answer data when there is an increment in the answer data. As an example, the grouping results are stored in the result storage unit 300.

[0033] The data grouping method of this invention allows for the grouping of answer texts based on grading results from a large amount of answer data, without relying on manual intervention or requiring prior training. Therefore, the time required for preparing data for model creation and training the model can be significantly reduced.

[0034] As a result, variations in scoring results can be suppressed, and the burden on graders can be reduced. Furthermore, this invention can also be used to support the scoring of answers submitted by respondents in exercises, etc., and to help improve scores if they are low.

[0035] As described above, according to one embodiment of the present invention, it is possible to provide a grouping device for graded written exam answers, a method for grouping graded written exam answers, and a program that contribute to suppressing variations in grading results.

[0036] [First Embodiment] Next, the first embodiment will be described in detail with reference to the drawings. Figure 1 is a block diagram showing an example of the configuration of the grouping device 200 for graded written examination answers according to this disclosure and an example of the overall system configuration. Figure 2 is a block diagram showing an example of the configuration of the grouping implementation unit 213 of the grouping device 200 according to this disclosure.

[0037] Figure 3 shows an example of the structure of graded answer sheet data for written examinations related to this disclosure. Referring to Figure 3, the graded answer sheet data for written examinations that the answer sheet data reading unit 210 reads from the graded answer sheet data storage unit 100 includes answer data that has the examinee ID, answer text, and scoring result (score). Figure 4 shows an example of the structure of the rule definition related to this disclosure.

[0038] Next, an example of the processing operation of the grouping device according to this disclosure will be described with reference to Figures 5 to 9. Figure 5 is a flowchart showing an example of the processing operation of the grouping device according to this disclosure. Referring to Figure 5, the processing starts in step S100. The answer data reading unit 210 of the grouping device 200 reads data from the graded answer data storage unit (also called DB) 100 (step S101). The format of the data to be read includes examinee ID, answer text, and grading result (score), as shown in an example in Figure 3.

[0039] The morphological analysis unit 211 performs morphological analysis on the graded answer sheet data (step S102). If any of the resulting morphemes contain parts of speech (such as particles or auxiliary verbs) that are deemed unnecessary, these parts of speech may be excluded. The results of the morphological analysis are stored in the result storage unit 300.

[0040] The vectorization unit 212 vectorizes the morphemified answer text (step S103). At this time, weights may be assigned to specific words that must be included in the answer as defined in the scoring criteria. Smaller values ​​may be assigned to words judged to be unimportant for grouping, and larger values ​​to words judged to be important. The weighted vectors may be normalized. Any method may be used for vectorization, such as TF (Term Frequency), TF-IDF (Term Frequency-Inverse Document Frequency), BM25 (Best Matching 25), Word2Vec, and Doc2Vec, but in the first embodiment, TF-IDF will be explained as an example. The vectorized answer text will be referred to as the answer text vector. The results of vectorization are stored in the result storage unit 300.

[0041] Next, the grouping unit 213 performs group analysis and grouping (step S104). Referring to Figure 2, the grouping unit 213 includes a rule definition reading unit 220, a rule-based group division unit 221, a similarity calculation unit 222, a high-similarity answer data extraction unit 223, a transition relationship-considering group division unit 224, a community detection unit 225, and a group representative answer data generation unit 226.

[0042] Figure 6 is a flowchart showing an example of the rule-based grouping process of the grouping device according to this disclosure. In the grouping performed in step S104 of Figure 5, as an example, the rule-based grouping process shown in Figure 6 is performed first.

[0043] Referring to Figure 6, the rule-based grouping process begins in step S110. The rule definition reading unit 220 reads rule definitions from a rule definition file that defines scoring criteria, etc. (step S111). Figure 4 shows an example of a rule definition. A rule definition is data consisting of a rule ID in column 500 and a rule content in column 501, and multiple rules may be written across multiple lines, as shown in rule ID=3 in Figure 4. In this case, as an example, the rule content of each line is used as an AND condition to determine whether the rule definition of rule ID=3 is satisfied. That is, if the rule definition of all lines is satisfied, it is determined that it matches the rule definition. It is determined whether the answer data matches the defined rule. If the answer data matches the rule definition (Yes in step S112), it is assigned to a group based on the rule ID (step S113). In addition, groups based on rule IDs are generated after the answer data has been assigned to a group (step S114). Even if there are 0 answer data assigned to a rule ID, groups based on rule IDs are generated. The process ends in step 115.

[0044] In the group analysis (step S104), answer data that does not match the rule definitions (No. in step S112) based on group assignment according to the rule definitions shown in steps S110 to 115 of Figure 6 will be processed in the group analysis (step S104) as an example.

[0045] [Handling when the rule definition does not match] Figure 7 is a flowchart showing an example of the processing performed in the group analysis (step S104) shown in Figure 5 of the grouping device according to this disclosure. The processing starts in step S120. The similarity calculation unit 222 calculates the similarity between all answer data (step S121). That is, it calculates the similarity between all answer text vectors. At this time, any method such as cosine similarity or Euclidean norm may be used to calculate the similarity, but in the first embodiment, cosine similarity will be used as an example.

[0046] Figure 12 shows an example of the calculation of cosine similarity and contribution of the grouping device according to this disclosure. As an example, the cosine similarity between the TF-IDF vector of answer data 1 and the TF-IDF vector of answer data 2 in column 600 is calculated. Columns 601 to 603 are elements obtained by morphological analysis. As an example, the product of the values ​​of each element in the TF-IDF vector of answer data 1 and the TF-IDF vector of answer data 2 is calculated as the contribution, and the cosine similarity is calculated as the sum of these contributions. Referring to Figure 12, as an example, if the answer sentence is "I don't know how to write the answer," then "answer," "description," and "how" are obtained through morphological analysis. The same applies if the TF-IDF vector of answer data 1 and the TF-IDF vector of answer data 2 are the centroid vector of group 1 and the centroid vector of group 2, respectively. Figure 13 shows an example of the relationship between answer data and the calculation of cosine similarity of the grouping device according to this disclosure. Since there is no need to calculate the similarity between each answer data 1 to N and the similarity between each answer, we calculate the cosine similarity of the shaded combinations. Note that if the similarity is above a certain threshold, a similar relationship is considered to exist.

[0047] The high-similarity answer data extraction unit 223 in Figure 2 extracts answer data where the similarity between each answer data is equal to or greater than a threshold (step S122). In the first embodiment, as an example, the target is answer text vectors with a similarity of 0.4 or higher.

[0048] The transition relationship-considering group division unit 224 in Figure 2 groups the network of high-similarity answer data extracted in step S104 in Figure 5 as a transition relationship-considering group (step S123). Even if there is a similarity relationship between answer data A and B, and a similarity relationship between answer data B and C, but no similarity relationship between answer data A and C, it is considered that there is a relationship between A and C and it is treated as a transition relationship-considering group. In other words, the network of similarity relationships is treated as a transition relationship-considering group and considered as one group.

[0049] Figure 14 shows an example of an overview of a transitional relationship-considering group in the grouping device according to this disclosure. The numerical values ​​shown on the lines connecting each answer data represent the similarity between those answer data. When answer data with a similarity above a threshold are extracted, a network like the one shown in Figure 14 is formed. Here, there is no direct similarity relationship between answer data 2 and answer data 5, but they belong to the same network via answer data 1. Such a network with indirect similarity relationships is called a "transitional relationship-considering group".

[0050] The community detection unit 225 in Figure 2 divides the transitional relationship-considering groups using community detection (step S124). Community detection methods include those based on edge via centrality and those based on random walks, but in the first embodiment, community detection based on a greedy algorithm will be explained as an example. If one transitional relationship-considering group becomes too large, it may have different meanings at the ends of the network. This is divided into communities using community detection, a technique that divides groups into communities based on the shape of the graph. The groups after division are considered the final groups. Figure 15 shows an example of the group division image by the grouping device according to this disclosure. The circles represent answer data vectors, and the lines represent connections. Connections will be described later.

[0051] The group representative answer data generation unit 226 in Figure 2 generates representative data for the group's answer data and combines it into a single answer data (step S125). The generation method may involve selecting one representative example from the group, or it may be a text summarized by an LLM (Large Language Model). Alternatively, the centroid vector of the group may be calculated, and the answer data closest to the centroid vector may be selected. The process ends in step S126.

[0052] Returning to Figure 5, the group feature analysis unit 214 described in Figure 1 performs the group feature analysis in step S105 of Figure 5, analyzing words characteristic of a particular group and words common to many groups. Figure 8 is a flowchart showing an example of the group feature analysis process of the grouping device according to this disclosure, and describes the detailed contents performed in the group feature analysis process in step S105 of Figure 5. The group feature analysis process starts in step S130.

[0053] Referring to Figure 8, the centroid vector of each group is calculated (step S131). The distance between groups is calculated from the centroid vector of each group (step S132). The contribution, which will be described later, is calculated (step S133). Based on the contribution, statistical information such as words that appear commonly between groups, words that differ greatly between groups, words unique to each group, and words common to all groups is output as the analysis result (step S134). Figure 16 shows an example of the case when the space of answer data vectors is reduced to two dimensions. Figure 16 shows an example of a figure displayed as statistical information.

[0054] Each point in Figure 16 represents an answer data vector, and points connected by lines indicate that they belong to the same group. The centroid vector of each group is represented by a rectangle. The contribution between centroid vectors is checked, and words with a high contribution between groups are displayed as common words (common vocabulary). Words with a low contribution to any group are displayed as group-specific words. In other words, dimensionality reduction is performed to display common words between groups and group-specific words.

[0055] The contribution is calculated by vectorizing the centroids of the groups and then measuring the contribution between these vectors (centroid vectors). For example, if the TF-IDF vectors of each answer data shown in Figure 12 are used as the group centroid vectors, then each term in the dot product of the vectors used to calculate the cosine similarity represents the contribution of each word.

[0056] Words with a high contribution between these groups are those that appear commonly in both groups, while words with a contribution of 0 appear in only one of the groups. Words that appear in only one of the groups are group-specific words. Based on the analysis results, the aforementioned vector weighting for each word may be applied, and the grouping may be repeated. The same analysis may also be performed across communities.

[0057] Returning to Figure 5, in the group feature analysis of step S105 in Figure 5, another method of intergroup analysis performed by the group feature analysis unit 214 in Figure 1 could be considered, for example, using the IDF (Inverse Document Frequency) vector. The IDF vector may be calculated when calculating the centroid vector of the groups. Words with a large IDF value are words that exist in only a small number of groups, while words with a small IDF value are words that exist in many groups.

[0058] Referring to Figure 5, the scoring result analysis unit 216 described in Figure 1 performs the scoring result analysis in step S106 of Figure 5. Figure 9 is a flowchart showing an example of the scoring result analysis process of the grouping device according to this disclosure, and corresponds to the scoring result analysis in step S106 of Figure 5. The scoring result analysis process starts in step S140.

[0059] The scoring result analysis unit 216 reads the grouping results from the result storage unit 300 (step S141). From the scoring results (scores) of all answer data within the group, it calculates statistical information such as the mean, median, standard deviation, maximum value, minimum value, and interquartile range (step S142). Next, it detects outliers from the calculated statistical information (step S143). In the first embodiment, as an example of a method for calculating outliers, a method using the interquartile range is described. Let Q1 be the first quartile (boundary of the lower 25% of data points) and Q3 be the third quartile (boundary of the upper 25% of data points). In this case, the interquartile range can be obtained by Q3-Q1.

[0060] Using these, the outlier boundary is calculated using the following formula to detect whether an outlier exists. Lower limit outlier = Q1 - 1.5 × interquartile range Upper limit outlier = Q3 + 1.5 × interquartile range Figure 17 shows an example of scoring result analysis, illustrating an example of detecting outliers within a specific group, namely outliers exceeding the upper limit and outliers exceeding the lower limit.

[0061] Data points exceeding the above range are detected as outliers. If an outlier is detected, an outlier flag is set for the examinee ID of the target response data in the grouping results of the result storage unit 300, and stored in the result storage unit 300 as an example.

[0062] The output unit 217 in Figure 1 refers to the grouping results in the result storage unit 300 and outputs the groups for which the outlier flag is set and the examinee ID information.

[0063] The data grouping method of the first embodiment of the present invention allows for the grouping of answer texts corresponding to grading results from a large amount of answer data without relying on manual intervention or requiring prior training. Therefore, the time required for data preparation for model creation and model training can be significantly reduced. Furthermore, community detection allows for further division within the groups to create more precise groups. Unlike existing clustering methods, grouping can be performed without specifying the number of clusters in advance.

[0064] According to the first embodiment of the present invention, the characteristics of a group can be visualized by calculating the centroid of the group and outputting statistical information of the centroid vector. Based on these characteristics, the parameters can be set again and grouping can be performed again, resulting in more accurate grouping.

[0065] As a result, variations in scoring results can be suppressed, and the burden on graders can be reduced. Furthermore, this invention can also be used to support students in scoring answers submitted by test takers in exercises, etc., and to help them improve their scores if their scores are low.

[0066] Accordingly, according to the first embodiment of the present invention, a grouping device for graded written exam answers, a method for grouping graded written exam answers, and a program can be provided that contribute to suppressing variations in scoring results.

[0067] [Second Embodiment] Next, a second embodiment will be described in detail with reference to the drawings. The second embodiment is an embodiment relating to incremental data screening. Figure 10 is a flowchart showing an example of the regrouping process of the grouping device according to this disclosure. Figure 11 is a flowchart showing an example of the regrouping analysis process of the grouping device according to this disclosure. For an example of the configuration of the grouping device and an example of the overall system configuration, please refer to Figure 12, and for an example of the configuration of the grouping implementation unit of the grouping device, please refer to Figure 2.

[0068] After performing grouping using answer data once, the system addresses situations where answers from similar tests are administered (for example, validating the results of grading past exam questions), or where a nationwide test is administered and grading is done regionally, resulting in staggered collection times for the grading results. In other words, it ensures the overall validity of grading when the timing of answer data collection and retrieval differs. Incremental data screening involves adding incremental data to existing answer data and regrouping it appropriately.

[0069] Referring to Figure 10, the process of starting the regrouping begins in step S150. Next, the answer data reading unit 210 in Figure 1 reads incremental data from the graded answer data storage unit (also called DB) 100 (step S151). The incremental data refers to the graded answer data accumulated since the previous grouping was performed.

[0070] The morphological analysis unit 211 shown in Figure 1 performs morphological analysis on the augmented response text (step S152). Of the resulting morphemes, parts of speech deemed unnecessary (such as particles and auxiliary verbs) may be excluded, but they must be the same parts of speech that were excluded during the morphological analysis of existing data. The results of the morphological analysis are stored in the result storage unit 300.

[0071] The vectorization unit 212 vectorizes the morphemified answer text (step S153). At this time, weights may be assigned to specific words. Smaller values ​​are assigned to words deemed unimportant for grouping, while larger values ​​are assigned to important words. The weighted vectors may be normalized. Any method can be used for vectorization, such as TF, TF-IDF, BM25, Word2Vec, or Doc2Vec, but it must be the same method as the original answer text vector. Below, TF-IDF will be used as an example. The vectorized answer texts will be referred to as augmented answer text vectors. The vectorization results are stored in the result storage unit 300.

[0072] Next, the grouping unit 213 performs a regrouping analysis (step S154).

[0073] Figure 11 is a flowchart showing an example of the regrouping analysis process of the grouping device according to this disclosure. The regrouping analysis process will be explained with reference to Figure 11. The regrouping analysis process starts in step S160.

[0074] The grouping unit 213 in Figure 1 reads the answer text vectors of existing answer data from the result storage unit (also called DB) 300 (step S161).

[0075] Figure 18 shows an example of the operation for calculating cosine similarity when regrouping incremental data. In Figure 18, it is assumed that similarity checks have already been performed between existing data, indicated by reference numeral 1201. Next, the similarity calculation unit 222 in Figure 2 performs similarity calculations between all incremental data, as indicated by the shaded area of ​​reference numeral 1202 in Figure 18 (step S162). That is, it calculates the similarity between all incremental answer sentence vectors. At this time, any similarity can be used, such as cosine similarity or Euclidean norm, but it is necessary to use the same method as the similarity calculation of existing data described above.

[0076] Next, as indicated by reference numeral 1203, the similarity calculation unit 222 calculates the similarity between all incremental data and existing data (step S163). At this time, any similarity can be used, such as cosine similarity or Euclidean norm, but it is necessary to use the same method as the similarity calculation of the existing data described above.

[0077] The high-similarity answer data extraction unit 223 in Figure 2 extracts answer data where the similarity between each answer data is equal to or greater than a threshold (step S164). This threshold must be the same as the threshold for the existing data mentioned above. As an example, the target is those with a similarity of 0.4 or higher between the answer text vectors.

[0078] The transition relationship-considering group division unit 224 groups the network of high-similarity answer data extracted in step S164 as a transition relationship-considering group (step S165).

[0079] At this point, the following six types of similarity relationships are possible between the incremental data and the existing data. (1) Incremental data that has not been grouped is joined with one existing group. (2) Incremental data that has not been grouped is combined with multiple existing groups. (3) None of the incremental data that have not been grouped are joined to any existing groups. (4) The incremental data that makes up the group is combined with one existing group. (5) The incremental data that makes up the group is joined with multiple existing groups. (6) The incremental data that makes up the group is not joined with any existing group.

[0080] Here, "combined" refers to cases where the similarity between each answer data is above a threshold, and "not combined" refers to cases where the similarity between each answer data is below a threshold, but this is not the only definition.

[0081] The group join adjustment unit 215 in Figure 1 adjusts the joins for each of the six patterns described above (step S166). In the case of similarity relationship (1), incremental data that is joined to an existing group should be added. In the cases of similarity relationships (3) and (6), since the data is not joined to an existing group, there is no need to consider regrouping, and it can be left as is. In the cases of similarity relationships (2), (4), and (5), it can be said that one incremental data is joined to multiple groups. In this case, the incremental data is added to the group with the most joins, and the joins with other groups are deleted. If there are groups with the same number of joins, the incremental data is added to the group with the higher similarity, and the joins with other groups are deleted. The group after this operation is considered the group to which the incremental data has been added. Figures 19 to 22 show the operation when incremental data and existing groups are joined.

[0082] Figures 19 to 24 illustrate an example of the operation when regrouping transitional relationship groups. Referring to Figure 19, data 1321 of incremental data group 1320 is joined with data 1301 of existing group A1300, and data 1322 of incremental data group 1320 is joined with data 1301 of existing group A1300, and data 1311, 1312, and 1313 of existing group B1310.

[0083] Referring to Figure 20, data 1321 in incremental data group 1320 has one join with existing group A1300 and three joins with incremental data group 1320. Therefore, data 1321 is assigned to incremental data group 1320.

[0084] Referring to Figure 21, data 1322 in incremental data group 1320 has one join with existing group A1300, three joins with existing group B1310, and two joins with incremental data group 1320. Therefore, data 1322 is assigned to existing group B1310.

[0085] Referring to Figure 22, data 1321 of incremental data group 1320 belongs to incremental data group 1320, and data 1322 belongs to existing group B1310.

[0086] Referring to Figure 23, if the incremental data 1400 is joined to two different groups, for example, existing group 1 and existing group 2, then it is added to existing group 1, which has the higher similarity of joins 1401 and 1402.

[0087] Referring to Figure 24, if the incremental data 1400, which has not been grouped, is joined to existing group 1, for example, then it will be added to existing group 1.

[0088] The community detection unit 225 in Figure 2 divides the transition relationship-considering groups created in step S166 using community detection (step S167). Community detection methods include those based on edge betweenness centrality and those based on random walks, but as an example, we will explain community detection based on a greedy algorithm.

[0089] The group representative answer data generation unit 226 in Figure 2 generates representative data for the group's answer data and combines it into a single answer data (step S168). The generation method may involve selecting one representative example from the group, or it may be a text summarized by LLM. Alternatively, the group's centroid vector may be calculated, and the answer data closest to the centroid vector may be selected.

[0090] The regrouping analysis process ends in step S169, and the process returns to step S155 in Figure 10.

[0091] Step S155 in Figure 10 is a group feature analysis process, which corresponds to step S105 in Figure 5, in which the group feature analysis unit 214 analyzes words characteristic of a specific group or words common to many groups. However, the detailed processing content is the same as steps S121 to S124 in Figure 7, which were explained in the first embodiment, so the explanation will be omitted here.

[0092] Although the explanation has focused on the processing flow of components such as the graded answer data storage unit 100, the grouping device (grouping function unit) 200, and the result storage unit 300, the present invention may also be comprised of a graded support device, a graded support system, a virtual server on the cloud, etc.

[0093] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and further modifications, substitutions, and adjustments can be made without departing from the basic technical idea of ​​the present invention. For example, the network configuration, the configuration of each element, and the message representation form shown in each drawing are examples to aid in understanding the present invention, and are not limited to the configurations shown in these drawings. Also, "A and / or B" is used to mean at least one of A or B.

[0094] Furthermore, the procedures described in the first and second embodiments above can be implemented by a program that enables a computer (9000 in Figure 25) that functions as a grouping device for graded written exam answers according to the present invention to perform the function of a grouping device for graded written exam answers. Such a computer is exemplified by a configuration comprising a CPU (Central Processing Unit) 9010, a communication interface 9020, a memory 9030, and an auxiliary storage device 9040 as shown in Figure 25. That is, the CPU 9010 in Figure 25 executes a control program for the grouping device for graded written exam answers and performs the update process of each calculation parameter held in the auxiliary storage device 9040, etc.

[0095] Memory 9030 refers to RAM (Random Access Memory), ROM (Read Only Memory), etc.

[0096] In other words, each part (processing means, function) of the grouping device for graded written exam answers, as described in the first and second embodiments above, can be realized by a computer program that causes the computer's processor to execute each of the above-described processes using its hardware.

[0097] Finally, preferred embodiments of the present invention are summarized. [First form] The grouping device for graded written exam answers may include a reading unit that reads the graded written exam answer data. The grouping device may include a morphological analysis unit that performs morphological analysis on the answer data. The grouping device may include a vectorization unit that vectorizes the results of the morphological analysis into answer text vectors. The grouping device may include a grouping implementation unit that groups the answer data that matches the rule definition, and groups the answer data that does not conform to the rule definition based on the answer text vector to generate groups. The grouping device may include a group feature analysis unit that analyzes statistical information of the characteristics of the generated groups. The grouping device may include a scoring result analysis unit that analyzes statistical information of the scoring results for each group. The grouping device may include an output unit that outputs statistical information on the characteristics of the group and statistical information on the scoring results. [Second form] In the grouping device for graded written examination answers described in the first embodiment, it is preferable that the grouping unit calculates the similarity between all the answer data that do not match the rule definition, and groups the answer data whose similarity is equal to or greater than a predetermined threshold to generate the group. [Third form] In the grouping device for graded written examination answers described in the first embodiment, it is preferable that the grouping unit groups the transition relationship consideration group as one group. [Fourth form] In the grouping device for graded written examination answers described in the second embodiment, it is preferable that the grouping unit calculates the similarity based on the answer text vector. [Fifth form] In the grouping device for graded written examination answers described in the fourth embodiment, it is preferable that the similarity is cosine similarity or the Euclidean norm. [Sixth form] In the grouping device for graded written examination answers described in the first embodiment, it is preferable that the group feature analysis unit analyzes statistical information of words characteristic of each group and words common to multiple such groups. [Seventh form] The grouping device for graded written exam answers described in the first embodiment may further include a group merging adjustment unit that adjusts the merging of the incremented answer data for each group of processed answer data when there is an increment in the answer data. [Eighth form] In the grouping device for graded written examination answers described in the seventh embodiment, the group combination adjustment unit is: If incremental data that has not been grouped is joined to only one existing group, the incremental data is added to the existing group. If one incremental data point is joined with multiple existing groups, the incremental data point is added to the existing group with the largest number of joins. It is preferable to adjust the combination of the incremented answer data. [Ninth form] The grouping method for graded essay-type exam answers may involve a computer reading the graded essay-type exam answer data. The computer may perform morphological analysis on the answer data. The computer may vectorize the results of the morphological analysis into a solution text vector. The computer may group the answer data that matches the rule definition and group the answer data that does not conform to the rule definition based on the answer text vector to generate groups. The computer may analyze statistical information of the characteristics of the group it has generated. The computer may analyze statistical information of the scoring results for each group. The computer may also output statistical information on the characteristics of the group and statistical information on the scoring results. [Tenth form] The program is for the computer, The program may cause the computer to execute a process to read data of graded answer sheets for written examinations. The program may cause the computer to perform a process of morphological analysis on the answer data. The program may also cause the computer to perform a process of vectorizing the results of the morphological analysis into a solution text vector. The program may cause the computer to perform a process of grouping the answer data that matches the rule definition, and grouping the answer data that does not conform to the rule definition based on the answer text vector, thereby generating groups. The program may cause the computer to perform a process to analyze statistical information of the characteristics of the generated group. The program may cause the computer to perform a process to analyze statistical information of the scoring results for each group. The program may also cause the computer to perform a process that outputs statistical information on the characteristics of the group and statistical information on the scoring results. Furthermore, the ninth and tenth forms described above can be expanded into the second through eighth forms, similar to the first form.

[0098] Furthermore, the disclosures in the above-mentioned patent documents are incorporated into this work by reference. Within the framework of the full disclosure of the present invention (including the claims), further modifications and adjustments to the embodiments or examples are possible based on the fundamental technical concept. Also, within the framework of the disclosure of the present invention, various combinations or selections of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that a person skilled in the art could make in accordance with the technical concept. In particular, with respect to the numerical ranges described in this work, any numerical value or sub-range included within that range should be interpreted as being specifically described, even if not otherwise stated. Furthermore, each disclosure item of the above-mentioned cited documents may, if necessary, be used in combination with the items described in this work as part of the disclosure of the present invention, in accordance with the spirit of the present invention, in part or in whole, and this is also considered to be included in the disclosure of this application. [Explanation of Symbols]

[0099] 100 Graded Answer Sheet Data Storage Section 110 CBT exam data (scored answer sheets) 120 Scanners 130 graded answer sheets 140 Rule Definitions 200 Grouping device 210 Answer Data Reading Unit 211 Morphological analysis section 212 Vectorization section 213 Grouping Implementation Department 214 Group Characteristics Analysis Department 215 Group coupling adjustment unit 216 Scoring Results Analysis Department 217 Output section 220 Rule definition reading unit 221 Rule-based group division section 222 Similarity calculation part 223 High Similarity Answer Data Extraction Unit 224 Transitional Relationship Consideration Group Division Section 225 Community detection unit 226 Group Representative Answer Data Generation Department 300 Result storage section 9000 Computers 9010 CPU 9020 Communication Interface 9030 memory 9040 Auxiliary storage device

Claims

1. A reading unit that reads graded answer sheet data for written exams, A morphological analysis unit that performs morphological analysis on the aforementioned answer data, A vectorization unit that vectorizes the results of the morphological analysis into a solution text vector, A grouping implementation unit groups the answer data that matches the rule definition, and groups the answer data that does not conform to the rule definition based on the answer text vector to generate groups. A group feature analysis unit analyzes statistical information of the characteristics of the generated group, A scoring result analysis unit analyzes statistical information of the scoring results for each of the aforementioned groups, Includes an output unit that outputs statistical information on the characteristics of the group and statistical information on the scoring results. A grouping device for graded answer sheets from written exams.

2. The grouping device for graded written exam answers according to claim 1, wherein the grouping implementation unit calculates the similarity between all the answer data that do not match the rule definition, and groups the answer data whose similarity is equal to or greater than a predetermined threshold to generate the group.

3. The grouping device for graded written test answers according to claim 1, wherein the grouping implementation unit groups the transition relationship consideration group as one group.

4. The grouping device for graded written exam answers according to claim 2, wherein the grouping unit calculates the similarity based on the answer text vector.

5. The group feature analysis unit analyzes statistical information of words characteristic of each group and words common to multiple such groups, in the grouping device for graded written exam answers according to claim 1.

6. A grouping device for graded descriptive exam answers according to claim 1, further comprising a group combination adjustment unit that adjusts the combination of the incremented answer data for each group of the processed answer data when there is an increment in the answer data.

7. The group coupling adjustment unit is, If incremental data that has not been grouped is joined to only one existing group, the incremental data is added to the existing group. If one incremental data point is joined with multiple existing groups, the incremental data point is added to the existing group with the largest number of joins. A grouping device for graded written exam answers according to claim 6, which adjusts the combination of the incremented answer data.

8. Computers Read the data of graded answer sheets for the written exam, The aforementioned answer data is subjected to morphological analysis, The results of the morphological analysis are vectorized into a solution text vector. The answer data that matches the rule definition is grouped, and the answer data that does not conform to the rule definition is grouped based on the answer text vector to generate groups. Analyze the statistical information of the characteristics of the generated group, Analyze the statistical information of the scoring results for each of the aforementioned groups. This includes outputting statistical information on the characteristics of the group and statistical information on the scoring results. Method for grouping graded answer sheets from written exams.

9. On the computer, The process of reading graded answer sheet data for written exams, The process involves performing morphological analysis on the aforementioned answer data, The process of vectorizing the results of the morphological analysis into a solution text vector, The process involves grouping the answer data that matches the rule definition, and grouping the answer data that does not conform to the rule definition based on the answer text vector to generate groups, A process for analyzing statistical information of the characteristics of the generated group, A process for analyzing statistical information of the scoring results for each of the aforementioned groups, A program that executes a process to output statistical information on the characteristics of the aforementioned group and statistical information on the scoring results.

Images