Automatic scoring apparatus
The automatic scoring device efficiently scores new questions by dividing and vectorizing document data into semantic units, clustering similar units, and generating pyramid scores, addressing the inefficiencies of requiring pre-defined model answers in existing systems.
Patent Information
- Application Number
- JP2023086780
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing automatic scoring systems require the creation of model answers and scoring criteria in advance, making it difficult to efficiently obtain scoring results for new questions.
An automatic scoring device that divides document data into semantic units, vectorizes them, clusters similar units, and generates pyramid scores based on frequency and similarity, allowing efficient scoring of new questions without pre-defined model answers.
Enables efficient and accurate scoring of new questions by reducing the need for pre-processing and model creation, leveraging vector similarity and frequency analysis to generate comprehensive scoring results.
Smart Images

Figure 2024169918000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an automatic marking device that outputs the marking results of document data that is an answer to a question. [Background technology]
[0002] Patent Document 1 describes a system for automatically marking answers to descriptive questions, which breaks down the answer into two layers, verifies the accuracy and completeness of the sentence structure, verifies whether the two layers are effectively connected, and calculates a score by adding up the scores of the verification results. Patent Document 2 describes a device for automatically marking answers to explanation questions, which analyzes the answer text to extract features, compares the features of the answer text with those of the model answer, calculates a score for each item, and determines the score by combining the scores for each item. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2017-129830 A [Patent Document 2] JP 2020-160159 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the system described in Patent Document 1, it is necessary to create model answers for descriptive questions in advance and to set various grading criteria in advance. Also, in the device described in Patent Document 2, it is necessary to create model answers for explanation questions in advance. Therefore, when trying to obtain a grading result for an answer to a new question, it is necessary to create a model answer each time, and it is difficult to efficiently obtain a grading result for an answer to a new question.
[0005] SUMMARY OF THE PRESENT EMBODIMENTS Accordingly, an object of the present invention is to provide an automatic marking device that can efficiently obtain marks for answers to new questions. [Means for solving the problem]
[0006] The automatic marking device of the present invention is an automatic marking device that outputs marking results for document data that is the answer to a question, and is equipped with at least one processor, which divides multiple document data for a specific target question into semantic units, vectorizes the divided multiple semantic units to generate multiple semantic unit vectors, compares the multiple semantic unit vectors in order and adds up the frequencies of multiple semantic unit vectors that are similar to each other to generate pyramid data representing a score based on the frequency of each of multiple representative vectors among the multiple semantic unit vectors, and calculates and outputs the marking result for the document data for the specific target question by adding up the scores in the pyramid data for the representative vectors that are similar to the multiple semantic unit vectors generated from the document data. Effect of the Invention
[0007] According to the present invention, the results of grading answers to new questions can be obtained efficiently. [Brief description of the drawings]
[0008] [Figure 1] FIG. 2 is a block diagram showing the functions of a server 100 according to the present disclosure. [Diagram 2] FIG. 13 is a diagram showing a processing image of generating EDU vectors and clustering the EDU vectors by the server 100. [Diagram 3] FIG. 2 is a diagram showing an example of the configuration of frequency data generated by the server 100. [Figure 4] 10 is a diagram showing an example of the configuration of weighting information stored in a weighting information storage unit 111. FIG. [Diagram 5] FIG. 13 is a diagram showing a processing image of generating EDU vectors and pyramid scores by the server 100. [Figure 6] FIG. 2 is a diagram showing an example of the configuration of pyramid scores stored in a pyramid data storage unit 112. [Figure 7] 13 is a flowchart showing a procedure of pre-processing by the server 100. [Figure 8] 13 is a flowchart showing the procedure of a scoring process by the server 100. [Figure 9] 13 is a flowchart showing a procedure of pre-processing by the server 100. [Figure 10] 13 is a flowchart showing the procedure of a scoring process by the server 100. [Figure 11] 13 is a flowchart showing the procedure of a scoring process by the server 100. [Figure 12] 1 is a diagram illustrating an example of a hardware configuration of a server 100 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] The embodiments of the present disclosure will be described with reference to the accompanying drawings. Whenever possible, the same parts are designated by the same reference numerals and redundant description will be omitted.
[0010] FIG. 1 is a diagram showing a functional configuration of a server 100 that is an automatic marking device in the present disclosure. In this diagram, the server 100 is configured to be able to perform data communication with a plurality of terminal devices 10 via a communication network (not shown). The server 100 has a function of outputting a marking result for answer data (document data) for a predetermined question provided by the terminal device 10, as described later. The predetermined question is a descriptive question that requires an answer in a language such as English, and the answer data is an essay that includes a sentence in that language for the descriptive question. However, the predetermined question is not limited to the above example, and may be a descriptive question that requires an answer in another language, or may be another type of question such as a grammar question that requires an answer in various languages. The answer data may also be an essay that includes a sentence in another language, or answer data other than an essay that includes a sentence in various languages.
[0011] In the present disclosure, the terminal device 10 is a device that provides answer data for a predetermined question to the server 100, and is a device such as a personal computer, a smartphone, a tablet terminal, a feature phone, or a server device.
[0012] As shown in FIG. 1, the server 100 is configured to include, as functional components, a data receiving unit 101, an analysis unit 102, a vector acquisition unit 103, a clustering unit 104, a weight generation unit 105, a pyramid data generation unit 106, a score calculation unit 107, a score output unit 108, a vector storage unit 109, a cluster information storage unit 110, a weight information storage unit 111, and a pyramid data storage unit 112.
[0013] The data accepting unit 101 accepts a plurality of pieces of answer data for a plurality of questions of predetermined content from one or more terminal devices 10. The number of questions is preset to, for example, 10. At this time, the data accepting unit 101 may accept the plurality of pieces of answer data separately from the plurality of terminal devices 10, or may accept the plurality of pieces of answer data collectively from one or more terminal devices 10. In detail, the data accepting unit 101 accepts a plurality of combinations of answer data and question identification information (for example, question number) that identifies the question corresponding to the answer data.
[0014] The analysis unit 102 divides a plurality of sentences included in each answer data accepted by the data accepting unit 101 into semantic units. A semantic unit is a unit expressing one meaning in a sentence, for example, an elementary discourse unit (EDU) equivalent to a phrase obtained by discourse structure analysis. For example, the analysis unit 102 inputs the answer data to a sequence labeling model such as a BiLSTM-CRF model, recognizes the division positions of EDUs in phrase units in the answer data based on the output of the sequence labeling model, and divides the answer data into EDUs using the division positions. Here, models such as CRF, BiLSTM, and BERT+CRF may be used in addition to the BiLSTM-CRF model. For example, if the answer data is, "Rose is caring. Her conversation with Anna shows this. If the essay is “Rose asks Anna if she feels okay,” the analysis unit 102 divides the essay into the following five EDUs: "Rose is caring." “Her conversation with Anna” “shows this.” "Rose asks Anna" “if she feels okay.”
[0015] The vector acquisition unit 103 generates a plurality of semantic unit vectors by vectorizing each semantic unit divided by the analysis unit 102. For example, when the semantic unit is an EDU, the vector acquisition unit 103 generates an EDU vector based on the EDU. More specifically, the vector acquisition unit 103 adds a CLS token to the beginning of each EDU and an SEP token to the end, and inputs each EDU to which the CLS token and SEP token are added to a large-scale language model (BERT: Bidirectional Encoder Representations from Transformers). Then, the vector acquisition unit 103 acquires, as an EDU vector, a fixed-length data string obtained from an output sequence corresponding to the CLS token in the output of BERT.
[0016] When the data receiving unit 101 receives a plurality of answer data for all of a predetermined number of questions, the clustering unit 104 clusters a plurality of semantic unit vectors generated based on the answer data. That is, the clustering unit 104 clusters a plurality of answer data corresponding to a plurality of questions into a predetermined number of clusters (for example, 5). For example, the clustering unit 104 clusters a plurality of EDU vectors by using the k-means method to group EDU vectors that are close to each other into a predetermined number of clusters. In addition, the clustering unit 104 refers to the clustering result to obtain information (cluster information) about the predetermined number of clusters, and stores the cluster information in the cluster information storage unit 110. The cluster information includes an average value of a plurality of EDU vectors clustered into each of the predetermined number of clusters, and is stored in the cluster information storage unit 110 together with a cluster number that identifies each cluster.
[0017] Furthermore, the clustering unit 104 refers to the clustering results, counts the frequency of the clustered EDU vectors for each of the questions and for each of the predetermined number of clusters, and generates frequency data indicating the counting results.
[0018] 2 shows an image of the process of generating EDU vectors and clustering EDU vectors by the server 100. Assuming that ten questions Q1 to Q10 are set in advance, the server 100 acquires a plurality of answer data A1 for the question Q1, acquires a plurality of answer data for each of the questions Q2 to Q9, and acquires a plurality of answer data A10 for the question Q10. Next, the server 100 generates a plurality of EDU vectors V1, ..., V10 from the acquired plurality of answer data A1, ..., A10. Furthermore, the server 100 executes a clustering process on a set VSET of all the generated EDU vectors V1, ..., V10, and clusters all the EDU vectors V1, ..., V10 into five clusters CL1 to CL5.
[0019] FIG. 3 shows an example of the configuration of frequency data generated by the clustering unit 104. In this configuration example, it is assumed that the number of questions is 10 and the number of clusters is 10. In the frequency data, the frequency is arranged for each question number that identifies 10 questions and for each cluster that identifies 10 clusters. For example, in association with a question identified by question number "8", the frequency "11" of cluster number "0", the frequency "3091" of cluster number "1", the frequency "4297" of cluster number "2", the frequency "16" of cluster number "3", the frequency "0" of cluster number "4", ..., the frequency "2202" of cluster number "9" are arranged in this order. Note that in FIG. 3, the frequency values for each cluster number associated with question numbers between question numbers "2" and "8" and question numbers between question numbers "8" and "10" are omitted, but this means that they can take any value.
[0020] Returning to FIG. 1, the weight generation unit 105 calculates the importance value for EDU for each of a predetermined number of clusters and each of a predetermined number of questions using the TF-IDF method based on the frequency data generated by the clustering unit 104. First, the weight generation unit 105 calculates the importance TF of each cluster in one question as follows. That is, the weight generation unit 105 calculates the sum of the frequencies of all clusters in a certain question, and divides the frequency of each cluster in the question by the sum of the frequencies to calculate the importance TF of each cluster in the certain question. The importance TF calculated in this way is the ratio of the frequency of EDU vectors clustered in each cluster in one question. The weight generation unit 105 calculates the importance TF of each cluster in each question by repeating the above calculation for each question.
[0021] Furthermore, the weight generation unit 105 calculates the importance IDF of each cluster in all questions as follows. That is, the weight generation unit 105 sums up the frequencies of all clusters in the frequency data, and sums up the frequencies of each cluster in all questions. The weight generation unit 105 then divides the sum of the frequencies of all clusters by the sum of the frequencies of each cluster, and adds 1 to the logarithm (log) of the divided value to calculate the importance IDF of each cluster. The importance IDF calculated in this way is a value based on the reciprocal of the sum (sum) of the frequencies of EDU vectors in each cluster across multiple questions.
[0022] Furthermore, the weight generation unit 105 calculates weights for each of the predetermined number of clusters and each of the predetermined number of questions, taking into account the importance TF of each cluster in each question and the importance IDF of each cluster. As an example, the weight generation unit 105 calculates weights by multiplying the importance TF of each cluster in each question by the importance IDF corresponding to each cluster. Then, the weight generation unit 105 stores weight information indicating the calculated weights for each cluster and each question in the weight information storage unit 111.
[0023] FIG. 4 shows an example of the weight information stored in the weight information storage unit 111. In this example, it is assumed that the number of questions is 10 and the number of clusters is 10. In the weight information, weight data is arranged for each question number that identifies 10 questions and for each cluster that identifies 10 clusters. For example, in association with a question identified by question number "8", the weight "0" of cluster number "0", the weight "0.64" of cluster number "1", the weight "0.89" of cluster number "2", the weight "0" of cluster number "3", the weight "0" of cluster number "4", ..., the weight "0.41" of cluster number "9" are arranged in this order. In FIG. 4, the weight values of the cluster numbers associated with the question numbers between question number "2" and question number "8" and the question numbers between question number "8" and question number "10" are omitted, but this means that they can take any value.
[0024] Returning to FIG. 1, when multiple answer data are received by the data receiving unit 101 for one specific target question among a predetermined number of multiple questions, the pyramid data generating unit 106 calculates a pyramid score (pyramid data) based on the answer data and stores it in the pyramid data storage unit 112. That is, the pyramid data generating unit 106 sequentially compares multiple EDU vectors generated from multiple answer data, and adds up the frequency of EDU vectors that are similar to each other. At this time, the similarity of the EDU vectors is determined, for example, by whether the distance between the EDU vectors is equal to or less than a threshold. In detail, the pyramid score records the score for each representative vector that was previously added, and the pyramid data generating unit 106 determines the similarity between a new EDU vector and the representative vector recorded in the pyramid score, and if a similar representative vector exists, adds "1" to the score of the representative vector, and if a similar representative vector does not exist, records a new score of "1" in the pyramid score with the EDU vector as the representative vector.
[0025] 5 shows an image of the generation of EDU vectors and pyramid scores by the server 100. Assuming that a specific question Q1 is set in advance as the processing target in this manner, the server 100 acquires multiple answer data A1 for the question Q1. Next, the server 100 generates multiple EDU vectors V1 from the acquired multiple answer data A1. Furthermore, the server 100 sequentially compares the generated multiple EDU vectors V1, and generates pyramid scores including scores "10", "5", "2", "1", ... for each of the multiple representative vectors RV1, RV2, RV3, RV4 ....
[0026] 6 shows an example of pyramid scores stored in the pyramid data storage unit 112. In this example, "V1" indicating the vector value of representative vector RV1 and its score, which is its frequency "10", "V2" indicating the vector value of representative vector RV2 and its score, which is its frequency "5", "V3" indicating the vector value of representative vector RV3 and its score, which is its frequency "2", and "V4" indicating the vector value of representative vector RV4 and its score, which is its frequency "1" are stored in correspondence with each other.
[0027] Returning to Fig. 1, the score calculation unit 107 calculates the score of the marking result of the answer data for one target question among the multiple questions using at least one of the following first calculation method and the following second calculation method. First, based on the answer data for which the score is to be calculated and accepted by the data acceptance unit 101, the score calculation unit 107 acquires multiple EDU vectors by having the analysis unit 102 and the vector acquisition unit 103 process the corresponding answer data.
[0028] Furthermore, when the first calculation method is used, the score calculation unit 107 has the clustering unit 104 process each of the multiple EDU vectors to cluster them into one of a predetermined number of clusters. Then, the score calculation unit 107 refers to the weight information stored in the weight information storage unit 111, and reads out the weights associated with the clusters to which each of the multiple EDU vectors is clustered and with one target question. Furthermore, the score calculation unit 107 calculates the first score of the corresponding answer data by the first calculation method by adding up the weights read for each of the multiple EDU vectors. For example, when the target question is a question with question number "8" and the EDU vectors of the answer data are clustered into two clusters, the cluster with cluster number "1" and the cluster with cluster number "2", the weight information shown in FIG. 4 is referred to, and the first score is calculated as 0.64×1+0.89×1=1.53.
[0029] Furthermore, when the second calculation method is used, the score calculation unit 107 reads out the pyramid score for the target question from the pyramid data storage unit 112. Then, the score calculation unit 107 sequentially compares each of the multiple EDU vectors with the multiple representative vectors included in the pyramid score. Furthermore, the score calculation unit 107 calculates the second score of the corresponding answer data by the first calculation method by adding up the frequencies associated with the representative vector most similar to each of the multiple EDU vectors. The similarity between the EDU vector and the representative vector is determined, for example, by the distance between the vectors. For example, when the representative vectors similar to the two EDU vectors generated from the answer data are the two representative vectors, the representative vector "V1" and the representative vector "V2", the pyramid score shown in FIG. 6 is referenced, and the second score is calculated as 10×1+5×1=15.
[0030] The score output unit 108 calculates the final score of the answer data based on at least one of the first score and the second score calculated by the score calculation unit 107 for the answer data, and outputs the final score to the terminal device 10. That is, the score output unit 108 may output either the first score or the second score, or may output a weighted sum taking into account the first score and the second score. For example, if the first score is "4", the second score is "7", the weighting coefficient of the first score is "W1", and the weighting coefficient of the second score is "W2", the final score is calculated as 4×W1+7×W2. According to such a calculation, the calculation becomes simple and the calculation process can be performed independently.
[0031] On the other hand, the score output unit 108 may calculate a final score for each of a plurality of EDU vectors by weighting and adding the weights obtained by the first calculation method and the frequencies obtained by the second calculation method. For example, if weights "3" and "1" are obtained by the first calculation method and frequencies "5" and "2" are obtained by the second calculation method for each of two EDU vectors, the final score is calculated as (3×W1+5×W2)+(1×W1+2×W2). According to such a calculation, the importance of each EDU is emphasized, and a clear scoring result can be obtained.
[0032] On the other hand, the score output unit 108 may calculate a final score for each of a plurality of EDU vectors by multiplying the weight obtained by the first calculation method by the frequency obtained by the second calculation method. That is, the score output unit 108 may add up the frequency obtained by the second calculation method using the value obtained by the second calculation method as a weighting. For example, when weights "3" and "1" are obtained by the first calculation method and frequencies "5" and "2" are obtained by the second calculation method for each of two EDU vectors, the final score is calculated as (3 x 5) + (1 x 2).
[0033] The operation of the server 100 configured as above will be described.
[0034] First, the operation of server 100 when the first calculation method is used will be described. Fig. 7 is a flowchart showing the procedure of pre-processing by server 100, and Fig. 8 is a flowchart showing the procedure of scoring processing by server 100.
[0035] When the pre-processing is started, the data receiving unit 101 acquires a plurality of pieces of answer data for a plurality of questions (step S101). Next, the analysis unit 102 divides the acquired plurality of pieces of answer data into a plurality of EDUs (step S102).
[0036] Thereafter, a plurality of EDU vectors are generated based on the plurality of EDUs divided by the vector acquisition unit 103 (step S103). Furthermore, the plurality of generated EDU vectors are clustered into a predetermined number of clusters by the clustering unit 104 (step S104).
[0037] Next, the clustering unit 104 generates frequency data based on the clustering results of the multiple EDU vectors (step S105). After that, the weight generating unit 105 generates weight information indicating the weight for each cluster and each question based on the generated frequency data, and stores the weight information in the weight information storage unit 111 (step S106).
[0038] When the marking process of the answer data for the target question is started after the pre-processing is completed, the multiple EDU vectors generated based on the answer data are clustered in the same procedure as steps S101 to S104 (steps S201 to S204). Next, the score calculation unit 107 obtains weights associated with the clusters to which the multiple EDU vectors are clustered and the target question from the weight information storage unit 111 (step S205).
[0039] Then, the score calculation unit 107 calculates a first score for the corresponding answer data by summing up the weights acquired for each of the EDU vectors (step S206). Finally, the score output unit 108 outputs the first score to the terminal device 10 as the final score for the answer data (step S207).
[0040] Next, the operation of the server 100 when the second calculation method is used will be described. Fig. 9 is a flowchart showing the procedure of the pre-processing by the server 100, and Fig. 10 is a flowchart showing the procedure of the scoring process by the server 100.
[0041] When the pre-processing is started, multiple EDU vectors are generated sequentially based on multiple answer data for the target question in the same procedure as steps S101 to S103 (steps S301 to S303). Next, the pyramid data generation unit 106 calculates the similarity between the EDU vectors generated sequentially from the multiple answer data and multiple representative vectors included in the pyramid score of the target question (step S304). Then, the pyramid data generation unit 106 determines whether or not there is a representative vector with a high calculated similarity (step S305).
[0042] As a result of the above judgment, if a representative vector with high similarity exists (step S305; YES), the pyramid data generation unit 106 adds "1" to the score of the representative vector in the pyramid score of the target problem (step S306). On the other hand, if a representative vector with high similarity does not exist (step S305; NO), the pyramid data generation unit 106 adds a score of "1" to the pyramid score of the target problem, with the corresponding EDU vector as a new representative vector (step S307).
[0043] Then, the pyramid data generating unit 106 determines whether all EDU vectors generated based on multiple answer data have been processed (step S308). If all EDU vectors have been processed (step S308; YES), the pre-processing is terminated, and if all EDU vectors have not been processed (step S308; NO), the process returns to step S303, and the comparison of the EDU vectors with the representative vectors is repeated.
[0044] When the marking process of the answer data for the target question is started after the pre-processing is completed, multiple EDU vectors are generated based on the answer data in the same procedure as steps S101 to S103 (steps S401 to S403). Next, the score calculation unit 107 searches the pyramid score for representative vectors that are highly similar to each of the multiple EDU vectors, and obtains the scores associated with the representative vectors that are highly similar from the pyramid data storage unit 112 (step S404).
[0045] Thereafter, the score calculation unit 107 sums up the scores acquired for each of the EDU vectors to calculate a second score for the corresponding answer data (step S405). Finally, the score output unit 108 outputs the second score to the terminal device 10 as the final score for the answer data (step S406).
[0046] Here, the server 100 may operate to calculate a final score by taking the first score and the second score into consideration. Fig. 11 is a flowchart showing another procedure of the scoring process by the server 100.
[0047] When the marking process of the answer data for the target question is started after the pre-processing shown in Figures 7 and 9 is completed, multiple EDU vectors are generated based on the answer data in a procedure similar to steps S101 to S103 (steps S501 to S503). Next, similar to step S404, the scores of the representative vectors that are highly similar to each of the multiple EDU vectors are obtained (step S504). In parallel with this, similar to steps 204 and S205, weights are obtained for each of the multiple EDU vectors (steps S505 and S506).
[0048] After that, the score output unit 108 performs a weighted addition of the scores obtained for each of the EDU vectors and the weights obtained for each of the EDU vectors (step S507).Then, the score output unit 108 sums up the weighted sums for all EDUs to calculate a final score (step S508).Finally, the score output unit 108 outputs the final score obtained for the answer data to the terminal device 10 (step S509).
[0049] Next, a modified example of the present disclosure will be described.
[0050] In the above disclosure, the server 100 functions to obtain a marking result for answer data to a predetermined question among a plurality of predetermined questions. As a modified example, the server 100 may function to obtain a marking result for answer data to an unknown question.
[0051] That is, the server 100 functions to execute a marking process on answer data for questions for which frequency data and weight information have not been generated in advance by pre-processing. The server 100 estimates questions that have a similar cluster frequency distribution in the frequency data based on the clustering results of multiple EDUs included in multiple answer data for a new question, determines the questions as pseudo question numbers, and executes a marking process.
[0052] More specifically, the server 100 generates multiple EDU vectors based on multiple answer data for unknown questions, in the same manner as in the above-described embodiment. Thereafter, the clustering unit 104 clusters the multiple EDU vectors and counts the frequency for each of a predetermined number of clusters, in the same manner as in the above-described embodiment. Furthermore, the clustering unit 104 calculates the similarity between the counted frequency for each of the predetermined number of clusters and the frequency data for each question generated by the clustering unit 104 in pre-processing. Then, the clustering unit 104 determines the question number of the question with the highest similarity among all questions included in the frequency data as the pseudo question number.
[0053] Furthermore, the score calculation unit 107 uses the determined pseudo question number to calculate a first score using the target answer data in the same manner as in the above-described embodiment.
[0054] According to the above modification, when answer data to an unknown question is targeted, there is no need to execute the pre-processing again, and the amount of calculation can be reduced.
[0055] Next, the effect of the server 100, which is the automatic scoring device of the present disclosure, will be described. According to the server 100 of the present disclosure, multiple document data for a specific target question are divided into EDUs, and the divided multiple EDUs are vectorized into multiple EDU vectors. Then, the multiple EDU vectors are compared in order, and the frequencies of multiple EDU vectors that are similar to each other are added, thereby generating a pyramid score that represents a score based on the frequency of each of multiple representative vectors. Furthermore, for document data for a specific target question, the scores in the pyramid score for representative vectors that are similar to the EDU vector based on that document data are added up to calculate the scoring result of the document data.
[0056] By performing such operations, when document data for a new target question is set as a target for scoring, multiple document data that are answer data for the target question are acquired in advance, and a pyramid score is acquired using these multiple document data, thereby efficiently obtaining a scoring result for the document data for the target question. In particular, by generating a pyramid score by aggregating multiple EDU vectors based on multiple document data for the target question into multiple representative vectors and aggregating the frequency, the importance of EDUs contained in the document data can be accurately reflected in the scoring result. As a result, when document data for a new target question is targeted, there is no need to acquire a model answer for the target question, and therefore a highly accurate scoring result can be efficiently obtained.
[0057] Here, it is also assumed that a mechanism for automatic grading using machine learning is used. In other words, a combination of answer data for a question and its grading result is used as teacher data, and a learning model is constructed by machine learning, and ungraded answer data is input to the constructed learning model to obtain a grading result. However, in this case, in order to obtain an accurate grading result, it is necessary to prepare a grading standard, obtain grading results for multiple answer data, and prepare a lot of training data, which is time-consuming. In addition, in order to accurately realize automatic grading of answer data for a new question, it is also necessary to construct a learning model corresponding to the content of the question each time. In the present disclosure, it is only necessary to obtain multiple document data for the target question, so the time and effort required for preparing training data and constructing a learning model is reduced.
[0058] Here, in the present disclosure, the server 100 divides the document data for the target question into multiple EDUs, vectorizes the divided multiple EDUs to generate multiple EDU vectors, and calculates and outputs the grading result of the document data by using the multiple EDU vectors. According to the present disclosure, when the document data for the target question is targeted, a grading result that comprehensively evaluates the importance of multiple EDUs included in the document data can be obtained. This makes it possible to efficiently obtain a highly accurate grading result for the document data for the target question.
[0059] In the present disclosure, the server 100 judges the similarity between a plurality of EDU vectors and a representative vector based on the distance between the vectors. This makes it possible to obtain highly accurate scoring results for document data for a target question.
[0060] In the present disclosure, according to the server 100, multiple document data for multiple questions are divided into EDUs, and the divided multiple EDUs are vectorized into multiple EDU vectors. Then, the multiple EDU vectors are clustered into a predetermined number of clusters, and frequency data is generated by counting the frequency of the clustered EDU vectors for each of the multiple questions and for each of the predetermined number of clusters. Furthermore, a weight (importance) for each of the predetermined number of clusters is calculated based on the frequency data, and a scoring result for the document data for one question is calculated by adding up the weights of the clusters corresponding to the EDU vectors based on the document data.
[0061] With this operation, even when document data for an arbitrary question among a plurality of questions is to be graded, by acquiring a plurality of document data that are answers to the plurality of questions in advance and acquiring the importance of each cluster using the plurality of document data, it is possible to efficiently obtain the graded result of the document data for the arbitrary question. In particular, by clustering a plurality of EDU vectors based on a plurality of document data for a plurality of questions and acquiring weights for each of the plurality of clusters, it is possible to accurately reflect the importance of the EDUs contained in the document data in the graded result. As a result, when document data for an arbitrary question is the target, there is no need to acquire model answers to the plurality of questions, so it is possible to efficiently obtain a graded result with high accuracy.
[0062] Here, it is also assumed that a mechanism for automatic grading using machine learning is used. That is, a combination of answer data for a question and its grading result is used as teacher data, and a learning model is constructed by machine learning, and ungraded answer data is input to the constructed learning model to obtain a grading result. However, in this case, in order to obtain an accurate grading result, it is necessary to prepare a grading standard, obtain grading results for a plurality of answer data, and prepare a lot of training data, which is time-consuming. In addition, in order to realize accurate automatic grading of answer data for a plurality of questions, it is also necessary to construct a learning model corresponding to the content of the question each time. In the present disclosure, it is only necessary to obtain a plurality of document data for a plurality of questions, so that the time and effort for preparing training data and constructing a learning model is reduced.
[0063] In addition, in the present disclosure, the server 100 divides document data for one question into multiple EDUs to be graded, vectorizes the divided multiple EDUs, clusters them into a predetermined number of clusters, and adds up the weights of the clusters to which the multiple EDUs are clustered, thereby calculating and outputting the graded result of the document data. According to the present disclosure, when document data for one of multiple questions is targeted, a graded result that comprehensively evaluates the importance of multiple EDUs included in the document data can be obtained. This makes it possible to efficiently obtain a highly accurate graded result for document data for one question.
[0064] In the present disclosure, the server 100 calculates the importance-based weight for each cluster using the frequency ratio of EDU vectors clustered to that cluster in one question based on frequency data.
[0065] This makes it possible to set the weight for each cluster so that a higher weight is assigned to a cluster of EDUs that are contained in a larger number in the answer data for a single question, thereby enabling more accurate scoring results to be obtained for document data that is the answer data for a single question.
[0066] In addition, in the present disclosure, the server 100 calculates the weight for each cluster using the inverse of the sum of the frequencies of EDU vectors in that cluster across multiple questions, based on frequency data.
[0067] This makes it possible to set the weight for each cluster so that a lower weight is assigned to a cluster of EDUs that are contained more frequently in answer data to multiple questions, thereby enabling more accurate scoring results to be obtained for document data that is the answer data to a single question.
[0068] Furthermore, in the present disclosure, the server 100 calculates the weight for each cluster by taking into account the rate of the frequencies and the inverse of the sum of the frequencies.
[0069] This makes it possible to set the weight for each cluster so that a cluster of EDUs that is contained more frequently in the answer data for one question has a higher weight, and a cluster of EDUs that is contained more frequently across answer data for multiple questions has a lower weight, thereby enabling more accurate scoring results to be obtained for document data that is the answer to a single question.
[0070] In addition, in the present disclosure, the server 100 calculates the scoring result of the document data for a specific target question by taking into account the weights for each of a predetermined number of clusters and the score in the pyramid score for the representative vector. This makes it possible to accurately obtain scoring results that correspond to both the case where a large amount of answer data is obtained for multiple questions and the case where a large amount of answer data is obtained only for a specific target question.
[0071] In addition, in the present disclosure, the server 100 calculates the scoring result of the document data for a specific target question by weighting and adding a value based on the weights for a predetermined number of clusters and a value based on the score in the pyramid score for the representative vector. This makes it possible to accurately obtain a scoring result that flexibly deals with both cases, that is, when a large amount of answer data for multiple questions is obtained and when a large amount of answer data for only a specific target question is obtained.
[0072] The automatic marking device, which is the server 100 of the present invention, has the following configuration.
[0073] [1] An automatic marking device that outputs a marking result of document data that is an answer to a question, the device comprising at least one processor, wherein the at least one processor divides a plurality of document data for a specific target question into semantic units, vectorizes the divided semantic units to generate a plurality of semantic unit vectors, sequentially compares the semantic unit vectors, and adds up the frequencies of the semantic unit vectors that are similar to each other to generate pyramid data that indicates a score based on the frequency of each of a plurality of representative vectors among the plurality of semantic unit vectors, and calculates and outputs a marking result of the document data for the specific target question by adding up the scores in the pyramid data related to representative vectors that are similar to the plurality of semantic unit vectors generated from the document data. Automatic scoring device.
[0074] [2] The at least one processor divides document data for the specific target problem into a plurality of semantic units, vectorizes the divided semantic units to generate a plurality of semantic unit vectors, and calculates and outputs a scoring result for the document data by using the plurality of semantic unit vectors. The automatic scoring device described in [1] above.
[0075] [3] The at least one processor determines similarity between a plurality of semantic unit vectors and the representative vector based on a distance between the vectors. The automatic scoring device described in [1] or [2] above.
[0076] [4] the at least one processor divides a plurality of document data for a plurality of questions into semantic units, vectorizes the divided semantic units to generate a plurality of semantic unit vectors, clusters the plurality of semantic unit vectors into a predetermined number of clusters, generates frequency data by counting the frequency of the clustered semantic unit vectors for each of the plurality of questions and for each of the predetermined number of clusters, calculates an importance for each of the predetermined number of clusters based on the frequency data, and calculates and outputs a scoring result for document data for one of the plurality of questions by adding up weights based on the importance for each of the predetermined number of clusters. The automatic scoring device according to any one of the above [1] to [4].
[0077] [5] the at least one processor divides document data for the one question into a plurality of semantic units, vectorizes the divided plurality of semantic units, and then clusters the plurality of semantic units into the predetermined number of clusters; and calculates and outputs a scoring result for the document data by adding up weights based on the importance of each of the clusters into which the plurality of semantic units are clustered. The automatic scoring device described in [4] above.
[0078] [6] The at least one processor calculates the weight based on the importance of each cluster using a frequency ratio of the semantic unit vectors clustered in the cluster in the one problem based on the frequency data. The automatic scoring device described in [4] or [5] above.
[0079] [7] The at least one processor calculates the weight based on the importance of each cluster using an inverse of a sum of frequencies of the semantic unit vectors in the cluster across the multiple problems based on the frequency data. The automatic scoring device described in [4] or [5] above.
[0080] [8] The at least one processor calculates the weight based on the importance of each cluster by taking into account the frequency ratio and an inverse of the sum of the frequency of the semantic unit vectors in the cluster across the multiple problems, based on the frequency data. The automatic scoring device described in [6] above.
[0081] [9] The at least one processor calculates a scoring result of the document data for the specific target problem by taking into account weights based on the importance of each of the predetermined number of clusters and the scores in the pyramid data for the representative vectors. The automatic scoring device according to any one of [4] to [8] above.
[10] The at least one processor calculates a scoring result of the document data for the specific target problem by weighting and adding a value based on the weight based on the importance of each of the predetermined number of clusters and a value based on the score in the pyramid data for the representative vector. The automatic scoring device described in [9] above.
[0082] The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wireless, etc.). The functional block may be realized by combining the one device or the multiple devices with software.
[0083] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.
[0084] For example, the server 100 in one embodiment of the present disclosure may function as a computer that performs processing of the automatic scoring method of the present disclosure. FIG. 12 is a diagram showing an example of a hardware configuration of the server 100 according to one embodiment of the present disclosure. The above-mentioned server 100 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. Note that the server 100 may be configured as a computer device including at least one processor, may be configured as a computer device including multiple processors, or may be configured to include multiple computer devices.
[0085] In the following description, the term "apparatus" may be replaced with a circuit, a device, a unit, etc. The hardware configuration of the server 100 may be configured to include one or more of the apparatuses shown in the drawings, or may be configured to exclude some of the apparatuses.
[0086] Each function of server 100 is realized by loading a specific software (program) onto hardware such as processor 1001 and memory 1002, causing processor 1001 to perform calculations, control communications via communication device 1004, and control at least one of reading and writing of data in memory 1002 and storage 1003.
[0087] The processor 1001 controls the entire computer by running an operating system, for example. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned data reception unit 101, analysis unit 102, vector acquisition unit 103, clustering unit 104, weight generation unit 105, pyramid data generation unit 106, score calculation unit 107, score output unit 108, etc. may be realized by the processor 1001.
[0088] Moreover, the processor 1001 reads out a program (program code), a software module, data, etc. from at least one of the storage 1003 and the communication device 1004 to the memory 1002, and executes various processes according to the programs. As the programs, a program that causes a computer to execute at least a part of the operations described in the above-mentioned embodiment is used. For example, the data receiving unit 101, the analysis unit 102, the vector acquisition unit 103, the clustering unit 104, the weight generation unit 105, the pyramid data generation unit 106, the score calculation unit 107, and the score output unit 108 may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may be similarly realized. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed by two or more processors 1001 simultaneously or sequentially. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via an electric communication line.
[0089] The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software module, etc. that can be executed to implement the automatic scoring method according to one embodiment of the present disclosure.
[0090] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.
[0091] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., in order to realize at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD). For example, the above-mentioned data acceptance unit 101, score output unit 108, etc. may be realized by the communication device 1004.
[0092] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).
[0093] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.
[0094] Furthermore, the server 100 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc., and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.
[0095] The notification of information is not limited to the aspects / embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination thereof. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.
[0096] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0097] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.
[0098] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).
[0099] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).
[0100] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.
[0101] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
[0102] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.
[0103] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0104] In addition, the terms described in this disclosure and the terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channel and the symbol may be a signal (signaling). Also, the signal may be a message. Also, the component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.
[0105] In addition, the information, parameters, etc. described in the present disclosure may be represented using absolute values, may be represented using relative values from a predetermined value, or may be represented using other corresponding information. For example, a radio resource may be indicated by an index.
[0106] The names used for the above-mentioned parameters are not limiting in any way. Moreover, the formulas using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not limiting in any way.
[0107] In this disclosure, the terms "Mobile Station (MS)", "user terminal", "User Equipment (UE)", "terminal", etc. may be used interchangeably.
[0108] A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.
[0109] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.
[0110] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.
[0111] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0112] Any reference to an element using a designation such as "first", "second" etc. used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.
[0113] When "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.
[0114] In this disclosure, where articles have been added due to translation, such as, for example, a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.
[0115] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]
[0116] 100...server (automatic scoring device), 1001...processor, 101...data receiving unit, 102...analysis unit, 103...vector acquisition unit, 104...clustering unit, 105...weight generation unit, 106...pyramid data generation unit, 107...score calculation unit, 108...score output unit, 109...vector storage unit, 110...cluster information storage unit, 111...weight information storage unit, 112...pyramid data storage unit, CL1 to CL5...clusters, Q1 to Q10...questions, A1 to 10...answer data (document data), V1 to V10...EDU (semantic unit) vectors, RV1, RV2, RV3, RV4...representative vectors.
Claims
1. An automatic marking device that outputs a marking result of document data which is an answer to a question, At least one processor; The at least one processor Divide multiple document data for a specific target problem into semantic units, vectorizing the divided semantic units to generate a plurality of semantic unit vectors; comparing the plurality of semantic unit vectors in order and adding up the frequencies of the plurality of semantic unit vectors that are similar to each other, thereby generating pyramid data representing a score based on the frequency of each of a plurality of representative vectors among the plurality of semantic unit vectors; calculating and outputting a scoring result for the document data for the specific target problem by summing up the scores in the pyramid data for representative vectors similar to a plurality of semantic unit vectors generated from the document data; Automatic scoring device.
2. The at least one processor Dividing document data for the specific target question into a plurality of semantic units, vectorizing the plurality of semantic units to generate a plurality of semantic unit vectors, and calculating and outputting a scoring result for the document data by using the plurality of semantic unit vectors. The automatic scoring device according to claim 1.
3. The at least one processor determining the similarity between a plurality of semantic unit vectors and the representative vector based on the distance between the vectors; The automatic scoring device according to claim 1.
4. The at least one processor Divide multiple document data for multiple problems into semantic units, vectorizing the divided semantic units to generate a plurality of semantic unit vectors; Clustering the plurality of semantic unit vectors into a predetermined number of clusters; generating frequency data by counting the frequency of the clustered semantic unit vectors for each of the plurality of questions and for each of the predetermined number of clusters; Calculating the importance of each of the predetermined number of clusters based on the frequency data; calculating and outputting a scoring result for document data for one of the plurality of questions by adding up weights based on the importance of each of the predetermined number of clusters; The automatic scoring device according to claim 1.
5. The at least one processor Dividing document data for the one question into a plurality of semantic units, vectorizing the divided plurality of semantic units and then clustering them into the predetermined number of clusters, and calculating and outputting a scoring result for the document data by adding up weights based on the importance of each of the clusters into which the plurality of semantic units are clustered.
5. The automatic scoring device according to claim 4.
6. The at least one processor Calculating the weight based on the importance of each cluster using a frequency ratio of the semantic unit vectors clustered into the cluster in the one question based on the frequency data; 6. The automatic scoring device according to claim 5.
7. The at least one processor The weight based on the importance of each cluster is calculated using the inverse of the sum of the frequency of the semantic unit vectors in the cluster across the multiple questions, based on the frequency data.
6. The automatic scoring device according to claim 5.
8. The at least one processor Calculating the weight based on the importance of each cluster by taking into account the frequency ratio and the inverse of the sum of the frequency of the semantic unit vectors in the cluster across the multiple questions, based on the frequency data; 7. The automatic scoring device according to claim 6.
9. The at least one processor Calculating a scoring result of the document data for the specific target problem by taking into account weights based on the importance of each of the predetermined number of clusters and the scores in the pyramid data for the representative vectors.
5. The automatic scoring device according to claim 4.
10. The at least one processor Calculating the scoring result of the document data for the specific target question by weighting and adding a value based on the weight based on the importance of each of the predetermined number of clusters and a value based on the score in the pyramid data for the representative vector. The automatic scoring device according to claim 9.
Citation Information
Patent Citations
Automatic sentence point rating system
JP2017129830A
Scoring device, scoring method, and program
JP2020160159A
