Open question answer scoring method, apparatus, device, and storage medium
By extracting features and parsing intent from open-ended questions and answers, and combining similarity and intent vector scoring models, the problems of low efficiency and poor accuracy in scoring open-ended answers are solved, achieving fast and accurate scoring results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI MIYUE ARTIFICIAL INTELLIGENCE INFORMATION TECH CO LTD
- Filing Date
- 2021-12-29
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, the grading of open-ended answers is inefficient and inaccurate, making it impossible to accurately determine whether an answer meets the requirements of the question, resulting in unfair scoring.
By acquiring target test questions and answers, word and sentence features are extracted, semantic similarity and character similarity are calculated, and pre-trained intent parsing and scoring models are used to determine the intent vector of the answer. Finally, the answer score is determined by combining multiple factors.
It enables rapid and accurate scoring of answers to open-ended questions, improves marking efficiency, and reduces the time and subjective bias of manual marking.
Smart Images
Figure CN114328874B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The embodiment of the present application relates to the technical field of intelligent education, and particularly relates to an open question answer scoring method and device, equipment and a storage medium. BACKGROUND
[0002] In daily teaching and examination, there are a large number of open answer questions, such as paragraph appreciation, summarizing the article intention and the like. At present, the correction of open answers mainly relies on manual work of teaching experts, which is extremely low in efficiency, and subjective factors of different scorers can lead to unfair scoring. The existing open question scoring model can only perform simple text recognition and logical judgment on the answers, and cannot accurately determine whether the answers meet the question, so the accuracy of scoring is not high. SUMMARY
[0003] The present application provides an open question answer scoring method, device, equipment and storage medium to realize fast and accurate scoring of open question answers.
[0004] In a first aspect, the embodiment of the present application provides an open question answer scoring method, comprising:
[0005] Obtaining a target question and a target answer, and performing word and sentence feature extraction processing on the target question and the target answer respectively to obtain a question feature set and an answer feature set;
[0006] Measuring the similarity of the question feature set and the answer feature set to obtain a semantic similarity and a character similarity;
[0007] Using a pre-trained target intention analysis model to determine a question intention vector of the question feature set and an answer intention vector of the answer feature set;
[0008] According to the semantic similarity, the character similarity, the question intention vector and the answer intention vector, and combining a pre-trained target scoring model, determining an answer score of the target answer.
[0009] Optionally, the word and sentence feature extraction processing on the target question and the target answer respectively to obtain the question feature set and the answer feature set comprises:
[0010] Performing sentence embedding operation on the target question and the target answer respectively to obtain a question sentence embedding vector and an answer sentence embedding vector;
[0011] Performing Chinese word segmentation operation on the target question to obtain a question Chinese word segmentation set and a question English word segmentation set, and performing word embedding operation on the question Chinese word segmentation in the question Chinese word segmentation set to obtain a question word embedding sequence;
[0012] The target answer is segmented to obtain a Chinese word segmentation set and an English word segmentation set. The Chinese word segments in the Chinese word segmentation set are then embedded to obtain an answer word embedding sequence.
[0013] The question sentence embedding vector, question word embedding sequence, Chinese word segmentation set, and English word segmentation set constitute the question feature set, and the answer sentence embedding vector, answer word embedding sequence, Chinese word segmentation set, and English word segmentation set constitute the answer feature set.
[0014] Optionally, the step of measuring the similarity between the question feature set and the answer feature set to obtain semantic similarity and character similarity includes:
[0015] The semantic similarity is determined based on the embedding vectors of the question sentences and the embedding vectors of the answer sentences;
[0016] The similarity of Chinese characters between the Chinese word segmentation set of the test questions and the Chinese word segmentation set of the answers is measured to obtain the Chinese character similarity. Similarly, the similarity of English characters between the English word segmentation set of the test questions and the English word segmentation set of the answers is measured to obtain the English character similarity.
[0017] The character similarity is determined based on the similarity between the Chinese characters and the English characters.
[0018] Optionally, the step of using a pre-trained target intent parsing model to determine the question intent vector of the question feature set and the answer intent vector of the answer feature set includes:
[0019] The target question intent parsing network in the target intent parsing model is used to parse the question word embedding sequence to obtain the question intent vector;
[0020] The target intent parsing network in the target intent parsing model is used to parse the intent of the answer word embedding sequence to obtain the answer intent vector.
[0021] Optionally, the training process of the target intent parsing model includes:
[0022] Construct a trainable intent parsing model that includes a question intent parsing network, an answer intent parsing network, and an intent classifier, and initialize the number of intent categories in the intent classifier to a preset number of categories;
[0023] The intent parsing model to be trained is trained based on the training word embedding data, and the model parameters of the intent parsing model to be trained are updated.
[0024] The updated intent parsing model to be trained is tested based on the test word embedding data, and the number of intent categories in the intent classifier is adjusted.
[0025] Return to retraining and testing the intent parsing model to be trained, and determine the adjustment trend of the number of intent categories. When the adjustment trend meets the adjustment stability condition, the last updated intent parsing model to be trained is determined as the target intent parsing model.
[0026] Optionally, training the intent parsing model based on training word embedding data and updating the model parameters of the intent parsing model includes:
[0027] Obtain a preset number of training word embedding data sets, input each training word embedding data set into the intent parsing model to be trained for training, and update the model parameters of the intent parsing model to be trained. The training word embedding data sets include training question word embedding sequences and training answer word embedding sequences.
[0028] Return to the operation of retrieving the preset number of training word embeddings and updating the model parameters until the preset number of updates is reached.
[0029] Optionally, the step of inputting the embedding data of each of the training words into the intent parsing model to be trained for training, and updating the model parameters of the intent parsing model to be trained, includes:
[0030] For each set of training word embedding data, the test question intent parsing network is used to determine the training test question intent vector of the training test question word embedding sequence, and the answer intent parsing network is used to determine the training answer intent vector of the training answer word embedding sequence. The training test question intent vector and the training answer intent vector are classified according to the intent classifier to obtain the training test question intent type and the training answer intent type.
[0031] Based on the intent type of each training question and the corresponding intent type of the training answer, the intent fitting loss function is obtained;
[0032] The intent fitting loss function is used to backpropagate the intent parsing model to be trained, thereby updating the model parameters of the intent parsing model to be trained.
[0033] Optionally, the step of testing the updated intent parsing model based on test word embedding data and adjusting the number of intent categories in the intent classifier includes:
[0034] Obtain a preset number of test word embedding data sets, and input each of the test word embedding data sets into the updated intent parsing model to be trained. The test word embedding data sets include test question word embedding sequences and test answer word embedding sequences.
[0035] For each set of test word embedding data, the test question intent vector of the test question word embedding sequence is determined using the test question intent parsing network, and the test answer intent vector of the test answer word embedding sequence is determined using the answer intent parsing network.
[0036] The distribution of vector distances is determined based on the Euclidean distance between the intent vectors of each test question and the corresponding intent vectors of test answers.
[0037] When the vector distance distribution meets the preset distribution conditions, the number of intent categories in the intent classifier is reduced by one; otherwise, the number of intent categories in the intent classifier is increased by one.
[0038] Optionally, the training process of the target scoring model includes:
[0039] The training feature combination data is scored and labeled to obtain the standard score corresponding to the training feature combination data, wherein the training feature combination data includes standard semantic similarity, standard character similarity, standard question intent vector and standard answer intent vector;
[0040] The training feature combination data is input into the scoring model to be trained to obtain the actual output score.
[0041] Based on the standard score and the actual score, a score fitting loss function is obtained;
[0042] The scoring model is obtained by backpropagating the training scoring model using the scoring fitting loss function.
[0043] Secondly, embodiments of the present invention also provide an open-ended test answer scoring device, the device comprising:
[0044] The language processing module is used to acquire the target test question and the target answer, and to perform word and sentence feature extraction processing on the target test question and the target answer respectively to obtain the test question feature set and the answer feature set;
[0045] The similarity determination module is used to measure the similarity between the question feature set and the answer feature set to obtain semantic similarity and character similarity.
[0046] The intent recognition module is used to determine the question intent vector of the question feature set and the answer intent vector of the answer feature set by using a pre-trained target intent parsing model.
[0047] The scoring determination module is used to determine the answer score of the target answer based on the semantic similarity, the character similarity, the question intent vector, and the answer intent vector, combined with a pre-trained target scoring model.
[0048] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the open-ended question answer scoring method as described in any embodiment of the present invention.
[0049] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the open-ended question answer scoring method as described in any embodiment of the present invention.
[0050] This invention acquires target test questions and target answers, and performs word and sentence feature extraction on both to obtain test question feature sets and answer feature sets. It then measures the similarity between the test question feature sets and answer feature sets to obtain semantic similarity and character similarity. A pre-trained target intent parsing model is used to determine the test question intent vector of the test question feature set and the answer intent vector of the answer feature set. Based on semantic similarity, character similarity, test question intent vector, and answer intent vector, combined with a pre-trained target scoring model, the answer score of the target answer is determined. The open-ended test question answer scoring method provided by this invention can be integrated into smart devices, allowing users to quickly and accurately score answers to open-ended test questions, saving marking time and improving marking efficiency. Attached Figure Description
[0051] Figure 1 This is a flowchart of an open-ended test answer scoring method provided in Embodiment 1 of the present invention;
[0052] Figure 2 This is a flowchart illustrating the training process of the target intent parsing model in an open-ended test answer scoring method provided in Embodiment 1 of the present invention.
[0053] Figure 3 This is a structural block diagram of an open-ended test answer scoring device provided in Embodiment 2 of the present invention;
[0054] Figure 4 This is a structural block diagram of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation
[0055] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not all structures. Moreover, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0056] Example 1
[0057] Figure 1 This is a flowchart of an open-ended question answer scoring method provided in Embodiment 1 of the present invention. This embodiment is applicable to scoring the answers to open-ended questions. The method can be executed by an open-ended question answer scoring device, which can be implemented by software and / or hardware.
[0058] like Figure 1 As shown, the method specifically includes the following steps:
[0059] Step 110: Obtain the target test questions and target answers, and perform word and sentence feature extraction processing on the target test questions and target answers respectively to obtain the test question feature set and the answer feature set.
[0060] In this embodiment, the open-ended answer that needs to be scored can be called the target answer, and the open-ended question corresponding to the answer can be called the target question.
[0061] In practical applications, target questions and answers may be entire segments of text data, the meaning of which cannot be directly read by a computer. Therefore, it is necessary to break down the target questions and answers into data that the machine can further use for calculation. Natural language processing methods such as word and sentence decomposition, word embedding, and sentence embedding can be used to extract a set of features from the target questions and answers that contain potential information beyond the meaning of the characters themselves. In this embodiment, the feature set extracted from the target questions can be called the question feature set, and the feature set extracted from the target answers can be called the answer feature set.
[0062] Optionally, step 110 can be implemented in the following specific ways:
[0063] S1101. Obtain the target question and the target answer. Perform sentence embedding operations on the target question and the target answer respectively to obtain the question sentence embedding vector and the answer sentence embedding vector.
[0064] In this embodiment, any sentence embedding model can be selected to perform sentence embedding operations on the target question and the target answer to obtain the question sentence embedding vector and the answer sentence embedding vector.
[0065] S1102. Perform word segmentation on the target test question to obtain a Chinese word segmentation set and an English word segmentation set. Perform word embedding on the Chinese word segmentation set to obtain a word embedding sequence.
[0066] In this embodiment, any word embedding model can be selected to perform word embedding operations on the Chinese word segmentation of the test questions. For example, the BERT (Bidirectional Encoder Representations from Transformers) model can be used to perform word embedding operations on the Chinese word segmentation of the test questions to obtain the word embedding sequence of the test questions.
[0067] S1103. Perform word segmentation on the target answer to obtain the Chinese word segmentation set and the English word segmentation set of the answer. Perform word embedding on the Chinese word segmentation set of the answer to obtain the word embedding sequence of the answer.
[0068] Similar to word embedding operations on Chinese word segmentation of test questions, any word embedding model can be selected to perform word embedding operations on Chinese word segmentation of the answer. For example, word embedding operations can be performed on Chinese word segmentation of the answer using the BERT model to obtain the answer word embedding sequence.
[0069] S1104. The question feature set is constructed by embedding the question sentence into a vector, embedding the question word into a sequence, and forming the Chinese word segmentation set and the English word segmentation set of the question. The answer feature set is constructed by embedding the answer sentence into a vector, embedding the answer word into a sequence, forming the Chinese word segmentation set and the English word segmentation set of the answer.
[0070] Step 120: Measure the similarity between the question feature set and the answer feature set to obtain semantic similarity and character similarity.
[0071] Specifically, the similarity between target questions and target answers can be measured from both sentence and text perspectives by calculating the semantic and character similarity between the question feature set and the answer feature set.
[0072] Optionally, step 120 can be implemented in the following specific ways:
[0073] S1201. Determine the semantic similarity based on the embedding vectors of the question sentences and the answer sentences.
[0074] In this embodiment, any semantic similarity calculation method can be used to calculate semantic similarity. For example, the cosine theorem can be used to calculate the similarity between two vectors to obtain semantic similarity.
[0075] S1202. Measure the similarity of Chinese characters between the Chinese word segmentation sets of the test questions and the Chinese word segmentation sets of the answers to obtain the Chinese character similarity. Measure the similarity of English characters between the English word segmentation sets of the test questions and the English word segmentation sets of the answers to obtain the English character similarity.
[0076] In this embodiment, any character similarity calculation method can be used to calculate the similarity between Chinese and English word segments. For example, the Jacard similarity between the Chinese word segmentation sets of the test questions and the Chinese word segmentation sets of the answers can be used to represent the similarity of Chinese characters; the Jacard similarity between the English word segmentation sets of the test questions and the English word segmentation sets of the answers can be used to represent the similarity of Chinese characters.
[0077] S1203. Determine the character similarity based on the similarity between Chinese characters and English characters.
[0078] Specifically, based on the specific usage scenario and the proportion of Chinese and English characters, the weights of Chinese character similarity and English character similarity in the overall character similarity can be preset. Based on the preset weights, the weighted sum of Chinese character similarity and English character similarity is obtained.
[0079] Step 130: Using a pre-trained target intent parsing model, determine the question intent vector of the question feature set and the answer intent vector of the answer feature set.
[0080] Specifically, the question feature set and answer feature set formed in step 110 can be input into a pre-trained target intent parsing model, which outputs a question intent vector and an answer intent vector. This proposal can determine whether a respondent's answer to a target question is off-topic by analyzing the intent of the target question and the target answer.
[0081] Optionally, step 130 can be implemented in the following specific ways:
[0082] The target question intent parsing network in the target intent parsing model is used to parse the intent of the question word embedding sequence to obtain the question intent vector; the target answer intent parsing network in the target intent parsing model is used to parse the intent of the answer word embedding sequence to obtain the answer intent vector.
[0083] The training process of the target intent parsing model in this embodiment may include:
[0084] A1. Construct a trainable intent parsing model that includes a question intent parsing network, an answer intent parsing network, and an intent classifier, and initialize the number of intent categories in the intent classifier to a preset number of categories.
[0085] Specifically, an intent parsing model to be trained can be constructed, and the model parameters of the question intent parsing network, answer intent parsing network, and intent classifier can be initialized. The number of intent categories can be a model parameter of the intent classifier, and the preset number of categories can be pre-set by the technician, for example, to 5 or 10.
[0086] A2. Train the intent parsing model based on the training word embedding data and update the model parameters of the intent parsing model.
[0087] In this embodiment, a large number of historical test questions and their corresponding reference answers or high-scoring answers can be selected, and word embedding operations can be performed on them to obtain corresponding word embedding sequences. Selecting reference answers or high-scoring answers can determine that the intent of the test questions and answers is consistent. The word embedding sequences of a large number of test questions and the word embedding sequences of the corresponding answers are used as training word embedding data and input into the intent parsing model to be trained, continuously optimizing the model and updating the model parameters of the intent parsing model to be trained.
[0088] Furthermore, step A2 can be implemented through the following specific steps: acquiring a preset number of training word embedding data sets, inputting each training word embedding data set into the intent parsing model to be trained for training, updating the model parameters of the intent parsing model to be trained, wherein the training word embedding data sets include training question word embedding sequences and training answer word embedding sequences; returning to the operation of acquiring a preset number of training word embedding data sets again and updating the model parameters, until the number of updates reaches the preset number of updates.
[0089] In this embodiment, the preset training count can be pre-set, such as 512. This means that 512 sets of training word embedding data are used to train the intent parsing model, completing one model optimization update. Each time parameters are updated, the update count is incremented by one, and then 512 sets of training word embedding data are acquired again to train the intent parsing model, until the preset update count is reached. Alternatively, the preset update count can be pre-set, such as 1000. In this case, a total of 512 * 1000 sets of training word embedding data are used to train the intent parsing model, and the current round of model training is complete.
[0090] Preferably, the process of inputting the embedded data of each training word into the intent parsing model to be trained for training and updating the model parameters of the intent parsing model can be specifically implemented in the following steps:
[0091] A21. For each set of training word embedding data, use the question intent parsing network to determine the training question intent vector of the training question word embedding sequence, and use the answer intent parsing network to determine the training answer intent vector of the training answer word embedding sequence. According to the intent classifier, perform intent classification on the training question intent vector and the training answer intent vector to obtain the training question intent type and the training answer intent type.
[0092] Specifically, during a model optimization and update process, the training question word embedding sequence from the training word embedding data can be input into the question intent parsing network to obtain the corresponding training question intent vector; similarly, the training answer word embedding sequence from the training word embedding data can be input into the answer intent parsing network to obtain the corresponding training answer intent vector. Inputting the training question intent vector and the corresponding training answer intent vector into the intent classifier will output the training question intent type and the training answer intent type.
[0093] A22. Based on the intent type of each training question and the corresponding intent type of the training answer, obtain the intent fitting loss function.
[0094] Specifically, since the selected training data consists of historical test questions and their corresponding reference answers or high-scoring answers, it can be assumed that the intent of the test questions and answers in the training data is basically consistent. Based on the intent types of each training test question and the corresponding intent types of the training answers output by the model, the existing errors are determined, and an intent fitting loss function is formed.
[0095] A23. Backpropagate the training intent parsing model using the intent fitting loss function to update the model parameters of the training intent parsing model.
[0096] Specifically, after obtaining the intent fitting loss function, the intent parsing model to be trained can be backpropagated using the intent fitting loss function to optimize the model parameters. A3. The updated intent parsing model to be trained is tested based on the test word embedding data, and the number of intent categories in the intent classifier is adjusted.
[0097] Furthermore, step A3 can be achieved through the following specific steps:
[0098] A31. Obtain the test word embedding data of the preset number of test groups, and input each test word embedding data into the updated intent parsing model to be trained. The test word embedding data includes the test question word embedding sequence and the test answer word embedding sequence.
[0099] Specifically, after one round of training on the intent parsing model, a batch of test word embedding data can be used to test the optimization level of the model. In this embodiment, the preset number of tests can be pre-set, such as 2000, meaning that 2000 sets of test word embedding data are used to test the intent parsing model, completing one model testing operation.
[0100] A32. For each set of test word embedding data, use the test question intent parsing network to determine the test question intent vector of the test question word embedding sequence, and use the answer intent parsing network to determine the test answer intent vector of the test answer word embedding sequence.
[0101] Specifically, during a model test, the test question word embedding sequence from the test word embedding data can be input into the test question intent parsing network to obtain the corresponding test question intent vector; the test answer word embedding sequence from the test word embedding data can be input into the answer intent parsing network to obtain the corresponding test answer intent vector.
[0102] A33. Determine the vector distance distribution based on the Euclidean distance between the intention vector of each test question and the intention vector of the corresponding test answer.
[0103] Specifically, the Euclidean distance between the intent vector of each test question and the intent vector of the corresponding test answer can be calculated. Based on the distribution of the Euclidean distances corresponding to all test word embeddings, the optimization status of the intent parsing model to be trained can be determined. When the vector distance distribution meets the preset distribution conditions, proceed to step A34; otherwise, proceed to A35. In this embodiment, the vector distance distribution can be compared with a normal distribution. When the vector distance distribution is more concentrated than a normal distribution, proceed to step A34; otherwise, proceed to A35.
[0104] A34. Decrease the number of intent categories in the intent classifier by one.
[0105] Specifically, the current number of intent categories is reduced by one to determine the new number of intent categories.
[0106] A35. Increment the number of intent categories in the intent classifier by one.
[0107] Specifically, the current number of intent categories is incremented by one to determine the new number of intent categories.
[0108] A4. Return to retrain and test the untrained intent parsing model, and determine the adjustment trend of the number of intent categories. When the adjustment trend meets the adjustment stability condition, determine the last updated untrained intent parsing model as the target intent parsing model.
[0109] Specifically, after each model testing operation, it can be determined whether the adjustment trend of the number of intent categories meets the adjustment stability condition. If not, return to step A2 to continue training the intent parsing model to be trained; if the adjustment stability condition is met, the last updated intent parsing model to be trained can be determined as the target intent parsing model. The adjustment stability condition can be that the number of intent categories repeatedly adjusts between two adjacent natural numbers to a preset number. At this point, the intent parsing model to be trained can be considered to have stabilized. For example, if the number of intent categories has been changing between 23 and 24 100 times, then the intent parsing model to be trained can be considered to have stabilized.
[0110] For example, Figure 2This is a flowchart illustrating the training process of the target intent parsing model in an open-ended test answer scoring method provided in Embodiment 1 of the present invention.
[0111] In this embodiment, there is no clear order between steps 120 and 130. In practical applications, steps 130 can be performed after step 110, followed by step 120, or steps 120 and 130 can be performed simultaneously after step 110.
[0112] Step 140: Based on semantic similarity, character similarity, question intent vector, and answer intent vector, and combined with the pre-trained target scoring model, determine the answer score of the target answer.
[0113] Specifically, the semantic similarity, character similarity, question intent vector, and answer intent vector obtained from the above steps can be input into a pre-trained target scoring model, and the output predicted score is the answer score of the target answer.
[0114] Optionally, the training process of the target scoring model in this embodiment may include:
[0115] B1. The training feature combination data is scored and labeled to obtain the standard score corresponding to the training feature combination data. The training feature combination data includes standard semantic similarity, standard character similarity, standard question intent vector, and standard answer intent vector.
[0116] Specifically, a large amount of training feature combination data can be obtained, and the data can be labeled according to the scores corresponding to the training feature combination data, that is, the scores of the answers corresponding to the training feature combination data are used as the standard scores.
[0117] B2. Input the training feature combination data into the scoring model to be trained to obtain the actual output score.
[0118] Specifically, all training feature combinations are input into the scoring model to be trained to obtain the corresponding actual scores.
[0119] B3. Based on the standard score and the actual score, obtain the score fitting loss function.
[0120] Specifically, since the standard score is pre-labeled and the actual score is predicted analytically by the scoring model to be trained, there will inevitably be an error between the standard score and the actual score. Based on the existing error, a scoring fitting loss function can be formed.
[0121] B4. The scoring model is obtained by backpropagating the training scoring model through the scoring fitting loss function.
[0122] Specifically, after obtaining the rating fitting loss function, the model parameters of the training rating model can be optimized by backpropagating the rating model to be trained using the rating fitting loss function, and the rating model can be obtained.
[0123] The technical solution of this embodiment involves acquiring the target test question and the target answer, extracting word and sentence features from both to obtain a test question feature set and an answer feature set, respectively; measuring the similarity between the test question feature set and the answer feature set to obtain semantic similarity and character similarity; using a pre-trained target intent parsing model to determine the test question intent vector of the test question feature set and the answer intent vector of the answer feature set; and determining the answer score of the target answer based on semantic similarity, character similarity, test question intent vector, and answer intent vector, combined with a pre-trained target scoring model. The open-ended test question answer scoring method provided by this embodiment can be integrated into smart devices, allowing users to quickly and accurately score answers to open-ended test questions, saving grading time and improving grading efficiency.
[0124] Example 2
[0125] The open-ended question answer scoring device provided in the embodiments of the present invention can execute the open-ended question answer scoring method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution. Figure 3 This is a structural block diagram of an open-ended test answer scoring device provided in Embodiment 2 of the present invention, as shown below. Figure 3 As shown, the device includes: a language processing module 310, a similarity determination module 320, an intent recognition module 330, and a score determination module 340.
[0126] The language processing module 310 is used to acquire the target test question and the target answer, and to perform word and sentence feature extraction processing on the target test question and the target answer respectively to obtain the test question feature set and the answer feature set;
[0127] The similarity determination module 320 is used to measure the similarity between the question feature set and the answer feature set to obtain semantic similarity and character similarity.
[0128] The intent recognition module 330 is used to determine the question intent vector of the question feature set and the answer intent vector of the answer feature set by using a pre-trained target intent parsing model;
[0129] The scoring determination module 340 is used to determine the answer score of the target answer based on the semantic similarity, the character similarity, the question intent vector, and the answer intent vector, combined with a pre-trained target scoring model.
[0130] The technical solution of this embodiment involves acquiring the target test question and the target answer, extracting word and sentence features from both to obtain a test question feature set and an answer feature set, respectively; measuring the similarity between the test question feature set and the answer feature set to obtain semantic similarity and character similarity; using a pre-trained target intent parsing model to determine the test question intent vector of the test question feature set and the answer intent vector of the answer feature set; and determining the answer score of the target answer based on semantic similarity, character similarity, test question intent vector, and answer intent vector, combined with a pre-trained target scoring model. The open-ended test question answer scoring method provided by this embodiment can be integrated into smart devices, allowing users to quickly and accurately score answers to open-ended test questions, saving grading time and improving grading efficiency.
[0131] Optionally, the language processing module 310 includes:
[0132] The test question and answer retrieval unit is used to retrieve the target test question and the target answer.
[0133] The sentence embedding vector determination unit is used to perform sentence embedding operations on the target question and the target answer respectively, to obtain the question sentence embedding vector and the answer sentence embedding vector;
[0134] The test question word segmentation unit is used to perform word segmentation on the target test question to obtain a Chinese word segmentation set and an English word segmentation set for the test question. The unit then performs word embedding on the Chinese word segmentation set to obtain a test question word embedding sequence.
[0135] The answer word segmentation unit is used to perform word segmentation on the target answer to obtain a Chinese word segmentation set and an English word segmentation set. The Chinese word segmentation set is then used to perform word embedding on the Chinese word segmentation set to obtain an answer word embedding sequence.
[0136] The feature set construction unit is used to construct a question feature set by embedding the question sentence embedding vector, the question word embedding sequence, the question Chinese word segmentation set, and the question English word segmentation set, and to construct an answer feature set by embedding the answer sentence embedding vector, the answer word embedding sequence, the answer Chinese word segmentation set, and the answer English word segmentation set.
[0137] Optionally, the similarity determination module 320 includes:
[0138] A semantic similarity determination unit is used to determine semantic similarity based on the question sentence embedding vector and the answer sentence embedding vector;
[0139] The Chinese-English character similarity determination unit is used to measure the Chinese character similarity between the Chinese word segmentation set of the test questions and the Chinese word segmentation set of the answers to obtain the Chinese character similarity, and to measure the English character similarity between the English word segmentation set of the test questions and the English word segmentation set of the answers to obtain the English character similarity.
[0140] A character similarity determination unit is used to determine the character similarity based on the Chinese character similarity and the English character similarity.
[0141] Optionally, the intent recognition module 330 includes:
[0142] The test question intent parsing unit is used to perform intent parsing on the test question word embedding sequence using the target test question intent parsing network in the target intent parsing model to obtain the test question intent vector;
[0143] The answer intent parsing unit is used to perform intent parsing on the answer word embedding sequence using the target answer intent parsing network in the target intent parsing model to obtain the answer intent vector.
[0144] Optionally, the training process of the target intent parsing model includes:
[0145] Construct a trainable intent parsing model that includes a question intent parsing network, an answer intent parsing network, and an intent classifier, and initialize the number of intent categories in the intent classifier to a preset number of categories;
[0146] The intent parsing model to be trained is trained based on the training word embedding data, and the model parameters of the intent parsing model to be trained are updated.
[0147] The updated intent parsing model to be trained is tested based on the test word embedding data, and the number of intent categories in the intent classifier is adjusted.
[0148] Return to retraining and testing the intent parsing model to be trained, and determine the adjustment trend of the number of intent categories. When the adjustment trend meets the adjustment stability condition, the last updated intent parsing model to be trained is determined as the target intent parsing model.
[0149] Optionally, training the intent parsing model based on training word embedding data and updating the model parameters of the intent parsing model includes:
[0150] Obtain a preset number of training word embedding data sets, input each training word embedding data set into the intent parsing model to be trained for training, and update the model parameters of the intent parsing model to be trained. The training word embedding data sets include training question word embedding sequences and training answer word embedding sequences.
[0151] Return to the operation of retrieving the preset number of training word embeddings and updating the model parameters until the preset number of updates is reached.
[0152] Optionally, the step of inputting the embedding data of each of the training words into the intent parsing model to be trained for training, and updating the model parameters of the intent parsing model to be trained, includes:
[0153] For each set of training word embedding data, the test question intent parsing network is used to determine the training test question intent vector of the training test question word embedding sequence, and the answer intent parsing network is used to determine the training answer intent vector of the training answer word embedding sequence. The training test question intent vector and the training answer intent vector are classified according to the intent classifier to obtain the training test question intent type and the training answer intent type.
[0154] Based on the intent type of each training question and the corresponding intent type of the training answer, the intent fitting loss function is obtained;
[0155] The intent fitting loss function is used to backpropagate the intent parsing model to be trained, thereby updating the model parameters of the intent parsing model to be trained.
[0156] Optionally, the step of testing the updated intent parsing model based on test word embedding data and adjusting the number of intent categories in the intent classifier includes:
[0157] Obtain a preset number of test word embedding data sets, and input each of the test word embedding data sets into the updated intent parsing model to be trained. The test word embedding data sets include test question word embedding sequences and test answer word embedding sequences.
[0158] For each set of test word embedding data, the test question intent vector of the test question word embedding sequence is determined using the test question intent parsing network, and the test answer intent vector of the test answer word embedding sequence is determined using the answer intent parsing network.
[0159] The distribution of vector distances is determined based on the Euclidean distance between the intent vectors of each test question and the corresponding intent vectors of test answers.
[0160] When the vector distance distribution meets the preset distribution conditions, the number of intent categories in the intent classifier is reduced by one; otherwise, the number of intent categories in the intent classifier is increased by one.
[0161] Optionally, the training process of the target scoring model includes:
[0162] The training feature combination data is scored and labeled to obtain the standard score corresponding to the training feature combination data, wherein the training feature combination data includes standard semantic similarity, standard character similarity, standard question intent vector and standard answer intent vector;
[0163] The training feature combination data is input into the scoring model to be trained to obtain the actual output score.
[0164] Based on the standard score and the actual score, a score fitting loss function is obtained;
[0165] The scoring model is obtained by backpropagating the training scoring model using the scoring fitting loss function.
[0166] Example 3
[0167] Figure 4 This is a structural block diagram of a computer device provided in Embodiment 3 of the present invention, as shown below. Figure 4 As shown, the computer device includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of processors 410 in the computer device can be one or more. Figure 4 Taking a processor 410 as an example; the processor 410, memory 420, input device 430, and output device 440 in a computer device can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.
[0168] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the open-ended question answer scoring method in this embodiment of the invention (e.g., the language processing module 310, similarity determination module 320, intent recognition module 330, and score determination module 340 in the open-ended question answer scoring device). The processor 410 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 420, thereby implementing the aforementioned open-ended question answer scoring method.
[0169] The memory 420 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 420 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include memory remotely located relative to the processor 410, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0170] Input device 430 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the computer device. Output device 440 may include display devices such as a display screen.
[0171] Example 4
[0172] Embodiment 4 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute an open-ended question answer scoring method, the method comprising:
[0173] Obtain the target test question and the target answer, and perform word and sentence feature extraction processing on the target test question and the target answer respectively to obtain the test question feature set and the answer feature set;
[0174] The similarity between the question feature set and the answer feature set is measured to obtain semantic similarity and character similarity;
[0175] A pre-trained target intent parsing model is used to determine the test intent vector of the test question feature set and the answer intent vector of the answer feature set.
[0176] Based on the semantic similarity, the character similarity, the question intent vector, and the answer intent vector, combined with a pre-trained target scoring model, the answer score of the target answer is determined.
[0177] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the open-ended question answer scoring method provided in any embodiment of the present invention.
[0178] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0179] It is worth noting that in the embodiments of the above-mentioned open-ended question answer scoring device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.
[0180] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.
Claims
1. A method for scoring open-ended test questions, characterized in that, include: Obtain the target test question and the target answer, and perform word and sentence feature extraction processing on the target test question and the target answer respectively to obtain the test question feature set and the answer feature set; The similarity between the question feature set and the answer feature set is measured to obtain semantic similarity and character similarity; A pre-trained target intent parsing model is used to determine the test intent vector of the test question feature set and the answer intent vector of the answer feature set. Based on the semantic similarity, the character similarity, the question intent vector, and the answer intent vector, combined with a pre-trained target scoring model, the answer score of the target answer is determined. The training process of the target intent parsing model includes: Construct a trainable intent parsing model that includes a question intent parsing network, an answer intent parsing network, and an intent classifier, and initialize the number of intent categories in the intent classifier to a preset number of categories; The intent parsing model to be trained is trained based on the training word embedding data, and the model parameters of the intent parsing model to be trained are updated. The updated intent parsing model to be trained is tested based on the test word embedding data, and the number of intent categories in the intent classifier is adjusted. Return to retraining and testing the intent parsing model to be trained, and determine the adjustment trend of the number of intent categories. When the adjustment trend meets the adjustment stability condition, the last updated intent parsing model to be trained is determined as the target intent parsing model. The step of testing the updated intent parsing model based on test word embedding data and adjusting the number of intent categories in the intent classifier includes: Obtain a preset number of test word embedding data sets, and input each of the test word embedding data sets into the updated intent parsing model to be trained. The test word embedding data sets include test question word embedding sequences and test answer word embedding sequences. For each set of test word embedding data, the test question intent vector of the test question word embedding sequence is determined using the test question intent parsing network, and the test answer intent vector of the test answer word embedding sequence is determined using the answer intent parsing network. The distribution of vector distances is determined based on the Euclidean distance between the intent vectors of each test question and the corresponding intent vectors of test answers. When the vector distance distribution meets the preset distribution conditions, the number of intent categories in the intent classifier is reduced by one; otherwise, the number of intent categories in the intent classifier is increased by one.
2. The open-ended question answer scoring method according to claim 1, characterized in that, The step involves extracting word and sentence features from the target question and the target answer respectively to obtain a question feature set and an answer feature set, including: Sentence embedding operations are performed on the target question and the target answer respectively to obtain the question sentence embedding vector and the answer sentence embedding vector; The target test question is segmented into Chinese and English words to obtain a Chinese word segmentation set and an English word segmentation set. Word embedding is then performed on the Chinese word segments in the Chinese word segmentation set to obtain a word embedding sequence. The target answer is segmented to obtain a Chinese word segmentation set and an English word segmentation set. The Chinese word segments in the Chinese word segmentation set are then embedded to obtain an answer word embedding sequence. The question sentence embedding vector, question word embedding sequence, Chinese word segmentation set, and English word segmentation set constitute the question feature set, and the answer sentence embedding vector, answer word embedding sequence, Chinese word segmentation set, and English word segmentation set constitute the answer feature set.
3. The open-ended question answer scoring method according to claim 2, characterized in that, The similarity measurement of the question feature set and the answer feature set to obtain semantic similarity and character similarity includes: The semantic similarity is determined based on the embedding vectors of the question sentences and the embedding vectors of the answer sentences; The similarity of Chinese characters between the Chinese word segmentation set of the test questions and the Chinese word segmentation set of the answers is measured to obtain the Chinese character similarity. Similarly, the similarity of English characters between the English word segmentation set of the test questions and the English word segmentation set of the answers is measured to obtain the English character similarity. The character similarity is determined based on the similarity between the Chinese characters and the English characters.
4. The open-ended question answer scoring method according to claim 2, characterized in that, The step of using a pre-trained target intent parsing model to determine the question intent vector of the question feature set and the answer intent vector of the answer feature set includes: The target question intent parsing network in the target intent parsing model is used to parse the question word embedding sequence to obtain the question intent vector; The target intent parsing network in the target intent parsing model is used to parse the intent of the answer word embedding sequence to obtain the answer intent vector.
5. The open-ended question answer scoring method according to claim 1, characterized in that, The step of training the intent parsing model based on training word embedding data and updating the model parameters of the intent parsing model includes: Obtain a preset number of training word embedding data sets, input each training word embedding data set into the intent parsing model to be trained for training, and update the model parameters of the intent parsing model to be trained. The training word embedding data sets include training question word embedding sequences and training answer word embedding sequences. Return to the operation of retrieving the preset number of training word embeddings and updating the model parameters until the preset number of updates is reached.
6. The open-ended question answer scoring method according to claim 5, characterized in that, The step of inputting the embedded data of each training word into the intent parsing model to be trained for training, and updating the model parameters of the intent parsing model to be trained, includes: For each set of training word embedding data, the test question intent parsing network is used to determine the training test question intent vector of the training test question word embedding sequence, and the answer intent parsing network is used to determine the training answer intent vector of the training answer word embedding sequence. The training test question intent vector and the training answer intent vector are classified according to the intent classifier to obtain the training test question intent type and the training answer intent type. Based on the intent type of each training question and the corresponding intent type of the training answer, the intent fitting loss function is obtained; The intent fitting loss function is used to backpropagate the intent parsing model to be trained, thereby updating the model parameters of the intent parsing model to be trained.
7. The open-ended question answer scoring method according to claim 1, characterized in that, The training process of the target scoring model includes: The training feature combination data is scored and labeled to obtain the standard score corresponding to the training feature combination data, wherein the training feature combination data includes standard semantic similarity, standard character similarity, standard question intent vector and standard answer intent vector; The training feature combination data is input into the scoring model to be trained to obtain the actual output score. Based on the standard score and the actual score, a score fitting loss function is obtained; The scoring model is obtained by backpropagating the training scoring model using the scoring fitting loss function.
8. An open-ended test answer scoring device, characterized in that, include: The language processing module is used to acquire the target test question and the target answer, and to perform word and sentence feature extraction processing on the target test question and the target answer respectively to obtain the test question feature set and the answer feature set; The similarity determination module is used to measure the similarity between the question feature set and the answer feature set to obtain semantic similarity and character similarity. The intent recognition module is used to determine the question intent vector of the question feature set and the answer intent vector of the answer feature set by using a pre-trained target intent parsing model. The scoring determination module is used to determine the answer score of the target answer based on the semantic similarity, the character similarity, the question intent vector, and the answer intent vector, combined with a pre-trained target scoring model. The training process of the target intent parsing model includes: Construct a trainable intent parsing model that includes a question intent parsing network, an answer intent parsing network, and an intent classifier, and initialize the number of intent categories in the intent classifier to a preset number of categories; The intent parsing model to be trained is trained based on the training word embedding data, and the model parameters of the intent parsing model to be trained are updated. The updated intent parsing model to be trained is tested based on the test word embedding data, and the number of intent categories in the intent classifier is adjusted. Return to retraining and testing the intent parsing model to be trained, and determine the adjustment trend of the number of intent categories. When the adjustment trend meets the adjustment stability condition, the last updated intent parsing model to be trained is determined as the target intent parsing model. The step of testing the updated intent parsing model based on test word embedding data and adjusting the number of intent categories in the intent classifier includes: Obtain a preset number of test word embedding data sets, and input each of the test word embedding data sets into the updated intent parsing model to be trained. The test word embedding data sets include test question word embedding sequences and test answer word embedding sequences. For each set of test word embedding data, the test question intent vector of the test question word embedding sequence is determined using the test question intent parsing network, and the test answer intent vector of the test answer word embedding sequence is determined using the answer intent parsing network. The distribution of vector distances is determined based on the Euclidean distance between the intent vectors of each test question and the corresponding intent vectors of test answers. When the vector distance distribution meets the preset distribution conditions, the number of intent categories in the intent classifier is reduced by one; otherwise, the number of intent categories in the intent classifier is increased by one.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the open-ended question answer scoring method as described in any one of claims 1-7.
10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the open-ended question answer scoring method as described in any one of claims 1-7.